Tier-2 Review: claude-academy-guide

What the harness actually showed

The test results are honest about what this skill is — and isn't. install-and-auth passed trivially because there's no CLI, no SDK, no auth flow, nothing to install. A dummy API key caused zero crashes, which is exactly what you'd expect from a static markdown file. list-or-read came back partial: the skill instructs fetching the catalog from academy.claude.com, but network unavailability is handled by falling back to a resources library. The response shape is undefined in SKILL.md — you get "recommend something" but not "here's the exact JSON structure." Both write-or-mutate and rate-limit-handling were skipped, correctly, because this skill describes no mutations and makes no API calls.

The real blocker, as the harness notes, is that this is a guide, not a tool. It has no executable surface. So the composite score of 4.4 is really measuring documentation quality, not runtime behavior.

Failure modes I can infer

The biggest risk isn't technical — it's behavioral. Rule 2 is explicit: "Only recommend on strong matches" and "Silence is better than noise." But the trigger description is aggressive: it fires on "how do I," "what can Claude do," "teach me," plus a long list of product names. That's a wide net. In practice, an LLM using this skill will over-trigger on casual questions that don't actually warrant an Academy link. The skill tries to counteract this with the "caveat as tell" heuristic, but that's a judgment call, not a deterministic check.

Second failure mode: stale catalog. The skill says to fetch the catalog at conversation time, but if the network is down or content shifts, you're stuck with whatever the resources library has — which may be outdated. There's no versioning, no cache invalidation strategy, no fallback beyond "point to hub pages."

Third: the skill composes with other skills, which sounds nice but creates ordering ambiguity. If another skill already answered "how do projects work?" with documentation, this one is supposed to add an Academy recommendation. But nothing defines when in the response that recommendation appears, or how to prioritize it against other appended content.

What genuinely works

The trigger clarity is excellent — 5.0 is earned. The description is specific about intent categories and explicitly excludes mid-task requests. The self-containment is strong: rules are concrete, the "no hallucination" constraint is absolute, and the "strong match" test is operationalized with a hedge-detection heuristic. That's rare in skill documentation.

The dependency observation is minimal: network access to academy.claude.com is the only real requirement. No runtime, no package, no version constraint. That's a plus for portability.

When I'd actually use this

I'd use this skill in a controlled educational context — a course assistant, an onboarding bot, a support triage tool — where the user is explicitly in "learning mode" and the cost of a wrong recommendation is low. I would not use it in a production assistant where hallucinated links or over-recommendation could erode trust, unless I added a hard post-filter that verifies every URL against a fetched catalog before sending.

The skill is well-written, but it's a supplement, not a solution. It needs a runtime guardrail to enforce its own rules. If you can add that, the 4.4 score holds up. If you can't, expect the failure modes above to surface in real traffic.

Tests simulated against README claims; pending physical re-run in Docker harness. Ran 2026-08-20.

Overall: ok. 1 test passed, 1 partial, 2 skipped; key blocker: skill is a static guide with no CLI/SDK or API, so write and rate-limit tests are not applicable.

Inferred dependencies: network access to academy.claude.com.

Test	Status	Notes
install-and-auth	pass	Skill is a prompt/guide with no CLI or SDK dependency; no auth required. Dummy key irrelevant; no crash.
list-or-read	partial	Skill instructs fetching catalog from academy.claude.com; network unavailable would cause fetch failure, but skill handles by pointing to resources library. Response shape not defined in SKILL.md.
write-or-mutate	skipped	Skill is read-only; no write or mutate operations described.
rate-limit-handling	skipped	No API calls or rate-limit handling described; skill is a static guide.