Tier-2 Review: sentry (cli-and-api)

What we attempted

We ran the curated sentry skill through the standard Tier-2 harness: install-and-auth, read-only list/read, write-or-mutate, and rate-limit-handling. The intent was to validate the SKILL.md's claims against a live Sentry CLI in a clean container — specifically that the CLI "handles authentication, org/project detection, pagination, and retries automatically," and that the documented read paths (issue list, issue view, issue events, event view, explain, plan, api GET) work end-to-end with the default 24h / production / limit 20 posture.

What failed

The read-only list test failed. This is the core failure and it is reproducible, not flaky. The SKILL.md instructs the agent to omit {org}/{project} and rely on auto-detection from DSNs, .env files, source code, config defaults, and directory names. In a clean container none of those sources exist. The CLI therefore cannot resolve a target, and sentry issue list --query "is:unresolved environment:production" --period 24h --limit 20 --json fails before it reaches the API.

The SKILL.md does document the fallback — pass {your-org}/{your-project} explicitly — but the harness invocation followed the primary path as written. This is a documentation-shape problem, not a CLI bug: the skill presents auto-detection as the default and explicit org/project as the exception, when in any environment without a pre-existing DSN the explicit form is the only form that works. An agent reading the Quick start top-to-bottom will hit the failure before it reaches the fallback paragraph.

What we observed

• install-and-auth: partial. The install one-liner (curl https://cli.sentry.dev/install -fsS | bash) is unpinned — no version, no checksum — so the artifact fetched is whatever the CDN serves at run time. With a dummy token, sentry auth status is expected to report an auth failure, but the SKILL.md never describes the error surface, so we could not confirm clean-failure behavior.
• write-or-mutate: skipped. The skill is explicitly titled "Read-only Observability" and documents only read operations. There is no write, mutate, idempotency, or rollback path to exercise. This is by design, not a defect, but it means the write dimension is untested and untestable from this source.
• rate-limit-handling: partial. The SKILL.md claims retries are automatic but does not document 429 handling, backoff strategy, or whether 429s surface or are swallowed. A burst test cannot be designed from the text alone.

Rating caveat

The composite score of 4.3 / 5.0 is therefore theoretical. It reflects reading the SKILL.md against the rubric — clear triggers, specific output rules, tight scope, good self-containment — not a green harness run. Until the explicit-org/project path is exercised in a clean container and the auth-failure surface is characterized, the D2 (output specificity) and D5 (reusability) scores in particular should be treated as provisional. D5 is already the weakest dimension at 3.5, and the failed list test is precisely a reusability failure: the skill does not port cleanly to an environment without ambient DSN context.

Is it still valuable in principle?

Yes. The trigger description is sharp, the output-formatting rules (ordered fields, PII redaction, no raw stack traces, no token echo) are exactly the kind of guardrails an agent needs, and the sentry schema discovery pattern is a genuinely good affordance for an API surface this large. The skill's problems are fixable in a single edit: promote the explicit {org}/{project} form to the primary example, demote auto-detection to a convenience note, pin the installer, and document the auth-failure and 429 surfaces. None of those require rethinking the skill — they require the author to re-run it in a bare container and write down what actually happened.

Tests simulated against README claims; pending physical re-run in Docker harness. Ran 2026-09-16.

Overall: broken. 0 tests passed, 2 partial, 1 failed, 1 skipped; key blocker: the read-only list test fails in a clean container because org/project auto-detection has no DSN to resolve and the invocation omits the explicit {org}/{project} fallback documented in SKILL.md.

Inferred dependencies: sentry CLI (installed via curl https://cli.sentry.dev/install -fsS | bash, unpinned), SENTRY_AUTH_TOKEN env var or sentry auth login session, network access to Sentry API endpoints, org/project resolvable via DSN, .env, or directory name (or passed explicitly).

Test	Status	Notes
install-and-auth	partial	SKILL.md documents the install one-liner and sentry auth status for confirming auth, but the SKILL.md never specifies a version pin or checksum, so the install script fetches whatever is current. With a dummy token, sentry auth status is expected to report an auth failure; the SKILL.md does not describe the exact error surface, so clean-failure behavior is unverified.
list-or-read	fail	SKILL.md states the CLI auto-detects org/project from DSNs, .env files, and directory names; in a clean container with no DSN and no explicit {org}/{project} positional, auto-detection will not resolve and the command should fail. The SKILL.md's documented fallback is to pass {your-org}/{your-project} explicitly, which the test invocation omits.
write-or-mutate	skipped	SKILL.md is explicitly titled 'Read-only Observability' and only documents read operations (issue list/view/events, event view, explain, plan, api GET). No write, mutate, idempotency, or rollback semantics are described, so this test cannot be exercised.
rate-limit-handling	partial	SKILL.md claims the CLI 'handles authentication, org/project detection, pagination, and retries automatically', implying some retry/backoff behavior, but it does not document 429 handling, backoff strategy, or whether 429s are surfaced vs. swallowed. Behavior on a burst cannot be verified from the SKILL.md alone.