Tier-2 Review: vercel-deploy (cli-and-api)

What we attempted

We ran the standard five-probe harness against vercel-deploy as documented in its SKILL.md: install-and-auth, list-or-read, write-or-mutate, rate-limit-handling, and the implicit fallback path. The goal was to confirm the skill's claims — that it detects the Vercel CLI, deploys as preview by default, falls back to an unauthenticated script when credentials are missing, and returns a usable URL — against observable behavior rather than against the prose in the skill file.

What failed

0 tests passed. 2 partial. 0 failed. 2 skipped. The harness could not resolve the skill to a testable state, and the reasons are structural, not incidental.

The two skipped probes are skipped because the surface does not exist. list-or-read has nothing to call: SKILL.md documents exactly three commands — vercel deploy [path] -y, vercel deploy [path] --prod -y, and bash scripts/deploy.sh — none of which are read-only. rate-limit-handling is skipped because the skill never mentions 429s, backoff, or retries; its only error guidance covers network-layer failures (timeouts, DNS, connection resets) and routes those to sandbox_permissions=require_escalated. There is no rate-limit contract to test.

The two partial probes are partial for the same underlying reason: the skill's real behavior lives in an artifact we cannot see. install-and-auth only checks command -v vercel; it documents no API-key environment variable, so a dummy token surfaces as a CLI auth error, which the skill explicitly reroutes to scripts/deploy.sh. That fallback script is referenced but not included in the SKILL.md text, and the harness cannot execute it. So the auth path is verified only up to the branch point, not through it. write-or-mutate is partial for the same reason: the CLI path fails on missing credentials, the fallback path is unexecutable from the skill text alone, and the skill documents no idempotency or rollback semantics anywhere — so the mutation itself is unverified end to end.

What we observed

The failure mode is not a broken skill. It is a skill whose contract is split across two files, only one of which is inspectable. SKILL.md is a router: it decides between a CLI path and a script path, then hands off. Everything that would make the skill testable — framework detection, packaging, the actual deploy call, the JSON shape (previewUrl, claimUrl) — lives behind scripts/deploy.sh. From the skill text alone we can confirm the routing logic is coherent and the preview-by-default default is explicit. We cannot confirm the deploy works.

Two secondary observations worth recording. First, the skill deliberately forbids curling the returned URL to verify it — a defensible design choice, but it removes the only cheap end-to-end signal a harness could otherwise use. Second, the timeout guidance (10 minutes / 600000ms) is documented but unexercised, since no deploy ever ran.

Rating caveat

The composite 4.3 / 5.0 and the dimension scores above are therefore theoretical. They reflect the quality of the SKILL.md as a specification — trigger clarity, scope discipline, self-containment of the decision logic — not the quality of the skill as a working artifact. D5 (reusability, 3.5) is the dimension most likely to move once the fallback script is exercised, in either direction: if deploy.sh is robust, reusability rises; if it silently depends on environment state the skill never documents, it falls. Until a physical re-run with real Vercel credentials (and the actual scripts/deploy.sh present) resolves the two partial probes, treat the score as provisional.

Is the skill still valuable in principle?

Yes. The routing design is sound: check cheaply without escalation, escalate only the network-bound deploy, prefer preview over production, and degrade to an auth-free fallback rather than failing hard. That is exactly the shape a deploy skill should have, and the trigger phrasing ("deploy my app", "push this live", "create a preview deployment") is unusually well-matched to how users actually ask. The gaps — no read surface, no rate-limit contract, no idempotency story — are real but narrow; they matter most in agentic loops that deploy repeatedly. The honest verdict: promising specification, unverified implementation. Re-run with credentials and the fallback script in place before trusting the number.

Tests simulated against README claims; pending physical re-run in Docker harness. Ran 2026-09-17.

Overall: broken. 0 tests passed, 2 partial, 0 failed, 2 skipped; key blocker: SKILL.md documents only deploy operations (no read/list surface, no rate-limit or idempotency semantics) and relies on an undocumented scripts/deploy.sh fallback that cannot be validated from the SKILL.md text alone.

Inferred dependencies: vercel CLI (checked via command -v vercel), bash (for scripts/deploy.sh fallback), network access to Vercel (may require sandbox_permissions=require_escalated), 10 minute (600000ms) command timeout for deploy.

Test	Status	Notes
install-and-auth	partial	SKILL.md only checks for the CLI via command -v vercel and does not document any API-key env var; a dummy token would surface as a CLI auth error ('No existing credentials found'), which the skill explicitly routes to the fallback deploy.sh script rather than crashing. The fallback path itself is not exercised by this test since it requires the skill's scripts/deploy.sh to exist.
list-or-read	skipped	SKILL.md exposes no list/get/search operation — its only documented commands are vercel deploy [path] -y, vercel deploy [path] --prod -y, and the fallback scripts/deploy.sh. No read-only surface exists to test.
write-or-mutate	partial	SKILL.md mandates preview-only deploys with a 10-minute timeout and explicitly says not to curl the returned URL to verify; idempotency and rollback semantics are not described anywhere in the skill, so those aspects cannot be validated. Without real Vercel credentials the CLI fails with 'No existing credentials found' and the skill falls back to scripts/deploy.sh, which is expected to return JSON with previewUrl and claimUrl.
rate-limit-handling	skipped	SKILL.md contains no mention of 429 handling, backoff, or retry logic; the only error-handling guidance is for network failures (timeouts, DNS errors, connection resets) which should be retried with sandbox_permissions=require_escalated. Rate-limit behavior is therefore untested and undocumented.