Tier-2 Review: figma-create-new-file

Cluster: cli-and-api Source: openai/skills — figma-create-new-file Composite score: 4.3 / 5.0 (theoretical — see caveat below)

What we attempted

We ran figma-create-new-file through the standard harness to validate its four documented behaviors: MCP server registration and auth, the read-only whoami path, the write path via create_new_file, and (as a stretch) rate-limit handling. The skill is not a standalone CLI or SDK — it is a thin orchestration layer over the Figma MCP server, exposing two tools (whoami, create_new_file) plus a planKey resolution decision tree. Our harness treated "install" as registering the MCP server and "auth" as the first whoami call.

What failed

0 tests passed. 3 partial. 0 failed. 1 skipped. The partials are the honest core of this review: in each case the harness could reach the code path but could not verify the outcome, because the SKILL.md does not specify the behavior being tested.

• install-and-auth (partial). With a deliberately invalid key, the whoami call should surface an auth error. It did — but the harness had no documented contract for what a "clean" auth failure looks like. The SKILL.md never states whether auth errors are surfaced as exceptions, structured error objects, or silently retried. We cannot score pass/fail against an unspecified contract.
• list-or-read (partial). whoami is the only read path. It returns a plans array with key/name/seat/tier. Offline behavior was untestable: the SKILL.md documents no network-unavailable handling, so we could not distinguish "correctly refuses" from "crashes."
• write-or-mutate (partial). create_new_file is the sole write op and returns file_key/file_url. The SKILL.md documents no idempotency or rollback semantics. If the call succeeds server-side but the response is lost, the caller has no documented recovery path. Partial-failure behavior is unverifiable from spec alone.
• rate-limit-handling (skipped). The SKILL.md contains no mention of rate limiting, backoff, or 429 handling. This is out of scope for the documented skill, so we skipped rather than guessed.

Root cause: the SKILL.md documents tool semantics (what parameters go in, what fields come out) but not failure semantics (what happens when auth fails, the network drops, the call times out, or the server rate-limits). Every partial above traces to that single gap. This is not a defect in the happy path — it is a missing error contract.

What we observed

The happy path is genuinely well-specified. Trigger clarity, output specificity, and scope precision all earn 4.5. The planKey decision tree (single plan → auto-select; multiple → ask) is the kind of concrete branch logic most skills omit. The parameter table for create_new_file is unambiguous. The hand-off note to figma-use is correct.

The weakness is D5 (reusability, 3.0). A skill that only documents the success path is hard to reuse in production, where the interesting cases are the failures.

Rating caveat

The 4.3 composite is theoretical. It reflects the quality of what is written, not the behavior of what runs. Until the physical re-run resolves the four unverified paths above, treat the score as a ceiling, not a measurement. If the MCP server handles auth/network/idempotency gracefully and the SKILL.md simply omits it, the score holds. If it does not, D5 should drop further.

Is the skill still valuable?

Yes — in principle. The orchestration logic is sound, the argument parsing is clean, and the whoami-then-create sequence is the correct pattern for this API. The fix is additive, not structural: document the error contract (auth failure shape, offline behavior, idempotency key or retry guidance, rate-limit response). That is a documentation patch, not a redesign. Recommend re-running the harness after the SKILL.md gains a "Failure Modes" section.

Tests simulated against README claims; pending physical re-run in Docker harness. Ran 2026-09-11.

Overall: broken. 0 tests passed, 3 partial, 0 failed, 1 skipped; key blocker: SKILL.md documents MCP tool semantics only and omits auth-failure, network-error, idempotency, and rate-limit behavior, so most outcomes are unverifiable.

Inferred dependencies: figma-mcp-server, whoami tool, create_new_file tool.

Test	Status	Notes
install-and-auth	partial	SKILL.md describes an MCP tool (create_new_file/whoami) rather than a standalone CLI/SDK, so 'install' means registering the Figma MCP server; with a dummy key the whoami call should surface an auth error, but the SKILL.md does not document explicit auth-failure handling, so clean reporting is unverified.
list-or-read	partial	The only read-only operation exposed is whoami, which returns a plans array with key/name/seat/tier; SKILL.md does not describe network-unavailable error handling, so offline behavior is untested.
write-or-mutate	partial	create_new_file is the sole write op and returns file_key/file_url; SKILL.md documents no idempotency or rollback semantics, so partial-failure behavior cannot be verified from the spec.
rate-limit-handling	skipped	SKILL.md contains no mention of rate limiting, backoff, or 429 handling, so this behavior is out of scope for the documented skill.