Tier-2 Review: gh-fix-ci

What we attempted

We picked up gh-fix-ci from the curated OpenAI skills repo because it targets a concrete, recurring developer pain: a PR with red GitHub Actions checks, and a developer who wants the failure summarized and fixed rather than spelunked through the Actions UI. The SKILL.md reads well on the first pass — clear trigger ("debug or fix failing GitHub PR checks that run in GitHub Actions"), explicit scope fence around non-Actions providers like Buildkite, and a bundled helper script (scripts/inspect_pr_checks.py) that is supposed to absorb gh field drift and job-log fallbacks.

We tried to run it end-to-end in a clean container: copy the skill directory, invoke the documented quick-start command, and see whether it produced a structured failure summary we could hand back to the user.

What failed

Two of the three harness steps did not complete, and the third was only partial.

1. Install (partial). There is no install step to run. SKILL.md references a bundled script and a gh auth login prerequisite, but ships no package manifest, no Makefile, no dependency declaration. Copying the skill directory is the whole "install." That is defensible for a skill that is essentially a prompt plus one Python script, but it means the harness has nothing to verify — no version pin, no pip install, no way to confirm the script's imports resolve. We recorded this as partial rather than pass because "no install needed" and "install undocumented" are indistinguishable from the outside.

2. Smoke invocation (fail). This is the real blocker. Step 1 of the workflow is gh auth status. In a clean container with no credentials, this fails immediately, and the script cannot resolve PR checks without an authenticated gh plus a live PR with Actions runs. SKILL.md is honest about this — it says "Prereq: authenticate with the standard GitHub CLI once" and instructs the agent to ask the user to run gh auth login with repo + workflow scopes. But there is no dry-run mode, no fixture, no --repo pointing at a sample that would let the harness exercise the script's parsing logic against canned gh JSON. The skill is therefore untestable in isolation: every meaningful code path requires network, credentials, and a real failing PR.

Net: 0 passed, 1 partial, 1 failed. The failure mode is not a bug in the skill — it is that the skill's contract assumes an authenticated, network-connected environment and offers no seam for offline verification.

What we observed

The SKILL.md itself is well-constructed. The trigger is unambiguous, the scope boundary (external providers out of scope, report the details URL only) is exactly the kind of precision that keeps an agent from wandering into Buildkite logs. The workflow is sequenced sensibly: verify auth → resolve PR → inspect checks → summarize → plan → implement after approval → recheck. The "implement only after explicit approval" gate is the right default for a skill that will edit a user's branch.

The gaps we can see without running it: the script's behavior is asserted, not demonstrated; the create-plan dependency is soft ("if a plan-oriented skill is available"), which is fine but means output shape varies; and D5 reusability suffers because the skill is welded to gh + GitHub Actions + a live repo.

Rating caveat

The composite of 4.2 / 5.0 should be read as theoretical until someone re-runs this in an environment with an authenticated gh, a target repo, and an open PR with failing Actions checks. Our harness could not get past step 1, so dimensions like D2 (output specificity) and D4 (self-containment) are scored from the document, not from observed output. If the script works as advertised, 4.2 is probably fair. If it drifts against current gh JSON fields, D2 and D4 will fall.

Is it still valuable in principle?

Yes. The problem is real, the scope fence is disciplined, and the approval gate is correct. The fix is small: add a --fixture or --json-from-file mode to the script so the parsing path can be exercised without credentials, and document a one-line install (uv pip install -r requirements.txt or equivalent) so the harness has something to verify. That would move this from "trust the doc" to "trust the test."

Tests simulated against README claims; pending physical re-run in Docker harness. Ran 2026-09-28.

Overall: broken. 0 tests passed, 1 partial, 1 failed; key blocker: no documented install command and the smoke invocation requires an authenticated gh CLI plus a live PR, neither available in a clean container.

Inferred dependencies: gh (GitHub CLI) authenticated via gh auth login with repo + workflow scopes, python (to run scripts/inspect_pr_checks.py), network access to GitHub API, a target repository with an open PR and GitHub Actions checks.

Test	Status	Notes
install	partial	SKILL.md documents no install command; it only references a bundled script at scripts/inspect_pr_checks.py and requires gh auth login. There is no package manifest or install step to execute, so installation is effectively a no-op copy of the skill directory.
smoke-invocation	fail	In a clean container without gh authenticated, step 1 (gh auth status) fails and the script cannot resolve PR checks; the SKILL.md explicitly requires prior gh auth login with repo + workflow scopes. Without a real PR and credentials the invocation cannot produce output.