iso42001-specialist

Tier-2 Review: ISO42001 Specialist (iso42001-specialist)

What we attempted:
We evaluated the ISO42001 Specialist skill, which claims to serve as an AI Management System (AIMS) specialist for compliance teams conducting internal audits against ISO/IEC 42001:2023. The skill defines three core decisions: (1) identifying gaps against Clauses 4-10 and prioritizing remediation, (2) populating an AI risk register with Annex A controls, and (3) generating a 12-month internal audit plan satisfying Clause 9.2. We attempted to install and invoke the skill programmatically to verify its functionality.

What failed:
Both automated tests failed completely. The install test failed because the SKILL.md contains no documented installation command—there is no package on PyPI, no requirements.txt, and no setup script. The smoke-invocation test failed because the skill defines no CLI binary, entry point, or executable script; attempting to invoke it via command line returned "command not found." The skill is described as a conversational artifact meant to be used within a chat interface (e.g., as a system prompt), not as a standalone tool or package.

What we observed:
The SKILL.md is well-structured, with clear boundaries (e.g., it explicitly excludes EU AI Act compliance and executive strategy), and the output specificity is high—it promises concrete audit artifacts. However, the skill is fundamentally a prompt/context definition for a large language model, not an installable software component. Our test harness expects a runnable package or CLI, which this skill does not provide. The composite score of 4.2 reflects theoretical quality: trigger clarity (4.5), output specificity (4.0), scope precision (4.5), self-containment (4.0), and reusability (3.5) are all rated based on the textual description alone, not on any executed behavior.

The rating is therefore theoretical until a physical re-run resolves the failures. The skill may function perfectly when pasted into a compliant LLM session and used conversationally, but our automated tests cannot validate that. A manual test with a human evaluator would be required to assess whether the generated audit plans and risk registers are accurate and useful.

Is the skill still valuable in principle?
Yes. ISO/IEC 42001 is a nascent standard, and compliance teams need structured guidance to operationalize it. The skill’s focus on three concrete decisions (gap analysis, risk register, audit plan) addresses real pain points. Its exclusion of adjacent domains (EU AI Act, executive strategy) demonstrates thoughtful scoping. If the skill were re-packaged as a Python library or CLI tool (e.g., with a pip install iso42001-specialist and a iso42001-audit command), it would likely earn a high execution score. In its current conversational form, it remains useful for teams already working within an LLM-based workflow, but the 4.2 rating should be understood as an upper-bound estimate of its design quality, not a validated performance metric.

Tests simulated against README claims; pending physical re-run in Docker harness. Ran 2026-06-28.

Overall: broken. 0 tests passed, 0 partial, 2 failed; key blocker: SKILL.md describes a conversational skill with no installable package or CLI, so both tests fail.