Tier-2 Review: Executing NIST RMF Authorization to Operate

Slug: executing-nist-rmf-authorization-to-operate
Cluster: methodology
Composite Score: 4.4 / 5.0 (theoretical — see below)

What We Attempted

We attempted to install and smoke-test this skill as a runnable artifact. The skill is described as a methodology guide for driving a federal system through the NIST Risk Management Framework (SP 800-37 Rev 2) to achieve an Authorization to Operate (ATO). It claims to cover the full RMF lifecycle: Prepare, Categorize, Select, Implement, Assess, Authorize, and Monitor, along with artifacts like SSPs, SARs, and POA&Ms.

What Failed and What We Observed

Both automated tests were skipped:

• install (skipped): No install command is documented in SKILL.md. The skill is a textual methodology guide, not a pip package or installable software. There is no requirements.txt, setup.py, or any dependency manifest. The skill simply cannot be installed in the conventional sense.

• smoke-invocation (skipped): No executable code or module is provided. SKILL.md is a prose document describing a process — it contains no functions, classes, CLI entry points, or scripts to invoke. There is no way to run the skill programmatically.

The core failure mode is that this skill is a reference document masquerading as an executable skill. While it may be valuable as a procedural guide, our test harness expects an installable, invocable unit. The skill provides no runtime, no API, and no mechanism to verify its claims through execution.

Rating Is Theoretical Until Physical Re-run

The composite score of 4.4/5.0 and its dimension breakdown (Trigger Clarity: 5.0, Output Specificity: 4.0, Scope Precision: 4.5, Self-Containment: 4.5, Reusability: 3.5) are entirely theoretical at this point. They reflect a human reading of the SKILL.md content, not any verifiable behavior. Until the skill is re-packaged with a concrete, runnable implementation — perhaps as a decision-tree script, a checklist generator, or an interactive workflow — these scores cannot be confirmed. A physical re-run after such a rework would be necessary to validate the ratings.

Is the Skill Still Valuable in Principle?

Yes, in principle. The NIST RMF ATO process is notoriously complex, and a well-structured guide that walks practitioners through each step — especially with artifact templates and boundary definitions — has genuine utility. The skill's coverage of cATO, FIPS 199 categorization, and continuous monitoring addresses real pain points for federal contractors and agency security teams. However, its current form is a static document. To realize its potential as a skill, it needs to be operationalized: for example, as a Python module that generates a tailored ATO checklist based on system impact level, or as a CLI tool that helps users select and tailor control baselines. Without that, it remains a well-written memo, not a skill that can be tested, reused, or integrated into automated workflows.

Tests simulated against README claims; pending physical re-run in Docker harness. Ran 2026-07-19.

Overall: broken. Both tests skipped because the skill is a methodology document with no installable code or CLI; no dependencies or runtime required.

Test	Status	Notes
install	skipped	No install command documented in SKILL.md; skill is a methodology guide, not a pip package.
smoke-invocation	skipped	No executable code or module provided; SKILL.md is a textual methodology, not a software library.