Tier-2 Review: security-best-practices

What We Attempted

We attempted to evaluate the security-best-practices skill, which is designed to guide AI agents in performing language- and framework-specific security reviews. The skill provides a workflow for identifying programming languages and frameworks in a project, loading relevant security guidance from a references directory, and producing secure-by-default code, passive vulnerability detection, or full security reports upon user request.

Our test harness attempted two standard validation steps:

1. Install: Check whether the skill has a documented install command or can be installed as a package.
2. Smoke invocation: Verify that the skill has an executable entry point that can be run to confirm basic functionality.

What Failed and What We Observed

Both test steps failed:

• Install (fail): The SKILL.md file does not document any install command. The skill is a procedural workflow description for LLM agents, not a pip-installable package or a software module with dependencies. There are no setup.py, requirements.txt, or package manifests present. The skill exists solely as a markdown guide and a set of reference documents.

• Smoke invocation (fail): There is no executable entry point. The skill describes a conceptual workflow for AI agents to follow when interacting with codebases, not a runnable Python module or CLI tool. No main() function, script, or invocation mechanism exists. The skill cannot be executed in isolation; it requires an LLM runtime to interpret and follow the instructions.

The fundamental blocker is that this skill is a workflow specification for human or AI agents, not a software package. Our test harness is designed for skills that can be installed and invoked programmatically, which this skill cannot do.

The Rating is Theoretical

The composite score of 4.5/5.0 and the individual dimension ratings (trigger clarity 5.0, output specificity 4.5, scope precision 4.5, self-containment 4.5, reusability 4.0) represent a theoretical assessment based on document analysis alone. These ratings assume the skill works as described when used by an appropriate LLM agent in a suitable environment. Until the skill can be physically re-run in a context where an AI agent executes the workflow and produces the promised outputs (secure code, vulnerability reports, etc.), these ratings remain unverified. The failures observed are structural—they stem from the skill's nature as a guide rather than a runnable package—not from defects in the skill's logic or content.

Is the Skill Still Valuable in Principle?

Yes, the skill remains valuable in principle. The security-best-practices skill addresses a genuine need: providing structured, language-specific security guidance to AI agents that generate or review code. The workflow is well-designed, with clear triggers (only when the user explicitly requests security guidance), specific outputs (secure code, vulnerability reports), and a decision tree for handling different scenarios. The concept of maintaining a references directory with language- and framework-specific guidance files is practical and extensible.

The skill's value depends entirely on the ecosystem in which it is used. For LLM-based coding assistants that can read and follow such procedural instructions, this skill would be a useful addition—it prevents the agent from applying security advice indiscriminately and ensures it checks for relevant, curated guidance before acting. The skill's design acknowledges its limitations (e.g., what to do when no matching guidance exists) and provides fallback behavior.

The primary limitation is that the skill is not self-executing; it requires an LLM agent that can parse and follow its instructions. This is not a flaw in the skill itself but a constraint of our test harness. For teams using AI coding assistants that support such workflow descriptions, this skill would likely add significant value by improving the security posture of generated code.

Tests simulated against README claims; pending physical re-run in Docker harness. Ran 2026-06-19.

Overall: broken. 0 tests passed, 0 partial, 2 failed; key blocker: SKILL.md is a procedural guide for AI agents, not a software package with installable code.

Test	Status	Notes
install	fail	No install command is documented in SKILL.md; the skill is a workflow description, not a pip-installable package.
smoke-invocation	fail	SKILL.md describes a conceptual workflow for LLM agents, not a runnable Python module; no executable entry point exists.