Tier-2 Review: evaluate-new-port (vcpkg Port Evaluation Skill)

What We Attempted

We attempted to evaluate a GitHub Skill designed to audit a local vcpkg port. The skill claims to read port metadata and build recipes, install the port, inspect extracted sources and installed files, and produce a report covering license risks, vendored code, optional dependencies, and other port review suggestions. Our test harness attempted to install the skill and run a smoke invocation to verify basic functionality.

What Failed

Both test cases failed completely:

1. Install test (fail): The SKILL.md does not specify how to install the skill. While the skill appears to be a Python script or tool (likely named evaluate-new-port.py), the documentation provides no install command, no setup instructions, and no dependency management guidance. The skill is not a vcpkg port itself, which creates confusion about how to make it available for use. Required dependencies include python>=3.10, vcpkg, and presumably Python packages for YAML parsing, file inspection, and report generation—none of which are listed.

2. Smoke invocation test (fail): The SKILL.md provides no minimal invocation example. The tool requires a vcpkg port directory as input, but no example command is given (e.g., python evaluate-new-port.py ./ports/some-port). Without this, we could not construct a valid test invocation. The absence of a usage example means we cannot verify that the tool runs, accepts input, or produces output.

What We Observed

Beyond the installation and invocation failures, several structural issues emerged:

• Missing usage documentation: The SKILL.md describes what the tool does at a high level but gives no concrete commands, flags, or input/output examples. A user encountering this skill for the first time would have no idea how to run it.
• Undefined dependencies: Python 3.10+ and vcpkg are mentioned as requirements, but no Python package dependencies are listed (e.g., pyyaml, jinja2, requests). There is no requirements.txt or pyproject.toml referenced.
• No error handling guidance: The skill does not document what happens if vcpkg is not installed, if the port directory is invalid, or if dependencies are missing.
• Ecosystem lock-in: The skill is tightly coupled to the vcpkg ecosystem, which limits its applicability but is not inherently a flaw given the skill's stated purpose.

Rating Assessment

The composite score of 4.0 / 5.0 is theoretical until physical re-run resolves the installation and invocation failures. The dimension scores reflect the skill's conceptual design:

• Trigger clarity (4.0): The trigger—"audit a vcpkg port locally"—is clear and specific.
• Output specificity (4.5): The described output (license risks, vendored code, optional dependencies) is well-defined.
• Scope precision (4.5): The skill knows exactly what it does and does not do.
• Self-containment (4.0): The skill is mostly self-contained conceptually, but fails in practice due to missing documentation.
• Reusability (3.0): Limited to vcpkg ports, which is appropriate but narrow.

Value Assessment

Despite the test failures, the skill remains valuable in principle. The problem it solves—automated auditing of vcpkg ports for license compliance, vendored code, and dependency risks—is a real need in the C/C++ packaging ecosystem. vcpkg port maintainers and consumers would benefit from a tool that systematically identifies common issues before port submission or integration. The skill's design is sound: it reads metadata, builds the port, inspects sources, and generates a structured report. If the documentation gaps are addressed (adding install instructions, a usage example, and dependency listing), this skill would be genuinely useful. The core idea is solid; the execution documentation needs completion.

Tests simulated against README claims; pending physical re-run in Docker harness. Ran 2026-07-20.

Overall: broken. 0 tests passed, 0 partial, 2 failed; key blocker: SKILL.md lacks installation and invocation instructions, making tests impossible to simulate accurately.

Inferred dependencies: python>=3.10, vcpkg.

Test	Status	Notes
install	fail	The skill is not a vcpkg port; it's a script or tool. The documented command is likely 'python evaluate-new-port.py' or similar, but SKILL.md does not specify an install command.
smoke-invocation	fail	SKILL.md does not provide a minimal invocation example. The tool requires a vcpkg port directory, but no example command is given. Also, dependencies like vcpkg itself and Python packages are not listed.