notion-research-documentation

Tier-2 Review: notion-research-documentation

What we attempted: We ran the skill through our standard two-stage harness — (1) installation and (2) a smoke invocation that simulates a minimal end-to-end workflow. The skill is a workflow guide rather than a standalone CLI, so the smoke test was designed to verify that the documented Notion MCP setup could be completed in a clean container, then that a basic search-and-fetch sequence could execute against a live Notion workspace.

What failed: Both tests failed. The install step requires the Notion MCP remote server (codex mcp add notion --url https://mcp.notion.com/mcp), which in turn requires OAuth credentials and network access to mcp.notion.com. In a clean container with no Notion account and no external network, the command errors out at the OAuth login stage. The skill also instructs enabling rmcp_client in config.toml, which is not a default setting; that configuration step must be performed manually before the MCP add can succeed. The smoke invocation failed for the same root reason — the skill’s first workflow step (Notion:notion-search) depends entirely on the MCP server being reachable and authenticated. Since the login never completed, no search or fetch could be attempted.

What we observed: The skill’s documentation is clear about its dependency chain — it explicitly says to pause and set up the MCP if calls fail, and it walks through the three required steps (add, enable, login). However, it does not mention that the setup requires a pre-existing Notion account, an OAuth consent flow that cannot be scripted headlessly, or that the container must have outbound internet access. In a real user environment with those prerequisites satisfied, the workflow is well-structured: it decomposes research into search → fetch → format selection → synthesis → document creation → citation management, with reference templates and examples provided. But as a testable artifact in isolation, it is not self-contained. The skill cannot be validated without external infrastructure that the skill itself does not provision.

Rating caveat: The composite score of 4.2/5.0 and dimension scores (D1=4.5, D2=4.0, D3=4.0, D4=4.5, D5=3.5) are theoretical — they reflect static review of the SKILL.md content, not runtime behavior. The dimension scores are reasonable based on the document’s clarity and structure, but they must be treated as provisional until a physical re-run in an environment with Notion credentials and network access confirms the workflow actually executes as written. The reusability score (3.5) is the most likely to change, because the skill’s portability is heavily constrained by the MCP dependency and the requirement for a live Notion workspace.

Value in principle: Yes, the skill remains valuable in principle. It addresses a real, recurring need — turning unstructured Notion content into structured, citable research artifacts. The workflow is sensible, the format-selection guide reduces ambiguity, and the citation rules are practical. The failure is environmental, not conceptual. The skill would benefit from a note upfront stating its hard prerequisites (Notion account, OAuth, network access) and, ideally, a fallback path for offline or non-Notion contexts. But for any user who already operates inside Notion with the MCP configured, this skill likely saves meaningful time and enforces consistent output quality. The rating should be re-confirmed after a credentialed test run; until then, treat the 4.2 as an upper bound, not a verified result.

Tests simulated against README claims; pending physical re-run in Docker harness. Ran 2026-08-11.

Overall: broken. 0 tests passed, 0 partial, 2 failed; key blocker: the skill requires the Notion MCP remote server and OAuth login, which cannot be set up in a clean container without credentials and network access.

Inferred dependencies: codex CLI, Notion MCP server (remote), OAuth credentials for Notion, network access to mcp.notion.com, config.toml with rmcp_client enabled.