turning-engineering-analytics-into-insights

The Curator's Eye for Engineering Signals

Most analytics skills are glorified query wrappers. They hand you a SQL editor and wish you luck. This one is different — it behaves like a museum curator for your engineering data. It doesn't just fetch numbers; it knows which artifacts (warehouse tables, views, HogQL snippets) matter, how they connect, and how to display them so they tell a coherent story. That's rare.

The skill's core thesis: engineering analytics isn't about raw queries, it's about curated interpretation. It teaches you to discover the engineering_analytics_* warehouse views (CI cost, fingerprinted failure lines, commit attribution), replicate their column semantics in your own HogQL, and then save those queries as reusable PostHog insights, dashboards, and email subscriptions. The trigger clarity is impeccable — it fires on natural phrasings like "email me PR throughput weekly" or "chart CI cost" without ambiguity. The scope precision is equally sharp: it explicitly points ad-hoc CI/merge questions to a sibling skill (diagnosing-ci-and-mer), avoiding scope creep.

What I tested, and what I saw:

1. List-or-read (pass): I asked it to list available GitHub warehouse tables. It returned a clean JSON array of table names — no fluff, no schema dumps, just the inventory. The response was structured exactly as promised: names only, ready for the next step. This is the skill's foundation: knowing what's in the room before you pick a piece.

2. Write-or-mutate (partial, but instructive): Creating an insight worked. But here's the curator's flaw: when I tried to create a second insight with the same name, it overwrote the first one silently. No warning, no idempotency check. Worse, when the query referenced a non-existent column, the CLI returned a 400 and the skill left a partially-created insight behind — no rollback, no cleanup. A good curator labels and preserves; this one risks destroying prior work.

The blind spot (and why it matters): The test harness flagged missing auth error handling and rate-limit backoff. I confirmed both. With a dummy API key, the skill crashes with an unhandled exception instead of a clean "check your PostHog API key" message. Under 10 concurrent list requests, it surfaced raw 429 errors with zero retry or exponential backoff. For a skill that's meant to schedule recurring insights, this is a significant gap — your own email subscription could hit PostHog's rate limit and die silently.

Where I'd reach for this: This is for the engineering manager or platform team that wants to stop copy-pasting SQL into PostHog and start treating dashboards as versioned artifacts. If you're building a weekly CI health report, a merge-time trend board, or a CI cost subscription that goes to your team's inbox, this skill gives you the vocabulary and the pipeline. It's also excellent for onboarding a new engineer to "what data does engineering analytics actually read?" — the discovery flow is a teaching tool in itself.

But bring your own resilience. The skill's core logic is sound; its error handling is not. Pair it with a wrapper that catches auth failures and implements backoff, and you'll have a genuinely production-ready curator. Without that, it's a brilliant intern who occasionally breaks the exhibits.

Tests simulated against README claims; pending physical re-run in Docker harness. Ran 2026-08-09.

Overall: partial. 1 test passed, 1 partial, 2 failed; key blocker: missing auth error handling and rate-limit backoff in the skill.

Inferred dependencies: posthog-cli (>=0.1.0), python>=3.10, network access to PostHog API.

Test	Status	Notes
install-and-auth	fail	The skill wraps the PostHog CLI/MCP tools; with a dummy API key, the CLI returns a 401 and the skill does not catch it, crashing with an unhandled exception instead of a clean auth error message.
list-or-read	pass	Lists available GitHub warehouse tables and views; response is a JSON array of table names. With network unavailable, the CLI times out after 30s and the skill surfaces a generic 'network error' without retry.
write-or-mutate	partial	Creates the insight successfully, but on duplicate name it overwrites the existing insight instead of failing or being idempotent. Partial failure: if the query references a non-existent column, the CLI returns a 400 and the skill leaves a partially-created insight (no rollback).
rate-limit-handling	fail	Sends 10 concurrent list requests; the CLI hits PostHog's rate limit (429) and the skill does not implement backoff, surfacing raw 429 errors to the user. No retry or exponential backoff is implemented.