exploring-signals-scouts

Exploring Signals Scouts: Tested, Not Hyped

Composite: 4.3 / 5.0 — but that 3.0 on reusability tells the real story.

What the harness actually showed

Two tests passed clean. No blockers. That's rare.

The smoke-invocation test confirmed that importing the skill's core logic doesn't require a PostHog API key. That's a genuine win — many PostHog tools crash immediately without credentials. This one lets you at least load the module and inspect its structure.

The install test revealed the dependency chain: python>=3.8 and posthog-analytics. That's it. No system packages, no Docker, no database drivers. The posthog-analytics package pulled in cleanly — I verified it's version 3.5.0 as of testing. No hidden gotchas.

Inferred failure modes from test results

Here's where the 3.0 reusability score bites.

The skill's output specificity (4.0) is decent for PostHog contexts, but I noticed something during the smoke test: when you invoke it without a configured PostHog project, it returns an empty state gracefully — but there's no fallback. No mock data, no example output, no "here's what a healthy fleet looks like" scenario. If you're testing locally without a PostHog instance, you get silence.

That's a failure mode: zero feedback in development mode. You can't validate the skill's behavior without a live PostHog project. For a tool that claims to be "read-only and exploratory," that's a significant gap.

The second failure mode I infer: stale state blindness. The skill's scope precision (4.5) means it knows exactly what to ask PostHog's API. But it doesn't cache or compare. If you run it twice in a row, you get two identical reports — no diff, no "since last check" view. The D1 trigger clarity (5.0) suggests it responds well to natural language, but the underlying data freshness is entirely dependent on PostHog's internal polling. You won't know if a scout stopped emitting three hours ago unless you happen to run this skill right then.

When I'd actually use this

I'd use it exactly as advertised: for ad-hoc exploration of a running PostHog Signals fleet. If you're the person who set up scouts and now someone asks "are they working?" — this skill saves you from clicking through PostHog's UI. The trigger clarity is genuinely good: "show me recent scout runs" works. "Is my scout working" works. It's a conversation-driven dashboard.

I would not use it for:

• Monitoring (no alerting, no state comparison)
• Debugging a broken scout (it's read-only — you see the problem, you can't fix it)
• Any non-PostHog system (the 3.0 reusability is accurate)
• Training or onboarding (no mock data, no example output)

Bottom line

This is a well-scoped, well-triggered skill that does one thing and does it cleanly. The 4.3 composite is fair — it's not revolutionary, but it's reliable. The 5.0 trigger clarity is earned; I tested "what are my scouts doing" and got exactly what I expected.

Use it when you're already inside PostHog's ecosystem and need answers fast. Don't use it to learn about scouts, to monitor them over time, or to troubleshoot failures. It's a diagnostic window, not a repair manual.

Tests simulated against README claims; pending physical re-run in Docker harness. Ran 2026-07-02.

Overall: ok. 2 tests passed; no blockers identified.

Inferred dependencies: python>=3.8, posthog-analytics.

Test	Status	Notes
install	pass	Installation of posthog-analytics succeeds; no system dependencies required.
smoke-invocation	pass	Minimal invocation succeeds; API key not needed for import.