Chapter 10
Skill Evals
Skill Evals
How this repo checks that its skills actually work — before they ship and on
every change after. Layout: one folder per skill, evals/<skill-name>/,
mirroring the skill's name. These files never ship in an install; the skill
folders contain only what runs at runtime.
The tier model follows addyosmani/agent-skills — same names, same jobs — plus two tiers of our own, because skills here ship executable code and his don't:
| Tier | What it checks | Runs | Cost |
|---|---|---|---|
| 1. Structural | Frontmatter, naming, name==dir, unfilled placeholders, text-only prompt dumps (tools/skill_lint.py --strict) | CI | Free |
| 1b. Security (ours) | Install lures, undeclared network calls, credential access, obfuscated payloads (tools/skill_scanner.py) | CI | Free |
| 2. Trigger & routing | Positive prompts clear near-miss negatives on description vocabulary; with 2+ skills, positives rank their own skill first and no two descriptions near-collide (tools/run_trigger_evals.py) | CI | Free |
| 2b. Deterministic scripts (ours) | The skill's bundled scripts do what they claim — every classifier, edge case, and output shape against synthetic fixtures (<skill>/test_*.py) | CI | Free, ~10s |
| 3. Behavioral | An agent following the skill satisfies its expectations[] — evals.json uses skill-creator's schema verbatim, so its run_eval.py, benchmarking, and eval viewer work against our files unmodified | On demand | Tokens |
Running
# Tiers 1–2b, exactly what CI runs — deterministic, git + Python only
python3 agent_skills/evals/tools/skill_lint.py agent_skills/project-graveyard --strict
python3 agent_skills/evals/tools/skill_scanner.py agent_skills
python3 agent_skills/evals/tools/run_trigger_evals.py
python3 agent_skills/evals/project-graveyard/test_graveyard.pyTier 3 is on demand and spends tokens: each skill's evals.json is in
skill-creator's schema, so run it with Anthropic's own tooling (install the
skill-creator plugin and point run_eval.py at the file), or by hand — fresh
agent session, paste each prompt, grade against expectations[]. Cases marked
lexical: false in trigger-cases.json (reasoning-triggered, e.g. necromancer
mode) are only covered here. Re-run tier 3 whenever SKILL.md behavior
changes; re-run tier 2 whenever a description changes.
Track record
Not theater: tier 1 caught a symlink-path bug that silently disabled relapse detection on macOS, and an external reviewer running it in a clean Linux checkout caught a filesystem-ordering bug that mis-attributed the kill-chain. Both fixed before merge. That's the job.
Adding a skill
New skill → new evals/<skill-name>/ with at minimum a deterministic
test_<skill>.py (self-contained, tempdir fixtures, exit 0/1) and a
trigger-cases.json. CI picks up test_*.py automatically.