Skip to content

Evaluating Skills

Condensed from evaluating skill output quality. Load this when measuring whether a skill improves results, not merely whether it triggers. For trigger-only work, use descriptions.

Test Cases

Start with 2–3 cases. Each case needs a realistic prompt, a human-readable expected output, and optional input files.

Store them in evals/evals.json inside the skill directory. Vary phrasing and formality; include at least one boundary case. Vague prompts ("process this data") do not test the skill.

Add assertions only after seeing a first run. Good assertions are observable ("valid JSON", "chart has labeled axes"). Avoid "the output is good" and brittle exact-phrase checks.

Baseline

Run each case with the skill and without it (or against a snapshot of the previous version). Fresh context per run.

Record tokens and duration. A quality gain that triples cost is a different decision than a gain that is also cheaper.

Grading

PASS requires concrete evidence quoted from the output. Review the assertions themselves: always-pass checks inflate scores; always-fail checks are broken or too hard.

Mechanical checks (file exists, valid JSON, row count) belong in a script. Holistic quality can use a blind LLM comparison of two outputs without revealing which skill version produced them.

Patterns

  • Assertions that pass in both configurations do not measure the skill. Remove them.
  • Assertions that fail in both configurations are usually bad tests.
  • Pass-with / fail-without is the value of the skill. Identify which instruction or script caused it.
  • High variance across repeats means flaky evals or ambiguous instructions.
  • Time/token outliers belong in the execution transcript.

Human review catches correct-but-unhelpful output. Record specific complaints per case; empty feedback means that case is done.

Iteration

Failed assertions, human feedback, and transcripts are the input. Propose generalized fixes, not patches for one eval prompt.

  • Keep the skill lean. Plateauing pass rates after adding rules often means over-constraint.
  • Explain why an instruction exists.
  • If every run reinvented the same helper, bundle it under scripts/.

Stop when feedback is empty or iterations stop moving the metric.

skill-creator automates much of this loop.