Eval: speckit-extension

The regression net for the speckit-extension (the spec-kit-side half of SpecKit Companion). It verifies the assumptions behind each feature against a real spec the user created, combining a deterministic file checker with a conversational cross-check of what the user actually did this session. Re-run it whenever a lifecycle/auto-mode/future feature ships.

Eval areas (add a section per feature as the extension grows):

1. Lifecycle capture (shipped) — hooks → write-context.py → canonical history[]; per-task journaling; derive-from-files fallback.
2. Command quality (shipped) — check_quality.py: verbosity budgets on the spec artifacts, time-waste checks on history[] (untrusted spans, bursts, outliers), static prompting checks on the command-body sources.
3. Auto-mode (future) — add checks here when it lands.
4. …

Steps

1. Resolve the target spec

• If the user named a spec (NNN or a dir), use specs/<that>/.
• Else pick the spec the user most recently created/ran in this conversation (whose /speckit.* or /sdd:* commands appear above). Fall back to the most recently modified specs/*/ containing .spec-context.json.

2. Run the deterministic checker

python3 .claude/skills/eval-speckit-extension/check_capture.py specs/<NNN>-<slug>/

Read the PASS/FAIL/INFO rows and the timing breakdown. (--json for machine output; --strict to exit non-zero on any FAIL.)

2b. Run the command-quality checker

python3 .claude/skills/eval-speckit-extension/check_quality.py --feature-dir specs/<NNN>-<slug> --commands-dir speckit-extension/commands

Same report shape plus a WARN tier (judgment calls — budgets, duration outliers, untrusted spans; never affects --strict, which fails only on FAIL). Three dimensions: verbosity (spec/plan/tasks against line/char WARN/FAIL bands calibrated on the completed specs 484/509/510 — only oversize flags; a missing or lean artifact is fine), time-waste (each reached step's span trusted only with ordered extension-stamped boundaries; ≥3 by:ai task finishes inside one second FAIL as the pre-#509 burst shape; a step >8× the median of the others and >5m WARNs), and prompting (never-halts commands — the after-* hooks, living-drift/sync/coverage, mark-complete, status, resume, classify — must contain no ask-the-user instruction; the clarify carrier must ask; negated mentions and fenced templates don't count; a missing roster file fails loudly). CI runs all three strict over specs/509-timing-capture, specs/510-living-sync, and the shipped command sources.

3. Conversational cross-check

The checker can't know what the user did. Reconstruct it from this conversation and reconcile against the capture:

• List every /speckit.specify|plan|tasks|implement (or /sdd:*) the user ran this session, with rough timestamps.
• For each, confirm a matching history[] entry exists with the expected step/status and by: "extension" (hook fired) — or by: "ai" if the AI appended it, by: "derive" if reconstructed. (The canonical by vocabulary is extension/user/cli/ai/derive.)
• Timing reality: does each capture's at land just after the command ran (live hook), or are timestamps clustered/round (backfilled)? The timestamps-real check flags round-ms; you add the "fired at the right moment" judgement.
• Task execution: if /speckit.implement ran, confirm per-task substep entries appear. Read task-cadence's source: live (by:ai) means the AI journaled each task as it finished — non-zero gaps are the real-cadence signal; hook burst (by:extension) means the single end-of-step hook synced them — 0ms gaps are expected there and not a defect (the AI didn't journal live, so the hook backstopped). Only a mixed or duplicated picture is a smell.

4. Score the assumptions

Render a verdict table (PASS / PARTIAL / FAIL + one-line evidence):

#	Assumption (lifecycle capture)
A1	Each lifecycle step the user ran produced a capture (after_specify/plan/tasks/implement → a history[] entry).
A2	Writes use canonical history[] with explicit kind — no legacy transitions[]/stepHistory.
A3	Timing is real for deterministic writes (by:extension/derive/cli/user): ms-precision and monotonic (timestamps-real/timestamps-monotonic check these only). by:ai entries carry second precision (date -u +%SZ) and may burst — that's graded by task-cadence, not failed. See docs/capture-and-timing.md.
A4	/speckit.implement journaled per-task progress as implement substeps (substep == task id), matching tasks.md completed markers.
A5	No-backward-clobber held — no step regressed; an advanced/terminal spec was never dragged back.
A6	(On demand) derive-from-files.py reconstructs the same state from artifacts when a hook didn't fire. Test: back up .spec-context.json, delete it, run python3 speckit-extension/scripts/derive-from-files.py --feature-dir specs/<NNN>-<slug>, diff, restore.
A7	Per-task journaling is deduped — each task carries one start + one complete, so dedup is checked per (task, kind) (per-task-no-duplicates); a repeated (task, kind) means the after_implement hook re-added an already-journaled entry. Live AI entries (by: ai, real date -u timing) are the cadence source. Real per-task cadence requires task-cadence source live (by:ai) with non-zero gaps; a hook burst source means the AI didn't journal live and the hook backstopped (correct final state, coarse timing — not a defect).

End with a one-paragraph plain verdict: did the extension work, and where reality diverged from the assumptions.

Extending

When a feature ships:

• add a deterministic assertion to check_capture.py (run_checks) and, if it's a new area, a new eval section + assumption block above,
• keep VALID_BY / CANONICAL_STEPS / CANONICAL_STATUSES in check_capture.py in sync with src/core/types/spec-context.schema.json (the jest drift guard locks schema↔TS; this is the eval side).