omcustom:agent-eval-framework

Purpose

Provides quantitative, trajectory-based evaluation for oh-my-customcode agents. Fills the measurement gap not covered by existing skills:

Skill	Coverage	Gap
harness-eval	SE benchmark task quality (15 tasks)	No efficiency metrics
evaluator-optimizer	Qualitative rubric loop	No quantitative gate
deep-verify	Release quality (structural/correctness)	No step/latency ratios
multi-model-verification	Code correctness across models	No trajectory comparison

This skill adds efficiency measurement — not just "did the agent succeed?" but "how efficiently did it succeed relative to an ideal trajectory?"

The 4-Metric Framework

Derived from LangChain's deep agent evaluation methodology.

Metric	Formula	Direction	Use
correctness	pass / fail per task	binary	Phase 1 gate — must pass before efficiency
step_ratio	observed_steps / ideal_steps	lower is better	Measures unnecessary reasoning hops
tool_call_ratio	observed_tool_calls / ideal_tool_calls	lower is better	Measures redundant tool invocations
latency_ratio	observed_latency_s / ideal_latency_s	lower is better	Measures wall-clock efficiency

Thresholds (recommended starting points):

• step_ratio ≤ 1.5 — acceptable
• tool_call_ratio ≤ 1.3 — acceptable
• latency_ratio ≤ 2.0 — acceptable (parallelism headroom)

Ideal Trajectory Annotation

Each task requires a hand-annotated ideal trajectory stored in .claude/outputs/evals/trajectories/.

task_id: example-001
capability: file_operations
ideal:
  steps: 4
  tool_calls: 4
  latency_seconds: 8
description: "Refactor user.py — read, parse, edit, verify"

Annotation guidelines:

• steps: count of distinct reasoning/action steps in an expert run
• tool_calls: minimum tool invocations required (no redundant reads)
• latency_seconds: median of 3 expert runs
• capability: one of the six taxonomy categories below

Capability-Categorical Taxonomy

Maps LangChain capability categories to oh-my-customcode tools and task types.

Capability	Tools	Example Tasks
file_operations	Write, Edit	Refactor, create files, patch configs
retrieval	Glob, Grep, Read	Code search, dependency analysis, symbol lookup
tool_use	Agent, Skill, Bash	Multi-tool workflows, pipeline execution
memory	Read/Write to .claude/agent-memory*/	Context recall, R011 patterns, cross-session refs
conversation	routing skills (secretary/dev-lead/de-lead/qa-lead)	Multi-turn user interaction, intent routing
summarization	result-aggregation skill	Multi-agent synthesis, parallel result merge

Use this taxonomy to select representative tasks per category when building an eval suite. Aim for ≥3 tasks per capability category.

Tool: Writing artifacts under .claude/outputs/

Under mode: "bypassPermissions", direct Write/Edit/Bash on .claude/** paths is permitted (CC v2.1.121+, #1101) — no /tmp/*.sh wrapping is needed.

To write eval trajectories or result reports under .claude/outputs/evals/:

1. Write the artifact body directly to .claude/outputs/evals/{trajectories,sessions}/... with the Write tool (every Agent tool call includes mode: "bypassPermissions", R010)
2. Read-only Bash on .claude/outputs/ (e.g., cat, head, wc) is allowed for verification

Reference: R006/R010 sensitive-path handling (CC v2.1.121+), #1101.

Phased Opt-in Gate Workflow

Phase 1: Correctness Gate (MUST pass before Phase 2)

1. Run agent against task set
2. Score each task pass/fail
3. Require correctness ≥ 0.80 (80%) before proceeding
4. If below threshold: diagnose failure modes, fix agent, re-run

Phase 2: Efficiency Comparison

1. Compute step_ratio, tool_call_ratio, latency_ratio per task
2. Aggregate by capability category
3. Compare against baseline (previous agent version or reference agent)
4. Flag regressions (ratio increase > 20%)

Consumers of this workflow:

• mgr-creator: candidate consumer — opt-in for new agents requiring quantitative gating before deployment
• worker-reviewer-pipeline: gate review cycles — efficiency regression = re-review trigger
• deep-verify: optional quantitative dimension for release quality checks

Tracing Infrastructure (LangSmith Alternative)

oh-my-customcode uses existing infrastructure for trajectory capture:

Component	Role	How
native MEMORY.md (R011)	Persist eval metrics across sessions	Write ## Metrics section with task_id + observed metrics to agent-memory MEMORY.md
statusline.sh (R012)	Real-time step counter during eval runs	Extend statusline with STEPS=n segment
.claude/outputs/evals/	Artifact storage for eval results	Per-session eval reports in sessions/{YYYY-MM-DD}/

Trace capture pattern:

task start → record tool_calls[] + timestamps → task end
→ compute ratios against ideal trajectory
→ write to native MEMORY.md ## Metrics: {task_id, capability, correctness, step_ratio, tool_call_ratio, latency_ratio}

Persistent Storage (added v0.116.0, #1036)

Baseline annotations and observed trajectories can be persisted to eval-core's SQLite database (evalBaselines + agentTrajectories tables). This complements the YAML file approach for cross-session analysis. The eval-core query API for the evalBaselines/agentTrajectories tables is not yet implemented — use the YAML annotation approach meanwhile.

Integration with Existing Skills

Skill	Integration Mode	How
harness-eval	Additive	After harness-eval runs 15 SE tasks, apply 4-metric layer to each result
evaluator-optimizer	Additive	After rubric loop converges, run efficiency gate as final check
deep-verify	Optional	Add --quantitative flag awareness; deep-verify can invoke this skill
mgr-creator	Gate	New agent creation includes Phase 1 correctness check before agent is deployed

Usage Pattern

/omcustom:agent-eval-framework measure <agent-name> <task-id>
/omcustom:agent-eval-framework compare <variant-a> <variant-b>
/omcustom:agent-eval-framework gate <agent-name>  # correctness → efficiency

measure: Runs a single agent against a single task, outputs all 4 metrics.

compare: Runs two agent variants against the same task set, produces side-by-side ratio table.

gate: Full two-phase gate — Phase 1 correctness check, then Phase 2 efficiency if Phase 1 passes. Returns PASS or FAIL with metric breakdown.

Example Output

[agent-eval-framework] gate lang-golang-expert
Phase 1: correctness = 0.87 (13/15) ✓ threshold: 0.80
Phase 2: efficiency
  step_ratio:      1.12 ✓
  tool_call_ratio: 1.08 ✓
  latency_ratio:   1.31 ✓
Result: PASS — agent approved for deployment

R020 Linkage

Quantitative metrics provide [Done] gate evidence beyond binary completion checks (R020 MUST-completion-verification).

R020 Task Type	4-Metric Evidence
Implementation	correctness ≥ threshold + step_ratio ≤ 1.5
Agent/Skill Creation	Phase 1 gate PASS (mgr-creator workflow)
Code Review	tool_call_ratio as efficiency signal for review thoroughness

When declaring [Done] for agent creation or major workflow changes, include eval gate results as completion evidence.

See R020 "Optional: Quantitative Evidence" section for the consumer-side advisory pattern.