Harness Eval — Structured SE Task Benchmark

Purpose

Evaluate agent quality using 15 structured software engineering task definitions with quantitative scoring. Based on research from revfactory/claude-code-harness which demonstrated 60% improvement (49.5 → 79.3 points) through structured pre-configuration.

Usage

/harness-eval                    # Run all 15 benchmarks
/harness-eval --preset quick     # Run top 5 high-impact benchmarks
/harness-eval --task api-design  # Run specific task benchmark

Quality Dimensions

Dimension	Weight	Description
Test Coverage	30%	Unit test count, edge case coverage, assertion quality
Architecture Design	25%	Separation of concerns, dependency management, scalability
Error Handling	25%	Input validation, error propagation, recovery strategies
Extensibility	20%	Plugin points, configuration flexibility, API surface

15 SE Task Benchmark Suite

#	Task	Category	Key Evaluation Criteria
1	API Design	Architecture	RESTful conventions, versioning, error responses
2	Data Modeling	Architecture	Schema normalization, relationships, indexing
3	Authentication Flow	Security	Token management, session handling, OWASP compliance
4	Test Suite Creation	Quality	Coverage breadth, assertion quality, edge cases
5	Error Handler	Reliability	Error classification, recovery, user feedback
6	Logging System	Observability	Structured logging, levels, correlation IDs
7	Configuration Manager	Operations	Env-based config, validation, secrets handling
8	CLI Tool	UX	Argument parsing, help text, exit codes
9	Database Migration	Data	Reversibility, data preservation, zero-downtime
10	Cache Layer	Performance	Invalidation strategy, TTL, cache-aside pattern
11	Queue Consumer	Reliability	Idempotency, retry logic, dead letter handling
12	Middleware Chain	Architecture	Composability, ordering, short-circuiting
13	File Processor	I/O	Streaming, error recovery, format validation
14	Webhook Handler	Integration	Signature verification, retry tolerance, idempotency
15	Rate Limiter	Security	Algorithm choice, distributed state, fairness

Scoring Rubric

Each task is scored 0-100 across the 4 quality dimensions:

Score = (test_coverage × 0.30) + (architecture × 0.25) + (error_handling × 0.25) + (extensibility × 0.20)

Score Thresholds

Score Range	Grade	Interpretation
80-100	A	Production-ready, well-structured
60-79	B	Functional with minor gaps
40-59	C	Works but needs improvement
0-39	D	Significant structural issues

Presets

all (default)

Run all 15 tasks. Full evaluation ~45 minutes.

quick

Run top 5 high-impact tasks (1, 3, 4, 5, 12). Quick evaluation ~15 minutes.

Integration with evaluator-optimizer

This skill provides preset rubrics for the evaluator-optimizer pipeline:

/harness-eval → loads rubric → evaluator-optimizer executes → scoring → report

The evaluator-optimizer skill's pre_negotiation phase accepts harness-eval rubric dimensions as sprint contract criteria.

Output

Results saved to .claude/outputs/sessions/{YYYY-MM-DD}/harness-eval-{HHmmss}.md with per-task scores and aggregate grade.

Tool: Writing artifacts under .claude/outputs/

Under mode: "bypassPermissions", direct Write/Edit/Bash on .claude/** paths (including .claude/outputs/sessions/) is permitted (CC v2.1.121+, #1101) — no /tmp wrapping is needed.

Write harness-eval results directly to .claude/outputs/sessions/$(date +%Y-%m-%d)/harness-eval-$(date +%H%M%S).md. Read-only Bash on .claude/outputs/ (e.g., cat, head, wc) is allowed for verification. Catastrophic shell operations (e.g., rm -rf /) remain blocked by independent safety guards. For CC < v2.1.121, see git history for the legacy /tmp/*.sh bypass pattern.

Reference: R006 Sensitive Path Handling, R010 Universal bypassPermissions, #1101.

4-Metric Quantitative Layer (added v0.113.0, #1025)

The 15 benchmark tasks defined here measure task correctness (pass/fail). For agent efficiency comparison and trajectory analysis, layer the 4-metric framework on top:

• correctness — existing benchmark pass/fail (unchanged)
• step_ratio — observed_steps / ideal_steps (efficiency)
• tool_call_ratio — observed_tool_calls / ideal_tool_calls (efficiency)
• latency_ratio — observed_latency / ideal_latency (efficiency)

Workflow

1. Run benchmark task → collect correctness result + trajectory (steps, tool_calls, latency)
2. Compare against ideal trajectory annotation (see agent-eval-framework skill)
3. Phase 1 gate: correctness MUST pass; Phase 2 gate: efficiency comparison among passing variants

Ideal Trajectory per Benchmark

For each of the 15 benchmark tasks, an ideal trajectory should be authored. Annotation schema:

task_id: <benchmark-id>
capability: <category>
ideal:
  steps: <int>
  tool_calls: <int>
  latency_seconds: <float>

Cross-references

• Skill: agent-eval-framework (4-metric framework definition)
• Guide: guides/agent-eval/README.md (measurement methodology)
• Issue: #1025

Attribution

Evaluation framework based on research by revfactory/claude-code-harness. Adapted for oh-my-customcode's evaluator-optimizer pipeline with permission.

Related Guide

• guides/harness-engineering/ — 하네스 엔지니어링 통합 가이드 (Benchmark Evaluation Layer 관점에서 harness-eval 위치)