Eval-Design Forensics — does the evaluation measure what the paper claims?

Audit evaluation-design and reporting validity for: $ARGUMENTS (requires claims.json from /evidence-ledger). Emit span-anchored eval-design-forensics.findings.json. This skill computes no verdict.

🔒 Do not wrap this skill in /loop, /schedule, or CronCreate. It is verdict-bearing input — it proposes the findings the deterministic adjudicator turns into the report. Re-firing it on a wall-clock timer adds no signal: its output changes only when the paper / ledger changes (or a repo arrives, raising the observability level), not with the clock. Schedule the external wait that precedes it — ledger built (or artifacts released → L2) → audit once. (Mirrors ARIS's external-cadence doctrine.)

Adapted from the ML-evaluation-methodology literature — the leakage taxonomy of Kapoor & Narayanan (2023), the LLM-as-judge validity work (MT-Bench self-enhancement, self-preference, position bias), and the "Show Your Work" / reproducibility-checklist reporting norms — reframed to audit a third party's evaluation. A favourite autoresearch shortcut is to report a number that is arithmetically self-consistent (family A), runs real code against a real ground truth (family D), and still does not measure what it claims: the protocol leaks, the load-bearing metric is a conflicted/unvalidated LLM judge, or the reporting quietly drops a declared condition. This skill is the constraint that asks "is this a valid measurement of the claim?", pointed at a submission, and it stays honest — leakage and under-reporting are usually honest methodological errors, so every finding is a discrepancy to clarify, never an accusation.

Why this exists

An optimizing pipeline (or rushed human) treats the evaluation as a number to make go up, not a measurement to keep valid. The repeatable failure modes — distinct from "is the number real?" (family D) — are:

• Leakage — the train/test boundary is broken (preprocessing fit before the split, no held-out set, duplicates across splits, a random split over time-ordered data, the same subject in both splits, an evaluated LLM that saw the benchmark in pretraining), so the reported score may not measure generalization at all. HP-EVAL-LEAKAGE
• Judge validity — the headline rests on an automatic LLM judge that is conflicted (the same model/family as a compared system, so its preference for that system is the "evidence") or unvalidated (no human-agreement correlation, no position/length bias control). HP-JUDGE-VALIDITY
• Selective reporting — a dataset / baseline / metric / seed-count the setup explicitly declares is dropped from the results, the metric is switched across tables to keep the method ahead, or "we report the best run/prompt/ checkpoint" with no held-out selection set (selecting on the test set). HP-SELECTIVE-REPORTING

None of these is inherently misconduct — they are what an agent does when nothing forces a valid evaluation. The stated version is decidable at L0/L1 from the described protocol; the verified version (real split/preprocessing/result files) deepens at L2. What this skill will not do is guess: three leakage subtypes are undecidable even with the repo and are handed off as needs_external_check, not invented (see below).

Core principle

Ledger-anchored, span-verified, reviewer≠adjudicator, honest about what it cannot settle. Four properties:

1. Anchor to a PAPER claim. Every above-info finding cites a ledger claim_id and quotes a verbatim span of that claim's text_span (references/integrity-forensics-contract.md rules 1–2). The leak/judge/reporting tell lives in the protocol / setup-description — usually method and scope claims, with comparison / number for the judge metric and caption / table_cell / baseline for reporting. The anchor is whichever paper claim the finding undermines; a split-file file:line, a config, or a leaderboard date is forensic context for the description, never the anchor.
2. The executor assembles facts; the reviewer judges. At L2 the executor gathers mechanical split/preprocessing/judge/result facts (grep/hash — listing what exists is a fact, not a judgment) and may record one public-record date fact (a benchmark's release vs a model's cutoff, for the contamination FP guard). It passes paths + the ledger + those facts + the checklist to the reviewer and never pre-declares "this leaks" (references/reviewer-independence.md). The model proposes; tools/adjudicate_findings.py decides. This skill computes no verdict.
3. Undecidable leakage subtypes → hand off, don't guess. An illegitimate-proxy feature, sampling bias in the test set, and pretraining/benchmark contamination are domain / black-box judgments not settleable from the PDF or the repo. Emit verdict_local: needs_external_check + requires_external_check: true (contract rule 6); name the external methods a domain check would use — exchangeability (Oren 2023), Min-K% Prob (Shi 2023), Time-Travel (Golchin 2023), BIG-bench canary strings — and never run them.
4. Verdict-bearing at L0/L1; observability still caps the L2-confirm. Unlike experiment-forensics (no eval code at L0/L1 ⇒ info-only), a stated-tell here is decided from the described protocol and emits observability_level_required: 0. The L2 confirmation of the same leak/ omission is a separate finding with observability_level_required: 2 that is marked as needing L2 on a PDF-only run (references/observability-levels.md). So a PDF-only run keeps the stated-tell as a flag and the verification as an info "confirm-at-L2" pointer — never the reverse.

How this differs from the other auditors (route correctly)

This skill is the L0/L1-stated / L2-verified sibling of baseline-comparison-audit and proof-derivation-forensics (both verdict-bearing without a repo) — not the L2-only experiment-forensics.

Auditor	Question it answers	Level
eval-design-forensics (this)	Is the evaluation a VALID measurement of the claim, and is the reporting complete? (train/test leakage, conflicted/unvalidated LLM judge, declared-but-unreported / metric-switch / best-without-held-out)	L0/L1 stated · L2 verified
experiment-forensics	Are the reported numbers what the code computes? (fake/derived GT, self-norm, phantom, dead metric)	L2
consistency-audit	Does the paper contradict ITSELF / described method = evaluated method? (owns HP-AGG-DRIFT, HP-APPENDIX-CONTRA, text-only HP-SCOPE-INFLATE)	L0
baseline-comparison-audit	Are the right baselines present, fairly tuned, and is "SOTA" earned? (owns HP-MISSING-BASELINE, HP-SIG-OVERLAP)	L0 stated / L2 verified
citation-forensics	Do the cited papers exist and support the claim?	L0
presentation-signals	Surface "AI-flavor" hints (auxiliary, surface-class)	L0
adversarial-case-builder	Strongest evidence-bound rejection memo (no verdict weight)	any

Do NOT raise here (hand off instead):

• An LLM generating the GROUND-TRUTH labels/targets (not judging outputs) → experiment-forensics HP-FAKE-GT (L2). The clean split: a judge whose preference IS the reported metric is HP-JUDGE-VALIDITY (here, L0/L1 stated); a model that fabricates the reference the metric is computed against is HP-FAKE-GT (there, needs the code, L2). When unsure which, prefer the L2 route and set needs_external_check.
• best-reported-as-mean (the aggregation lies) → consistency-audit HP-AGG-DRIFT; thin overall scope with no comparison → consistency-audit HP-SCOPE-INFLATE; appendix-vs-main disagreement on the same quantity → consistency-audit HP-APPENDIX-CONTRA.
• A never-mentioned expected SOTA baseline (completeness) → baseline-comparison-audit HP-MISSING-BASELINE; a "consistently/across-the-board" comparison resting on one dataset → baseline-comparison-audit's single-dataset HP-SIG-OVERLAP.
• Whether a reported number matches the code (fake GT, self-norm, phantom) → experiment-forensics (L2); whether a cited paper exists / is used in context → citation-forensics; surface / AI-flavor → presentation-signals.

HP-SELECTIVE-REPORTING is scoped to declared-but-unreported / cherry-picked- among-shown — the gap between what the setup promised and what the tables deliver. It never re-emits the four patterns above.

The Kapoor & Narayanan leakage taxonomy (adopted — paraphrased)

HP-EVAL-LEAKAGE adopts the eight leakage types in three categories of Kapoor & Narayanan (2023), paraphrased. The reviewer maps each finding to one type and records it in the description.

K&N category (the leakage TYPE)	The tell (subtypes)	This repo's observability	Common false positive
L1 — no clean train/test separation	(a) no held-out test set at all; (b) preprocessing (scaling / imputation / resampling) fit on all data before the split; (c) feature selection fit before the split; (d) duplicate / near-duplicate records across splits	L0 stated / L2 verified	a transductive / semi-supervised design where overlap is intended and declared; preprocessing fit on train only, then applied to test (the correct pattern)
L2 — illegitimate (proxy) feature	a feature that stands in for the target, or would be unavailable at prediction time	needs_external_check (domain judgment)	a "proxy-looking" feature that is genuinely available at prediction time
L3 — test set not from the distribution of interest	(a) temporal leakage (random split over time-ordered data / training on the future); (b) non-independence (same subject / patient / group in both splits); (c) sampling bias in the test set	(a),(b) L0 stated / L2 verified; (c) needs_external_check	a correctly time-respecting split; a standard fixed benchmark split the field uses
(LLM-specific) pretraining / benchmark contamination	the evaluated model may have seen the public benchmark during pretraining	needs_external_check (black-box) — name Oren 2023 (exchangeability), Shi 2023 (Min-K%), Golchin 2023 (Time-Travel), BIG-bench canary; never run them	a benchmark released after the model's training cutoff, or a corpus documented to exclude it

⚠️ Two scales — do not conflate them. K&N's L1 / L2 / L3 are leakage-type labels (severity-ordered categories of leak). This repo's L0 / L1 / L2 are observability levels (what you can see: PDF / +source / +repo). They are orthogonal. A K&N-L1 preprocessing leak that is stated in the protocol is decidable at observability-L0. Every finding carries both: the K&N type in description, the observability in observability_level_required.

Constants & Reviewer Calling Convention

REVIEWER_MODEL        = gpt-5.6-sol                  # different family from executor (Claude)
REVIEWER_REASONING    = xhigh                    # always; effort never lowers reviewer quality
REVIEWER_SANDBOX      = read-only                # detect-only; never mutate the paper
REVIEWER_CWD          = <paper-dir>              # so it can read claims.json + the protocol/source directly
THREAD_POLICY         = fresh mcp__codex__codex per PASS (and per entry on fan-out);
                        NEVER mcp__codex__codex-reply across passes/entries (the bias guard)
TAXONOMY_VERSION      = 0.5                      # references/hack-pattern-taxonomy.md (family H)
LEAKAGE_TAXONOMY      = Kapoor & Narayanan 2023  # 8 types / 3 categories — adopted, paraphrased
PATTERNS_OWNED / ALLOWED = HP-EVAL-LEAKAGE, HP-JUDGE-VALIDITY, HP-SELECTIVE-REPORTING   # emit ONLY these
DIMENSION             = evaluation              # SKILL_TO_DIMENSION["eval-design-forensics"]
FINDINGS_FILE         = eval-design-forensics.findings.json
FINDING_ID_NAMESPACE  = ED###                    # distinct from F###/NUM###/HL### (consistency), EF### (experiment), BC### (baseline), PD### (proof)
VERDICT_BEARING_AT    = L0/L1 (stated-tells)     # NOT repo-gated; L2 only CONFIRMS
TRACE_POLICY          = forensic (never silently dropped)
TRACE_DIR             = .aris/traces/eval-design-forensics/<YYYY-MM-DD>_run<NN>/

• Executor (Claude) builds none of the judgment: it locates the ledger, extracts the evaluation surface (protocol / judge / declared-condition claims), at L2 gathers mechanical split/preprocessing/judge/result facts (grep/hash) and at most one public-record date fact for the contamination guard, passes **paths + the ledger
  • those facts + the checklist** to the reviewer, validates the reviewer's spans, and writes the findings file. It never summarizes the paper, pre-judges "this leaks", or leaks an opinion into the prompt (reviewer-independence.md). Passing a public release date (with its source) is the same allowed division citation-forensics (canonical metadata) and baseline-comparison-audit (leaderboard dates) use — reference facts, not hunches.
• Reviewer (codex / gpt-5.6-sol) reads claims.json and the source (and, at L2, the split/preprocessing/judge/result files) directly from its cwd, decides which evaluations leak / rest on a conflicted-or-unvalidated judge / under-report, applies the known false-positive cases, and self-reports false_positive_risk. It is the evidence-extractor, not the judge.
• Fresh thread per pass. Leakage (Step 3) and judge-validity + selective-reporting (Step 4) are separate fresh mcp__codex__codex calls. On — effort: max or many evaluation tracks, fan each track entry out into its own fresh call — never codex-reply carrying one entry's conclusion into another (the bias guard). codex-reply is intentionally absent from allowed-tools.

---

Step 0 — Preconditions: locate the ledger, read the run level

The ledger is the only structure this skill reasons over. Resolve it and read the observability level L and paper_id it was built at (each Bash block is self-contained — shell state does not persist, so re-derive paths every block):

ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
# $ARGUMENTS is a paper-dir OR a claims.json path:
LEDGER="$ARGUMENTS"; [ -d "$LEDGER" ] && LEDGER="$LEDGER/claims.json"
# Only the NO-ARGUMENT case defaults to the CWD ledger. An EXPLICIT argument that
# resolves to a missing claims.json must NOT silently fall back to $(pwd) — that
# could audit the wrong paper; let the NO_LEDGER check below fire instead.
[ -z "$ARGUMENTS" ] && LEDGER="$(pwd)/claims.json"
python3 - "$LEDGER" <<'PY'
import json, sys, os, collections
p = sys.argv[1]
if not os.path.isfile(p):
    sys.exit("NO_LEDGER: claims.json not found. Run /evidence-ledger FIRST "
             "(it writes artifact_manifest.json + claims.json).")
d = json.load(open(p, encoding="utf-8"))
claims = d.get("claims", [])
by = collections.Counter(c.get("type") for c in claims)
print("LEDGER       =", os.path.abspath(p))
print("PAPER_DIR    =", os.path.dirname(os.path.abspath(p)) or ".")
print("PAPER_ID     =", d.get("paper_id", "?"))
print("RUN_LEVEL_L  =", d.get("observability_level", 0))
print("CLAIMS       =", len(claims), dict(by))
# applicability signal — method/scope carry the protocol; comparison/number the judge
# metric; caption/table_cell/baseline the reported conditions:
rel = sum(by.get(t, 0) for t in ("method", "scope", "comparison", "number", "table_cell", "caption", "baseline"))
print("APPLICABLE   =", "yes" if rel else "low (no protocol/scope/comparison/table claims)")
PY

Failure handling. If NO_LEDGER is printed, stop and tell the user to run /evidence-ledger first — this skill never re-reads the raw PDF and invents its own structure (contract rule 1). Carry L, PAPER_ID, and the absolute LEDGER / PAPER_DIR into every step below.

Step 1 — Extract the evaluation surface from the ledger (decide whether to run)

Pull the claims this audit reasons over — the protocol (leakage anchors), the judge (validity anchors), and the declared conditions (reporting anchors) — and decide if there is anything to audit. This is a mechanical surface scan; the reviewer decides validity:

LEDGER="<abs path to claims.json from Step 0>"
python3 - "$LEDGER" <<'PY'
import json, re, sys, collections
d = json.load(open(sys.argv[1], encoding="utf-8"))
claims = d.get("claims", [])
LEAK = re.compile(r"\b(train(?:ing)?[\s/_-]*(?:and[\s/_-]*)?test|train[\s/_-]*test|split|held?[\s-]*out|"
                  r"cross[\s-]*validat|k-?fold|preprocess|standardi[sz]|normali[sz]|imput|"
                  r"resampl|oversampl|smote|feature[\s-]*select|leak|duplicat|de-?dup|"
                  r"temporal|time[\s-]*(?:series|order)|contaminat|pre-?train|data\s+split)\b", re.I)
JUDGE = re.compile(r"\b(LLM[-\s]*as[-\s]*a?[-\s]*judge|as\s+(?:a\s+)?judge|automatic(?:ally)?\s+(?:judg|evaluat|scor|rat)|"
                   r"GPT-?4o?|GPT-?3\.5|Claude|Gemini|win[\s-]*rate|pairwise|preference|"
                   r"rated\s+by|scored\s+by|judged\s+by|LLM\s+(?:judge|evaluator|grader))\b", re.I)
DECLARE = re.compile(r"\b(we\s+(?:evaluate|report|test|measure|use)|datasets?|benchmarks?|metrics?|"
                     r"seeds?|over\s+\d+\s+(?:seed|run)|best\s+(?:checkpoint|prompt|run|model|epoch)|"
                     r"five|four|three|\{[^}]*\})\b", re.I)
leak_a, judge_a, report_a = [], [], []
for c in claims:
    t, span = c.get("type"), c.get("text_span", "")
    sec = c.get("location", {}).get("section", "?")
    if t in ("method", "scope") and LEAK.search(span):
        leak_a.append((c["claim_id"], t, sec, span[:160]))
    if t in ("comparison", "scope", "method", "number") and JUDGE.search(span):
        judge_a.append((c["claim_id"], t, sec, span[:160]))
    if t in ("scope", "method", "caption", "table_cell", "baseline") and DECLARE.search(span):
        report_a.append((c["claim_id"], t, sec, span[:160]))
print(f"LEAKAGE anchors: {len(leak_a)}  JUDGE anchors: {len(judge_a)}  REPORTING anchors: {len(report_a)}")
for tag, rows in (("leak", leak_a), ("judge", judge_a), ("report", report_a)):
    for cid, t, sec, sp in rows[:30]:
        print(f"  [{tag}:{t}] {cid} [{sec}] {sp!r}")
print("APPLICABLE   =", "yes" if (leak_a or judge_a or report_a) else "no -> write [] and stop")
PY

Branch. If APPLICABLE = no (no protocol / judge / declared-condition claims), this skill is not applicable: write an empty eval-design-forensics.findings.json ([]), record a one-line NOT_APPLICABLE reason in the trace (Step 7), and stop. Silent skip is forbidden — the orchestrator globs *.findings.json and expects the file to exist. Otherwise record the three anchor lists for the prompts. A purely mechanical grep of the source helps surface the protocol/table language (do not judge validity here — that is the reviewer's job):

LEDGER="<abs path to claims.json from Step 0>"
grep -rInE 'train[ /_-]*test|split|held[ -]*out|cross[ -]*valid|preprocess|standardi|normali|impute|leak|duplicat|as a judge|win rate|pairwise|we (evaluate|report) on|best (checkpoint|prompt|run)' \
    "$(dirname "$LEDGER")" --include='*.tex' --include='*.txt' 2>/dev/null | head -60

Step 2 — Gather mechanical facts (L2 split/preprocessing/judge/result; optional date fact)

Create the run's trace dir now — its first use is the facts file written just below, so it must exist before Step 7. Reuse this exact RUNDIR in Steps 3–7 (do not create a second one):

DATE=$(date +%Y-%m-%d); N=1
while [ -d ".aris/traces/eval-design-forensics/${DATE}_run$(printf %02d $N)" ]; do N=$((N+1)); done
RUNDIR=".aris/traces/eval-design-forensics/${DATE}_run$(printf %02d $N)"; mkdir -p "$RUNDIR"
echo "RUNDIR = $RUNDIR"   # carry this exact path forward (shell state does not persist)

At L2 only (repo + result files present), gather raw, uninterpreted facts — paths

• grep/hash only, the same executor/reviewer division as experiment-forensics. Skip this block at L0/L1 (there is no code to read — the reviewer block gets "L<2: ..."):

ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
PAPER_DIR="<abs PAPER_DIR from Step 0>"; L="<L from Step 0>"; RUNDIR="<the RUNDIR above>"
if [ "$L" = "2" ]; then
  # (a) split / preprocessing / dedup ordering FACTS (the K&N-L1 / L3 tells) -> leakage_grep.txt
  grep -rInE 'train_test_split|StratifiedKFold|KFold|GroupKFold|TimeSeriesSplit|\.split\(|train/val|holdout|hold-out|'\
'StandardScaler|MinMaxScaler|fit_transform|\.fit\(|SimpleImputer|impute|SMOTE|resample|oversampl|'\
'SelectKBest|feature_select|drop_duplicates|duplicated\(|dedup|shuffle=True|random_state' \
      "$PAPER_DIR" --include='*.py' --include='*.ipynb' 2>/dev/null | head -80 > "$RUNDIR/leakage_grep.txt"
  # (b) LLM-judge calling code FACTS -> judge_grep.txt
  grep -rInE 'as_judge|llm_judge|judge_model|gpt-?4|gpt-?3\.5|claude|gemini|openai|anthropic|'\
'pairwise|win_rate|preference|rate_response|score_response|annotate' \
      "$PAPER_DIR" --include='*.py' --include='*.ipynb' --include='*.yaml' --include='*.json' 2>/dev/null | head -60 > "$RUNDIR/judge_grep.txt"
  # (c) which DECLARED conditions actually produced result files (the selective-reporting L2 confirm) -> reporting_grep.txt
  { find "$PAPER_DIR/results" "$PAPER_DIR/outputs" "$PAPER_DIR/logs" -type f \( -name '*.json' -o -name '*.csv' \) 2>/dev/null | sort | head -60
    echo "## metric/dataset keys present in result files:"
    grep -rIhoE '"(dataset|benchmark|metric|seed|split|task)"[^,}]{0,40}' \
      "$PAPER_DIR/results" "$PAPER_DIR/outputs" "$PAPER_DIR/logs" 2>/dev/null | sort -u | head -60; } > "$RUNDIR/reporting_grep.txt"
  # (d) reproducibility anchors: hash each discovered file (space-safe; tolerant of zero matches)
  { grep -rIlE 'split|scaler|judge|metric' "$PAPER_DIR" --include='*.py' 2>/dev/null | head -n 20
    find "$PAPER_DIR" -maxdepth 3 -path '*results*' -name '*.json' 2>/dev/null | head -n 20; } \
    | while IFS= read -r ff; do shasum -a 256 "$ff" 2>/dev/null; done > "$RUNDIR/hashes.txt"
  echo "L2 facts -> $RUNDIR/{leakage_grep,judge_grep,reporting_grep,hashes}.txt"
else
  echo "L<2: stated-tell pass only (no split/judge/result files to read)."
fi

Optional contamination date fact (the FP guard, not a detector). If a benchmark is named and an evaluated model's training cutoff is knowable, you MAY record one public-record date fact — WebSearch/WebFetch for " release date" and " training cutoff" — and write it (URL + access date) to $RUNDIR/contamination_dates.json. This is a fact that suppresses a false contamination flag (benchmark released after the cutoff → legitimate); it is never a contamination detector. The skill never runs Min-K% / exchangeability / Time-Travel.

Failure handling. No network → skip the date fact; the reviewer treats contamination as needs_external_check regardless. Empty L2 greps (a thin repo) → pass "L2 but no split/judge/result files found" to the reviewer so it does not invent a leak.

Step 3 — Leakage pass (cross-model, fresh thread) → HP-EVAL-LEAKAGE

Open a fresh mcp__codex__codex thread (Reviewer Calling Convention). The reviewer reads claims.json from its cwd for the described protocol and, at L2, the split/ preprocessing files; every finding anchors to a ledger claim_id. Send EXACTLY (fill every [ ... ]):