Experiment Forensics — are the reported results what the code computes?

Audit experiment integrity for: $ARGUMENTS (a paper-dir or repo-dir; use an ABSOLUTE path — it is referred to as TARGET below). Emit span-anchored experiment-forensics.findings.json.

🔒 Do not wrap this skill in /loop, /schedule, or CronCreate. It is verdict-bearing input — it proposes the findings the deterministic adjudicator turns into the report. Re-firing it on a wall-clock timer adds no signal: what unlocks new conclusions is a higher observability level (a repo / result files arriving → L2), not elapsed time. Schedule the external wait that precedes it — artifacts released → run once at the new level. (Mirrors ARIS's external-cadence doctrine.)

Adapted from ARIS experiment-audit (#57/#131), reframed for the reviewer side. The original audits your own experiment before you claim results; this audits a third party's submission. The crucial reframe: at L0/L1 (no code) these patterns are not decidable — they appear only as info-level "could-not-verify" signals. Code-level fraud requires L2. A PDF can never produce a fraud verdict.

Why this exists

LLM-driven research pipelines (and rushed human work) produce results that look computed but are not what the paper claims. The repeatable failure modes — ported from ARIS's experiment-integrity audit — are:

1. Fake ground truth — the eval "reference/target" is derived from model outputs and reported as performance, not as a labeled proxy. HP-FAKE-GT
2. Score self-normalization — a metric divided by the model's own max/min/ mean to approach 1.0; no raw score shown. HP-SELF-NORM
3. Phantom results — a paper number maps to a result file or metric key that does not exist (or a function never called). HP-PHANTOM-RESULT
4. Dead metric code — a metric defined in eval code, discussed in the paper, but never called / never present in any result file. HP-DEAD-METRIC
5. Scope inflation (verified) — "comprehensive/robust/SOTA" while the repo actually ran 1–2 datasets/seeds/configs. HP-SCOPE-INFLATE
6. Method drift (confirmed) — the method described differs from the method evaluated (A-lite, A+oracle, extra data, different backbone, test-time labels the method claims not to use). HP-METHOD-DRIFT
7. Synthesized-looking results — numbers across configs related by a too-clean arithmetic pattern ("不像跑出来的"). HP-SUSPICIOUS-REGULARITY
8. Placeholder / fake data in released code — the released code still ships placeholder/dummy/fake data (e.g. a # fake data for plotting annotation, a TODO: replace with real data, a hard-coded np.random.* array) and a reported figure/number is drawn from it rather than from a real run. HP-PLACEHOLDER-DATA (flag the checkable code marker; do not infer who wrote it)
9. Result ≠ artifact — the code / result artifacts, read or run as released, produce numbers different from the paper's reported values for the same experiment. HP-RESULT-ARTIFACT-MISMATCH (an implementation that computes a different loss/normalization/architecture than the equations state is HP-METHOD-DRIFT, not this)
10. Missing reproducibility artifacts — an empirical / agent / LLM paper ships neither code nor the prompts/configs/hyperparameters its results depend on, so the claim cannot be reproduced even in principle (the absence is L0-stated; what its results specifically need is L2-verified). HP-MISSING-REPRO-ARTIFACT

These are NOT inherently misconduct — they are failure modes of optimizing agents that lack an integrity constraint. This skill is that constraint, pointed outward, and it stays honest about what it can and cannot see.

Core principle (two independence axes)

The executor (Claude) collects paths + the ledger and passes them through; a fresh, different-family reviewer (codex) reads the code and proposes findings; a deterministic tool decides the verdict. Both axes from references/reviewer-independence.md hold:

• Layer 1 — cross-model (executor ≠ reviewer). The executor never summarizes, pre-judges, or leaks a hunch into the prompt — it ships only paths + claims.json
  • the checklist + the observability level. The reviewer is a different model family.
• Layer 2 — reviewer ≠ adjudicator. The reviewer is demoted from judge to evidence-extractor: it emits span-anchored findings; tools/adjudicate_findings.py computes overall_verdict by fixed rules. Same ledger + same findings → same verdict, with no model in the final decision.

How this differs from the other auditors (route correctly)

Auditor	Question it answers	Level
experiment-forensics (this)	Are the reported numbers what the eval code actually computes? (fake/derived GT, self-norm, phantom result, dead metric, verified scope, method drift, placeholder/fake data, code↔paper mismatch, missing repro artifacts)	L2 (L0/L1 → info-only; missing-repro absence is L0-stated, surfaced info here)
consistency-audit	Does the paper contradict ITSELF / does described method = evaluated method?	L0
baseline-comparison-audit	Are the right baselines present, tuned, and is "SOTA" earned?	L0 stated / L2 verified
citation-forensics	Do the cited papers exist and support the claim made?	L0
presentation-signals	Surface "AI-flavor" hints (auxiliary, surface-class)	L0
adversarial-case-builder	Strongest evidence-bound rejection memo (no verdict weight)	any

Do NOT raise here (hand off instead): pure text-vs-text contradiction or scope-vs-evidence-in-text → consistency-audit; "first / SOTA / beats prior work" external truth → baseline-comparison-audit + citation-forensics (emit needs_external_check); citation existence/context → citation-forensics; surface/AI-flavor → presentation-signals; evaluation-design validity (train/test leakage, a conflicted/unvalidated LLM judge, declared-but-unreported conditions) → eval-design-forensics (family H, L0/L1 stated-tells — distinct from this skill's L2 code/result-integrity); the rejection memo → adversarial-case-builder. (Note: an LLM that produces the GROUND-TRUTH labels stays here as HP-FAKE-GT, L2 — only the LLM-as-judge validity question hands off to eval-design-forensics.)

Pipeline role + the anchoring model (read before running)

This is an auditor skill in the integrity-forensics pipeline (references/integrity-forensics-contract.md):

/evidence-ledger  →  claims.json (+ artifact_manifest.json, observability level L)
                          │
       experiment-forensics  ── reads the ledger, PROPOSES findings ──►  experiment-forensics.findings.json
                          │
   tools/adjudicate_findings.py  (deterministic; the ONLY thing that computes a verdict)

• It reads the ledger, it never re-reads the raw paper to invent structure, and it never computes the overall verdict — the orchestrator runs tools/adjudicate_findings.py for that. This skill stops at emitting findings.
• The anchor is always a PAPER claim. Every above-info finding cites a ledger claim_id and quotes a verbatim span of that claim's text_span (the paper number/scope/method sentence it undermines). The eval-code smoking gun (src/eval.py:88) is not a ledger claim, so it lives in the finding's description / recommended_reviewer_action, never as the anchor. No paper claim to anchor to ⇒ the finding cannot rise above info. (See worked examples.)
• Observability caps severity. Findings declare observability_level_required. Every code/result-level pattern is decidable only at L2; at L0/L1 it is emitted as info (Step 2). tools/adjudicate_findings.py is the structural backstop — any observability_level_required above the run's level is reported with that shortfall marked in its own column — visible, and visibly unconfirmed.

Constants & Reviewer Calling Convention

• REVIEWER = mcp__codex__codex — model gpt-5.6-sol, config: {"model_reasoning_effort": "xhigh"}, sandbox: read-only, cwd = TARGET (the repo/paper dir, where the code + results live). A different model family from the executor (Claude). One fresh thread per audit pass; never mcp__codex__codex-reply across passes (the bias guard — reply is deliberately absent from allowed-tools). See references/reviewer-independence.md.
• PATTERNS_OWNED (must match references/hack-pattern-taxonomy.md, taxonomy_version 0.5): HP-FAKE-GT, HP-SELF-NORM, HP-PHANTOM-RESULT, HP-DEAD-METRIC, HP-SCOPE-INFLATE (verified form), HP-METHOD-DRIFT (L2 confirm), HP-SUSPICIOUS-REGULARITY (L2 confirm), HP-PLACEHOLDER-DATA (L2), HP-RESULT-ARTIFACT-MISMATCH (L2), HP-MISSING-REPRO-ARTIFACT (verdict-bearing at L2 — absence noticeable at L0/L1 but surfaced as info there, confirmed at L2).
• ROOT = $(git rev-parse --show-toplevel 2>/dev/null || pwd); L = observability_level from artifact_manifest.json; OUTPUT = experiment-forensics.findings.json (a bare JSON array).
• FILES (all under TARGET, next to claims.json): the only output is experiment-forensics.findings.json; the reviewer handoff is .aris/last_reviewer_response.txt; traces live in .aris/traces/experiment-forensics/<date>_run<NN>/.
• ⚠️ Shell state does not persist between Bash calls (cwd + env reset each call). Every block below re-resolves ROOT and TARGET at its top and reads L / paper_id from the manifest/ledger. Never rely on a variable from an earlier block, and never cd into TARGET (it would break ROOT resolution).

Division of labor (references/reviewer-independence.md):

• Executor (Claude) locates the ledger, lists artifact paths, gathers mechanical facts (file listings, literal-string greps, hashes), passes paths + the ledger + the checklist to the reviewer, validates the reviewer's spans, and writes the findings file. It never summarizes file contents, pre-judges, or leaks an opinion.
• Reviewer (codex / gpt-5.6-sol) reads ./claims.json and the code/result files directly from its cwd, proposes findings, and self-reports false_positive_risk. Told: the artifact paths, the ledger, the per-pass checklist, the level. Not told: any other auditor's findings, the executor's hunches, or "this looks AI-generated" — the tool audits integrity, not authorship.
• One fresh thread per pass. The A–F checklist is a single call; the optional G/H passes are each a NEW thread. If codex stalls (long sessions can hang), re-invoke the same prompt in a fresh thread — never codex-reply.

Step 1 — Preflight: resolve root, level, paper_id (self-contained)

ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
TARGET="$ARGUMENTS"          # paper-dir or repo-dir — ABSOLUTE path
case "$TARGET" in /*) ;; *) echo "FATAL: TARGET must be a non-empty ABSOLUTE path (got: '$TARGET'). Pass the paper/repo dir as an absolute path."; exit 1 ;; esac

# Toolchain must be reachable (invariant: ROOT = the Anti-Autoresearch checkout).
test -f "$ROOT/tools/build_manifest.py" || { echo "FATAL: Anti-Autoresearch tools not under $ROOT. Run this skill from inside the Anti-Autoresearch checkout (or point ROOT at it)."; exit 1; }

# The ledger is mandatory and is produced by /evidence-ledger. Do NOT invent it.
test -f "$TARGET/claims.json" || { echo "FATAL: $TARGET/claims.json missing. Run /evidence-ledger on $TARGET first (experiment-forensics reads the ledger)."; exit 1; }

# artifact_manifest.json derives the level. Build it with the real tool if absent
# (exact flags — confirm via: python3 "$ROOT/tools/build_manifest.py" --help).
if [ ! -f "$TARGET/artifact_manifest.json" ]; then
  PID=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1]))["paper_id"])' "$TARGET/claims.json")
  python3 "$ROOT/tools/build_manifest.py" --paper-id "$PID" --dir "$TARGET" --out "$TARGET/artifact_manifest.json"
fi

# Read the level (L) + paper_id. The manifest decides L; never override it.
python3 - "$TARGET" <<'PY'
import json, sys
t = sys.argv[1]
m = json.load(open(f"{t}/artifact_manifest.json")); c = json.load(open(f"{t}/claims.json"))
print(f"paper_id={c['paper_id']}  observability=L{int(m['observability_level'])}  "
      f"(rule: repo+results->L2, latex/no-results->L1, pdf/text-only->L0)")
PY
mkdir -p "$TARGET/.aris/traces/experiment-forensics"

Branch on L (read from the echo above; references/observability-levels.md): L < 2 → Step 2 (info-only) → Step 6. L == 2 (or 3 — treat as 2, never re-run code) → Steps 3–6.

Failure handling. No claims.json ⇒ stop (the FATAL above); the ledger has not been built and this skill never invents one — tell the user to run /evidence-ledger first. Never claim a higher level than the artifacts support: L is derived deterministically (repo + results → L2; latex, no results → L1; pdf/text only → L0), and the manifest decides it — you do not override it.

Step 2 — L0 / L1: info-only "could-not-verify" signals (the honesty backbone)

At L0/L1 there is no eval code and no result files, so you cannot decide any experiment-integrity pattern. Do not assert fraud, and do not duplicate the L0 text-scope check — consistency-audit owns scope from the manuscript. This skill's only job here is to mark, for the human, which numbers become checkable if a repo is released. Generate the signals deterministically from the ledger:

ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd); TARGET="$ARGUMENTS"
python3 - "$TARGET" <<'PY'
import json, re, sys
t = sys.argv[1]
claims = json.load(open(f"{t}/claims.json"))["claims"]
out, n = [], 1
def add(pat, title, desc, ev, act):
    global n
    out.append({"finding_id": f"EF{n:03d}", "skill": "experiment-forensics",
                "pattern_id": pat, "title": title, "description": desc,
                "severity": "info", "observability_level_required": 2,
                "evidence": ev, "verdict_local": "needs_external_check",
                "requires_external_check": True, "false_positive_risk": "high",
                "recommended_reviewer_action": act}); n += 1
def anc(c): return [{"claim_id": c["claim_id"], "span": c["text_span"], "location": c.get("location", {})}]

GT   = re.compile(r"reference|ground.?truth|\bgt\b|gold|agreement|target", re.I)
SCOPE= re.compile(r"comprehensive|extensive|robust|general|thorough|state[- ]of[- ]the[- ]art|\bSOTA\b", re.I)
nums = [c for c in claims if c.get("type") in ("number", "comparison")]

for c in nums:                                   # GT-provenance pointer
    if GT.search(c.get("text_span", "")):
        add("HP-FAKE-GT", "Ground-truth provenance not verifiable without the repo (L0 could-not-check)",
            "This number is reported against a 'reference/target/GT'. At this level it cannot be determined whether that reference is dataset-provided or derived from model outputs. NOT an allegation — verifiable only at L2 (eval code + result files).",
            anc(c), "Request the eval code + result files; verify GT provenance at L2 (HP-FAKE-GT).")
for c in nums:                                   # near-ceiling -> normalization pointer
    v = (c.get("value") or {}).get("normalized")
    if isinstance(v, (int, float)) and ((0.99 <= v <= 1.0) or (99.0 <= v <= 100.0)):
        add("HP-SELF-NORM", "Near-perfect score — normalization not verifiable from text",
            "A near-ceiling score with no raw value shown cannot be checked for self-normalization from a PDF.",
            anc(c), "At L2, check whether the metric is divided by the model's own output statistics (HP-SELF-NORM).")
for c in claims:                                 # verified-run-count pointer (defer text scope to consistency-audit)
    if SCOPE.search(c.get("text_span", "")):
        add("HP-SCOPE-INFLATE", "Scope language — actual run count not verifiable without the repo",
            "consistency-audit owns the L0 scope-vs-evidence-in-text check; experiment-forensics can only verify how many datasets/seeds/configs ACTUALLY ran at L2.",
            anc(c), "At L2, count the configs/seeds actually executed in the result files vs this wording.")
        break
add("HP-PHANTOM-RESULT", "Result existence not verifiable without backing files",
    "Whether each reported number maps to a real key in a real result file cannot be decided from a PDF.",
    (anc(nums[0]) if nums else []), "At L2, map each headline number to a result-file key (HP-PHANTOM-RESULT).")
add("HP-DEAD-METRIC", "Metric-code liveness not verifiable without the repo",
    "Whether any discussed metric is actually computed/called cannot be decided from a PDF.",
    [], "At L2, confirm each discussed metric is called and appears in a result file (HP-DEAD-METRIC).")
add("HP-PLACEHOLDER-DATA", "Placeholder / fake data in released code not verifiable without the repo",
    "Whether the released code still contains placeholder/dummy/fake data (e.g. a '# fake data for plotting' annotation or a hard-coded random array) feeding a reported figure/number cannot be decided from a PDF — flag the code marker, not who wrote it.",
    (anc(nums[0]) if nums else []), "At L2, grep the code for placeholder/dummy/fake markers and trace whether any reported figure/number is drawn from them (HP-PLACEHOLDER-DATA).")
add("HP-RESULT-ARTIFACT-MISMATCH", "Code-output vs paper-number agreement not verifiable without the repo",
    "Whether the released code / result artifacts actually produce the paper's reported numbers cannot be decided from a PDF. (A code-vs-equation implementation divergence is HP-METHOD-DRIFT, not this.)",
    (anc(nums[0]) if nums else []), "At L2, read the code's computation + result files and check each reported number against the code-produced value (HP-RESULT-ARTIFACT-MISMATCH).")
if nums:                                          # repro-artifact inventory pointer (absence is L0-observable)
    add("HP-MISSING-REPRO-ARTIFACT", "Reproducibility artifacts absent — empirical claims not checkable even in principle (absence noticeable at L0; verdict-bearing only at L2)",
        "This paper reports empirical/number results but the submission ships no eval code and no prompts/configs the results depend on. The ABSENCE is observable now (L0 'stated'); whether the SPECIFIC prompts/configs/hyperparameters its results need are present is verifiable only if a repo is released (L2). NOT a misconduct claim — a reproducibility gap. FP: a genuinely theoretical paper; double-blind submission norms (treat as a camera-ready expectation, lower severity).",
        anc(nums[0]), "Ask for the code + the exact prompts/configs/hyperparameters the reported numbers depend on; at L2 verify they are present and complete (HP-MISSING-REPRO-ARTIFACT).")

json.dump(out, open(f"{t}/experiment-forensics.findings.json", "w", encoding="utf-8"), indent=2, ensure_ascii=False)
print(f"L<2: wrote {len(out)} info 'could-not-verify' signals (severity=info, req:2 — the adjudicator keeps them at info).")
PY

Each emitted finding has the shape below (info-only, observability_level_required:2, empty evidence permitted only for info). Then go to Step 6 (no reviewer call needed at L<2 — these are deterministic pointers, not judgments):

{
  "finding_id": "EF001",
  "skill": "experiment-forensics",
  "pattern_id": "HP-FAKE-GT",
  "title": "Ground-truth provenance not verifiable without the repo (L0 could-not-check)",
  "description": "Claim C014 reports agreement against a 'reference'. At L0 (PDF only) it cannot be determined whether that reference is dataset-provided or derived from model outputs. This is NOT an allegation — verifiable only once the eval code + result files are available (L2).",
  "severity": "info",
  "observability_level_required": 2,
  "evidence": [
    {"claim_id": "C014", "span": "98% agreement with the reference",
     "location": {"file": "paper.txt", "section": "experiments"}}
  ],
  "verdict_local": "needs_external_check",
  "requires_external_check": true,
  "false_positive_risk": "high",
  "recommended_reviewer_action": "Request the evaluation code and result files; verify GT provenance at L2. Do not treat as a flag at this observability level."
}

Step 3 — L2: collect artifacts (executor — paths + mechanical FACTS only)

You (Claude) gather inputs and mechanical facts; you do not interpret, summarize, or pre-judge them (references/reviewer-independence.md). Listing what exists and grepping for a literal string are reproducible facts, not judgments — the same division citation-forensics uses for existence vs. context. Do not cd; use absolute $TARGET/... paths.

ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd); TARGET="$ARGUMENTS"
mkdir -p "$TARGET/.aris"

# (a) Eval / metric / test / score / benchmark / runner code + configs -> eval_paths.txt
find "$TARGET" -type f \( -name '*eval*.py' -o -name '*metric*.py' -o -name '*test*.py' \
   -o -name '*score*.py' -o -name '*benchmark*.py' -o -name 'run*.py' -o -name 'main.py' \
   -o -name '*.yaml' -o -name '*.toml' \) 2>/dev/null | sort > "$TARGET/.aris/eval_paths.txt"

# (b) Result files the paper's numbers should live in -> result_paths.txt
find "$TARGET/results" "$TARGET/outputs" "$TARGET/logs" -type f \( -name '*.json' -o -name '*.csv' \) 2>/dev/null | sort > "$TARGET/.aris/result_paths.txt"

# (c) Likely GT / reference loaders (FACT for the reviewer; do NOT judge) -> gt_grep.txt
grep -rInE 'ground.?truth|reference|target|label|gold|gt_|normaliz' --include='*.py' "$TARGET" 2>/dev/null | head -60 > "$TARGET/.aris/gt_grep.txt"

# (d) Phantom-result FACTS: grep EACH headline number from the ledger (no hardcoded token) -> number_grep.txt
python3 - "$TARGET" > "$TARGET/.aris/number_grep.txt" <<'PY'
import json, subprocess, sys
t = sys.argv[1]
claims = json.load(open(f"{t}/claims.json"))["claims"]
toks, seen = [], set()
for c in claims:
    if c.get("type") not in ("number", "comparison"): continue
    v = c.get("value") or {}
    for tok in (str(v.get("raw") or ""), ("" if v.get("normalized") is None else repr(v["normalized"]))):
        tok = tok.strip().rstrip("%").strip()
        if tok and tok not in seen:
            seen.add(tok); toks.append((c["claim_id"], tok))
for cid, tok in toks:
    hits = subprocess.run(["grep", "-rIn", "--", tok, f"{t}/results", f"{t}/outputs", f"{t}/logs"],
                          capture_output=True, text=True).stdout.strip()
    print(f"### {cid} token={tok!r}: {'FOUND' if hits else 'NOT FOUND under results/outputs/logs'}")
    if hits: print(hits)
PY

# (e) Reproducibility anchors: hash EVERY discovered eval script + result file -> hashes.txt
{ while IFS= read -r f; do [ -n "$f" ] && shasum -a 256 "$f"; done < "$TARGET/.aris/eval_paths.txt"
  while IFS= read -r f; do [ -n "$f" ] && shasum -a 256 "$f"; done < "$TARGET/.aris/result_paths.txt"; } 2>/dev/null > "$TARGET/.aris/hashes.txt"

# (f) Ledger claim subset the reviewer must anchor to -> claim_subset.json
python3 - "$TARGET" > "$TARGET/.aris/claim_subset.json" <<'PY'
import json, sys
t = sys.argv[1]
KEEP = {"number", "comparison", "scope", "method", "artifact_ref"}
claims = json.load(open(f"{t}/claims.json"))["claims"]
print(json.dumps([{"claim_id": c["claim_id"], "type": c["type"],
                   "text_span": c.get("text_span", ""), "location": c.get("location", {})}
                  for c in claims if c.get("type") in KEEP], indent=2, ensure_ascii=False))
PY