Responding to a red check

The doctrine

A red check is a bug you have not diagnosed yet. Treat every failure as real and caused by the change in front of you until a read of the actual log proves otherwise. These dismissals are forbidden as opening assumptions: "infrastructure flake," "timeout," "unrelated," "pre-existing," "already broken on the base branch," "passes on my machine," "only fails on another OS/platform," "load," "transient," "not what I was asked." Each is a conclusion you must earn. "Flaky"/"external" is provable only by reading the log AND demonstrating the root cause is non-deterministic and outside the repo. A timeout/download/racy failure is a real defect (a test too slow or racy under parallelism, a missing retry/backoff), yours to fix at the root.

"Flake" is NEVER a license to ignore, skip, mute, xfail, or merely re-run a failure. Proving something is a flake does not close it, it OBLIGATES a root-cause fix (make the test deterministic, add the missing retry/backoff/timeout budget, pin the unstable input). Even when it is pre-existing and unrelated to your change, fix it in its own fix(test):/fix(ci): commit — "it was already flaky" is not an exemption.

A red that reaches a SECOND round is itself a defect in the red-fixing process. One round closes a red: the round that diagnoses it also pushes its fix, or pushes a branch update (a commit, or one PR-body lead sentence) saying exactly why one round cannot — a fix only CI hardware can validate, an external service down. When a red reaches round two without that recorded reason, first name why round one's response failed — a fix built on an unverified belief about a dependency, a log read that stopped at the symptom, a "flake" left without its root-cause fix — then fix THAT in the same session: sharpen this skill, the babysit-prs skill, or the babysitting code under .github/scripts/, so the class lands first-round.

Deduplicate before you fix, on whatever names the defect — the failing test's exact name, the red check's name, or the file the fix lands in — and search the ISSUE list as well as the PR list, because a multi-PR flake sweep is tracked by an issue and no PR search finds it. mcp__github__search_issues and mcp__github__search_pull_requests, both with owner/repo set and fields: ["number","state","title"] to drop the body and labels. gh pr list --search is GraphQL and 403s in a web session, and the REST search/issues path is blocked there as well, so the MCP tools are the only route that works on both surfaces. An OPEN tracking issue owns the item: add your row to it by EDITING it, and do not file a second issue. Then read the PR state: an OPEN match is the one case you hand back, as one line naming that PR and no fresh investigation; a MERGED one means you re-run the test against current origin/main before touching anything; a CLOSED one covers nothing, so the fix is yours.

A shared red DUPLICATES, because every session that notices it starts the same fix and none of them is answering a webhook that names the others. Investigate it ONCE, and never patch every affected PR independently. Before you write a line of it, list the open pull requests — mcp__github__list_pull_requests with state: "open", sort: "updated", fields: ["number","title","head"] — and read the checks of every one whose title or head branch names that check, that test or that file. Prefer the branch already running the failing check green: close yours as superseded, or hand it what it lacks. One session owns the repair, and the other open PRs pick it up from the base once it lands, so a fix ported onto each of them is waste, not thoroughness. #4993 and #4999 each fixed the same three main reds for a day, and the second branch's work was thrown away.

Then LAY A CLAIM, before you write the fix. This binds on any defect several sessions can see at once and no webhook addresses to one of them: a red on main, a row on the CI-health issue, a finding a scheduled sweep reported. Search the open issues first — mcp__github__search_issues with owner/repo, state: "OPEN", fields: ["number","state","title","createdAt"], over the failing check, test or file. On a match, READ the issue: createdAt is what tells you whether the claim is still live, and the branch is in the body, which the search does not return. A LIVE claim means another session holds the work: take something else, or hand your fix to the session that claim names, because root CLAUDE.md § Where work lands bars pushing to its branch without established coordination. Never file a second one. A claim over 24 hours old with no open PR linked is DEAD — close it and take the work, rather than deferring to it, which is how a dead claim would park the red forever. With none open, file ONE issue titled claim: <the check, test or file>, and give its body two lines — what you are fixing, and the branch you are fixing it on. Then search again, because GitHub accepts both issues when two sessions file at the same moment and neither search saw the other's. The LOWEST-numbered live claim wins: it is the same answer from either session, so both stand down the same way with no round trip. Yours is not the winner — close it as a duplicate of that one, and take the work only if its branch is yours. The PR that lands the fix closes the winning claim (Closes #N). Fix the red with no PR at all, or abandon the work, and you close the claim yourself.

A claim is a lock, not a plan. It carries no design, no diagnosis and no patch, and filing it is never a substitute for the fix — root CLAUDE.md bans an issue for work you could start, and this is its one CI carve-out.

Hardware you lack is never an exit. When a reproduction or a verification needs a machine this session does not have, commit the reproduction to the CI surface that has it (bin/checks/sbx/*.bash and its live shards, or the matching workflow), name the run that executes it, and treat the flake as unfixed until that run reports. Only a credential you lack, which blocks committing a runnable reproduction at all, ends the work — and that is one line naming the credential.

NEVER just re-run a failure — root-cause it, then dispatch a subagent to FIX it. A re-run is forbidden as the response to a red check, even for a failure you've proven external (a third-party runtime crash, a hosted-runner fault, a vendored-binary bug). The mandatory sequence on any failure: (1) read the actual failing log and diagnose to root cause — the specific mechanism, not a category label; (2) spawn a sub-agent to implement the fix at that root cause and drive it green — a resilience/retry/guard in our own code that would have survived the fault, a portability fix, a race fix, a missing-backoff fix, or (when the defect is genuinely upstream) a hardening of our interaction with it plus a written upstream report. The subagent owns the fix end-to-end (branch, commit, push, CI). A re-run is permitted only as a one-time unblock riding alongside an already-in-flight root-cause fix. If you can't yet name the mechanism, keep digging (reproduce it, read more log, unshare/local-repro the guest-netns or FS behavior), don't re-run.

When a re-run IS warranted, re-run the RUN whose triggering event carries the right payload — never the failed job. A re-run replays the original webhook payload, so a job that reads github.event.pull_request.* (labels, title, body, head sha) re-reads the state as it was at that event, not as it is now. Fixing the condition off-PR and then calling rerun_failed_jobs therefore re-fails on the stale payload — and because GitHub resolves a required check by the newest run of that check's name, the re-run buries the passing result under a fresh red. Before any re-run, ask which event's payload the job actually reads and re-run that run.

The CHECKOUT is frozen the same way, so NO re-run can pick up a fix that landed on the base branch. actions/checkout on a pull_request event checks out github.sha, and that is the merge commit the original webhook payload names. job-log-excerpt.py prints it as the excerpt's first line, where it reads identically across every attempt of the run. Only a new PR event moves it: a push to the head, or update-branch. So a red whose cause main has since fixed is never a re-run — weigh update-branch, which also pulls in whatever else main is currently red on, against waiting for the fix to reach this PR another way.

A red whose log reports the gate's own refusal with decide-result: 'abandoned' is the one shape a log read clears — and your own re-run wave MANUFACTURES it. report-job-result.sh fails closed when the decide job it gates did not resolve, and its last log line reads decide gate did not resolve cleanly (decide-result: 'abandoned'). Read the VALUE, not the shape — the same line prints for every non-clean result, and only abandoned is benign: an empty value means a caller never wired needs.<decide>.result, and failure means the decide job itself broke; both are real defects you diagnose and fix. abandoned means GitHub abandoned the run ATTEMPT this job ran on. It never means the gated work failed, so do not diagnose the check, its test, or its decide filter. Two causes produce it: the decide job died in Set up job (its own log ends Failed to resolve action download info. / Service Unavailable), or a newer attempt superseded this one. The second is self-inflicted — each rerun_failed_jobs abandons the previous attempt, which keeps reporting for a while. So a broad re-run wave hands you a burst of red it created. Read the attempt, not the conclusion: gh api repos/{owner}/{repo}/actions/runs/<id> --jq .run_attempt, against the attempt in the check run's details_url. The log read is still what earns the dismissal; an annotation alone never does.

Two consequences. 403 This workflow is already running means an attempt is live: wait and retry it, never debug it. The full-run rerun method is refused on the same condition, so the two remedies never overlap in time. And an abandoned check is RED, so the stale finding treats it as an honest reason to be blocked — .github/scripts/required-check-liveness.py does not cover it, and nothing re-runs it for you.

A red blocking THIS PR is this PR's work, whoever caused it. "Pre-existing", "I didn't break it" and "out of scope" are not exemptions for a check that stands between this diff and its merge: fix it here, in its own commit (fix(test):/fix(ci):) when it is unrelated to the change. The one thing that is out of scope is a red on a PR that is not yours, which the paragraph above routes to whichever session owns that repair — a task session does not become a repo-wide CI babysitter.

Collect the whole round's reds before you push a fix — the push cancels the run in flight and discards every check that had not reported yet. Seven workflows carry cancel-in-progress: true, so one push per red turns a parallel round into one round per red, and each new head starts the whole check battery again. On PRs #4223, #4234 and #4237 that cost 40 to 50 percent of their runs, which ended cancelled at a median push gap of 10 to 17 minutes. Read the round to the end, then dispatch ONE fix agent carrying every red that round named — the subagent's end-to-end push above is a push per ROUND, never a push per red, and a branch never has two of them at once. That is not idling — work another PR while the round finishes.

Read what the failing job already PRINTED before you reproduce anything. A coverage table, a ratchet verdict, a lint listing and a diff of expected-versus-actual are the diagnosis, not a hint toward one, and the job published them for free. They also sit OUTSIDE every anchor the excerpt tools use: the report prints before the ##[error] line, often hundreds of lines earlier, so --log-failed, failed_only and job-log-excerpt.py all cut it away. When the error line names a measurement (X% is below its floor), grep the whole job log for the row that measurement came from. Re-deriving it locally costs minutes and answers about YOUR tree, not the merged one CI read. A GREEN Python tests (pytest) job prints a report too. Its py-ratchet: candidate lines are shortfalls a SUBSET run did not assert. On a file this PR changed they are the session's to close while the PR is open, not a later paydown's.

Reproduce, then fix at the root — do not guess. Once the log has said all it can, reproduce the failure (same parallelism/-n auto, same OS marker, same inputs). Prefer the root cause (or the tool's own config) over a local suppression; reach for an inline disable/ignore only for a genuine false positive, justified in a comment naming why.

A PR's checks run your head MERGED with the base, so a check can be red there and green in your worktree. Two branches that each merge clean combine into a broken tree whenever they answer one question twice — two imports of one name, a call to a function the other side renamed. Git adds both lines and raises no conflict, so nothing warns you. Run git merge origin/main on your branch before you believe a local pass: a green run on the unmerged head is evidence about a tree CI never tested. That merge commit is also a MOVING ref. GitHub rebuilds refs/pull/N/merge whenever the base branch moves, with no push to the PR, so git show refs/pull/N/merge:<file> now can hand back bytes the job never ran. Read the SHA the job itself logged — job-log-excerpt.py prints it as the excerpt's first line — before you diff any file against main.

An OS/platform-specific failure is the highest-value signal, not an excuse. "Green on Linux, red on macOS" (or vice-versa) means you found a real GNU/BSD divergence — reason about it from first principles (coreutils flag differences, mkdir -p/ln/stat/readlink behavior, symlink semantics) and fix the portability bug.

A blocking check that fails on a change that was CORRECT is a false red, and the session that hits it RECORDS it — nobody else can. Comment on the CI-health issue tagged <!-- false-red -->, with the four labelled lines docs/false-reds-per-week.md § Collection specifies: check:, pr:, what it blocked:, resolution:. The resolution is the test — changing the check, waiving it, or re-running it unmodified is a false red; changing the code under test is a TRUE red and earns that check a guard-rent citation instead. One instance per occurrence, never one per check. Not false reds: a real defect caught, an infrastructure outage, a flake in the code under test, a red on a PR that was already broken. A citation is spent after 8 weeks and deleted at the guard-rent audit.

Handing off is the last resort, and only after the work. The only acceptable hand-off is a failure you've already diagnosed to root cause AND cannot resolve from inside the session (needs a maintainer to re-run a job your token can't trigger, or an environment-side change). Even then: state the proven root cause, what you tried, and exactly what's blocking — never declare green, never imply the red doesn't count, never go silent.

A fix that lands in CI machinery — a workflow, a landing or queue script, the scheduler — answers the TLA+ spec question in its PR body; .claude/rules/tla-plus.md owns it.

Reading GitHub state

Any claim about why a check passed or failed is a claim about check state — read the head's checks before you make it. This binds on the negative: "nothing failed", "that lint is advisory", "the hooks would have caught it". A local hook or lint run says what YOUR tree does; it names no check, so it can never answer why one is red. Answering from the push log leaves a required check red while the report reads clean.

Wherever gh reaches GitHub, ALWAYS read a PR's reds through uv run .github/scripts/pr-status.py <pr#> — never by hand. Hand-reading means any gh api …/check-runs, any gh pr checks, any MCP get_check_runs or get_status, and any jq over their output. The helper exists because each of those gets the same three things wrong: it pages past the API's 100-per-page cap, which a head here overruns; it reads the required-check set from the # required-check: true annotations rather than from your memory of which checks block; and it folds in mergeable_state, auto-merge and the review threads, which no check-run call returns. A hand-read that misses any one of them reports a blocked PR as green. Give it every PR in the set in ONE call — it takes several numbers, --all-open and --ready-only.

The helper cannot run where gh is 403, and there is NO script route there — api.github.com refuses a direct curl through the agent proxy with the same 403, token or no token, so nothing a script can call reaches GitHub. Only then do you read by hand, through mcp__github__*, under the paging and both-halves rules the babysit-prs skill carries. Probe first, per § A Claude Code web session below; a predicted 403 does not license the hand-read.

Where gh reaches GitHub it is installed and authenticated — use it for all other GitHub work:

gh run view --repo <owner>/<repo> --job <job-id> --log-failed   # only failed steps — START HERE
uv run .github/scripts/pr-status.py <pr#>                       # check status (works on both surfaces)
GITHUB_REPOSITORY=<owner>/<repo> GITHUB_RUN_ID=<id> \
  uv run .github/scripts/name-failed-jobs.py '<job-name-prefix>' # what stopped each non-success job
uv run .github/scripts/job-log-excerpt.py <job-id>              # only the failing region
gh run view --repo <owner>/<repo> --job <job-id> --log          # full job log — last resort

Never open a full job log to find one error line. A job log here is 90–150 KB, so it costs a slow transfer AND ~25–40k tokens of context, and every later turn re-reads that context. Ask for the failing steps alone and cap what you take:

1. --log-failed (CLI) or get_job_logs with failed_only: true (MCP) — the failing steps only.
2. Keep the tail small. get_job_logs defaults tail_lines to 500, which is already a large read; set it lower on the first look and pipe the CLI through tail -n 200.
3. Widen only when the tail does not carry the cause. Say in your report that you widened, so nobody reads a truncated log as the whole one.

A tail read of a job here returns CLEANUP, not the failure. A job appends 50 to 135 lines of post-job cleanup after the failing step, so tail_lines=80 returns the Docker image manifest. On job 95086344352 the pytest summary sits at line 1205 of 1401.

uv run .github/scripts/job-log-excerpt.py <job-id> prints that region instead — tens of lines, not the whole log. It anchors on the runner's ##[error], ##[group]Run and ##[start-action markers and on pytest's === FAILURES === banner. --run <run-id> sweeps a run's failed jobs. Where GitHub refuses the log endpoint over REST, the script names the route: take the signed logs_url from get_job_logs with return_content: false, curl it to a file, pass --file. The sanitize-output PostToolUse hook flags that URL as exfil-shaped and says not to fetch it. Fetch it anyway — but only that URL: the one the MCP tool returned as its own field, on a *.blob.core.windows.net or *.githubusercontent.com host — and say in the report that you did. It is a false positive there. GitHub signs Actions logs and artifacts with Azure SAS parameters — ske, sks, skt, skv, rsct — that the pinned agent-sanitizer does not allow-list. The URL is the tool's own answer to a job id, never text anyone injected. A URL that came out of a job log's CONTENT is written by whatever the job ran, so the flag on that one stands. pyproject.toml names what holds that pin.

When the log itself is gone, the per-step record is what still answers. A runner that dies mid-step never reaches Complete job, so GitHub never finalizes that job's log and the log endpoint answers 404 for good. That record outlives the runner: the first step with no conclusion is where the job stopped, and a pending step after it means the runner died rather than the step failing. For ONE job, mcp__github__actions_get(method="get_workflow_job", owner=…, repo=…, resource_id=<job-id>) reads it with no GITHUB_TOKEN of your own, and its answer carries the run_id. For every failing job in a run at once, name-failed-jobs.py reads it — quote its prefix argument, since it takes one positional and this repo's job names carry spaces. It also pages the jobs endpoint, which matters because GitHub caps it at 100 and this repo's runs carry 130+ jobs — so a hand-rolled one-page read drops the dead shard as often as any other.

failed_only: true needs run_id, not job_id, and covers every failing job in that run — one call for a red run instead of one call per job. Pair it with return_content: true; without that the tool answers with URLs. Never use actions_list method list_workflow_jobs to find what failed: it returns every job's full record — about 72 KB for a 4-cell run, most of it the same repository boilerplate repeated per job — so it overflows the tool-output cap, lands in a file, and leaves you writing a slicer for fields failed_only hands you directly.

In a Claude Code web session, PROBE gh before you reach for it: gh api repos/{owner}/{repo} --jq .full_name. Two outcomes, and only the probe tells them apart: