monitor-cron-sweep

Deploy each cron sweep to produce ONE comprehensive update across all active clusters.

⚠ STEP 0 — READ .agents/ops/<cluster>/ops.md FIRST, every sweep, for each cluster you'll touch. It carries the binding gotchas: the GPFS find/du ban (stat-walks stall SSH for minutes — locate logs via scontrol show job <id> -o StdOut=/%Z + depth-1 ls), the login01 fork-saturation false-drain (re-check via login02/03/04), cleanup-isn't-done-until-rm'd, the SIF/Ray-actor/NCCL debugging tooling (ptrace blocked → faulthandler; the opCount dead false-positive), the sig53/EDQUOT traps, and shell idioms (sacct -S now-Nhours, simple single-string ssh).

Local clone = ground truth. Any code/config fix this sweep performs — or dispatches a subagent to perform — is edited in the local Mac checkout → commit → push → git pull on the cluster. NEVER hand-edit, git commit, or leave divergent/untracked changes on a cluster; no patch-by-rsync (vLLM built from source per-cluster). Bake this rule into EVERY subagent prompt you dispatch.

Tables → the monitor-job-tables skill (box-drawing ┌─┬─┐, NOT markdown), bucketed RL · SFT · Datagen · Eval · Catch-all, with the mandatory metric columns, thresholds, and benign-noise-vs-real-fault rules per bucket. This skill is the process that fills those tables. Cluster particulars (ssh, paths, RL concurrency cap, gpu-mem ceiling, dotenv) live in .agents/ops/<cluster>/ops.md — no cluster-specific values inlined here.

1. Gather (per cluster)

Scope = Leonardo + CoreWeave(iris) + TACC(Vista) — all three each sweep. Jupiter is SKIPPED (MDC downtime until ~2026-07-12 — re-add as a 4th cluster when it returns); Perlmutter DROPPED 2026-06-05 (do NOT ssh). Leonardo + TACC are SLURM (squeue/sacct); CoreWeave is a k8s/iris controller backend with NO ssh — state-poll the iris lifecycle.

SLURM clusters (Leonardo, TACC):

• squeue -u <user> -t RUNNING + sacct -u <user> -S <-Nh> -X (terminal states since last sweep). TACC <user>=penfever via ssh TACCVista; Leonardo <user>/ssh → ops/leonardo.
• Validate squeue succeeded before trusting a 0-count. A slurmctld timeout prints slurm_load_jobs error: Socket timed out with NO job lines → a naive grep -c reads 0 → false "drained". Treat an errored squeue as UNKNOWN (keep waiting); prefer a positive done-signal via sacct -j <ids> --format=State (slurmdbd survives slurmctld outages). login01 fork-saturation is a second false-empty cause → re-check via login02/03/04. Mandatory before any destructive datagen consolidate+delete.

Leonardo gather/triage → ops/leonardo/ops.md ("Sweep / gather particulars"). The Leonardo layer (GPFS find/du ban, scontrol log location, eval log-path trap, standard-eval results-JSON shape, active campaigns, flawed_summ POLICY.md/STATE.md, HF-upload sbatch-tunnel, step-ca cert, $WORK vs $SCRATCH_FAST) lives there. Cross-cluster rules that DO apply: AgentTimeoutError / ContextLengthExceeded are EXPECTED passthrough exceptions in agentic eval (still scored) — never the cause of a hang. Drive named campaigns (flawed_summ et al.) off their OWN tracker docs — do NOT restate their rules here (restated rules drift): each sweep READ the campaign's POLICY.md/STATE.md and drive off them. Eval launch/listener mechanics live in eval-agentic-launch + .agents/projects/ot-agent/, not here.

CoreWeave(iris) — STATE-POLL, not squeue, not a log-string watch:

• export KUBECONFIG=~/.kube/coreweave-iris-gpu in the same shell FIRST (the Mac default kubeconfig points at a DIFFERENT cluster → wrong-context "0 pods/not found"). Use the otagent-env iris binary /Users/benjaminfeuer/miniconda3/envs/otagent/bin/iris (the marin .venv iris has a broken kubernetes import). All iris/kubectl calls SYNCHRONOUS (never background).
• Per active job, poll the authoritative lifecycle: PY=/Users/benjaminfeuer/miniconda3/envs/otagent/bin/python; $PY scripts/iris/iris_ops.py /benjaminfeuer/<job> --once --json and/or iris --cluster=cw-us-east-02a job summary --json (authoritative). iris … query over the jobs table (state 1=PENDING 2=BUILDING 3=RUNNING) lists live jobs. Treat "running-but-0-pods / record disappeared" as TERMINAL — the silent-wedge signature (a clean kill/eviction/preempt emits no terminal log line + reaps pods). Log-content greps (scripts/iris/analyze_iris_harbor_job.py, sel_rows/EPDIAG) are for SCIENCE/throughput ONLY, never liveness. Full log: iris … job logs --since-ms <submitted_at_ms> --no-tail (finelog keeps the whole log; only --tail caps lines).
• RL bring-up signals (fresh launches): gang/leafgroup Kueue admission (pods SchedulingGated until atomically admitted = normal), apply_ep/mesh-load, weights resolving. shm_broadcast …60s + a transient ghcr-EOF ImagePullBackOff self-heal are BENIGN bring-up noise. --max-retries ≥1 re-brings-up the gang on a transient HF-weight-resolution flake.

TACC(Vista): ssh TACCVista (ControlMaster live, hardened single-string ssh). salloc is BLOCKED → sbatch; uv/builds go in a CPU -p gg sbatch, never the shared login node. Compute nodes have FULL internet → NO proxy/SOCKS/step-ca cert (contrast Leonardo). GPUs are NOT a SLURM gres (whole-node alloc); RealMemory misreported. Agentic eval runs through the front door python -m hpc.launch --job_type eval_listener --cluster-config tacc (sbatch_script=eval/tacc/eval_harbor.sbatch, eval_jobs_dir=/scratch/10635/penfever/eval_jobs) — newly integrated, currently validated by a canary, so sanity-check the canary's traces uploaded + registered before relying on it. Harvest finished TACC evals the same way as Leonardo (eval-agentic-cleanup if auto-register failed).

Cross-cluster liveness + inode checks (apply per cluster):

• LIVENESS — RUNNING is NOT proof of progress (catches silent wedges). A job can hold its allocation for hours while hung (engine deadlock, NCCL stall, generation-buffer wedge) — squeue still says RUNNING. For EVERY RUNNING job, stat -c%y <StdOut> and compare log mtime to "now": if a job has emitted NOTHING materially longer than its expected cadence (RL step / SFT log interval / eval trial — minutes, not hours), treat as a suspected silent hang → investigate (tail the log, grep ray-worker logs for EngineDead/NCCL-timeout/Watchdog/RPC-timeout around the last-output timestamp). A multi-hour-stale log on a multi-node job is a wedge burning nodes → diagnose + (with permission, since it's RUNNING) kill+relaunch. Never report a RUNNING job as "healthy" without confirming its log is live. Put the log-mtime ("last output N min ago") in the table.
• INODE HEADROOM (each sweep) — jutil project dataquota -p <project> + df -i. Inodes (file COUNT), not bytes, are the binding constraint; the shared datasets project on /e/data1/datasets (where …/playground/ot-baf lives) runs chronically near/over its soft limit. (DORMANT while Jupiter is down, re-arm when it returns; Leonardo's bind is disk quota, not inodes → ops/leonardo; CoreWeave artifacts go to HF/R2, no POSIX tree to reap.) See ops/jupiter/ops.md → "Inode allocations" (#inode-allocations). At/over the inode soft limit → sweep red-flag → trigger the cleanup-reclaim step (§4).

2. Bucket every job by type

By job-name prefix / run-tag: rl__* → RL, sft__* → SFT, datagen__* → Datagen, eval-* / eval run-tags → Eval, everything else (consolidate, pretokenize, hf_upload, SIF build, DCP/CP/GPU-CI smoke, measurement/grid probes) → Catch-all.

3. Render per monitor-job-tables (unify cross-cluster per type)

Unify all clusters' runs of a type into ONE table. Extraction pointers:

• RL — Step (.out tqdm Training Step Progress: N/M or trainer/global_step) + reward/grad/entropy/ TIS from the WANDB_MIRROR lines (chain-restart logs may have step but not the dict — scan the chain's logs). Apply the collapse-signal rule. For any RL job in a NEW/UNTESTED setting (new config/geometry/model/image, a "debug"/"smoke-test" run, or the first launch after a code/config change), the table row is NOT enough — dispatch a subagent armed with rl-job-health-deep-dive this tick to deep-probe it (sync trace_jobs + logs, live-poll GPUs vs the serving LUT, read the literal rollouts) → a KILL/NO-KILL recommendation. State-poll + metrics can read "healthy" on a run that is silently dead (weight-sync garbage, engine-starvation wedge, all-reward-0). Carry the verdict into §4; the supervisor owns the actual kill.
• SFT — Step + {'loss','grad_norm'} from the .out (NOT trainer_log.jsonl); total steps from the config/banner.
• Datagen — chunks done/total (squeue+sacct) + result.json count + avg_turns (realness gate: ≈1.0 = dead) + exc%.
• Eval — result.json/total + pass-rate + top exception + the 4 infra checks (eval-agentic-launch §4 for greps).
• Catch-all — one line each: State / Elapsed / human note.

4. Flag + hand off

• Completion → the matching cleanup skill: RL by flavor — agentic (Harbor/Daytona/terminal_bench) → rl-agentic-job-cleanup; standard / non-agentic GRPO (the Delphi/rlvr/dapo math cells from rl-standard-launch-leonardo; no trace_jobs/) → rl-standard-job-cleanup (model + metric CSVs only, no trace dataset). SFT → sft-job-cleanup (upload + DB register). datagen (all chunks done) → datagen-job-cleanup (consolidate + advance the tracker). eval → eval-agentic-cleanup (only if auto-upload/register failed). For RL, recognize resume-overshoot: a clean COMPLETED at max_steps means done → cleanup; spurious past-max chain links should be cancelled. CLEANUP IS NOT DONE UNTIL THE ARTIFACT DIR IS rm'd. Uploading to HF then leaving the experiment's trace_jobs//tasks//already-pushed-exports/ subtrees on disk is the #1 inode leak. Every cleanup handoff (and every cleanup subagent prompt) MUST: confirm the artifact is on HF, then delete the on-disk trees (detached rm per the GPFS-delete discipline in ops/jupiter/ops.md), and verify inode reclaim (df -i / jutil).
• HF-only SFT chain (Delphi #6279 + any enable_db_registration: false series) — "move the chains", 3 legs, autonomous every sweep, no asking:
  1. SFT completes → HF upload via sft-cleanup-hf-only (NOT sft-job-cleanup; upload, no DB).
  2. upload completes → eval-standard-launch for the newly-uploaded cell(s).
  3. eval completes → record scores in the tracker — Delphi midtrained-cell grid → main_sft_evals/SCORES.md; base-model SFT grid (#6279 rows 2&4) → base_sft_evals/grid.md (one row per base×recipe cell). Each sweep advance whichever leg is pending (catch up backlog). Idempotent — skip done legs. Applies to BOTH main_sft_evals (27 midtrained cells) AND base_sft_evals (9 base × 2 recipes = 18 cells).
• Standalone eval-grid trackers (self-describing — harvest pending rows every sweep): any tracker markdown holding ⏳ pending rows with a recorded eval job id — e.g. experiments/active/delphi/rl-scaling-laws-6279/baseline_evals/grid.md, …/base_sft_evals/grid.md, …/pass_at_k_sft_evals/grid.md, …/main_sft_evals/SCORES.md. For each pending row: sacct -j <jobid> --format=State → on COMPLETED, harvest per EVAL_CONVENTION.md §5.2 D/E (rsync the per-task results_*.json to the tracker's <RUN>/ dir, verify the JSON has numeric scores — a COMPLETED job can carry an empty results:{}, extract MATH500/AIME24-mean±se/gsm8k, fill the row, flip to ✅). On failure, diagnose per EVAL_CONVENTION.md §3.3 + log. The tracker carries the jobids — read the grid each sweep.
• Cluster working-tree hygiene (every sweep, Leonardo + TACC only — CoreWeave has NO clone: the iris launcher uploads the local Mac workspace to /app per launch, so a local commit takes effect on the next launch with no push/pull and no on-cluster tree to drift): run git -C <cluster repo> status --short. If untracked/modified files have piled up (ad-hoc launch scripts, priority lists, configs, .baks, manifests, stray &1 junk), triage them back to the local ground-truth clone: rsync local, TRACK the reusable/canonical ones (commit locally → supervisor pushes; place at the path matching tracked siblings, e.g. eval/lists/*.txt, sft/lf_configs/, launchers under scripts/), GITIGNORE the recurring transient/generated set (*.bak, *_manifest.txt, &1, ephemeral reeval_priority_*). Diff any tracked-but-modified vs origin first (identical → stale HEAD). Reconcile the cluster with git pull (fast-forward) — NEVER git reset --hard while live jobs depend on uncommitted working-tree state. Dispatch a triage subagent if the set is large.
• Chain-restart TIMEOUT (12h/24h wall) with a successor RUNNING/PENDING → normal, not a failure — note the successor.
• Genuine FAILED (exit≠0, not a wall TIMEOUT) → diagnose (read the first real traceback, often masked by the elastic summary) + a dated agent_logs/ entry; recurring identical failures ≠ transient.
• RL collapse rule (≥2 signals fire same step) → flag for cancel+salvage per rl-agentic-job-cleanup. Spike-mitigation ablations OVERRIDE this: a job_name containing zclip/staleclip/stale_clip/ z_clip/maxgn09_hint/shaped_entropy (or any spike-mitigation tag) is NOT auto-scancelled on 2/4 collapse signals — observing whether the mechanism damps the spike IS the experiment (still REPORT the signals, marked "ablation observation, not actionable"). Standard runs (a3/a2/a1-base, no tag) DO follow cancel+salvage. A real crash/NaN/SIGSEGV is still a genuine failure → diagnose.
• New/untested RL → act on the rl-job-health-deep-dive verdict: NO-KILL → note it + the watch-signal that would flip it; KILL → it's our own doomed/wedged job, so (with the standing kill-permission in mind) the supervisor cancels + relaunches on the corrected setting per rl-agentic-launch-iris/rl-*-launch-*, logging the probe + verdict to a dated agent_logs/ entry. Don't sit on a confirmed-garbage run for another 3h.
• Eval stall/zombie/instant-fail red-flags → act per the monitor-job-tables Eval section; DCAgent2/* measurement runs are EXEMPT (report as calibration, not production).
• Writing discipline (agent_logs + ops edits): lead with WHAT (the fact/state/change), concise, no speculation. Ops docs hold validated ground truth only — a doubted/unvalidated claim goes to a dated agent_logs/ entry with a ⚠ pointer in the ops doc, NOT asserted as fact.

5. Respect the standing constraints (reference, don't relitigate)

• RL concurrency cap per cluster (value in ops/<cluster>); a3 series CONCLUDED — do NOT launch/refill a3 rows; enable_db_registration: false in YAMLs → DB registration is the manual cleanup step only; Daytona snapshot org cap is HARD — at the cap clean STALE snapshots, never raise it; cross-user FK safety before ANY Supabase delete/mutate (restrict to rows you own). Full policy → the cron sweep directive + the cleanup skills.

6. Output + record

Post the bucketed tables (RL/SFT/datagen/eval/catch-all) + a short "actions taken / flagged" summary (completions cleaned/handed off, failures diagnosed, health flags, anything launched). Log a standalone dated file under ~/Documents/agent_logs/ (YYYY-MM-DD_<topic>.md) + update the relevant tracker (a3 / MiniMax datagen / Delphi). Skip unreachable clusters (note it) rather than blocking.