agami save-golden

You are writing to the answer key. Goal: get a question — or a bank of them — into the profile's golden dataset in the state it has actually earned, and never in a better one. A dataset is what /agami-eval gates on, so an item that claims to be confirmed when nobody vouched for the answer turns the whole suite green on nothing — and equally, a question the sheet already carries a validated answer for should not sit unconfirmed for no reason beyond which door wrote it down.

This skill has three doors and they write different things. Everything below is organized around them, because confusing them is the one failure that matters:

Door	Input	What it writes	Can it gate a run?
Import	A CSV of questions, or a table pasted into chat	sql_confirmed: true for a row that already carries a statement, false for a bare question	Yes, for the rows that carried an answer
Save	One question, the statement that answered it, the result the person accepted	One item with sql_confirmed: true and its confirmed_by	Yes

An imported row with no statement has none until somebody runs it and saves the answer through the save door. That is the intended shape for a bare question, not a gap to close: it is an honest in-progress case that reports and cannot gate. A row that arrived already carrying a statement is different — it is somebody's own pre-validated answer, brought into the dataset already vouched for, and the import door confirms it on the spot rather than asking for a second confirmation of something it was just told to trust.

This skill never generates SQL for an imported question. Not to be helpful, not "as a starting point", not even marked as a draft. Filling one in would fabricate ground truth, which is the one thing an answer key may not contain — the answer key is what everything else is measured against, so a statement nobody verified corrupts every future verdict rather than just one item. If the user wants an imported question answered, run it (/agami-query), let them look at the result, and come back through the save door with what they accepted.

Spec for the deterministic half: scripts/golden_author.py (column matching, id derivation, the append-only funnel and the write-then-re-read gate). The file shape is shared/golden-dataset-shape.md's.

Conversation style

• One question per turn. This is a tool, not a tutorial. At most two sentences of prose between phases.
• Show before you write. Nothing reaches disk that the user has not seen rendered first — parsed rows as a table, a replacement as a before and an after.
• Never volunteer SQL for a question that has none. See the rule above; it is the reason this skill exists in the shape it does.

---

Phase −1: Plan-mode check

Run the detection + ask logic from shared/plan-mode-check.md. agami-save-golden needs Write (the pasted-table CSV, the item JSON) and Bash (the helper) — both are blocked in plan mode.

If plan mode is active and the user picks Stay in plan mode (or this skill is invoked under an active plan-mode context): refuse and end the turn. DO NOT write a plan file. DO NOT call ExitPlanMode. Refusal text (verbatim):

I can't save a golden item in plan mode — every door here writes to the model tree. Switch to Auto or Edit Automatically mode (Shift+Tab to cycle) and re-invoke me with the dataset name.

If plan mode is not active, skip this phase silently and go to Phase 0.

---

Phase 0: Preflight

1. Plan-mode check — Phase −1 above. Do it first; a refusal halfway through a parse is a confusing partial state.
2. Model present — <artifacts_dir>/<profile>/datasource.yaml must exist. If not, invoke /agami-connect. A dataset belongs to a profile, and a profile with no model has nothing to be scored against.
3. See what exists — ls <artifacts_dir>/<profile>/golden_datasets/. It writes nothing, and it is how you find out whether this is a new dataset or an existing one you are appending to. A missing directory is ordinary: the first write creates it.
4. Which dataset — the one named in $ARGUMENTS, the only one that exists, or ask with AskUserQuestion listing what is there plus "a new one". The name is the filename stem (orders → orders.yaml); the file never declares its own name and the writer never puts one in. A stem is one plain name — letters, digits, dots, dashes and underscores, nothing else. --profile is held to the same rule. Both are joined straight into the path the writer truncates and rewrites, so anything with a separator or a .. in it is refused with exit 2 before a single byte is read; never "helpfully" pass a path where a name is asked for.

Never glob for a dataset outside this profile. shared/golden-dataset-shape.md is the authority on every field and it carries the hard rule verbatim: never read another profile to learn the file shape. A golden dataset is the business definitions and the answer key in one file, so a glob across <artifacts_dir> returns another customer's questions together with the SQL that correctly answers them — a tenant-data leak in a hosted deployment and a lift of somebody's business definitions even locally. The reference has every field; a sibling profile never is the reference.

---

Phase 1: Which door

Route on what the user brought, and say which door you are opening:

• A file path, a spreadsheet, a pasted table, "here are our questions" → the import door (Phase 2).
• A question plus a statement plus a result they just accepted → the save door (Phase 3).
• Both ("import these, and the first one I've already checked") → import first, then the save door for the one they verified. The save door replaces the imported item by id, which is the append-only path in Phase 4 and is exactly right here — say so and confirm it rather than treating it as a surprise.

---

Phase 2: The import door

2a — Get the file

The parse reads the question bank straight from the file, and there is still one parser: a workbook is read into rows and then parsed exactly as a CSV is, so there is one place a column can be misread.

• A .csv path — use it as given.
• A .xlsx path — use it as given. Do not open it with the Read tool (it cannot read a workbook) and do not guess at its contents; the parse reads it. A workbook often holds several sheets — a cover page, a schema reference, the questions — and which sheet holds the questions is the user's call, not yours. Run the parse without --sheet first: a workbook with more than one sheet stops with exit 2 and lists every sheet. Show the list, suggest the one whose name reads like the question bank, and re-run with --sheet "<name>" once they confirm.
• A .xls path (Excel's older binary format) — it cannot be read. Ask the user to save it as .xlsx (or CSV) from Excel and re-invoke.
• A table pasted into chat — write it out as a CSV with the Write tool and then parse that file. One parser, one code path, and the file is also the thing the user can fix and re-run. Per shared/invocation-conventions.md: never a heredoc, never python3 -c, never a shell variable — quoting mangles the commas and quotes that are the whole point of a CSV. Write it to /tmp/agami-golden-pasted-<ts>.csv and tell the user where it went.

The table needs a header row with a question column — question, query, nl question, prompt or ask (case, underscores and hyphens all fold). It does not have to be the first row: a title or a note above the table is fine, and the parse takes the first row within the top 20 that names a question column. Optional columns: id, expected / expected value / answer, sql / statement, tags. A header the contract does not know is left alone — matching is exact, never fuzzy, so an analyst's note column costs nothing. The same exactness means a column is only picked up under one of those names — but a column it can't read that looks like a field (a header with sql, statement, question, prompt, id, answer or tag in it, for a field the sheet hasn't already supplied) is never dropped silently. The parse lists it under unrecognized, and Phase 2c asks about it. Nobody has to rename anything: --column sql="Warehouse SQL" reads that column as the statement.

2b — Parse (this writes nothing)

python3 "$AGAMI_PLUGIN_ROOT/scripts/golden_author.py" parse \
  --file <path-to-csv-or-xlsx> [--sheet "<sheet name>"] [--column <field>="<header>"] \
  > /tmp/agami-golden-parse-<ts>.json

Then Read the file. parse is inert by construction — it has no write in it at all, which is what makes "the rows are confirmed before anything is written" a fact about the software rather than a promise about how you behave.

The payload carries columns (the header as found), header_row (the sheet's own row number for that header), rows, skipped (each with its row number in the user's own sheet and why) and summary. unrecognized lists each unread column that looks like a missing field, with the field it could be. A workbook adds sheet (the tab that was read) and above_header (every row above the header that had anything in it — usually a title — none of which was read). Exit 2 means the parse could not start: no column could be identified as the question — the stderr line names every header it read, which is the whole of what the user needs to rename one and re-invoke — or, for a workbook, the sheet has to be named or the file can't be read, and the stderr line says which.

2c — Render the rows and get an explicit yes

Say what was read, before the rows. For a workbook, name the sheet and the header row ("Read the Questions sheet, header on row 4.") and quote every above_header row, so the user can see nothing they meant as a question sat above the header. A right-looking table from the wrong tab is the one mistake the rows alone won't show.

Ask about every unrecognized column, before anything else. Each is a column the parse did not read whose header looks like a field the sheet hasn't supplied — {"column": "Warehouse SQL", "could_be": "sql"}. Ask about each by its header: "The sheet has a column Warehouse SQL that wasn't read. Is that the SQL for each question? If so, rows that have it will import as confirmed." On a yes, re-run the parse with --column sql="Warehouse SQL" (one --column per field) and show the new payload instead; on a no, carry on without it. Wait for the answer and never map a column on your own, even when the header plainly says SQL, because a statement column the person didn't mean becomes an answer key every future run is scored against.

Show the rows as a markdown table. Not a count, not a summary — the rows:

| id | Question | Expected | SQL in the sheet? | Tags |
|---|---|---|---|---|
| how-many-orders-have-been-placed | How many orders have been placed? | 1,329 | — | orders, smoke |
| revenue-by-channel-2024 | What was revenue by channel in 2024? | — | yes (confirmed on import) | revenue |

Ids are derived from the question when the sheet has none, so they are stable across re-imports — say that, because it is why re-running the same sheet lands on the duplicate path instead of doubling the dataset.

If skipped is non-empty, list every skipped row with its number and reason before asking. A question missing from a dataset is one nobody notices is missing.

Derived ids are unique by construction, but a sheet with its own id column can repeat one. The import refuses the whole batch and names the id, because an id is the key a result is stored under and the second row would otherwise overwrite the first inside one write. The table you just rendered is where to spot it — ask the user which row keeps the id.

Then ask, plainly, naming both counts when they differ: "Import these <N> questions into <dataset>? <K> of them already carry an answer in the sheet and will be written confirmed, able to gate a run immediately; the rest have no answer yet and stay unconfirmed until someone saves one." Wait for an explicit yes. Never skip this and never infer it from the user having handed you the file.

2d — Import

Pass the same parse payload to --rows — the file you just showed them, not a re-derivation of it:

python3 "$AGAMI_PLUGIN_ROOT/scripts/golden_author.py" import \
  --profile <profile> --dataset <dataset> \
  --rows /tmp/agami-golden-parse-<ts>.json \
  > /tmp/agami-golden-import-<ts>.json

--description "<prose>" sets the dataset's description, and is worth passing on a new dataset. A row whose sheet carried a statement is written sql_confirmed: true, with confirmed_by.method naming the sheet as the source; a row with no statement is written sql_confirmed: false, since there is no answer yet for anyone to have confirmed.

Read the exit code before the payload (Phase 4). On 0, report summary.added and where the file is, then say what is still missing: "<N> questions are in <dataset>. None can gate a run yet — ask one of them, and say 'save this as a golden question' when the answer is right."

---

Phase 3: The save door

This is the only door that can produce an item able to gate a run, and it needs three things the user has actually got:

1. The question, as a user would ask it.
2. The statement that answered it — the SQL that ran, not one you are writing now.
3. The result they accepted, and how they checked it.

If any of the three is missing, say which one and stop. Most often this skill is invoked straight after /agami-query returned a result the user is happy with, and all three are in the conversation already; take them from there rather than asking again.

Write the item with the Write tool as JSON — the same rule as any JSON file this plugin passes to a script, and for the same reason (shared/invocation-conventions.md):

{
  "query": "How many paid orders came through each channel in 2024?",
  "sql": "SELECT channel, COUNT(*) AS order_count FROM orders WHERE status = 'paid' AND placed_at >= '2024-01-01' AND placed_at < '2025-01-01' GROUP BY channel",
  "confirmed_by": { "method": "read on screen and accepted by the analyst who asked" },
  "tags": ["orders"]
}

• id is optional — derived from the question when absent, the same way the import door derives it, so saving an answer to an imported question lands on that item rather than beside it.
• recorded is stamped by the script itself and never taken from this payload — omit it. The golden-datasets directory has no gitignore exclusion, so the door does not write a query's result rows to it; a run is judged against expected, re-executed live at score time, never against anything recorded here.
• confirmed_by.method is free text and is required — an answer key whose provenance is blank cannot be audited later, and the script refuses without it. Say how it was checked ("read on screen and accepted", "cross-checked against the finance close"), not who in a way that names a person.
• match is optional and defaults to exact, which is almost always the right answer. Column pairing is by value at every level — neither a column's name nor its position is ever consulted — so "the names or the order might differ" is never a reason to loosen. values forgives exactly two things: a floating-point tail, and an extra column the question did not ask for. Reach for it when the answer carries a real float, and not for a count or a result whose columns the question already names — over-answering is the most common way a generated statement is wrong, and values is the level that forgives it. bounded takes a bounds block, for an answer that legitimately moves — see shared/golden-dataset-shape.md. bounded with no band is refused with exit 2: a level with nothing to consult would keep passing forever.
• must_filter gates how the answer was reached. Add it when the question only means what it says with a filter in place (must_filter: [status]).

match, bounds and must_filter all reach the written item exactly as sent — which is why a replacement has to repeat whatever the existing item carried (Phase 4).

python3 "$AGAMI_PLUGIN_ROOT/scripts/golden_author.py" save \
  --profile <profile> --dataset <dataset> \
  --item /tmp/agami-golden-item-<ts>.json \
  > /tmp/agami-golden-save-<ts>.json

A statement that departs from this profile's own examples stops here too, with exit 1 and a needs_confirmation_convention payload. The library of curated examples is where a team's conventions live — which of two equally correct date columns they answer with, how they phrase a join — and it is what agami reads when it answers the question for real. An answer key that contradicts it will fail every future run and blame the model.

The payload carries the nearest example's question and statement, and the claims that differ. Show both statements and ask, in one line:

Your examples for this kind of question filter on created; this one uses opened. Save it anyway?

It is a question and never a refusal. A golden item may legitimately depart from convention — that is sometimes exactly why one is written. On an explicit yes, re-run the same command with --confirm-convention appended. On a no, nothing was written, and the fix is usually to the statement rather than to the dataset.

Only three claims can stop a save this way — the tables read, the filters written, and the resolved date window — because those are the ones that change which rows are counted. Two statements answering one question differ in ordering and limit all the time, and stopping for those would make this a prompt people learn to click through.

A relative question over a frozen answer key is refused here, with exit 2: "how many orders in the last 7 days?" names a window that slides forward and SQL pinned to a fixed date does not. Either anchor the statement to the current date (CURRENT_DATE - INTERVAL '7 days', the dialect's spelling) or rewrite the question to name the window it means (…in Q1 2024?), then re-invoke. Do not save it and plan to fix it later — the whole point of refusing at save time is that the one person who can still fix it is here.

---

Phase 3b: The curation door — the explorer page's queued actions

The explorer page (/agami-eval, or rendered directly) shows a profile's datasets and lets the reader queue changes against them. It writes nothing itself; it hands back one block. This is the door that applies it.

Render the page:

python3 "$AGAMI_PLUGIN_ROOT/scripts/render_golden_datasets.py" \
  --profile <profile> \
  --out "<artifacts_dir>/local/eval/<profile>/datasets-<ts>.html"

--out must land under local/, the gitignored half — the page carries every confirmed answer key in full, and that is only safe where it is not committed. The script refuses any other destination.

When the user pastes the block back, parse it before you read it:

python3 "$AGAMI_PLUGIN_ROOT/scripts/parse_golden_feedback.py" \
  --block-file /tmp/agami-golden-block-<ts>.txt \
  > /tmp/agami-golden-ops-<ts>.json

Read needs_judgment first. If it is confirmation_cannot_be_granted, the block tried to set an answer key from the page. Nothing in it applies — not the offending op and not its neighbours. Tell the user what it names and stop; confirming an item means running it and accepting the result through the save door above, never editing a page. anomalies is the softer list: ops that were dropped and the rest carried on.

Then apply, one dataset at a time. The block's profile: line names its target — use that, not the active profile, or a page rendered for one model applies to another:

python3 "$AGAMI_PLUGIN_ROOT/scripts/golden_author.py" apply \
  --profile <the block's own profile> --dataset <stem> \
  --ops /tmp/agami-golden-ops-<ts>.json \
  > /tmp/agami-golden-apply-<ts>.json

Exit codes are Phase 4's, unchanged. On 1 the payload carries needs_confirmation for the edits and needs_confirmation_removals for the deletions — the latter has a before and no after, because the whole of what somebody is agreeing to is what disappears. Render the removals' questions and answer keys, not their ids. An id is a slug; nobody can decide from it.

What may be queued, and the one thing that may not. Six actions: add or remove a tag, set match, edit the question, remove the item, and withdraw confirmation. Nothing here grants confirmation. The page has no control for it, the parser refuses a block that asks for it, and this door knows no verb that does — three layers, because a queued action that could confirm an item is the cheapest possible way to make a failing suite green.

---

Phase 4: Exit codes, and the append-only stop

Read the exit code before the payload. Both write doors share it:

• 0 — written. Report added / replaced and the path.

• 1 — needs confirmation. Nothing was written. Not a failure and not a success; a pipeline that treated it as either would be wrong in both directions. This is the only thing that produces 1 — every other outcome, expected or not, is 2 with a sentence on stderr — so a 1 is never a crash.

  Read which key the payload carries before you answer it. There are three, and they take different flags:

  Key	What it means	Clears with
  needs_confirmation	A write would replace an item that already exists. Carries {id, before, after}.	--confirm-replace
  needs_confirmation_removals	A queued action would delete an item. Carries {id, before} and no after.	--confirm-replace
  needs_confirmation_convention	The statement departs from the profile's own examples. Carries the nearest example and the claims that differ.	--confirm-convention

  A payload may carry more than one — a replacement whose statement also departs needs both flags. If you ever see 1 with none of the three, stop and report it rather than re-running with a flag.

• 2 — cannot start. The stderr line (prefix agami-save-golden:) says why. Nothing was written, or a failed write was rolled back to the bytes that were there before.

On a needs_confirmation payload — a list of {id, before, after} — render both sides — a markdown table or two fenced blocks per id, the existing item and the one that would take its place. "This id already exists" is not enough for anyone to decide with: the thing being overwritten is an answer key, and they have to see what they would lose.

A replacement is wholesale — the item that is written is the item you sent, so anything the existing one carries and yours does not (its tags, a must_filter, a match) is gone. Read the before and carry forward what still applies before you ask; that is also what the user is agreeing to when they look at the two sides.

Then ask, and only after an explicit yes re-run the same command with --confirm-replace appended:

python3 "$AGAMI_PLUGIN_ROOT/scripts/golden_author.py" save \
  --profile <profile> --dataset <dataset> \
  --item /tmp/agami-golden-item-<ts>.json --confirm-replace \
  > /tmp/agami-golden-save-<ts>.json