add-gdp-scatter

Bulk-add a scatter view to a set of OWID charts, with the same defaults the admin UI's applyDefaultsForScatter applies — plus extra parity checks against a reference scatter chart.

When to invoke

The user pastes (TSV or CSV) a table like:

chart_admin_url	target_chart_admin_url	gdp_source
https://admin.owid.io/admin/charts/1035/edit	https://admin.owid.io/admin/charts/7842/edit	Maddison Project Database
https://admin.owid.io/admin/charts/6305/edit	https://admin.owid.io/admin/charts/6918/edit	World Bank

• chart_admin_url — the existing reference scatter chart used as the source of parity hints (yAxis log, y display.name, color/size override, tolerance, exclusions).

• target_chart_admin_url — the chart the user wants to gain a scatter view.

• gdp_source — one of (case-insensitive, substring matches accepted):

  • World Bank / WDI → variableId 1294305 (WDI 2026-07-27)
  • Maddison / Maddison Project Database → 900793
  • PWT / Penn World Table → 1108541

  GDP_SOURCES in the script is the authority on these ids; the WDI one goes stale on every WDI update — see the version check below.

The admin host that gets written to is OWID_ENV.admin_api, which auto-resolves to staging-site-<branch> on a feature branch. Confirm the branch before running.

Pre-flight: GDP version check

Before processing any rows, the script queries variables for the latest id matching each canonical GDP-per-capita catalogPath pattern (worldbank_wdi/.../ny_gdp_pcap_pp_kd, ggdc/.../maddison_project_database#gdp_per_capita, ggdc/.../penn_world_table#rgdpo_pc). If the latest id differs from the one hardcoded in GDP_SOURCES, a WARN is printed at the top of stdout with the newer id and catalogPath. The script does NOT auto-switch — update GDP_SOURCES (and the feedback_scatter_gdp_picker memory) deliberately if the new version is the one we want.

Read that WARN before the run, not after. These ids go stale on every WDI update, and the "target already has x → leave it" guard means a re-run will not repair a target that already received the stale variable — the x dimension has to be rewritten directly, chart by chart. Cross-check against the id the source scatters actually plot: if the sources are on a newer version than GDP_SOURCES, the newer one is what you want, and bumping first saves a repair pass. (2026-08-04: every source plotted WDI 2026-07-27 1294305 while the script was still pinned to 2026-02-27 1204826.)

What the script does (per row)

Mirrors the admin's applyDefaultsForScatter and the extra moves we agreed on:

1. Adds ScatterPlot to chartTypes, preserving existing tabs. Seeds the schema default [LineChart, DiscreteBar] when chartTypes is unset.

2. Appends x (the chosen GDP variable), color, size dimensions if absent.

   • color: if the source uses a non-default color variable (e.g. World Bank income groups), mirror it; otherwise use CONTINENTS_ID=900801.
   • size: the rule is always use the default Population indicator (POPULATION_ID=953899) for any population-type size. If the source sizes by any population variant (regular, historical, WPP, …), the target gets the default Population. A genuinely non-population size (e.g. GDP, area) is mirrored as-is but raises a WARN so the bubble sizing gets a manual review. If the source has no size dim at all, the target also gets none — the script won't add sizing the curator deliberately omitted. Population variants are detected by the variable's name starting with "Population" or its catalogPath living under a /population/ dataset; the action note records any normalization.

3. Sets matchingEntitiesOnly: true.

4. Sets xAxis to scaleType: log + canChangeScaleType: true.

5. Y-axis log toggle (not forced): when the source scatter is scaleType: log, only enable the toggle (canChangeScaleType: true) and leave the default linear. yAxis is shared across all views, so forcing log would flip the line/bar views too. Mirrors explicit yAxis min/max bounds the source sets (each bound copied independently) — except a non-zero min is NOT mirrored when the target has a Marimekko or Stacked* view, because those draw from a baseline and a scatter-tuned non-zero min would make them start above zero (misleading). DiscreteBar is not in that set: DiscreteBarChart.yAxisConfig hardcodes min: undefined and anchors at zero, so it ignores yAxis.min outright — withholding the min from a DiscreteBar target protects nothing and costs the scatter a well-fitted axis. A degenerate min: 0 + max: 0 (collapsed axis) has its max stripped. Note: y-axis bounds affect all views, not just scatter.

   On y-axis bounds, prefer removing to pinning. Axis.updateDomainPreservingUserSettings takes min(config.min, data.min) and max(config.max, data.max), so an authored min is a hard floor across every view — which is why {min: 0, max: 0} renders identically to {min: 0} (the data max always wins) and is not the inert junk it looks like. When a reviewer says the scatter's axis is wasting space at zero, dropping min usually beats mirroring the source's: each view then fits its own data, and mirroring a non-zero min can clip a LineChart whose series run below it. (2026-08-04, chart 2201: source min was 5, but the line data reaches 0.92; dropping min gave the scatter 6.94–13.97 while the line chart moved only 0 → 0.92, and the DiscreteBar was unaffected either way.)

The scatter's single-year default needs no config (usually)

A recurring question: the scatter should show only the latest year, without affecting the other views — is that possible? Yes, and Grapher already does it — do not set minTime/maxTime for this.

minTime/maxTime are global. Only the map has its own time (map.time / map.startTime in MapConfigInterface); there is no per-chart-type time override, so pinning them to the latest year would collapse a LineChart to a single point.

Grapher handles it at runtime instead. checkSingleTimeSelectionPreferred returns true for the ScatterPlot tab whenever the scatter is not the primary chart type and the chart is not in relative mode, and adjustStateForTab → ensureTimeHandlesAreSensibleForTab then collapses both time handles onto the end (latest) time. That is runtime state, so the line/bar/map views keep their full range. Since the applier appends ScatterPlot, it is never chartTypes[0] and the condition holds by construction.

But it only fires when the reader CLICKS the tab — see the shared caveat below.

Also: hideTimeline: true breaks it even on a tab click. With a hidden timeline, timelineHandleTimeBounds reads the authored minTime/maxTime on every chart tab and ignores the runtime handles, so the collapse never takes effect — and the reader has no slider to fix it. Authored minTime == maxTime is then the only fix, and it is only safe when every other tab is single-time anyway (DiscreteBar/StackedDiscreteBar/Marimekko). With a LineChart, SlopeChart or single-indicator Dumbbell in the mix, one global time cannot serve both — un-hide the timeline or accept the range. The script emits a WARN for each case. (2026-08-04: chart 1253, DiscreteBar + hideTimeline, needed minTime/maxTime = latest; the other 16 targets in that batch needed nothing.)

adjustStateForTab fires on a tab CLICK only — not on a direct URL load

Both scatter adjustments — collapsing the time handles and clearing the entity selection — live in the same function behind the same guard:

if (!this.isEditor) {
    this.ensureEntitySelectionIsSensibleForTab(tab)
    this.ensureTimeHandlesAreSensibleForTab(tab)
}

So they always happen together, or not at all. adjustStateForTab has exactly one production caller, onTabChange, which in turn has exactly one: the ContentSwitchers tab control. A tab supplied in the URL takes a different path — populateFromQueryParams → setTab, which only assigns this.tab. Three consequences:

• Clicking the scatter tab: time collapses to the latest year and the selection is cleared. The scatter matches the old standalone chart.
• Landing directly on ?tab=scatter: neither happens. The scatter opens on the authored time range with the authored entities highlighted — unless the URL says otherwise. This is the path Part 2's redirect uses, which is why every part of its stored tab=scatter&time=latest&country= is load-bearing: each param hand-supplies one adjustment the click would have made. time=latest stands in for ensureTimeHandlesAreSensibleForTab, and country= (present, empty) for ensureEntitySelectionIsSensibleForTab — parseCountryParam returns valid([]) for an empty value and setSelectedEntities([]) clears, so the scatter shows every entity unhighlighted. Whenever a new tab-click adjustment is added upstream, a matching param has to be added here or the two paths drift apart again. country= has not yet been exercised through a live redirect — confirm in a browser on the first --apply run (see "Verifying Part 2").
• The admin editor shows neither, because of the isEditor guard — deliberate, so switching tabs cannot mutate the authored config on save (grapher #6794). A scatter that looks wrong in /admin/charts/<id>/edit may be fine for readers. Verify on the chart page.

The target's entity selection highlights the scatter, it does not filter it

Targets normally carry a selectedEntityNames list for their line/bar view (4–20 entities is typical) while the source scatters carry none — they show every country. That asymmetry does not hide data on the new scatter view:

• ScatterPlotChartState.seriesNamesToHighlight uses the selection to highlight only; every entity is still plotted.
• Axis domains narrow to the selection only via pointsForAxisDomains, and only when zoomToSelection is set. Check that field — with it, a scatter's axes really would zoom to the highlighted subset.
• On a tab click, ensureEntitySelectionIsSensibleForTab clears the selection entirely (CHART_TYPES_THAT_SHOW_ALL_ENTITIES is [ScatterPlot, Marimekko]) so long as it is still the authored one — the scatter then looks exactly like the old standalone chart. On a direct URL load it does not, for the reason in the section above; the authored entities render highlighted.

That second case is what Part 2's redirect produces, so a reader arriving by a retired scatter's URL sees the same data and axes but with a few countries emphasized — visually unlike the chart they used to get, and unlike what a reader who clicks the tab gets. Decide per batch whether that is acceptable, and see the country= note above for the fix.

Cross-view safety (which fields are global)

yAxis (scaleType, min, max) is the only config the skill writes that meaningfully bleeds into the non-scatter views — hence the log-toggle and zero-baseline handling above. The others were checked and are safe: xAxis.scaleType: log is ignored by Line/DiscreteBar (they hardcode a linear time axis) and has no visible effect on Slope; the color dimension does not recolor line/bar (they color by entity); size is scatter-only (not even in the table tab); matchingEntitiesOnly is honored only by Scatter and Marimekko. 6. Mirrors source's manually-set y display.name when present. 6b. Mirrors source comparisonLines when the target has none. A scatter's reference line (e.g. yEquals: 1 on a ratio-to-a-benchmark indicator) is often the whole point of its framing, so dropping it makes the migrated view say less than the chart it replaces. Never overwrites an existing set. 7. Emits warnings (no action) for:

• Target has no selectedEntityNames — line/bar/slope views will fall back to Grapher defaults.
• Target stackMode: relative — on scatter this is the "Display average annual change" mode; we want the toggle available but off by default, so a relative default is flagged for review.
• Source excludedEntityNames — never applied to the target (they would hide the entity from all views, not just the scatter), so each one reappears on the migrated scatter. Graded per entity by classify_exclusions into y-OUTLIER / aggregate / high-GDP-material / unclear / ungradeable (a decision is needed) vs high-GDP / no data (benign), with the numbers in the EXCLUDED ENTITIES table. Only the first group makes the note a WARN — the group is EXCLUSION_WARN_CLASSES, which the table's own footer prints, so the two cannot drift. Note that a high GDP per capita is benign only while it stays inside X_MATERIAL_DECADES; past that it grades high-GDP-material and warns like the rest.
• Source y axis is log — the target's scatter tab opens linear, and only a URL carrying yScale=log restores it. See "A log y axis and an exclusion list are the two things the migration cannot carry".
• GDP coverage mismatch — if y-indicator's earliest year predates the chosen GDP's coverage (WDI≈1990, PWT≈1950, Maddison≈year 1), suggest a deeper-history alternative.
• Few entities on default scatter view — counts entities with both a y- and an x-value within tolerance at the default time; if fewer than ~15 AND source uses higher tolerance, recommends bumping target's y display.tolerance.

Push uses apps.chart_sync.admin_api.AdminAPI.update_chart(id, cfg).

A log y axis and an exclusion list are the two things the migration cannot carry

Everything else on the source is either mirrored onto the target or left behind for a reason that holds. These two are different — they are lost, and the only channel that gives either back is a query string:

• A log y axis stays behind because yAxis is global (step 5). Part 2's redirect and a hand-updated article link carry yScale=log; a reader who clicks the scatter tab does not, and neither does any surface that has no URL of its own.
• excludedEntityNames is never applied to the target (exclusions are global too, so they would hide the entity from its line/bar/map views), so every excluded entity reappears on the migrated scatter. Nothing, anywhere, puts it back.

The surfaces with no query string are what decide whether the retirement is worth doing, and there are three:

• a key-chart slot has nowhere to put one — GdocPost.loadRelatedCharts selects only chartId, slug, title, variantName, keyChartLevel, and RelatedCharts renders <GrapherWithFallback slug={activeChartSlug}>;
• a gdoc embed resolves the chart itself and renders its default tab (makeGrapherLinkedChart builds no query string);
• a featured metric is worse still: it names a chart, an MDIM view or an explorer view and never a chart's tab, so the scatter view cannot be featured at all (see "Featured metrics").

On a featured or embedded source, a log axis is therefore gone for good and no amount of re-pointing recovers it. That is what makes "is this migration worth doing?" a real question rather than a formality, and why the answer depends on how the old chart is referenced.

Leaving the standalone chart alone is a legitimate outcome. The skill reports the loss and the topic owner decides: the applier WARNs on a log source, the reviewer HTML asks the question with both shapes side by side, and Part 2's audit prints a RECONSIDER block weighing the loss against the blast radius. None of them blocks — see "RECONSIDER" in Part 2 for why not.

Exclusions are graded, not listed, because the two usual reasons for one have opposite consequences here. The target's x axis is log, so a very high GDP per capita — the classic Ireland / Luxembourg / Qatar exclusion — costs almost nothing: on chart 6305, Ireland's $131,338 against a pack topping out at $95,173 is +0.14 of a decade of extra axis width, i.e. invisible — though "benign" there is a claim about the axis, and the note says so: a very high GDP per capita can also be excluded because the figure itself is distorted (Ireland's profit shifting, a Gulf state's expat denominator), which a log axis does not fix. A y outlier is the opposite: on chart 5029, Australia's 3,243 ha average farm size against a pack of 0.35–582.5 stretches the y axis 5.6x, and yAxis.max is global so the scatter cannot cap it alone. That is why grade_exclusion measures each axis in the units it is drawn in rather than testing for statistical outlierness — a symmetric IQR fence gets skewed indicators badly wrong (chart 1131: Cape Verde's 40.3 kg/ha cereal yield sits well inside a ±3·IQR fence while being 14.3x below the lowest of the other 88 countries).

The two pack tests (Y_PACK_FACTOR, "N× above the highest / below the lowest") are ratios, so they only run against a positive bound. On an indicator whose values are negative, hi_y × 2 sits below the pack, which would make an ordinary in-range value read as an outlier, and a pack topping out at exactly 0 would divide by zero. Those indicators are graded on the span stretch alone, which is sign-agnostic and still catches a genuine outlier.

One case needs its own test rather than a ratio: an excluded value at or below zero while every peer is positive. That is the limiting case of "below the lowest" — infinitely far below — so no ratio expresses it, and the span stretch does not cover it either, because a broad pack absorbs the extra width (y=0 against a pack of 1–100 stretches the axis only 1.01×). It is graded y-OUTLIER on the sign alone. This matters because the pack tests, not the stretch, are what catch this whole shape of case: chart 1131's Cape Verde is caught by "below the lowest" alone, its stretch being just 1.04×.

Both sides of that comparison are read at the same year. An entity with no value at the target's default year is graded at its latest year with both indicators instead, and the peer pack is rebuilt at that year rather than held at the default one — otherwise a trending indicator has the point and the pack drifting apart, and the verdict measures the trend rather than the entity. The measured year is printed whenever it is not the default.

That fallback year honours the target's tolerance, and is not a raw-year intersection: at a non-zero tolerance Grapher pairs a y value with a GDP value from a neighbouring year, so an entity whose two observations never share a year can still be a point the reader meets by dragging the timeline. Requiring the same year graded it a benign no data and dropped it out of the warning altogether. The value is then read back at that same tolerance, since the observation itself may sit a year or two off the timeline year it was found under.

The candidates for that year are every year the two variables cover, not just the entity's own observation years, because the year that pairs them can be one where the entity has neither: y in 2000 and GDP in 2002 at tolerance 1 meet at 2001, a year on the timeline because other entities have data there and a year the reader reaches by dragging the handle. Searching only the entity's own years missed it and sent the entity back as a benign no data.

no data is also checked before the OWID_ code, so an excluded aggregate that has no pairable year is no data rather than aggregate. The aggregate note claims the entity "renders as one point among the countries"; an entity with no pair renders nowhere, since matchingEntitiesOnly hides it. Testing the code first raised a warning — and a RECONSIDER row — on the strength of a sentence that was not true of that entity.

(2026-08-19, production: of 22 published GDP scatters carrying excludedEntityNames, 8 exclude World or another OWID aggregate — the single commonest case, which is why aggregate is its own class, detected by the OWID_ prefix on entityCode rather than by a name list. Charts 1131 and 5029, both of which exclude a genuine y outlier, are also key charts on their topic pages — the compound case where the loss lands where no query string reaches.)

Run this as a checklist in the chat

Create a TodoWrite list covering the WHOLE migration on the first step, before touching anything — not just the part being worked on now. This migration's failure mode is not getting a step wrong, it is losing a step: the work spans two scripts, a human review round, a merge, and a production run, with days between them. Anything not on the list from the start gets discovered later by a reader hitting a 404 or an article rendering the wrong tab.

So the reference sweep and Part 2 go on the list as pending from the very beginning, even when the request is only "add the scatter views". They are the two that get forgotten, and they are the two that break things for readers.

The canonical items, in order:

1. Confirm the branch / which admin host OWID_ENV resolves to (on master that is production).
2. Pre-flight every row (preflight_targets.py); report and drop the blocked ones.
3. Act on the GDP version WARN — bump GDP_SOURCES before applying if the sources plot a newer id.
4. Apply (apply_scatter_defaults.py).
5. Verify every target: ScatterPlot present, log x-axis, and the current GDP id on x.
6. Display-name follow-up — after the final applier run, or the next run re-mirrors it.
7. Build the review HTML (build_review.py) and hand it to the topic owner.
8. Apply the reviewer's flagged notes; regenerate the HTML and re-import their JSON.
9. Chart-diff sign-off on staging, then merge.
10. Confirm the scatter views actually reached production. A merged PR is not evidence that they did: chart-sync only carries chart edits whose diffs were approved in Chart Diff, so a PR can merge green with every row ✅ on staging and leave production untouched. An abandoned first attempt (PR #6173, merged 2026-06-24) left production untouched on all seven of its pairs — deliberately: the target_query_param needed for Part 2 did not exist yet, so it was dropped and the migration restarted from scratch rather than left half-done. Whatever the reason, check production directly rather than inferring it from the merge.
11. Reference sweep on the old charts — find-chart-references over each source slug and its aliases, then scripts/build_reference_handoff.py to turn it into the handoff (it keeps the sweep's 📄 doc / 👁 preview / 🔗 page links and its "Find in the doc" search string — see below). Re-point embeds and links at the target's scatter view before retiring anything: an embed is never fixed by a redirect, and a link that works only via a 301 outlives everyone's memory of why. Do not skip this because the Part 2 audit reports few references — it counts a narrower set; see the key-chart and featured-metric traps below. Settle the ⭐ featured-metric rows in the same pass: they are the only ones that cannot be repaired after the unpublish.
12. Narrative charts on the sources: replace where the parent is being retired (create → re-point articles → delete; never delete first).
13. Part 2 audit — redirect_to_scatter.py with no --apply. Read every verdict, and resolve every RECONSIDER row with the topic owner before item 14. That block is the one verdict here that does not block on its own (a lossy retirement is an editorial call, not a broken page), so it is the one that gets applied past if nobody answers it.
14. Part 2 --apply on staging, then the browser checks in "Verifying Part 2".
15. Part 2 --apply --allow-production once the scatter views are live on production, then the same checks against the live site.