Add provider regions

Add an external data provider's regional grouping (World Bank, WHO, UN, IEA, Pew, Maddison, WID, ILO, …) to OWID's shared regions dataset. The regions are defined once in regions.yml; the grapher step then turns each provider into a categorical country→region map indicator ({defined_by}_region) that powers the world-region-map-definitions page.

There are two repos involved: the etl repo (Steps 1–8 — definitions, indicators, metadata, optional chart migration) and the owid-grapher repo (Step 9 — registering the provider in the frontend so its regions show up in entity selectors, tooltips, and admin presets). The owid-grapher work is a separate PR done after the ETL regions are merged and published to the catalog.

The defining principle of this skill: the country composition of each region must come from the source — either the provider's own dataset or a reference document the provider publishes — and be verified by set-equality. Never hand-type or guess which countries belong to a region.

Inputs

• Provider — name and a short slug for defined_by (e.g. wb, who, maddison). Region names will be suffixed (Provider).
• Dataset path (if one exists) — the provider's garden/grapher dataset, e.g. data/garden/<ns>/<version>/<short_name>. Many providers ship their regional grouping inside their own dataset.
• Reference URL (optional) — where the provider publishes the classification, used only if the dataset doesn't encode it.

All paths below are repo-relative. Always use .venv/bin/ for python/etl/etlr.

The files you'll touch in etl:

• etl/steps/data/garden/regions/2023-01-01/regions.yml — region definitions (the header comment documents every field).
• etl/steps/data/grapher/regions/2023-01-01/regions.py — builds the {defined_by}_region indicators (only edit for a cross-tier back-fill, Step 3).
• etl/steps/data/grapher/regions/2023-01-01/regions.meta.override.yml — origin anchors + per-indicator metadata.
• Not regions.codes.csv — that file lists countries and OWID historical codes only; provider aggregates never go there.

And in owid-grapher (Step 9, separate PR): the auto-generated packages/@ourworldindata/utils/src/regions/regions.data.ts, a few hand-maintained label registries, and the two region-color dictionaries in packages/@ourworldindata/grapher/src/color/CustomSchemes.ts. Reference PRs: owid/owid-grapher#6465 (IEA regions) and #6852 (regions colored by name on categorical maps).

---

Step 1 — Find the region composition (the key check)

Inspect the provider's dataset for an embedded aggregation, in this priority order:

1. A categorical region / subregion column mapping each country to its region.
2. Region entities present as rows in the data (e.g. "East Asia (Provider)" alongside countries).
3. A table-of-contents / dictionary table with tier columns (some providers ship *_region, *_subregion_broad, *_subregion_detailed style columns — these are authoritative tier definitions).

from owid.catalog import Dataset
ds = Dataset("data/garden/<ns>/<version>/<short_name>")
print(ds.table_names)
tb = ds.read("<table>", safe_types=False)
print([c for c in tb.columns])
# region column -> country mapping:
print(tb.dropna(subset=["region"]).groupby("region")["country"].unique())
# or region entities present in the data:
print(sorted(c for c in tb["country"].unique() if "(" in str(c)))

If the composition is in the data: derive each region's member set directly from it. This is the source of truth.

If it is NOT in the data: ask the user for a reference — a link, PDF, or doc where the provider publishes the classification (e.g. a "regional groupings" page or methodology annex). Fetch it with WebFetch and derive membership from there. Keep the reference URL; it becomes the origin url_main in Step 6.

Lesson: membership comes from the source and is verified by set-equality — not from judgment or memory.

---

Step 2 — Resolve members to OWID region codes

Each member must map to a code that exists in regions.yml. Use regions.codes.csv for ISO alpha-2/alpha-3, with a name/alias fallback for non-standard codes (microstates, Kosovo-style cases).

import csv, yaml
regions = yaml.safe_load(open("etl/steps/data/garden/regions/2023-01-01/regions.yml"))
by_code = {r["code"]: r for r in regions}
name_to_code = {}
for r in regions:
    name_to_code.setdefault(r["name"], r["code"])
    for a in r.get("aliases", []):
        name_to_code.setdefault(a, r["code"])
alpha2, alpha3 = {}, {}
for row in csv.DictReader(open("etl/steps/data/garden/regions/2023-01-01/regions.codes.csv")):
    if row["iso_alpha2"]: alpha2[row["iso_alpha2"]] = row["code"]
    if row["iso_alpha3"]: alpha3[row["iso_alpha3"]] = row["code"]

def resolve(member):  # member is a provider country name or code
    return alpha2.get(member) or alpha3.get(member) or name_to_code.get(member)

unresolved = [m for m in provider_members if resolve(m) is None]
assert not unresolved, f"Resolve these before continuing: {unresolved}"

Handle deliberately:

• Historical entities the provider assigns to a region (OWID_USS, OWID_CZS, OWID_YGS, OWID_SDN, …) — include them where the source does.
• OWID aggregate codes (e.g. Channel Islands OWID_CIS) — if the provider lists a sub-territory that OWID models as an aggregate, decide once where it lives and avoid listing it twice across regions (the garden step's duplicate-member check will catch a double-count). Document any such choice with a # NOTE: in the YAML.
• A region's members may reference other aggregate codes (not just countries); the garden step's replace_aggregate_members expands them recursively (see UN M49 in regions.yml for the pattern).

---

Step 3 — Decide the tier structure

• Single flat partition (most providers): one defined_by: <provider>, one indicator. Done.
• Hierarchical provider (broad regions split into subregions): use one defined_by per level — <provider>_1 (broadest), <provider>_2, … Look at the existing un_m49_1/2/3 and ilo_1/ilo_2 sections in regions.yml as templates.

Why split: the grapher step inverts all aggregates sharing a defined_by into a single country→region column. If two tiers share one defined_by, every country lands in 2+ regions, producing "belongs to multiple regions" warnings and a scrambled, order-dependent indicator. One defined_by per level keeps each indicator a clean partition.

Two rules for multi-tier providers:

• Completeness: each kept tier must partition the provider's covered world. You cannot keep a sub-breakdown of one parent without its siblings — e.g. if you keep one parent's sub-regions, keep every parent's, so the tier still covers everyone. Drop intermediate levels that don't form a complete partition; keep only tiers that are both useful and complete.
• Region shared across tiers: when one region exists at two levels (a broad region with no finer breakdown), it carries a single defined_by, so it appears natively in only one indicator. Tag it at one level, then back-fill it into the other level's indicator in the grapher step — mirror the existing process_un_definitions pattern in grapher/regions/2023-01-01/regions.py (one extra fillna/masked assignment after the inversion loop).

---

Step 4 — Edit regions.yml

Append a section, delimited like the others:

##########
# <Provider full name>
##########
- code: PROVIDER_XXX
  name: "<Region> (<Provider>)"
  region_type: "aggregate"
  defined_by: <provider>            # or <provider>_1 / <provider>_2 for tiers
  members:
    - ISO3
    - ISO3
    - PROVIDER_SUB                  # may reference another aggregate code

• Names carry the (Provider) suffix and should match the entity names the provider's own dataset publishes (so charts line up).
• Codes are PROVIDER_XXX, uppercase, unique.
• For composite levels whose members are sub-aggregates, you can either list the sub-aggregate codes (expanded recursively) or list the union of countries directly. When you need the country union, compute it programmatically rather than hand-typing:

def expand(code):
    out = []
    for m in by_code[code]["members"]:
        out.extend(expand(m) if m in by_code and m.startswith("PROVIDER_") else [m])
    return sorted(set(out))

For anything beyond a couple of regions, regenerate the whole provider section with a small script (compute members, emit the YAML block, splice it into the file) rather than many manual edits — it's less error-prone and keeps formatting uniform.

---

Step 5 — Build and verify the garden step

.venv/bin/etlr data://garden/regions/2023-01-01/regions

(No --force — editing the YAML is enough to trigger a rebuild.) The step runs sanity checks: unique codes/names, unique members within a region, all referenced codes exist, and cycle detection during aggregate expansion.

Then verify against the source and the partition property:

from owid.catalog import Dataset
import json
tb = Dataset("data/garden/regions/2023-01-01/regions").read("regions").reset_index()

# 1) set-equality vs the source mapping (expected = region -> set of OWID codes from Step 1/2)
m = {r["code"]: set(json.loads(r["members"])) for _, r in tb[tb["defined_by"].str.startswith("<provider>")].iterrows()}
for code, exp in expected.items():
    assert m[code] == exp, f"{code}: missing {exp - m[code]}, extra {m[code] - exp}"

# 2) each tier partitions the same set and is pairwise-disjoint
for tier, codes in {"<provider>_1": [...], "<provider>_2": [...]}.items():
    union = set().union(*(m[c] for c in codes))
    for i, a in enumerate(codes):
        for b in codes[i+1:]:
            assert not (m[a] & m[b]), f"{a} & {b} overlap in {tier}"
    print(tier, len(union), "countries")

If a sanity check fails, fix the upstream logic or the membership — don't suppress the assertion.

---

Step 6 — Grapher indicators + metadata

The grapher step auto-creates a {defined_by}_region column for every defined_by. Each needs a metadata block or the build fails on a missing title (grapher_checks).

6a. Origin anchor — add to the definitions: block in regions.meta.override.yml, taken from the provider dataset's actual origin:

from owid.catalog import Dataset
ds = Dataset("data/garden/<ns>/<version>/<short_name>")
tb = ds.read(ds.table_names[0], safe_types=False)
o = tb[[c for c in tb.columns if c not in ("country", "year")][0]].metadata.origins[0]
print(o.producer, "|", o.title, "|", o.url_main, "|", o.date_accessed, "|", o.attribution, "|", o.attribution_short)

  origins_<provider>: &origins_<provider>
    producer: <producer>
    title: <title>
    url_main: <url_main>              # or the reference URL from Step 1
    date_accessed: "<YYYY-MM-DD>"
    attribution: <attribution>        # only if the source defines it
    attribution_short: <short>        # only if the source defines it

Omit date_published. Region definitions are time-invariant; a publication year would render next to the source line below the chart, where it's meaningless.

6b. Indicator block — one per {defined_by}_region, under tables.regions.variables:

      <provider>_region:                       # or <provider>_1_region / <provider>_2_region
        title: World regions according to <Provider>
        description_short: |-
          Regions as defined by <Provider full name>.
        type: ordinal
        sort:                                    # legend/map order — see ordering rule below
          - <Region> (<Provider>)
          - ...
        origins:
          - *origins_<provider>
        presentation:
          grapher_config:
            hideAnnotationFieldsInTitle:
              time: true                        # hide the placeholder data year in titles
            map:
              tooltipUseCustomLabels: true          # tooltip shows the stripped label too — see below
              colorScale:
                baseColorScheme: OwidCategoricalMap    # name-keyed region colors — see Step 9
                customCategoryLabels:
                  # one entry per region in `sort` — drops the suffix from the
                  # legend, and from the tooltip via the flag above
                  "<Region> (<Provider>)": "<Region>"
                customHiddenCategories:
                  "No data": true
                # No customCategoryColors. Colors live in MapContinentColors (Step 9),
                # keyed by region name; a block here would override them and fork the
                # source of truth.

customCategoryLabels alone only fixes the legend. Hovering a country still shows the raw "<Region> (<Provider>)", because the map tooltip falls back to the unformatted value unless map.tooltipUseCustomLabels: true is set (MapChartState.formatValueForTooltip — it looks up the bin's label only behind that flag). Set both, always, and give every region in sort a label entry: a region you miss keeps its suffix in the legend and the tooltip while its siblings lose theirs, which reads as a data error rather than a missing config line.

Set the palette, don't hardcode the colors. A categorical map with no baseColorScheme falls back to BuGn — a sequential green ramp (MapChartState.ts:53). Region colors are only looked up by name when the map is on OwidCategoricalMap, the scheme that carries colorMap: MapContinentColors. Setting it on the indicator means every chart built on it inherits the palette (the same inheritance that already carries customCategoryLabels). A chart's own patch still wins over the inherited value, so once the chart exists, confirm on staging that it isn't patched to something else (world-regions-according-to-pew is patched to continents — a chart palette on a map, which pulls the strong colors instead of the muted ones).

Ordering rule for sort: the map legend renders as a single row in sort order, so order the regions to read left-to-right across a world map. The house sweep is:

(North/Northern) America → Latin America / Caribbean → Africa (north to south within the slot) → Middle East / North Africa → Europe → CIS / Russia / Central Asia → South Asia → East and South-East Asia → Australia and New Zealand → Oceania

Drop what the provider doesn't have, keep the rest in this relative order. Three wrinkles worth knowing. Europe sits after Africa and the Middle East, because Europe and Africa share the same longitudes (Europe north, Africa south) so west-to-east alone can't separate them — the convention sweeps the southern band first. At sub-region granularity, a "Western Asia" that the provider models as part of Asia stays in the Asia block rather than moving up to the Middle East slot (compare un_m49_2 with ei). And which Africa regions land in the Africa slot depends on how the provider splits the continent: where Africa has its own sub-regions they run north to south inside the slot — Northern Africa then Sub-Saharan Africa (un_m49_2, ilo_2), or Northern Africa then the compass sub-regions (fao_2). Where instead North Africa is folded into a Middle East and North Africa bucket, that bucket is not part of the Africa slot at all — it takes the Middle East slot — so Sub-Saharan Africa is alone in the Africa slot and therefore comes first (wb, unsdg, pew, wid, fao_sdg).

The order is defined twice — keep the two equal. This sort drives the published map's legend; customRegionDisplayOrder[<provider>] in owid-grapher's RegionTooltipData.ts (Step 9) drives the legend of the mini-map in the region hover. When they diverge, the same provider lists its regions in two different orders on the same page. Treat customRegionDisplayOrder as the reference and copy it into sort verbatim; if you're adding a new provider, write the order once and paste it into both.

Reordering is only color-safe once the regions are pinned. For a region with a MapContinentColors entry, sort moves legend positions and nothing else — the color follows the name. For a region without one, OwidCategoricalMap falls back to handing out palette colors by position, so reordering silently recolors the map. Check every region of the tier against MapContinentColors before touching sort: if any are unpinned, pin them first (Step 9, with the design sign-off) and reorder in the same change, or leave the order alone and say why in a # NOTE: beside it. The two edits look independent and are not.

6c. Cross-tier back-fill — if Step 3 found a region shared across tiers, add the masked back-fill to grapher/regions/2023-01-01/regions.py after the inversion loop (see the existing process_un_definitions example for the shape).

6d. Build and verify:

.venv/bin/etlr data://grapher/regions/2023-01-01/regions

from owid.catalog import Dataset
tb = Dataset("data/grapher/regions/2023-01-01/regions").read("regions")
for col in ["<provider>_region", ...]:
    o = tb[col].metadata.origins[0]
    print(col, tb[col].notna().sum(), "countries,", tb[col].nunique(), "categories | attr:", o.attribution_short, "| date_published:", o.date_published)

Confirm: the new columns exist with titles, attribution carried, date_published is None, no "belongs to multiple regions" warning in the build log, and (multi-tier) each indicator covers the full partition.

---

Step 7 (optional) — Migrate an existing in-dataset region chart

Do this only if the provider's own dataset already has a region variable powering a region-definition map chart that should now read the shared indicator (an "indicator upgrade"). Skip otherwise.

Find the old variable and its chart on the staging DB for the current branch:

from etl.config import OWID_ENV
# variables of the provider dataset (note: query % LIKE with params=)
OWID_ENV.read_sql("""
  SELECT v.id, v.shortName FROM variables v JOIN datasets d ON v.datasetId=d.id
  WHERE d.shortName=%(s)s AND v.shortName='region'
""", params={"s": "<short_name>"})
# charts using it — slug lives on chart_configs, not charts
OWID_ENV.read_sql("""
  SELECT c.id, cc.slug, cc.config->>'$.title' AS title
  FROM charts c JOIN chart_dimensions cd ON cd.chartId=c.id
  JOIN chart_configs cc ON cc.id=c.configId
  WHERE cd.variableId=%(v)s
""", params={"v": OLD_VAR_ID})

Repoint the chart at the new {provider}_region variable:

from etl.config import OWID_ENV
from apps.chart_sync.admin_api import AdminAPI
api = AdminAPI(OWID_ENV)
cfg = api.get_chart_config(CHART_ID)
cfg["dimensions"] = [{"property": "y", "variableId": NEW_VAR_ID}]
cs = cfg.setdefault("map", {}).setdefault("colorScale", {})
# re-key the labels onto the NEW "(Provider)"-suffixed category values (the new
# variable's categories carry the suffix):
cs["customCategoryLabels"] = {f"{name} (<Provider>)": name for name in OLD_REGION_NAMES}
# put the map on the name-keyed palette and drop any hardcoded colors — the old
# chart's customCategoryColors would override MapContinentColors (Step 9):
cs["baseColorScheme"] = "OwidCategoricalMap"
cs.pop("customCategoryColors", None)
api.update_chart(CHART_ID, cfg)

Staging admin writes work behind Tailscale without ADMIN_API_KEY. The change surfaces in chart-diff for review before it reaches production. Verify by re-reading the config: dimensions point at the new variable, labels are re-keyed, customCategoryColors is gone, and the map renders in the muted map colors (chart-diff will show the color change — that's expected, and it's what Step 9 pins).

---

Step 8 — Commit and open a PR

make check
git add etl/steps/data/garden/regions/2023-01-01/regions.yml \
        etl/steps/data/grapher/regions/2023-01-01/regions.meta.override.yml \
        etl/steps/data/grapher/regions/2023-01-01/regions.py   # if back-fill added
git commit -m "📊🤖 Add <Provider> regions to regions dataset"

If not already on a feature branch, create one and a PR with etl pr "Add <Provider> regions" data, then push. In the PR body, open with the disclosure blockquote (> _Written by Claude <model name> — @<handle> at the wheel._, model name = the model actually generating the content) and keep any reviewer attribution out of committed code/YAML.

Heads-up: once this merges, the post-merge deploy is slow — editing the regions dataset invalidates much of the DAG, so it can take hours for the new regions to reach the production catalog. The owid-grapher follow-up (Step 9) can't start until they do, so don't expect to chain straight into it.

---

Step 9 — Register the provider in owid-grapher (separate repo + PR)

The grapher frontend keeps its own copy of the regions and a few hand-maintained registries. The provider must be added there too, or its (Provider) entities won't be grouped/labelled correctly in entity selectors, map tooltips, and admin presets. Reference: owid/owid-grapher#6465 (IEA).

Sequencing — and expect a long wait: the frontend's regions.data.ts is regenerated from the production catalog (https://catalog.ourworldindata.org/external/owid_grapher/latest/regions/regions.csv). So do Step 9 after the ETL PR (Step 8) is merged and the data://external/owid_grapher/latest/regions step has rebuilt on prod. That rebuild is slow — often hours, not minutes — because editing the regions dataset invalidates a huge swath of the DAG (almost everything that aggregates by region or merges population/regions depends on it), so the post-merge deploy has a lot to rebuild before the new regions reach the catalog. Don't run yarn runRegionsUpdater until the regions are actually live there, or it'll regenerate from stale data. Verify first:

curl -s "https://catalog.ourworldindata.org/external/owid_grapher/latest/regions/regions.csv?nocache" | grep -c "PROVIDER_"

A non-zero count means it's ready. If you're waiting, poll this every few minutes rather than running the updater blind.

Preview from your ETL branch's staging catalog while you wait. The updater reads ETL_REGIONS_URL if it's set (devTools/regionsUpdater/update.ts), and every ETL staging server publishes its own catalog on port 8881 with the same schema as prod. So you can regenerate regions.data.ts — and everything downstream of it, including the colors and the test page — before the ETL PR is anywhere near merged:

The host name is not the branch name — derive it, don't hand-write it. etl.config.get_container_name() replaces /, . and _ with -, strips a leading staging-site-, truncates what's left to 28 characters, then drops any trailing hyphen. Substituting the raw branch (or truncating it yourself) silently produces a host that is either unreachable or, worse, another branch's staging server. Get it from the etl repo:

.venv/bin/python -c "from etl.config import get_container_name; print(get_container_name('<etl-branch>'))"

Then, in owid-grapher, use that value (it already carries the staging-site- prefix):

ETL_REGIONS_URL="http://<container-name>.tail6e23.ts.net:8881/external/owid_grapher/latest/regions/regions.csv" \
  yarn runRegionsUpdater

Use this for previews and for the color review only — before merging the grapher PR, re-run yarn runRegionsUpdater with no env var so the committed file is prod-derived (see Renaming existing regions for why a stale/hand-edited regions.data.ts is a trap).