Authoring CI workflows

Before you propose a change to CI, check things already tried for the idea. It records what was measured, and why some good-sounding changes were reverted or rejected.

Conventions for .github/workflows/** and .github/actions/**. The linters own the mechanical rules (below); this skill is the judgment calls they can't enforce.

Before you write

• Copy from a canonical file rather than from memory. ci-paths-filter.yml is the smallest complete example (triggers, concurrency, timeout, app token, Depot runner); ci-backend.yml is the reference for the heavy patterns (bounded-depth checkout, per-SHA concurrency, draft/ready, sharding).
• Related skills — reach for these instead of duplicating them here:
  • /gating-production-deploys — any job that pushes a prod image or dispatches a Charts deploy.
  • /managing-github-actions-secrets — creating the GitHub App / secret a workflow reads.
  • /depot-github-runners — Depot runner labels and sizing.
  • /debugging-ci-failures — CI is red and you need to know why.

What the linters already enforce

Run bin/hogli lint:workflows and actionlint before pushing — they gate CI, and they (not this list) are the source of truth for what's enforced. Today that's: timeout-minutes on every job, the canonical PR concurrency block, a repo-wide budget for unscoped PR event dispatches, dorny/paths-filter negation safety, justification for full-depth checkouts, cache-write gating, semgrep service coverage, MCP path-filter coverage of the trees the MCP build compiles, required-check gate hygiene, secrets a reusable workflow reads being declared and passed by its callers, and generic GHA correctness (bad secrets.* / needs: refs, deprecated ::set-output, unknown runner labels). Third-party action digests are bumped by Renovate.

The dispatch budget (500 runs / 10s / repo)

GitHub caps workflow-run dispatch at 500 runs per 10s per repo; overflow fails as startup_failure and takes unrelated runs in the same window down with it (a stack restack pushing many branches is the usual trigger). Minimize runs dispatched, not just work done — draft status doesn't help, runs dispatch before skip logic applies.

• A reusable-workflow call counts as one run. Small always-fire PR workflows should be jobs under a single workflow_call parent, not their own dispatches (see pr-updated.yml / pr-opened.yml folded behind their parent — fold pr housekeeping into one dispatch). Event-type scoping moves to job-level if: guards:

  jobs:
    turbo:
      if: contains(fromJSON('["opened", "synchronize", "reopened"]'), github.event.action)
      uses: ./.github/workflows/ci-turbo.yml
  

• Prefer a trigger-level paths: filter over dispatch-then-skip: a run that only starts to no-op still spends a dispatch (gate container workflows on trigger paths).

  on:
    pull_request:
      paths:
        - '.github/workflows/ci-x.yml'
        - 'path/to/product/**'
    workflow_dispatch:
  

• Judgment call — trigger paths: vs a runtime dorny/paths-filter job. Use trigger paths: for a workflow that is skippable as a whole. Never put a trigger paths: on a workflow whose check is required by branch protection: a required check that doesn't dispatch on a PR leaves it stuck "waiting for status" and unmergeable. Keep those firing on every PR and gate internally with a changes job (also the right call when several jobs branch on different path sets). Heavy matrices (ci-backend, ci-nodejs) do exactly this — deliberate.

• Delete dead dispatchers outright. A disabled-but-still-triggered workflow keeps dispatching no-op runs against the cap — remove the trigger, don't just disable it.

• A filter has to cover what the job actually compiles, not just where the code lives. A build that reaches out of its own tree — through a tsconfig paths alias, a workspace dependency, a generated file — breaks when a filter lists only the service directory. The PR touching the imported tree skips the job, the merge-queue run skips it too, and the break lands on master, where it then fails every later PR whose diff does match. Cover the enclosing directory, not the imported file: the module drags its own relative imports along. When the imported tree is busy, give it a second filter output and gate only the jobs it can actually break, rather than widening the one that also fires an expensive suite. WF009 derives the MCP filters' required trees from the MCP sources and fails when one is uncovered; a build with the same shape wants the same treatment.

Concurrency

Every PR-triggered workflow gets the canonical block:

concurrency:
  group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

• Cancel superseded PR runs; never cancel across master pushes. WF002 rejects a bare cancel-in-progress: true on any push-triggered workflow. Where latest-wins is genuinely right (a cache warmer), say so with # hogli-lint: allow-master-cancel -- <reason>.

• Use github.ref as the fallback, never github.run_id — run_id is unique per run, so it silently gives every push its own group and dedup is lost.

• Publish-on-push workflows must not let two master pushes race :latest / a deploy dispatch. Key the push arm per-SHA (see ci-backend.yml):

  group: ${{ github.workflow }}-${{ github.event_name == 'push' && github.sha || github.head_ref || github.ref }}
  

Required-check gates

The "gate" is the collate job that emits the required status check by reading needs.*.result. By convention its display name ends in Pass (Django Tests Pass, Visual regression tests pass), but WF007 also finds gates structurally when a step reads needs.<dep>.result, because the convention is not universally followed. A job that inspects results without gating anything opts out with # hogli-lint: not-a-required-gate — <reason> above the job key. Gates and the workers they inspect share the same condition:

Job	Condition	Why
Gate	if: ${{ !cancelled() }}	It emits an explicit verdict on every completed run, and a superseded run records cancelled, not failure.
Workers	if: !cancelled()	So a superseded run actually stops instead of holding the concurrency slot.

The gate condition must contain !cancelled(), with optional ${{ }} wrapping. always() is rejected: it is identical to !cancelled() on any run that is not cancelled, but on a superseded run it runs the gate after the cancel and reports failure, which inflates every CI failure-rate metric with runs a developer merely pushed over.

Extra predicates may only be OR-ed on, never AND-ed. This is a correctness rule, not a style one. A conjunction gives the gate a second way to be false, and a job skipped by its own condition records skipped, which branch protection reads as a pass. Both conclusions occur in the same cancelled run: in run 33496887370 Calculate running time recorded cancelled while Backend coverage report recorded skipped, because an AND-ed predicate of its own was already false. A disjunction cannot be false while !cancelled() is true, so cancellation stays the gate's one false predicate and the conclusion stays cancelled.

Cancellation still fails closed. A gate on !cancelled() that never starts records conclusion cancelled, never skipped. Measured on a superseded run (evidence): the gate recorded cancelled with zero steps, while the always() control ran after the cancel and recorded failure. GitHub's status checks reference lists success/neutral/skipped as passing and never places cancelled among them, and a commit whose only checks are cancelled rolls up to FAILURE (evidence). That is inference rather than a documented guarantee, which is the reason for the next rule.

A workflow that cancels its own run must OR that signal onto its gate. ci-backend cancels itself when repo checks or OpenAPI types fail deterministically, to stop paying for runners on a failure a retry cannot fix. Under a bare !cancelled() those real failures would report cancelled too, which both hides them from the failure-rate metric and rests merge safety on the inference above. OR-ing the deterministic-failure output back on keeps the honest verdict, because the disjunct is true, so the gate dispatches despite the cancel:

if: >
  !cancelled()
  || needs.repo-checks.outputs.deterministic_failure == 'true'
  || needs.check-openapi-types.outputs.deterministic_failure == 'true'

Measured on a self-cancelled run (evidence): the bare !cancelled() gate recorded cancelled, the OR-ed gate ran and recorded failure. Only superseded runs then report cancelled, and every real failure keeps a failure conclusion.

A failure-rate metric keyed on a gate job must exclude cancelled. Only success and a decisive failure are a verdict, so a denominator that counts cancelled measures push behavior, not test health. Find those rows through the run's conclusion, not the gate job's. The gate job's own conclusion changed on 2026-09-04: a superseded gate recorded failure before that date and records cancelled after it. A metric that drops the superseded rows from the numerator and the denominator stays comparable across that date. One that filters on the gate job's conclusion alone does not. The run's conclusion lives in the warehouse table github_workflow_runs. The posthog-ci-running-time event cannot supply it: the action fills that event's conclusion property from the job named in its status-job input, and every caller passes a gate job name. It writes the same value to the workflow_run group, so both of those fields carry a gate conclusion under a run-shaped name. A metric keyed on jobs joins github_workflow_jobs to that table on run_id, then scopes the job side to a single run_attempt. The runs snapshot keeps one row per run id, at its newest attempt, so an unscoped read stamps that conclusion onto every earlier attempt's gate and counts the gate once per attempt. Do not enforce that scope by joining run_attempt equality: it blanks or drops every earlier attempt, which is the population that actually ran after a partial re-run (products/engineering_analytics/backend/logic/views/job_costs.py records that decision). Reuse the canonical predicates instead of writing a new denominator: CONCLUSIVE_RUN_CONDITION in products/engineering_analytics/backend/logic/queries/_workflow_filters.py, and computeHealthSummary in products/engineering_analytics/frontend/lib/runHealth.ts. That run-level key identifies superseded runs only where the workflow never cancels its own run. Where it does (the rule above), a deterministic failure records run conclusion cancelled too, so the canonical predicates drop that honest failure together with the superseded rows. Measured on Backend CI run 34204389260: the run recorded cancelled while the Django Tests Pass gate recorded failure. Keep those rows in the numerator and the denominator, and find them through the cancel jobs: each one dispatches only on its deterministic-failure signal, so a success from any of them marks that population on both sides of 2026-09-04.

Four rules for the gate body:

1. Allowlist every dependency, never denylist. Assert success/skipped and fail everything else. A dependency tested only against == 'failure' lets cancelled through, and one bad dependency is enough — a gate that clears four correctly and one with a bare failure test is still wrong. The trap is the changes detector: clearing it with == 'failure' and then reading needs.changes.outputs.* reports green on cancellation, because those outputs are empty and the gate takes its "nothing to test" exit.
2. needs every job that produces coverage, and every upstream that can skip one. If a job's failure would only cascade into a downstream job being skipped, the gate reads that as a pass and you get a green check with zero tests run. The usual shape is a changes detector one step above the suite: it fails, the suite skips, and the gate reports success having run nothing (measured on ci-nodejs). Name every job whose failure would skip one you do test, not only the direct ones. Stop there. A test selector whose failure leaves the suite running in full is not gate-critical, and demanding it turns a recovered run into a red required check.
3. Legitimate skips must still pass. A frontend-only PR skips backend jobs by design.
4. Every dependency's result must reach a fail-closed allowlist guard. One inline if per dependency is the clearest form, but a shared shell helper or an env: block is equally fine: WF007 traces each result through assignments, ${!var} indirection, and helper argument positions within that step. The guard must compare with !=, join multiple allowed values with &&, and unconditionally exit 1 when entered. Comparisons in another step, comments, logs, or branches that do not exit nonzero prove nothing and are rejected. A result whose guard WF007 cannot follow is reported rather than assumed safe, so an unusual routing may need the checks moved inline.

WF007 enforces 1, 2, 4, and the !cancelled() condition, and it takes the dependency list from needs: as well as the step body, so a job you wired into needs: and then forgot to test is reported rather than silently trusted. For rule 2 it walks the needs: graph above each dependency and reports any job the gate does not test, which is the half a linter can see. It follows an edge only when the upstream's failure would actually skip the job below it. A job whose if calls no status function is skipped by any failed upstream, and so is one held behind success() or cancelled(), since neither is true in that state. A job that reaches always(), !cancelled() or failure() keeps running, and is skipped only where its own condition compares against the failed job and goes false: an output reads back empty, while result reads back failure, so a recovery path testing result == 'failure' still runs. The half it cannot see is a coverage job with no needs: edge into the gate at all: "reporting job" and "coverage job" look identical from outside the graph, so that one is on you and the reviewer.

What GitHub does with each conclusion

Rule 3 works because a skipped check run satisfies a required context, in the Trunk queue as well as on GitHub. Build Docker image rides on that: it concludes skipped on most PRs and they merge anyway.

Three cases behave in ways the name does not suggest:

• A required context that no check run reports stays pending and blocks. A trigger-level paths: filter that silences the whole workflow produces exactly this.
• A job-level continue-on-error: true posts a failure check run even though dependents read success and the run goes green. Use step-level continue-on-error plus an explicit verdict step instead.
• A matrix that expands to zero cells fails its dependents on GitHub Actions and posts no check run at all. Guard any fromJSON matrix with an if: that skips the job when the list is empty.

Required contexts are pinned to one app: every entry in this repo's master ruleset carries integration_id: 15368, the github-actions app, so a check run from any other app never satisfies one however exactly the name matches. /depot-ci has the measurements, the rulesets query, and how Depot CI differs.

Checkout / clone — sparse first, then shallow

This repo is 45k tracked files and 4.6 GiB of packed objects, so what you materialize costs more than how much history you fetch. Measured checkout-step durations, from the GitHub API on real runs:

Pattern	depot-ubuntu-24.04	GitHub-hosted ubuntu
sparse-checkout of a few paths, cone mode off	0–7s	0–7s
plain checkout (depth 1)	11–13s	22–44s
fetch-depth: 1000 + filter: blob:none	53–59s	—

• Biggest lever: check out only the paths the job reads. Sparse-checkout is not just for single files — a job that runs a local composite action, reads a JSON config, or lints one directory should name those paths and nothing else.

  - uses: actions/checkout@<sha> # v6
    with:
      sparse-checkout: |
        .github/actions/paths-filter
        .github/clickhouse-versions.json
      sparse-checkout-cone-mode: false
  

• Always set sparse-checkout-cone-mode: false. Cone mode additionally materializes every file in the repo root — here 70 files and 21.5 MB, .test_durations alone 18.5 MB — which is most of what you were trying to avoid. Cone mode also only takes whole directories, so it drags in all of bin/ when you wanted one script.

• filter: blob:none is counterproductive if the job then materializes the tree. It removes blobs from the fetch, but git checkout immediately lazy-fetches every blob in HEAD in a second round trip, which is slower than having fetched them in the pack. That lazy fetch also intermittently fails its per-blob credential lookup with could not read Username for github.com (#59779, blocked a merge until retried). Pair blob:none with sparse-checkout so the lazy fetch is a handful of blobs, or drop it and take the plain depth-1 checkout.

• Default: plain actions/checkout (depth 1). Add nothing.

• Diffing against the PR base: you need real history, so bound the depth, filter blobs, and sparse-checkout the files the job reads:

  - uses: actions/checkout@<sha> # v6
    with:
      fetch-depth: 1000
      filter: blob:none
      sparse-checkout: .github/actions/paths-filter
      sparse-checkout-cone-mode: false
  - name: Fetch PR base for affected diff
    if: github.event_name == 'pull_request'
    env:
      BASE_REF: ${{ github.event.pull_request.base.ref }}
    run: git fetch --no-tags --depth=1000 --filter=blob:none origin "$BASE_REF:refs/remotes/origin/$BASE_REF"
  

  A sparse working tree does not affect git merge-base, git diff <a>...<b>, git log --name-status, git ls-tree, git ls-files, or git show <rev>:<path> — those read the object database or the index. Only commands that compare against the worktree (git diff HEAD, git status) see the skip-worktree entries. One caveat when blob:none is also set: git show <rev>:<path> still needs that blob, and a sparse checkout never downloaded it, so the read becomes a lazy fetch that can fail. Name any file a step reads that way in the sparse set — ci-dagster.yml does this for docker-compose.base.yml, whose contents feed a cache key.

• changes / paths-filter gating jobs: on pull_request the vendored .github/actions/paths-filter diffs via the GitHub API and never touches the tree. The only reason to check out is that a local action must exist on disk, so sparse-checkout .github/actions/paths-filter plus any file the job's own steps read. Never pass base: HEAD to paths-filter from a sparse job — that routes it to git diff HEAD, which a sparse worktree makes return nothing, so every downstream job silently skips green.

• Foot-gun: git fetch --deepen=N with no refspec falls back to the wildcard refs/heads/* and pulls every branch. Always pass an explicit, --no-tags, --filter=blob:none refspec scoped to the base ref. (Bumping actions/checkout's own fetch-depth is safe — it uses a scoped refs/pull/N/merge refspec.)

• The linter rejects fetch-depth: 0 unless you add filter: blob:none, use sparse-checkout, or justify it with # hogli-lint: allow-full-depth-checkout -- <reason>. Genuinely full-history jobs: repo mirroring (foss-sync.yml), tag/submodule version math (release-cli.yml, desktop-tag.yml). Most base-diff jobs should use bounded 1000 + blob:none plus a sparse set.

Pinning and tool versions

• Pin every third-party action to a full 40-char commit SHA with a # vX.Y.Z comment. A moved tag can ship malicious code; pinning is also reproducible and skips a per-run GitHub-API version lookup. The only sanctioned exception is a debug-only action. In-repo composites use a local path with no ref (uses: ./.github/actions/pnpm-install).
• Node version comes from .nvmrc — node-version-file: .nvmrc, never a hardcoded node-version:. Sparse-checkout .nvmrc if the job has no checkout.
• Pin setup-uv's version: — an unpinned setup-uv calls the GitHub API on every job and burns the rate limit.

Network fetches

Downloads from outside the runner need retries, or a transient reset becomes a red check with no findings (actionlint died on curl: (35)).

curl -fsSL --retry 5 --retry-all-errors --retry-max-time 60 --connect-timeout 10 -o "$out" "$url"

• --retry-all-errors is the part that catches a reset; plain --retry covers only timeouts and 408/429/5xx, and --retry-connrefused adds ECONNREFUSED, not ECONNRESET.
• Drop it on GitHub API calls: with -f it also retries 403 and 404, spending five more requests on an already-empty token bucket.
• No --retry-delay (it replaces exponential backoff with a fixed wait). Keep -f, or an error page lands in your output file at exit 0.
• Don't retry anything non-idempotent (webhook posts, telemetry), or where a shell loop or readiness wait already retries.

Tokens — dedicated App tokens for high-volume calls

GITHUB_TOKEN shares one ~15k req/hr bucket across every job of every run in the repo; it goes hot at merge peaks and change-detection jobs fail before real work starts. A dedicated GitHub App installation is its own bucket — rate-limit headroom plus blast-radius isolation.

- uses: actions/create-github-app-token@<sha> # v3.1.1
  id: app-token
  # forks can't read org secrets — fall back to github.token
  if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
  with:
    client-id: ${{ vars.GH_APP_POSTHOG_PATHS_FILTER_APP_ID }}
    private-key: ${{ secrets.GH_APP_POSTHOG_PATHS_FILTER_PRIVATE_KEY }}

# a later step consumes the token (falling back to github.token on forks):
- uses: some-action@<sha>
  with:
    token: ${{ steps.app-token.outputs.token || github.token }}