Skill: /audit-consumers

Discovers every github.com/atlanhq/ consumer of application_sdk, applies a user-defined check specification across each consumer, and produces a markdown report that accounts for every candidate repo — including those with no relevant usage — so the audit is demonstrably holistic.

Optionally raises ready-for-review migration PRs against each consumer with confirmed findings, with fixes applied (deterministically from a fix: block in the spec, or LLM-generated from the prose recommendation:).

Typical use cases:

• "Which apps still import deprecated symbol X that we are removing in v4.0?"
• "Who catches exceptions from BaseSQLClient.load() narrowly, by class or message string?"
• "Which consumers pin the SDK to a version before feature Y shipped?"
• "What is the blast radius if we change the signature of ObjectStore.upload_file()?"
• "Raise migration PRs against all consumers that will break when we ship PR #1234."

---

Invocation

/audit-consumers                                  # prompts for a spec inline
/audit-consumers --spec ./my-spec.md             # reads spec from file
/audit-consumers --report ./out.md               # custom report path
/audit-consumers --sdk-major 3                   # explicitly set the target major version
/audit-consumers --sdk-release v4.0.0            # target SDK release for PR bodies
/audit-consumers --raise-prs                     # run Phase E and open PRs after Phase C
/audit-consumers --raise-prs --dry-run           # Phase E through E6 only — no push, no PR

Argument parsing (left-to-right, first match wins):

• --spec <path> — read check specification from this file instead of prompting
• --report <path> — write the markdown report here; default: <repo-root>/<slug>-consumer-audit.md where <slug> is the spec title lowercased, spaces replaced with hyphens
• --sdk-major <N> — integer target major version to audit against; repos pinned to other majors are recorded as skipped-major-mismatch without file analysis (see Phase 0b)
• --sdk-release <ref> — free-form string naming the SDK release (version tag, PR URL, or commit SHA); used in the report and PR bodies; if absent, Phase 0 prompts for it
• --raise-prs — after Phase C, enter Phase E (generate and raise migration PRs)
• --dry-run — used with --raise-prs; runs through Phase E6 but does not push or create PRs

---

Phase 0 — Capture the check specification

If --spec <path> was provided: read the file. Validate that it contains at least one anchor pattern and at least one named check. If malformed, report the exact problem and stop.

Otherwise: ask the user to describe what they want to check. Convert the description into a structured spec by echoing it back in the format below for confirmation.

Gate: confirm the spec with the user before running any GitHub queries. This is the only interactive step before Phase C — once the spec is locked, Phases A–C run to completion without further prompts.

The check-spec format is defined in references/spec-format.md. The short form for Phase 0 confirmation:

# Check spec — <Short title (becomes the report filename slug)>

## Context
<Why this audit exists and what SDK change prompted it.>

## target_release
<Version tag, PR number, or "pre-release". Prompted here if --sdk-release was not passed.>

## Anchors
- pattern: `<regex>`

## Checks
### check: R1 — <Name>
- impact: definite-break | silent-break | review
- patterns:
  - `<regex>`
- recommendation: <one-line fix>
- fix:           # optional — see references/spec-format.md
  - pattern_index: 0
    replace_regex: `…`
  add_import: `from …`

Additional validation when --raise-prs is set: Every fix: block is validated before Phase A runs. See references/spec-format.md for the validation rules. Failures are reported with the exact check ID and field; stop before Phase A.

---

Phase 0b — Determine target major version

This phase runs once, before Phase A. Its output is a single integer target_major that gates the per-repo analysis in Phase B.

Resolution order (first hit wins)

1. CLI flag: if --sdk-major <N> was passed, use that integer directly.
2. Local SDK tree: read pyproject.toml in the current working directory (normally the application-sdk checkout) and extract [project].version. Take the first numeric component before the first .:

   import re, tomllib
   with open("pyproject.toml", "rb") as f:
       version = tomllib.load(f)["project"]["version"]
   target_major = int(re.match(r"(\d+)", version).group(1))
   

3. Fallback: if neither source is available, prompt the user once for the target major during Phase 0 spec confirmation: "Which major version of the SDK are you auditing against? (e.g. 3)"

Surface the resolved value in chat: Target major version: 3 (from pyproject.toml).

Pin-parsing rules

When processing each repo in Phase B, parse the discovered SDK pin string into a set of compatible major versions using the following rules:

Pin string	Compatible majors
3.8.0, ==3.0.0, 3.0.0a1	{3}
2.7.4, ==2.5.0	{2}
>=3.0.0,<4.0.0, >=3.5.0,<4	{3}
>=3.6.0,<5	{3, 4}
>=2.8.0 (no upper bound)	{2, 3, 4, …} — treat as current major and later; always includes target_major
git tag v3.7.0	{3}
git branch release/v2	{2}
git branch main	{target_major} — tip of main tracks current major
git branch (any other name, incl. master, develop)	unknown — skip analysis; see [tool.uv.sources] dict rules below
path editable ../application-sdk	{target_major} — local dev, assume current major
missing / unparseable (string pin)	unknown — fall through to analysis; status becomes analyzed-pin-unknown

When the pin comes from a [tool.uv.sources] TOML dict (see Phase B Step 1), apply these additional rules (evaluated before the string-form rules above):

[tool.uv.sources] entry	Compatible majors	Coverage display string
{git, tag = "vN.M.P"}	{N}	tag vN.M.P
{git, branch = "release/vN"}	{N}	branch release/vN
{git, branch = "main"}	{target}	branch main
{git, branch = <any other>} (incl. master, develop, feature branches)	None	branch <name>
{git, rev = "<hash>"}	None	rev <hash[:8]>
{path = "...", editable = true} or {path = "..."}	{target}	editable <path>
anything else	None	the raw repr

When compatible_majors() returns None for a dict-form pin, record status skipped-pin-source-unknown and stop — do not analyze the repo. This is distinct from string-form None (which falls through to analyzed-pin-unknown with conservative analysis) because a rev hash or non-standard branch makes it impossible to know which SDK code the consumer is actually running against.

Implementation pseudocode (inline, not stored as a separate file):

import re

def compatible_majors(pin, target: int):
    """Return (set_of_compatible_majors_or_None, display_str).

    `pin` is one of:
      • str  — a PEP 440 specifier, branch name, git tag string, or path string
      • dict — a parsed [tool.uv.sources] entry, e.g.
               {"git": "...", "branch": "release/v2"}
               {"git": "...", "rev":    "91e9c4a9..."}
               {"git": "...", "tag":    "v3.7.0"}
               {"path": "../application-sdk", "editable": True}

    Returns:
      (set_or_None, display_str)
      • set  — compatible major versions
      • None — major undetermined (behaviour depends on pin type: str → analyze
                conservatively; dict → skip; see Phase B Step 1 gate table)
    """
    # --- Dict form: [tool.uv.sources] entry ---
    if isinstance(pin, dict):
        tag    = pin.get("tag")
        branch = pin.get("branch")
        rev    = pin.get("rev")
        path   = pin.get("path")

        if tag:
            m = re.match(r"v(\d+)\.", str(tag))
            if m:
                n = int(m.group(1))
                return {n}, f"tag {tag}"
            return None, f"tag {tag}"

        if branch:
            if branch == "main":
                return {target}, f"branch {branch}"
            m = re.match(r"release/v(\d+)", branch)
            if m:
                n = int(m.group(1))
                return {n}, f"branch {branch}"
            return None, f"branch {branch}"

        if rev:
            short = str(rev)[:8]
            return None, f"rev {short}"

        if path:
            return {target}, f"editable {path}"

        return None, repr(pin)

    # --- String form: [project].dependencies / requirements.txt ---
    pin = str(pin).strip()
    display = pin

    if pin.startswith(("../", "./", "/")):
        return {target}, display
    if pin == "main":
        return {target}, display
    m = re.match(r"release/v(\d+)", pin)
    if m:
        return {int(m.group(1))}, display
    m = re.match(r"v(\d+)\.", pin)
    if m:
        return {int(m.group(1))}, display
    if pin.startswith("rev:"):
        return None, display
    first = pin.split(",")[0].strip()
    op_m = re.match(r"^([=!<>~^]+)", first)
    operator = op_m.group(1) if op_m else ""
    first_clean = re.sub(r"^[=!<>~^]+", "", first).strip()
    m = re.match(r"(\d+)", first_clean)
    if not m:
        return None, display
    lower = int(m.group(1))
    if operator == "==":
        return {lower}, display
    ub = re.search(r",\s*<\s*(\d+)", pin)
    if ub:
        upper = int(ub.group(1))
        if upper > lower:
            return set(range(lower, upper)), display
        else:
            return {lower}, display
    return set(range(lower, target + 1)), display

Gate decisions after calling compatible_majors(pin, target_major):

Result	Pin source	Action
Set contains target_major	either	Proceed to Step 2
Set is non-empty, excludes target_major	either	Status skipped-major-mismatch — stop
None	dict ([tool.uv.sources] rev or unknown branch)	Status skipped-pin-source-unknown — stop
None	str ([project].dependencies / requirements.txt)	Status analyzed-pin-unknown — proceed conservatively
API call failed	—	Status analyzed-error — stop

---

Phase A — Discover candidate repos

Run four discovery signals in parallel, union the results:

gh search code "from application_sdk"   --owner atlanhq --limit 100 \
  --json repository --jq '.[].repository.nameWithOwner' | sort -u >  /tmp/audit-repos.txt

gh search code "import application_sdk" --owner atlanhq --limit 100 \
  --json repository --jq '.[].repository.nameWithOwner' | sort -u >> /tmp/audit-repos.txt

gh search code "application-sdk" --filename pyproject.toml --owner atlanhq --limit 100 \
  --json repository --jq '.[].repository.nameWithOwner'             >> /tmp/audit-repos.txt

gh search code "application-sdk" --filename requirements.txt --owner atlanhq --limit 100 \
  --json repository --jq '.[].repository.nameWithOwner'             >> /tmp/audit-repos.txt

sort -u /tmp/audit-repos.txt -o /tmp/audit-repos.txt

For each anchor pattern in the spec, also run:

gh search code "<anchor>" --owner atlanhq --limit 100 \
  --json repository --jq '.[].repository.nameWithOwner' >> /tmp/audit-repos.txt
sort -u /tmp/audit-repos.txt -o /tmp/audit-repos.txt

This catches repos whose code directly uses the anchored symbol even if their dependency declarations are non-standard.

Connector-map cross-check

Read the curated list of all connector/app repos:

gh api repos/atlanhq/integration-studio-aisdlc/contents/context/connector-map.md \
  --jq '.content' | base64 -d > /tmp/connector-map.md

Extract every atlanhq/<repo-name> that appears in that file and diff against /tmp/audit-repos.txt. For each repo in the connector map that is not in the candidate list, add it and log a warning:

⚠ Gap: atlanhq/<repo> appears in connector-map.md but was not found by gh search code.
  Added to candidate list. Cause: repo may not have searchable code indexed, or the SDK
  dependency uses a non-standard declaration format.

Record every repo in the final candidate list verbatim. The Phase C coverage table must account for every entry, even ones that turn out to have no SDK usage at all.

---

Phase B — Per-repo audit

For each candidate repo in /tmp/audit-repos.txt:

Step 1 — Read SDK pin and apply major-version gate

Fetch pyproject.toml first; fall back to requirements.txt if absent. Parse in two stages using the Python helper that is already running in this phase:

import base64, re, subprocess

# --- Fetch pyproject.toml ---
raw = subprocess.run(
    ["gh", "api", f"repos/atlanhq/{repo}/contents/pyproject.toml",
     "--jq", ".content"],
    capture_output=True, text=True,
)
text = base64.b64decode(raw.stdout.strip()).decode() if raw.returncode == 0 else ""

# --- Stage 1: [tool.uv.sources] (authoritative when present) ---
uv_source = None
src_block = re.search(
    r"\[tool\.uv\.sources\](.+?)(?:\n\[|\Z)", text, flags=re.DOTALL,
)
if src_block:
    for line in src_block.group(1).splitlines():
        line = line.split("#", 1)[0].strip()
        if not line:
            continue
        m = re.match(
            r'(?:atlan[-_]application[-_]sdk|application[-_]sdk)\s*=\s*\{(.+)\}',
            line, flags=re.IGNORECASE,
        )
        if m:
            uv_source = {}
            for kv in re.finditer(
                r'(\w+)\s*=\s*(?:"([^"]*)"|(true|false))', m.group(1)
            ):
                uv_source[kv.group(1)] = (
                    kv.group(2) if kv.group(2) is not None
                    else (kv.group(3) == "true")
                )
            break

# --- Stage 2: [project].dependencies or requirements.txt (fallback) ---
project_pin = None
if uv_source is None and text:
    m = re.search(
        r'["\']atlan[-_]application[-_]sdk[^"\']*["\']', text, re.IGNORECASE
    )
    if m:
        ver = re.search(r'((?:[><=!~^,\s\d\.\*]+)+)', m.group().split("[")[0].split('"')[-1])
        project_pin = ver.group(1).strip() if ver else None

if uv_source is None and project_pin is None and not text:
    req = subprocess.run(
        ["gh", "api", f"repos/atlanhq/{repo}/contents/requirements.txt",
         "--jq", ".content"],
        capture_output=True, text=True,
    )
    if req.returncode == 0:
        req_text = base64.b64decode(req.stdout.strip()).decode()
        m = re.search(r'atlan[-_]application[-_]sdk([^\n]*)', req_text, re.IGNORECASE)
        project_pin = m.group(1).strip() if m else None

pin = uv_source if uv_source is not None else project_pin
majors, pin_display = compatible_majors(pin, target_major)

Key rules for the two-stage parse:

• [tool.uv.sources] overrides [project].dependencies — when an entry for the SDK is found in the sources table, use it exclusively.
• Match the SDK package under either atlan-application-sdk or application-sdk (case-insensitive, hyphen or underscore).
• Commented-out lines are stripped before matching — only the active, uncommented source is used.
• pin_display is what goes in the Coverage table's SDK pin column.

Apply the gate decisions from Phase 0b after calling compatible_majors.

Step 2 — Anchor check (cheap, via GitHub API)

gh search code "<anchor>" --repo atlanhq/<repo> --json path,textMatches --limit 20

Run this for each anchor pattern in the spec. If all return empty → status analyzed-no-usage. If gh api fails → status analyzed-error. Never silently drop.

Step 3 — Fetch anchored files

For each unique file path surfaced by the anchor search:

gh api repos/atlanhq/<repo>/contents/<path> --jq '.content' | base64 -d > /tmp/audit-file.py

Clone fallback: if more than 5 files need individual fetching, OR if the spec requires cross-file analysis (e.g., subclass tracing), do a shallow clone instead:

git clone --depth 1 git@github.com:atlanhq/<repo>.git /tmp/sql-audit/<repo>

Step 4 — Run checks

For each check in the spec, against each fetched file:

rg -nP '<pattern>' /tmp/audit-file.py

Record every hit as an extended record:

(file_path, line_number, matched_line, check_id,
 context_before,      # 3 source lines preceding the match
 context_after,       # 3 source lines following the match
 enclosing_symbol,    # nearest def/class header above the match, if any
 try_body_excerpt)    # for `except …` patterns: lines between `try:` and the matched
                      # `except` (best-effort); empty string otherwise

For hits where require_anchor: false, run the patterns against all Python files in the repo.

Step 4b — Triage each finding

For each captured hit, assign a triage field and a one-line triage_reason:

Value	Meaning
confirmed	Real exposure to the SDK change being audited.
false_positive	Pattern matched, but code is not exposed to the SDK change.
needs_review	Insufficient signal — flag for human review.

Default heuristics (conservative — when in doubt, return needs_review)

For except <SpecificException> patterns (definite-break checks):

• false_positive if try_body_excerpt contains none of the spec's anchor patterns AND contains at least one of: urllib, urlparse, os.path, int(, float(, json.loads, CredentialRef, re.compile, datetime., uuid., Decimal(, pathlib., yaml.safe_load, Enum(.
• confirmed if try_body_excerpt contains an anchor pattern or a method call on a variable typed as the SDK class.
• needs_review otherwise.

For literal message-string patterns (silent-break checks):

• false_positive if the match is inside a """...""" docstring, a # comment, or a regex literal whose surrounding code does not invoke str(e), e.args[0], .message, re.search, re.match, or string in against an exception.
• confirmed otherwise.

For broad-catch patterns (review checks):

• false_positive if try_body_excerpt contains only the consumer's own raise <X>(...).
• confirmed if try_body_excerpt calls an SDK anchor method.
• needs_review otherwise.

Spec-supplied triage: overrides are evaluated before these defaults. See references/spec-format.md.

Surfacing the triage outcome

• Per-repo findings (Phase C) MUST list every hit, annotated inline: [confirmed], [false_positive — <reason>], or [needs_review — <reason>].
• false_positive hits do NOT count toward the action summary metrics — but are listed so reviewers can challenge a triage call.

Step 5 — Classify status

Status	Condition
analyzed-no-usage	major-match; no anchor match anywhere in the repo
analyzed-no-findings	major-match; anchor present, zero check hits — OR all hits false_positive
analyzed-has-findings	major-match; at least one confirmed or needs_review hit after triage
skipped-major-mismatch	pin parsed; compatible majors do not include target_major
skipped-pin-source-unknown	[tool.uv.sources] pins to rev hash or non-release/vN/non-main branch
analyzed-pin-unknown	string-form pin missing or unparseable; analyzed conservatively
analyzed-error	GitHub API call failed (log the error)

---

Phase C — Synthesise and write the report

Write the report to the path specified by --report (or the default slug-based path).

Required report sections

# <Spec title> — Consumer Audit

**Audit date:** <ISO date>
**Target SDK release:** <spec.target_release>
**Audited by:** automated gh search + per-file content inspection

---

## Executive summary

| Metric | Count |
|---|---|
| Candidate repos discovered | N |
| Skipped (major mismatch — not on v`<target_major>`) | N |
| Skipped (uv source pin — major undetermined) | N |
| Pin unknown (analyzed conservatively) | N |
| Repos with anchor-matched files | N |
| Repos needing action (after FP exclusion) | N |
| False-positive hits filtered out | N across M repos |
| <check_id> confirmed hits (<impact>) | N across M repos |
| … | … |

<2–3 sentence narrative.>

---

## Spec used

<Verbatim copy of the check spec from Phase 0, so the report is self-contained.>

---

## Coverage — every repo we looked at

| Repo | SDK pin | Compatible majors | Status |
|---|---|---|---|
| atlanhq/<repo-a> | `>=3.0.0,<4.0.0`  | {3} | analyzed-has-findings |
| atlanhq/<repo-b> | `2.7.4`           | {2} | skipped-major-mismatch |
| atlanhq/<repo-c> | `branch release/v2` | {2} | skipped-major-mismatch |
| atlanhq/<repo-d> | `rev 91e9c4a9`    | ?   | skipped-pin-source-unknown |
| atlanhq/<repo-e> | `tag v3.7.0`      | {3} | analyzed-no-usage |
| atlanhq/<repo-f> | (none)            | ?   | analyzed-pin-unknown |
| atlanhq/<repo-g> | —                 | —   | analyzed-error: 403 from gh api |

---

## Action summary — repos that need a migration PR

| Repo | SDK pin | Definite breaks | Silent breaks | Review items | Needs review | Fix sites | Priority | Recommended action |
|---|---|---|---|---|---|---|---|---|
| atlanhq/<repo-a> | `>=3.0.0,<4.0.0` | 3 | 1 | 0 | 0 | `clients/foo.py:42`, … | P0 | Catch `AppError` instead of `ClientError`; … |

---

## Per-repo findings (only repos with hits)

### atlanhq/<repo-name>

- **SDK pin:** `<version>`
- **Anchor files:**
  - `<path/to/file.py>` (line N: `<matched line>`)

#### <check_id> — <check name> (impact: <impact>)

- `<file>:<line>` — `<matched line>` [confirmed]
  - Recommendation: <from spec>
- `<file>:<line>` — `<matched line>` [false_positive — try body calls only urllib.parse.urlparse]
- `<file>:<line>` — `<matched line>` [needs_review — try body context ambiguous]

---

## Repos added from connector-map cross-check

<List any repos added from the connector-map but absent from gh search results.
Omit this section if there were no gaps.>

Post-Phase-C interactive prompt (when --raise-prs was not set)

After writing the report and printing the inline chat summary, if ALL of the following hold:

• --raise-prs was not set at invocation
• --dry-run was not set
• At least one repo has status analyzed-has-findings with at least one confirmed hit

Then prompt:

"Audit found repos with confirmed findings. Raise migration PRs against these consumers now? (yes/no)"

A yes answer enters Phase E exactly as if --raise-prs had been set at invocation. A no answer continues to Phase D (or exits).

Inline summary in chat

After writing the report, always print to chat:

1. The executive summary table.
2. The Action summary table (only confirmed/needs-review repos).
3. The top 3–5 highest-impact confirmed hits (definite-break first, then silent-break, then review), with file:line and recommendation inline.
4. The report file path.

---

Phase E — Generate and raise migration PRs

Runs after Phase C when --raise-prs is set or the user accepts the post-Phase-C prompt. Only operates on repos with status analyzed-has-findings and at least one confirmed hit. needs_review findings become PR-body TODOs; false_positive findings are skipped. analyzed-pin-unknown, skipped-major-mismatch, and skipped-pin-source-unknown repos are excluded entirely.