Verify a Candidate PR Actually Fixes the Issue

Goal: take a PR number, check it out cleanly, re-run the same tool call that failed on main, compare the outputs, and produce a verdict — "fixes", "fixes but with concerns", "does not fix", or "cannot run". Pair with /2-repro-issue (which produces the baseline). No edits, no merges, no pushes.

Phase 1 — Resolve the PR and its linked issue

# Accept "498", "#498", or "https://github.com/.../pull/498" — extract the digits only
PR=$(echo "$ARGUMENTS" | sed -E 's|.*/||; s|#||g' | grep -oE '^[0-9]+' | head -1)
[ -z "$PR" ] && { echo "Invalid input: '$ARGUMENTS'. Pass a PR number or URL." >&2; exit 1; }
REPO=stickerdaniel/linkedin-mcp-server

gh pr view $PR --repo $REPO --json title,body,baseRefName,headRefName,headRepositoryOwner,mergeable,mergeStateStatus,additions,deletions,changedFiles,maintainerCanModify,statusCheckRollup

Extract the linked issue number from the PR body (Closes #, Fixes #, Resolves #, or plain #N). Call it ISSUE. If multiple, ask the user which one is the verification target.

Phase 2 — Pick up the baseline + request metadata

/2-repro-issue writes two files for each issue:

• /tmp/repro-issue-<ISSUE>-main.json — the on-main response baseline
• /tmp/repro-issue-<ISSUE>-meta.json — the {tool, arguments} used to call it

Both must exist. The meta file is how this skill replays the exact same call instead of guessing tool and args from the response body.

ls -la /tmp/repro-issue-$ISSUE-main.json /tmp/repro-issue-$ISSUE-meta.json 2>&1
# Both files must exist AND be non-empty. ls alone never fails the script, so
# guard each path before we touch the PR branch or LinkedIn.
[ -s /tmp/repro-issue-$ISSUE-main.json ] || { echo "Missing or empty response baseline. Re-run /2-repro-issue $ISSUE." >&2; exit 1; }
[ -s /tmp/repro-issue-$ISSUE-meta.json ] || { echo "Missing or empty meta file. Re-run /2-repro-issue $ISSUE." >&2; exit 1; }
TOOL=$(jq -r .tool /tmp/repro-issue-$ISSUE-meta.json)
ARGS_JSON=$(jq -c .arguments /tmp/repro-issue-$ISSUE-meta.json)
echo "Replaying: $TOOL($ARGS_JSON)"

If either file is missing, stop and tell the user: "No baseline for #$ISSUE. Run /2-repro-issue $ISSUE first so we have an on-main reference to diff against." Do not silently re-run the reproduction, /2-repro-issue is the canonical source of "what's the call, what's the failure mode".

If the baseline is older than 24 h, warn, LinkedIn DOM/data may have shifted. Offer to re-run /2-repro-issue first.

Phase 3 — Check out the PR

git status --porcelain | head -5     # must be clean
git stash list | head -3             # warn if there are stashes the user may forget

# Remember where to return. `--abbrev-ref HEAD` returns the literal string "HEAD"
# in detached state (CI worktrees, prior PR checkouts), so fall back to the SHA.
CURRENT_REF=$(git symbolic-ref -q --short HEAD || git rev-parse HEAD)

# Fetch the PR head into FETCH_HEAD only and check it out as a detached HEAD.
# Skipping a named local branch avoids clobbering a maintainer's existing
# `pr-$PR` work and removes the cleanup-time `git branch -D` foot-gun.
# Guard the fetch: without it a network/auth failure or deleted pull ref would
# leave a stale FETCH_HEAD and the verdict would run against the wrong commit.
git fetch origin "pull/$PR/head" || { echo "git fetch for PR #$PR failed, aborting before checkout." >&2; exit 1; }
PR_SHA=$(git rev-parse FETCH_HEAD)
git checkout --detach "$PR_SHA"

# Any early exit after this point must still return to $CURRENT_REF + clear
# /tmp scratch files. Install a single trap so the fail-fast guards in Phase 4
# do not strand the user on the PR commit.
cleanup_verify() {
  rc=$?
  trap - EXIT INT TERM
  kill $SERVER_PID 2>/dev/null
  wait $SERVER_PID 2>/dev/null
  git checkout "$CURRENT_REF" 2>/dev/null
  rm -f /tmp/verify-pr-$PR.json /tmp/verify-pr-$PR-headers /tmp/verify-pr-$PR.log /tmp/pr-$PR-meta.json
  exit $rc
}
trap cleanup_verify EXIT INT TERM

# Scope sanity check
gh pr view $PR --repo $REPO --json files --jq '.files[].path' | head -20
git diff --stat $(git merge-base "$PR_SHA" origin/main)..$PR_SHA | tail -10

If the PR has merge conflicts with main (mergeStateStatus: DIRTY), continue anyway — local checkout still works — but note it in the verdict.

If maintainerCanModify: true, mention that to the user; it unlocks the "take over and push a small follow-up commit" path later.

Phase 4 — Re-run the same MCP call on the PR branch

$TOOL and $ARGS_JSON came from the meta file in Phase 2. Restart the server because old workers hold the old code. Probe a free port (default 8000 is commonly taken):

PORT=8765
while lsof -nP -iTCP:$PORT -sTCP:LISTEN >/dev/null 2>&1; do PORT=$((PORT+1)); done
uv run -m linkedin_mcp_server --transport streamable-http --port $PORT --log-level INFO > /tmp/verify-pr-$PR.log 2>&1 &
SERVER_PID=$!

# Wait for the port to actually start LISTENing (cap 30s) — a blind sleep would
# let a startup crash silently become a "does not fix" verdict.
for i in $(seq 1 30); do
  lsof -nP -iTCP:$PORT -sTCP:LISTEN >/dev/null 2>&1 && break
  kill -0 $SERVER_PID 2>/dev/null || { echo "Server died during startup. Tail of /tmp/verify-pr-$PR.log:" >&2; tail -20 /tmp/verify-pr-$PR.log >&2; exit 1; }
  sleep 1
done
lsof -nP -iTCP:$PORT -sTCP:LISTEN >/dev/null 2>&1 || { echo "Server never bound port $PORT after 30s" >&2; tail -20 /tmp/verify-pr-$PR.log >&2; exit 1; }

curl -s -D /tmp/verify-pr-$PR-headers -X POST http://127.0.0.1:$PORT/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"verify-pr","version":"1.0"}}}' > /dev/null

SESSION_ID=$(grep -i 'Mcp-Session-Id' /tmp/verify-pr-$PR-headers | awk '{print $2}' | tr -d '\r')
[ -z "$SESSION_ID" ] && { echo "MCP initialize returned no Mcp-Session-Id. Tail of /tmp/verify-pr-$PR.log:" >&2; tail -20 /tmp/verify-pr-$PR.log >&2; kill $SERVER_PID 2>/dev/null; exit 1; }

# notifications/initialized often replies with a pydantic validation error.
# Harmless, same session ID still works for tools/call.
curl -s -X POST http://127.0.0.1:$PORT/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "Mcp-Session-Id: $SESSION_ID" \
  -d '{"jsonrpc":"2.0","id":2,"method":"notifications/initialized","params":{}}' > /dev/null

# Same tool + args as baseline, pulled verbatim from /tmp/repro-issue-$ISSUE-meta.json:
curl -s -X POST http://127.0.0.1:$PORT/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "Mcp-Session-Id: $SESSION_ID" \
  -d "{\"jsonrpc\":\"2.0\",\"id\":3,\"method\":\"tools/call\",\"params\":{\"name\":\"$TOOL\",\"arguments\":$ARGS_JSON}}" \
  | tee /tmp/verify-pr-$PR.json | head -200

kill $SERVER_PID 2>/dev/null; wait $SERVER_PID 2>/dev/null

Phase 5 — Diff and classify

diff -u /tmp/repro-issue-$ISSUE-main.json /tmp/verify-pr-$PR.json | head -200

Classify the PR outcome:

• Fixes cleanly ✓ — the failing field/section/error from the baseline is gone, no new failures visible, output structure matches CLAUDE.md → Tool Return Format ({url, sections, ...}).
• Fixes with concerns ⚠ — works on the issue's target but: (a) introduces new keys/sections not in the documented return format, (b) fails on a second target (different locale, edge case), (c) regresses an unrelated tool, (d) scope creep into unrelated files.
• Does not fix ✗ — same failure mode persists.
• Cannot run — merge conflicts that affect runtime, dependency drift (e.g. PR pins a Python version we don't have), or unrelated breakage.
• Cannot verify fix (skill limitation) — /2-repro-issue produced a success-baseline, not a failure-baseline (the bug only manifests under crash conditions, stdio transport, specific environments, or was already shipped-fixed). The diff therefore cannot show "before/after fix". Fall back to: (a) regression check — same baseline call must still succeed on PR branch, (b) code audit against the rules in Phase 6. Report this explicitly as a skill limitation, not as a PR weakness.

For locale-sensitive bugs, run one extra call against a deliberately non-English target before declaring "fixes cleanly". The detection logic must not regress on a German/RTL profile.

Phase 6 — Audit the diff against CLAUDE.md scraping rules

Read the actual diff, not just the file list:

git diff $(git merge-base "$PR_SHA" origin/main)..$PR_SHA

Hard flags (any one = downgrade from ✓ to ⚠):

• Locale-dependent detection: any == "Connect", in ["Pending", "Follow"], contains("1st"), aria-label="..." with translated text. The verb is locale-dependent; attribute presence is not. See CLAUDE.md → detection must be locale-independent.
• LinkedIn-class-name selectors: .entity-result__item, .artdeco-button__text, etc. Only minimal generic selectors are acceptable (a[href*="/jobs/view/"]).
• Multiple navigations behind one section: any new entry in PERSON_SECTIONS / COMPANY_SECTIONS (scraping/fields.py) must map to exactly one URL.
• Missing tests: bug fixes should add or update a test in tests/test_scraping.py. Pure-DOM fixes without test coverage are a yellow flag.

Phase 7 — Report

**PR #<PR>** (fixes #<ISSUE>) — <one-line PR title>
**Mergeable:** <CLEAN | DIRTY conflicts | BLOCKED — reason>
**Scope:** <+X/-Y, N files; flag if unrelated>
**Verdict:** <Fixes ✓ | Fixes with concerns ⚠ | Does not fix ✗ | Cannot run | Cannot verify fix (skill limitation)>
**Evidence:** 2–4 lines of the diff between baseline and PR-branch output that prove the verdict.
**Audit flags:** <locale-dependent | DOM-class selectors | section-mapping violation | missing tests | none>
**Cleaner alternative:** <other PR # | refactor sketch | none — PR is good as-is>
**Recommended next step:** <merge | request changes citing flag X | take over via maintainer-edits (maintainerCanModify=true) | close as not-needed>

Then one short paragraph: what the PR actually changes, why it does/doesn't address root cause, and what the locale-independent / DOM-minimal version would look like if the verdict is ⚠ or ✗.

Phase 8 — Cleanup

# The EXIT trap installed in Phase 3 runs automatically — it returns to
# $CURRENT_REF and cleans up /tmp/verify-pr-$PR.* on any exit (success, error,
# Ctrl-C). No local branch was ever created (Phase 3 uses a detached HEAD on
# FETCH_HEAD), so nothing needs deleting either.
# Keep /tmp/repro-issue-$ISSUE-main.json + meta.json so the user can re-verify against another PR later.

Leave the user on the branch they started on, with a clean working tree.

Non-negotiables

• Real LinkedIn, same profile, same call as the baseline. Anything else isn't a verification.
• Restart the server after git checkout — running workers hold stale code.
• Do not push, do not edit the PR, do not merge. This skill verifies and reports.
• Locale and DOM-class flags are non-overridable: if they're present, the verdict is ⚠ at best, regardless of whether the diff "works on the target".
• If the baseline never captured a failure (e.g. crash-only bug, stdio-only bug, env-specific bug), name it "Cannot verify fix (skill limitation)" and lean on code-audit + regression check. Never escalate a missing failure into a PR criticism.