Feedback Synthesizer

Takes raw numbers and raw comments and produces one connected picture, not two separate reports that happen to sit in the same doc.

Most teams either report a score or summarize comments. This skill's whole job is to make the two inform each other — a score drop with no thematic explanation isn't a finding yet.

---

Step 0: What are you looking at?

Identify the input type before doing anything:

• From survey-architect (a /research/<study-name>/01-survey-spec.json exists) → read it for instrument, scoring formula, and expected n. Use it, don't re-derive.
• Standalone raw data (pasted CSV, app store reviews, support ticket export, no spec file) → identify the instrument from the data itself if it's a known format (SUS items, an 0–10 NPS column, etc.), or treat it as unstructured qualitative feedback if there's no scale attached. Say explicitly which case you're in — don't silently assume a scale that isn't there.

---

Step 1: Quantitative scoring

Run scripts/scoring.py — do not compute these by hand. Every instrument formula, the confidence interval, and the significance tests are exact, deterministic calculations with no judgment involved, which is exactly the kind of task that should never depend on an LLM doing arithmetic in its head. Call the matching function for the instrument in use:

Instrument	Function
SUS	sus_score(responses)
UMUX	umux_score(responses)
UMUX-Lite	umux_lite_score(item1, item2)
NPS	nps_score(responses)
CSAT	csat_top_two_box(responses) or csat_mean(responses) — state which
SEQ / CES	seq_score(responses) / ces_score(responses)
SUPR-Q	supr_q_score(five_point_items, nps_item_0_to_10)

If the instrument doesn't map to one of these (a custom item set), say so explicitly rather than forcing it through the nearest function.

Confidence intervals and significance testing

Also not optional, also run through the script — call confidence_interval(scores) and, when comparing against a benchmark or prior wave, significance_vs_benchmark(scores, benchmark) or compare_waves(scores_a, scores_b). Never report a headline score without its CI, and never call a difference "significant" without having actually run the test. The confidence level itself (scoring.DEFAULT_CONFIDENCE_LEVEL, currently 90%) is centralized in the script, not restated here — if it ever changes, it changes in one place.

State the result in plain language alongside the numbers ("the 6-point drop is outside the margin of error — this is a real change" vs. "the 3-point drop is within the CI — could be noise").

Three separate questions, never collapsed into one:

• Statistical significance — did significance_vs_benchmark / compare_waves actually return significant=True? This is the only question those functions answer.
• Practical significance — does the size of the difference matter for the decision the study was run for? A statistically significant 2-point SUS movement on an n=400 study is real but may not be worth acting on; state this explicitly when it applies, don't let a small p-value stand in for "this matters."
• Evidence confidence — see the dedicated section below. A narrow CI does not by itself mean the overall conclusion is trustworthy, and a wide CI does not by itself mean it isn't.

A statistically significant result is not automatically important. A non-significant result — especially at a small n — is not automatically "no real difference"; it may just mean the study was underpowered to detect one. Say which situation you're in rather than defaulting to either reading.

If scripts/scoring.py isn't available in the execution environment for some reason, say so explicitly before falling back to manual calculation — don't silently compute by hand and present it as if the script ran.

Small-sample handling

If n falls below the sample-size floor survey-architect specified (or below the general rule-of-thumb minimums if there's no spec file):

• Compute the score anyway. Never refuse.
• Widen the CI accordingly and say so explicitly.
• Flag it loudly at the top of the synthesis output, not buried in a footnote: ⚠️ LOW CONFIDENCE — n=<X>, below recommended minimum of <Y>. Treat this score as directional, not conclusive.
• If the gap is large enough that the score is essentially unusable for the stated decision (see survey-architect's learning goal), kick back to survey-architect with a note to extend collection — this is the loop-back condition, not a silent proceed.

---

Step 2: Qualitative coding

Treat this as a protocol, not a single read-through-and-label pass — initial impressions from comment 1 shouldn't silently become the frame everything else gets sorted into:

1. Read the full dataset once before coding anything. Don't start labeling on comment 1.
2. Generate initial codes — short, descriptive, close to the data.
3. Review code consistency — the same underlying complaint shouldn't end up under two different code names.
4. Merge/split codes where the initial pass was too fine- or coarse-grained.
5. Identify candidate themes by grouping related codes.
6. Actively look for disconfirming cases — comments that don't fit the emerging theme, or that would complicate it. Don't just note supporting examples.
7. Test each theme against the full dataset, not just the comments that inspired it.
8. Quantify themes only after coding has stabilized — counting before the taxonomy settles bakes in whatever grouping happened to come first.
9. Assign a per-theme coding confidence (see the Evidence Confidence section below) — some themes are obvious, some rest on a handful of ambiguous comments; say which.
10. Cross-reference against the quantitative data (Step 3).

Taxonomy source:

• New product or first study: code comments with an emergent/open approach — let themes surface from the data, don't force them into a predefined bucket.
• Product with prior studies: check /research/_benchmarks/<product>.md for the existing theme taxonomy. Reuse those theme labels where the data fits; only add a new theme label if genuinely nothing existing captures it. This is how the taxonomy converges over time instead of resetting every study.
• For each theme, report: frequency (n and %), representative paraphrased examples (never verbatim quotes — see copyright note below), whether it skews toward low or high scorers, and the coding confidence from step 9 above.

Never quote respondents verbatim in synthesis output — paraphrase every comment. This matters for two reasons: respondent privacy in smaller samples, and because verbatim quotes get copy-pasted into decks and lose the aggregation context.

---

Step 3: Cross-reference qual and quant

This is the step most synthesis work skips. For every notable score change (a wave-over-wave drop, a below-benchmark score, a segment that scores lower than others):

1. Pull the subset of open-text comments from respondents in that score band.
2. Check whether a theme is disproportionately represented in that subset vs. the overall theme distribution.
3. If yes, state the connection explicitly: "SUS dropped 8 points this wave (95% CI: −11 to −5, statistically significant vs. wave 2). 60% of respondents scoring below 60 mentioned the new checkout step, vs. 12% of respondents scoring above 80."
4. If no clear theme explains a score change, say that too — an unexplained score movement is itself a finding, not a gap to paper over.

---

Step 4: Severity and priority scoring

This skill owns severity/priority — not the reporter. Once you have the raw counts (how many respondents in each score band mentioned the theme), run scoring.severity_tier(...) rather than eyeballing the percentages — this is the exact calculation that was wrong in v1.0.0 (diluted by whole-sample frequency instead of within-band frequency), and it's now a tested function specifically so that bug can't quietly reappear in prose form. Getting the raw counts right — which respondents go in which band, whether a comment belongs to the theme — is still your judgment call; the arithmetic on top of those counts is not.

Score each theme on two axes:

• Frequency: what % of respondents in the affected score band raised it — not what % of the whole sample raised it. A theme mentioned by 3 of 4 low scorers is high-frequency even if it's only 3 of 14 respondents overall. Measuring frequency against the full sample dilutes a concentrated signal and can demote a real driver of low scores to a lower tier than it deserves. Also report the plain overall % for context, but don't let it drive the tier.
• Impact: does it correlate with low scores / high effort / low completion — i.e., is the theme's presence skewed toward one score band vs. another, per the cross-reference in Step 3

Combine into a simple priority tier (not a false-precision numeric score):

• Critical — high frequency within the affected band (roughly ≥50% of respondents in that band) and a clear skew vs. the unaffected band (that band's rate is meaningfully higher than the rest of the sample's rate for the same theme)
• Notable — meets one of the two Critical conditions, not both (e.g., mentioned by a majority of low scorers but also shows up somewhat among high scorers, or a clear skew but only a small minority of the affected band mentions it)
• Minor — low frequency within the affected band and no meaningful skew vs. the rest of the sample

State the reasoning for each tier assignment explicitly, including both the within-band and overall frequency numbers — don't just output the label, and don't let a low overall frequency read as an argument against a tier the within-band numbers support.

The frequency/skew tier is a default, not a law. Pass scoring.severity_tier's override argument (one of scoring.VALID_OVERRIDE_REASONS: safety, accessibility, legal_compliance, severe_user_harm, critical_task_blockage) when one of those considerations should outrank the computed tier — e.g. a low-frequency accessibility barrier can legitimately outrank a high-frequency cosmetic complaint. An override is never silent: report both computed_tier (what the frequency math actually produced) and the override reason side by side, never just the final "Critical" label. An override is for one of those five named reasons — not for a stakeholder who'd simply prefer a different number. severity_tier raises on any other reason string rather than accepting it.

Coding themes from judgment, not keyword matching. A comment can belong to a theme without using any of that theme's obvious keywords — e.g., "had to re-enter my card info twice" belongs in a payment/checkout friction theme even though it never says "payment." Read each comment for what it's actually describing, not for whether it contains a specific word. A literal string search will systematically undercount themes and produce frequency numbers that look lower than reality.

---

Step 4.5: Evidence confidence, claim strength, and uncertainty

Three fields belong on every finding, distinct from anything computed above:

Evidence confidence — how much to trust the overall conclusion, not any one statistic. One of scoring.EVIDENCE_CONFIDENCE_LEVELS: HIGH, MEDIUM, LOW, INSUFFICIENT. Base it on sample quality, source quality, per-theme coding confidence (Step 2), and whether qual and quant actually triangulate (Step 3) — not on how narrow the CI happens to be. State the drivers, not just the label, e.g.: {"overall": "MEDIUM", "drivers": {"sample_quality": "high", "coding_confidence": "medium", "triangulation": "low"}}. Don't collapse these into a single number — a category is honest about how fuzzy this judgment is; a number pretends otherwise.

Claim strength — tag each finding with one rung of scoring.CLAIM_STRENGTH_LEVELS: observed (reported/measured, no relationship claimed), associated (co-occurs with something else, direction untested), correlated (a statistical relationship was actually tested), or causal (one thing was shown to produce another). Run scoring.validate_claim_strength(level) before writing a claim down — it raises if causal is used without an experimental or strong quasi-experimental design behind it, which a survey/feedback synthesis never has. Default to correlated when tempted to say causal.

Alternative explanations and "cannot determine." For findings above Minor severity, name at least one plausible alternative reading of the same data (e.g. "navigation terminology may be unfamiliar" as an alternative to "navigation is broken") — this is what stops a single plausible story from being reported as the only one. Separately, list anything the data genuinely can't answer (e.g. "whether this causes account abandonment") rather than implying an answer the evidence doesn't support.

---

Step 5: Output

Write /research/<study-name>/02-synthesis.md and 02-synthesis.json — the .json is the canonical, machine-readable artifact research-reporter consumes; the .md is the human-readable copy of the same facts. Before writing, run scoring.validate_synthesis(synthesis) and fill in anything it reports missing. Required shape: study, product, instrument, n, score (value + CI), significance, low_confidence_flag, themes (each with frequency, severity [tier, computed_tier, override_reason], claim_strength), cross_references, evidence_confidence, alternative_explanations, cannot_determine.

Before appending to the benchmark file, check comparability against the prior wave's recorded instrument/wording/scale/population/sampling method/trigger (see research-loop's check_benchmark_comparability) — if they don't match closely enough, report the two waves separately rather than implying a trend. When they do, append the wave's headline score to /research/_benchmarks/<product>.md: date, instrument, score, CI, n, and the same measurement-condition fields, so the next study's comparability check has something to check against.

---

Rules

Priority when rules interact: STOP > MUST NOT > MUST > SHOULD > MAY — see ux-research/README.md for the full precedence explanation.

• STOP if the sample is so small the stated learning goal can't be answered at all — kick back to survey-architect instead of proceeding.
• MUST NOT report a score without its confidence interval.
• MUST NOT call a numeric difference "significant" without having run the actual test.
• MUST NOT refuse to score a small sample — flag it loudly and proceed instead.
• MUST NOT quote respondents verbatim.
• MUST NOT force new data into an existing theme taxonomy when it genuinely doesn't fit.
• MUST NOT code themes by keyword/string matching instead of reading each comment for what it actually describes.
• MUST NOT measure a theme's frequency only against the whole sample when deciding severity — check frequency within the affected score band first.
• MUST NOT label a finding causal without an experimental or strong quasi-experimental design — validate_claim_strength enforces this; don't work around it by skipping the call.
• MUST NOT apply a severity override for a reason outside scoring.VALID_OVERRIDE_REASONS, or apply one silently without stating computed_tier alongside the override.
• MUST NOT treat two benchmark waves as a trend without checking comparability first.
• MUST report a score movement and a theme together only after checking (Step 3) whether they're actually connected — an unexplained movement is itself a finding, never papered over.
• MUST state the frequency + impact reasoning behind every severity tier, not just the label.
• SHOULD name at least one alternative explanation for any finding above Minor severity, and list what genuinely cannot be determined from the data.

---

Quick reference

Situation	Action
n below recommended minimum	Score it, widen CI, flag loudly
n so small the learning goal is unanswerable	Kick back to survey-architect
Score drop, no theme correlation found	Report the drop as unexplained, don't force a story
Score drop + strong theme correlation	State both together as one finding
New product, first study	Emergent coding
Existing product with prior studies	Reuse taxonomy from benchmark file, extend only if needed
Standalone data, no spec file	Say so explicitly, identify instrument from data structure