Debug an agent from its trace

A trace is a tree of spans (parent request → LLM/tool/retrieval children). Most agent failures are visible in it if you read it in the right order. Don't guess from the final output - walk the tree.

Triage by symptom

Symptom	Look here first
Wrong answer	The LLM span just before the bad output: was the prompt/context correct? Then the retrieval span that fed it.
Empty / truncated output	finish_reason (length? content_filter?), max_tokens, and any span that raised an exception then got swallowed.
Hallucinated facts	Retrieval spans - were the right docs retrieved (check scores/IDs)? If not, it's a retrieval bug, not an LLM bug.
Too slow	Span durations - find the critical path. Usually one slow retrieval, a serial loop that should be parallel, or a huge context.
Too expensive	Token counts per LLM span - find the span with the largest input_tokens (usually bloated context or a retry storm).
Silent failure	A span with an error/exception that was caught and returned empty. A missing subtree = a step that never ran.

The systematic walk

1. Start at the root, confirm the user input is what you expect.
2. Follow to the first LLM call. Read the actual rendered prompt + context (not the template). Most "model is dumb" bugs are actually "we fed it the wrong context".
3. Check each tool/retrieval span in order: inputs correct? output sane? error?
4. Find the divergence point - the first span where reality differs from intent. Fix there, not at the output.
5. Check token + latency on every LLM span to catch cost/perf issues even when the answer is right.

Common root causes (in order of frequency)

• Wrong/empty retrieved context → the LLM was set up to fail. (Retrieval bug.)
• Prompt/template rendering bug → a variable didn't interpolate; context is blank or duplicated.
• Swallowed exception → a try/except that logs and returns empty, dropping context or output silently. Grep for these.
• Retry storm → the same call repeated N times (rate limit / transient error) inflating cost + latency.
• Context bloat → the whole history re-sent each turn; input_tokens grows every step.

Turn the fix into a regression test

Once you find the divergence, capture that input as an eval case (see the add-llm-evals skill) so the bug can't come back. Debugging a trace and not adding a test means you'll debug it again next month.

Anti-patterns

• Re-prompting the model when the real bug is upstream retrieval/rendering.
• Reading only the final output and inferring the cause.
• Fixing the symptom without adding a regression test.