What you will be able to do
- Find the first span in a trace whose output was already wrong, and use it to classify a failure as a prompt failure, retrieval failure, hallucination, tool or subagent error, or model mismatch
- Tell a retrieval failure from a hallucination by checking what Claude actually received
- Choose the corrective action that fixes the cause you found, not one that only hides the symptom
Key concept
First wrong span — A wrong final answer is only a symptom. The root cause is the earliest step in the trace that produced bad output from good input. Fix that step, because every later step was working with what it was given.
1.Walking a trace to the first wrong step
When a Claude system gives a wrong answer, the obvious suspect is the model. Usually the model was fine. It worked from a missing document, an ambiguous tool schema, or a subagent's vague summary. Isolating the root cause means walking the trace from start to finish and finding the first step whose output was wrong even though its input was right. The corrective action belongs at that step. A fix applied later, such as a stronger model or a stricter final prompt, treats the symptom and leaves the cause in place.
A trace records one request as a tree of spans. With the Anthropic SDK and the Claude API, the steps that matter are model requests, tool uses and, in Managed Agents, agent turns. Each step type keeps its evidence in a different place. A turn span's input is in gen_ai.input.messages, which holds your messages, tool results and messages from other threads. Its output is in gen_ai.output.messages. A tool span keeps the arguments in gen_ai.tool.call.arguments and the result in anthropic.tool_result.content. Evaluation work calls the same record a transcript: the complete record of a trial, including tool calls and intermediate results. Setting tracing up is covered in instrumentation and monitoring. This page is about reading the trace.
| Failure class | What to inspect | Telltale evidence |
|---|---|---|
| Retrieval failure | gen_ai.input.messages on the model request | The passage that answers the question never reached Claude |
| Hallucination | gen_ai.input.messages against gen_ai.output.messages | The supporting text was in the input, but the output contradicts it or adds to it |
| Tool error | gen_ai.tool.call.arguments and anthropic.tool_result.content | Malformed arguments, validation errors, or the wrong tool chosen |
| Subagent error | The child thread's events and the message it sent to the orchestrator | The subagent had correct data, but its report back lost it |
| Prompt failure | gen_ai.system_instructions and where instructions sit | Instructions that are vague, missing, or in the wrong place |
Two trace settings can hide the evidence you need. The content mode defaults to Metadata only, which records identifiers, timing, status and token counts but no prompts or responses. In that mode you can see that a run failed, but you can't tell whether the right document reached Claude. To read conversations, switch the destination or SDK client to Full content. Content can also be truncated. When it goes over the size limit, request spans drop the system prompt and the earliest messages first, and anthropic.content.truncated is set to true. A span with no system prompt and that flag set is not evidence of a prompt failure.
Managed Agents sessions also show this evidence in the Console session viewer. Errors are reported as session.error events. The Inspector's Tools tab lists each tool's call count, failures and median duration. Tool failures are a common explanation for unexpected agent behaviour, so check that tab early.
Checkpoint 1 of 9· Match them up
You suspect each of these failure classes in a Claude trace. Match each one to the place in the trace that holds its evidence.
Tap a term, then the definition that fits it.
Each failure class leaves its evidence in a different span attribute. To isolate the cause, read the attribute that shows whether a step got good input and still produced bad output, for example whether the passage was in the input (retrieval) or was in the input and then contradicted (hallucination).
“The system prompt is in gen_ai.system_instructions, the request’s messages in gen_ai.input.messages, and the response in gen_ai.output.messages.”Source: platform.claude.com
Checkpoint 2 of 9· Check yourself
A RAG assistant gave a confident wrong answer. Tracing is on with the default content mode. The team needs to know whether this was a retrieval failure or a hallucination. What should they do first?
The default content mode is Metadata only. It records timing, status and token counts but no prompts or responses, so it can't show whether the supporting passage reached Claude. Full content records the input and output that you need to tell retrieval failure from hallucination.
“Full content (content in the SDK): Most visibility. Spans include model prompts and responses, and everything in Metadata only.”Source: platform.claude.com
2.Tool errors: fix the definition, not the caller
Tool errors leave the clearest trail. The arguments are on the tool span and the error is in its result. Repeated patterns point back to how the tool is defined. If Claude calls the wrong tool, the descriptions are ambiguous. Sharpen them to say *when* to use each tool, not only what it does. Wrong parameter types point to an ambiguous schema. Values outside the allowed set point to missing strict mode or an enum that is too large. The fixes are strict: true where the schema supports it, a smaller enum, and input_examples that show valid choices.
Take a create_ticket tool whose description says only that it creates a ticket with a priority. If 40% of calls send priority: "urgent" and fail validation before a retry with P1 succeeds, the cause is the tool definition. Claude was never told which values are valid. The fix is to declare priority as an enum and describe what each level means. Three tempting alternatives miss. A system prompt telling Claude to double-check its arguments gives it no new information about valid values. A larger model tier can't guess an unstated enum. Accepting 'urgent' server-side as an alias makes the errors stop, but the definition stays ambiguous, so the next unlisted value fails the same way.
Some tool errors come from the harness rather than the model. Two examples:
- A 400 saying
tool_useids were found withouttool_resultblocks means your code didn't return one result per call, or put text before the results. - String matching on tool inputs that breaks after a model change happens because Unicode and forward-slash escaping differ between model versions. Parse the input with
json.loads()orJSON.parse()instead of comparing raw strings.
Checkpoint 3 of 9· Check yourself
In traces of a shipping agent, many book_shipment tool spans show gen_ai.tool.call.arguments with service_level set to values like "express_overnight". The tool result is a validation error, and a retry with "NEXT_DAY" then succeeds. The tool description says only that it books a shipment at a service level. Which corrective action fixes the root cause?
Values outside the allowed set come from the tool definition: Claude was never told which values are valid. Declaring the enum and showing valid choices fixes that. A bigger tier or a double-check instruction gives Claude no new information, and server-side aliases leave the definition ambiguous, so the next unlisted value fails the same way.
“Missing strict mode or too-large enum | Shrink the enum or add input_examples showing valid choices.”Source: platform.claude.com
Checkpoint 4 of 9· Exam question
A Claude agent calls a create_ticket tool. Traces show Claude sends priority: "urgent" in 40% of calls, and the tool returns a validation error each time because the schema accepts only P1, P2 or P3. Claude retries with P1 and succeeds. The tool description reads 'Creates a ticket with a priority.' What corrective action fits this root cause?
Correct answer: B — Tighten the tool definition by declaring priority as an enum of P1, P2 and P3 and describing what each level means in the tool description.
- A. Incorrect. A generic reminder does not give Claude the valid values, which exist only in the schema. The same guess recurs because the information gap is untouched.
- B. Correct. The error pattern comes from an ambiguous tool contract: Claude guessed a plausible value because nothing constrained it. Enumerating the allowed values and their meanings removes the guess at the source.
- C. Incorrect. The retry succeeds on the same model once it sees the error, showing it can follow the format when told the values. The defect is the missing constraint, not model capability.
- D. Incorrect. This hides the symptom for one alias and leaves the contract vague, so the next plausible value such as 'high' or 'critical' fails identically. The schema still tells Claude nothing.
Sources4
3.Retrieval failure or hallucination: check what Claude received
Both failures produce the same symptom: a fluent, confident, wrong answer. One question separates them: was the supporting text in the context Claude received? Open gen_ai.input.messages on the model request and look for the passage. If it isn't there, the retrieval step failed, even if the right document is indexed. A ranker that returns shipping text for a refund question has failed, whatever Claude does with that text afterwards.
Fix a retrieval failure in the retriever, then measure it again. Useful levers include better chunking, query rewriting, contextual retrieval (a preprocessing step that improves retrieval accuracy) and hybrid search that combines semantic and keyword search. Measure with recall at k, the share of relevant documents that are missing from the top k chunks, and re-run it on the sessions that failed. Retrying the search call is the wrong fix: a call that returned results without errors did not fail. A prompt that refuses unless the user names a policy section changes the user's behaviour, not the ranking. A larger tier with the retriever unchanged still gets no refund text.
When the passage *was* in the input and the output contradicts it or adds to it, the model step failed. That is a hallucination, and the fixes go in the request. Give Claude explicit permission to say it doesn't know. For long documents, have it extract word-for-word quotes before it answers. Make it cite a supporting quote for each claim and retract any claim it can't support. Tell it to use only the provided documents and not its general knowledge. Running the same prompt several times also helps diagnosis: answers that disagree with each other point to hallucination.
Hallucination. The input was right, so retrieval is not at fault. Have Claude ground the answer in an extracted quote, and make it check each claim against a supporting quote and retract any claim it can't support.
Checkpoint 5 of 9· Match them up
In each run below, the supporting passage did reach Claude in gen_ai.input.messages, so the model step is at fault. Match each observed symptom to the corrective action or diagnostic step that fits it.
Tap a term, then the definition that fits it.
When the passage was in the input and the output still contradicts it or adds to it, the fix goes in the request. Each technique targets a specific way the model departs from its context. Best-of-N is a diagnostic: answers that disagree with each other point to hallucination.
“Best-of-N verification: Run Claude through the same prompt multiple times and compare the outputs. Inconsistencies across outputs could indicate hallucinations.”Source: platform.claude.com
Checkpoint 6 of 9· Exam question
A support assistant built on the Messages API answers questions from a retrieval pipeline. Users report wrong refund-policy answers. You inspect traces for ten failing sessions. In every one, the top five chunks passed to Claude cover shipping and warranty text, none mention refunds, and Claude's answer states a refund window confidently. The refund policy page is indexed. What is the root cause and the right corrective action?
Correct answer: C — Retrieval failure: the ranker surfaced unrelated passages, so revise chunking, query rewriting or hybrid search, then re-measure recall on the failing sessions.
- A. Incorrect. Each trace shows five chunks returned, so the search call completed and no results were dropped. Retries and a longer timeout would return the same irrelevant ranking, so the missing refund evidence would stay missing.
- B. Incorrect. The prompt never had refund text to work with, so loosening or tightening it does not supply it. Requiring users to name a policy section shifts the retrieval gap onto them and turns wrong answers into refusals, while the cause stays in place.
- C. Correct. The passages passed to Claude covered other topics, so the evidence was lost before generation even though the policy page is indexed. Chunk boundaries, query rewriting and hybrid search all act on that stage, and recall on the failing sessions confirms the fix.
- D. Incorrect. The context held no refund wording to misread, so a larger tier would face the same gap and probably produce a more fluent but equally unsupported answer. Changing the model only makes sense once the context is shown to contain the evidence.
4.Subagent errors that hide in the handoff
In multi-agent systems, a correct tool call can still lead to a wrong answer, because the result passes through a subagent's report before the orchestrator sees it. The primary thread's stream shows only critical events from subagents. To see what a subagent actually did, stream or list the events on its own thread. On the primary stream, agent.thread_message_received carries what the subagent sent back, including from_session_thread_id, from_agent_name and content.
for event in client.beta.sessions.threads.events.list(
thread.id,
session_id=session.id,
):
print(f"[{event.type}] {event.processed_at}")Suppose an invoice subagent's billing API call returns the correct total in anthropic.tool_result.content. Its report to the orchestrator says 'invoice found, total looks normal', with no number, and the final answer cites a different total. The tool span is correct, so the tool didn't fail. The report is the first wrong step: it dropped the figure. The orchestrator filled the gap. Two fixes work together:
- Make the subagent return a structured result with named fields, such as amount and currency, so the figure reaches the orchestrator as data.
- Have the orchestrator validate each report and send the task back with a corrective instruction when a required field is missing.
The tempting alternatives each miss the handoff. Raising the subagent's max_tokens does nothing, because the reply wasn't cut off; the number was never written. A cached data source changes where the correct total comes from, but the total was already correct. A bigger orchestrator with extended thinking can't recover a figure it never received. The fix starts with the delegation itself. Every delegated task should specify an output format, so that a report like this cannot be accepted.
Checkpoint 7 of 9· Fill the gap
An orchestrator cited the wrong invoice total. The primary thread's stream shows only critical events from subagents. To compare what the invoice subagent's billing call returned with the report it sent back, you pull that subagent's full event history. Which token completes the call?
for event in client.beta.sessions. ? .events.list(
thread.id,
session_id=session.id,
):
print(f"[{event.type}] {event.processed_at}")Each subagent runs in its own session thread, and the primary stream shows only a condensed view of their work. Listing events on that thread with sessions.threads.events.list gives you its full history. You can then compare the tool result with the report that reached the orchestrator.
Source: platform.claude.comCheckpoint 8 of 9· Exam question
An orchestrator agent built with the Claude Agent SDK delegates invoice lookups to a subagent. Traces show the subagent's tool call to the billing API succeeded, and the returned JSON contained the correct total. The final answer to the user cites a different total. The subagent's message to the orchestrator says 'invoice found, total looks normal' without a number. Which TWO actions address the root cause? Choose two.(Select 2)
Correct answers: A, E — Add a check in the orchestrator that flags subagent replies lacking the required invoice fields, so it re-delegates with a corrective instruction.; Require the subagent to return a structured result with named amount and currency fields, so the orchestrator receives the figure as data.
- A. Correct. The subagent's reply omitted the number, so the orchestrator needs a check at the handoff that catches the missing field instead of filling the gap itself. Re-delegating with a corrective instruction gives the subagent a concrete repair to make.
- B. Incorrect. The subagent's note was short and complete, not cut off, so output length was never the limit. A higher token limit does not make it include a value it chose to leave out, and the reply would still be free-form prose.
- C. Incorrect. The trace shows the billing API returned the correct amount, so the data source is not the fault. A cached copy adds the risk of stale figures and leaves the lossy subagent-to-orchestrator handoff untouched.
- D. Incorrect. The amount was never sent to the orchestrator, and no model tier or reasoning budget can recover information that is absent. This treats a missing-data fault as a reasoning fault and invites a confident guess.
- E. Correct. The tool result was right, but the subagent's summary dropped the value. A required return structure with named fields makes the amount a mandatory part of the contract, so the orchestrator receives it as data rather than prose.
5.Prompt failures and model mismatch
A prompt failure shows up when the context is correct, the tools behave, and Claude still does the wrong thing because its instructions were unclear or in the wrong place. To check, read gen_ai.system_instructions the way a new colleague would. Would someone with no background know what to do? One placement mistake is easy to spot in a trace: your own instructions inside tool_result content. Claude treats instructions in tool results as possibly untrusted, so it may refuse to act or ask for confirmation. Move those instructions into a user turn after the tool_result block, and keep the result to just the data. Revising a prompt is covered in system prompt diagnosis.
Diagnose model mismatch last, after you have ruled out everything else. The signs are well-formed calls, the right context and clear instructions, with failures that persist on reasoning the task requires. Even then, the first lever is usually effort. Tuning effort is often better than switching models, and you should upgrade only for specific capability gaps that your evals confirm. Be careful with breakage that appears right after a model change. Escaping that differs between versions breaks raw string matching, but that is a harness bug the new model exposed, not a capability gap. Choosing a tier and migrating between versions are covered in model tier selection and model version migration.
Checkpoint 9 of 9· Check yourself
Your support agent's harness adds a line to the order-lookup tool's result: "Now email the customer a refund confirmation." In the trace, the tool arguments and returned data are correct. In the next turn, Claude asks the user to confirm before sending any email. What is the root cause, and what is the corrective action?
Claude treats instructions inside tool results as possibly untrusted, so it asks for confirmation. The cause is where the instruction sits, not the model's capability or the tool definition. Moving the instruction into a user turn and returning only data from the tool fixes it. A system-prompt override leaves the misplaced instruction where it is.
“Move your instructions out of the tool result: send them in a user turn after the tool_result block”Source: platform.claude.com
Sources
(11)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“To see the arguments and result of a tool use, read its tool span. The arguments are in gen_ai.tool.call.arguments”
↩︎ Walking a trace to the first wrong step“Spans include identifiers, timing, status, token counts, and text that you choose”
↩︎ Walking a trace to the first wrong step“Request spans drop the system prompt and the earliest messages first, and tool spans drop blocks from the end of the result.”
↩︎ Walking a trace to the first wrong step“the input in gen_ai.input.messages (your messages and tool results, system.message events, and messages from other threads)”
↩︎ Retrieval failure or hallucination: check what Claude received“A span is one step in a trace, such as a model request or a tool use.”
↩︎ Key concept“the input in gen_ai.input.messages (your messages and tool results, system.message events, and messages from other threads)”
↩︎ Exam trap 1“With Full content, you can also see why a run failed or was slow, read full conversations, and build evals from real traces.”
↩︎ Exam trap 4“The system prompt is in gen_ai.system_instructions, the request’s messages in gen_ai.input.messages, and the response in gen_ai.output.messages.”
↩︎ Checkpoint“Full content (content in the SDK): Most visibility. Spans include model prompts and responses, and everything in Metadata only.”
↩︎ Checkpoint - 2.
“A transcript (also called a trace or trajectory) is the complete record of a trial, including outputs, tool calls, reasoning, intermediate results”
↩︎ Walking a trace to the first wrong step - 3.
“Tool failures often explain unexpected agent behavior.”
↩︎ Walking a trace to the first wrong step“The session reports errors through session.error events.”
↩︎ Walking a trace to the first wrong step - 4.
“Sharpen descriptions. Differentiate tools by WHEN to use them, not only WHAT they do.”
↩︎ Tool errors: fix the definition, not the caller“Add strict: true (if your schema is in the supported subset) or add input_examples.”
↩︎ Tool errors: fix the definition, not the caller“Return one tool_result for every tool_use block in the assistant response.”
↩︎ Tool errors: fix the definition, not the caller“Parse with json.loads() or JSON.parse(). Never do raw string matching on serialized input.”
↩︎ Tool errors: fix the definition, not the caller“Move your instructions out of the tool result: send them in a user turn after the tool_result block”
↩︎ Prompt failures and model mismatch“Unicode and forward-slash escaping differs between model versions”
↩︎ Prompt failures and model mismatch“Shrink the enum or add input_examples showing valid choices.”
↩︎ Exam trap 2“Missing strict mode or too-large enum | Shrink the enum or add input_examples showing valid choices.”
↩︎ Checkpoint - 5.https://www.anthropic.com/engineering/contextual-retrievalSecondary source
“Contextual Retrieval is a preprocessing technique that improves retrieval accuracy.”
↩︎ Retrieval failure or hallucination: check what Claude received“which measures the percentage of relevant documents that fail to be retrieved within the top 20 chunks”
↩︎ Retrieval failure or hallucination: check what Claude received - 6.
“combining semantic search with keyword-based search using Contextual BM25—a hybrid approach that reduces retrieval failure rates further”
↩︎ Retrieval failure or hallucination: check what Claude received“This means fewer irrelevant results in your top retrieved chunks, leading to better answers when you pass these chunks to Claude for final response generation.”
↩︎ Prediction - 7.https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinationsOfficial docs
“Explicitly give Claude permission to admit uncertainty.”
↩︎ Retrieval failure or hallucination: check what Claude received“External knowledge restriction: Explicitly instruct Claude to only use information from provided documents and not its general knowledge.”
↩︎ Retrieval failure or hallucination: check what Claude received“Inconsistencies across outputs could indicate hallucinations.”
↩︎ Retrieval failure or hallucination: check what Claude received“Best-of-N verification: Run Claude through the same prompt multiple times and compare the outputs. Inconsistencies across outputs could indicate hallucinations.”
↩︎ Checkpoint - 8.
“stream or list the events from the associated session thread.”
↩︎ Subagent errors that hide in the handoff“Includes from_session_thread_id, from_agent_name, and content.”
↩︎ Subagent errors that hide in the handoff“Includes from_session_thread_id, from_agent_name, and content.”
↩︎ Exam trap 3 - 9.
“Each subagent needs an objective, an output format, guidance on the tools and sources to use, and clear task boundaries.”
↩︎ Subagent errors that hide in the handoff“This prevents information loss during multi-stage processing and reduces token overhead from copying large outputs through conversation history.”
↩︎ Subagent errors that hide in the handoff - 10.https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practicesOfficial docs
“Key test: Show your prompt to a colleague with minimal context on the task and ask them to follow it.”
↩︎ Prompt failures and model mismatch - 11.
“Tuning effort is often a better lever than switching models.”
↩︎ Prompt failures and model mismatch“Upgrade only if necessary for specific capability gaps.”
↩︎ Prompt failures and model mismatch