CertSafari
    Anthropic Claude Certified Architect – Professional (CCAR-P) - December 7 onwards· Lessons

    Domain 4 · Lesson 25/39

    Root-cause isolation from Claude logs and traces

    Root-Cause Isolation from Logs and Traces

    16 min read
    2.43% of exam
    11 sources
    Published 18 Nov 2026
    Docs as of 18 Nov 2026

    What you will be able to do

    • Find the first span in a trace whose output was already wrong, and use it to classify a failure as a prompt failure, retrieval failure, hallucination, tool or subagent error, or model mismatch
    • Tell a retrieval failure from a hallucination by checking what Claude actually received
    • Choose the corrective action that fixes the cause you found, not one that only hides the symptom

    Key concept

    First wrong span — A wrong final answer is only a symptom. The root cause is the earliest step in the trace that produced bad output from good input. Fix that step, because every later step was working with what it was given.

    1.Walking a trace to the first wrong step

    When a Claude system gives a wrong answer, the obvious suspect is the model. Usually the model was fine. It worked from a missing document, an ambiguous tool schema, or a subagent's vague summary. Isolating the root cause means walking the trace from start to finish and finding the first step whose output was wrong even though its input was right. The corrective action belongs at that step. A fix applied later, such as a stronger model or a stricter final prompt, treats the symptom and leaves the cause in place.

    A trace records one request as a tree of spans. With the Anthropic SDK and the Claude API, the steps that matter are model requests, tool uses and, in Managed Agents, agent turns. Each step type keeps its evidence in a different place. A turn span's input is in gen_ai.input.messages, which holds your messages, tool results and messages from other threads. Its output is in gen_ai.output.messages. A tool span keeps the arguments in gen_ai.tool.call.arguments and the result in anthropic.tool_result.content. Evaluation work calls the same record a transcript: the complete record of a trial, including tool calls and intermediate results. Setting tracing up is covered in instrumentation and monitoring. This page is about reading the trace.

    Where each failure class shows up in a Claude trace
    Failure classWhat to inspectTelltale evidence
    Retrieval failuregen_ai.input.messages on the model requestThe passage that answers the question never reached Claude
    Hallucinationgen_ai.input.messages against gen_ai.output.messagesThe supporting text was in the input, but the output contradicts it or adds to it
    Tool errorgen_ai.tool.call.arguments and anthropic.tool_result.contentMalformed arguments, validation errors, or the wrong tool chosen
    Subagent errorThe child thread's events and the message it sent to the orchestratorThe subagent had correct data, but its report back lost it
    Prompt failuregen_ai.system_instructions and where instructions sitInstructions that are vague, missing, or in the wrong place

    Two trace settings can hide the evidence you need. The content mode defaults to Metadata only, which records identifiers, timing, status and token counts but no prompts or responses. In that mode you can see that a run failed, but you can't tell whether the right document reached Claude. To read conversations, switch the destination or SDK client to Full content. Content can also be truncated. When it goes over the size limit, request spans drop the system prompt and the earliest messages first, and anthropic.content.truncated is set to true. A span with no system prompt and that flag set is not evidence of a prompt failure.

    Managed Agents sessions also show this evidence in the Console session viewer. Errors are reported as session.error events. The Inspector's Tools tab lists each tool's call count, failures and median duration. Tool failures are a common explanation for unexpected agent behaviour, so check that tab early.

    Checkpoint 1 of 9· Match them up

    You suspect each of these failure classes in a Claude trace. Match each one to the place in the trace that holds its evidence.

    Tap a term, then the definition that fits it.

    Checkpoint 2 of 9· Check yourself

    A RAG assistant gave a confident wrong answer. Tracing is on with the default content mode. The team needs to know whether this was a retrieval failure or a hallucination. What should they do first?

    Sources123

    2.Tool errors: fix the definition, not the caller

    Tool errors leave the clearest trail. The arguments are on the tool span and the error is in its result. Repeated patterns point back to how the tool is defined. If Claude calls the wrong tool, the descriptions are ambiguous. Sharpen them to say *when* to use each tool, not only what it does. Wrong parameter types point to an ambiguous schema. Values outside the allowed set point to missing strict mode or an enum that is too large. The fixes are strict: true where the schema supports it, a smaller enum, and input_examples that show valid choices.

    Take a create_ticket tool whose description says only that it creates a ticket with a priority. If 40% of calls send priority: "urgent" and fail validation before a retry with P1 succeeds, the cause is the tool definition. Claude was never told which values are valid. The fix is to declare priority as an enum and describe what each level means. Three tempting alternatives miss. A system prompt telling Claude to double-check its arguments gives it no new information about valid values. A larger model tier can't guess an unstated enum. Accepting 'urgent' server-side as an alias makes the errors stop, but the definition stays ambiguous, so the next unlisted value fails the same way.

    Some tool errors come from the harness rather than the model. Two examples:

    • A 400 saying tool_use ids were found without tool_result blocks means your code didn't return one result per call, or put text before the results.
    • String matching on tool inputs that breaks after a model change happens because Unicode and forward-slash escaping differ between model versions. Parse the input with json.loads() or JSON.parse() instead of comparing raw strings.

    Checkpoint 3 of 9· Check yourself

    In traces of a shipping agent, many book_shipment tool spans show gen_ai.tool.call.arguments with service_level set to values like "express_overnight". The tool result is a validation error, and a retry with "NEXT_DAY" then succeeds. The tool description says only that it books a shipment at a service level. Which corrective action fixes the root cause?

    Checkpoint 4 of 9· Exam question

    A Claude agent calls a create_ticket tool. Traces show Claude sends priority: "urgent" in 40% of calls, and the tool returns a validation error each time because the schema accepts only P1, P2 or P3. Claude retries with P1 and succeeds. The tool description reads 'Creates a ticket with a priority.' What corrective action fits this root cause?

    Sources4

    3.Retrieval failure or hallucination: check what Claude received

    Both failures produce the same symptom: a fluent, confident, wrong answer. One question separates them: was the supporting text in the context Claude received? Open gen_ai.input.messages on the model request and look for the passage. If it isn't there, the retrieval step failed, even if the right document is indexed. A ranker that returns shipping text for a refund question has failed, whatever Claude does with that text afterwards.

    Fix a retrieval failure in the retriever, then measure it again. Useful levers include better chunking, query rewriting, contextual retrieval (a preprocessing step that improves retrieval accuracy) and hybrid search that combines semantic and keyword search. Measure with recall at k, the share of relevant documents that are missing from the top k chunks, and re-run it on the sessions that failed. Retrying the search call is the wrong fix: a call that returned results without errors did not fail. A prompt that refuses unless the user names a policy section changes the user's behaviour, not the ranking. A larger tier with the retriever unchanged still gets no refund text.

    When the passage *was* in the input and the output contradicts it or adds to it, the model step failed. That is a hallucination, and the fixes go in the request. Give Claude explicit permission to say it doesn't know. For long documents, have it extract word-for-word quotes before it answers. Make it cite a supporting quote for each claim and retract any claim it can't support. Tell it to use only the provided documents and not its general knowledge. Running the same prompt several times also helps diagnosis: answers that disagree with each other point to hallucination.

    Checkpoint 5 of 9· Match them up

    In each run below, the supporting passage did reach Claude in gen_ai.input.messages, so the model step is at fault. Match each observed symptom to the corrective action or diagnostic step that fits it.

    Tap a term, then the definition that fits it.

    Checkpoint 6 of 9· Exam question

    A support assistant built on the Messages API answers questions from a retrieval pipeline. Users report wrong refund-policy answers. You inspect traces for ten failing sessions. In every one, the top five chunks passed to Claude cover shipping and warranty text, none mention refunds, and Claude's answer states a refund window confidently. The refund policy page is indexed. What is the root cause and the right corrective action?

    Sources1567

    4.Subagent errors that hide in the handoff

    In multi-agent systems, a correct tool call can still lead to a wrong answer, because the result passes through a subagent's report before the orchestrator sees it. The primary thread's stream shows only critical events from subagents. To see what a subagent actually did, stream or list the events on its own thread. On the primary stream, agent.thread_message_received carries what the subagent sent back, including from_session_thread_id, from_agent_name and content.

    Listing a subagent thread's full event history to compare what it found with what it reportedpython
    for event in client.beta.sessions.threads.events.list(
        thread.id,
        session_id=session.id,
    ):
        print(f"[{event.type}] {event.processed_at}")

    Suppose an invoice subagent's billing API call returns the correct total in anthropic.tool_result.content. Its report to the orchestrator says 'invoice found, total looks normal', with no number, and the final answer cites a different total. The tool span is correct, so the tool didn't fail. The report is the first wrong step: it dropped the figure. The orchestrator filled the gap. Two fixes work together:

    • Make the subagent return a structured result with named fields, such as amount and currency, so the figure reaches the orchestrator as data.
    • Have the orchestrator validate each report and send the task back with a corrective instruction when a required field is missing.

    The tempting alternatives each miss the handoff. Raising the subagent's max_tokens does nothing, because the reply wasn't cut off; the number was never written. A cached data source changes where the correct total comes from, but the total was already correct. A bigger orchestrator with extended thinking can't recover a figure it never received. The fix starts with the delegation itself. Every delegated task should specify an output format, so that a report like this cannot be accepted.

    Checkpoint 7 of 9· Fill the gap

    An orchestrator cited the wrong invoice total. The primary thread's stream shows only critical events from subagents. To compare what the invoice subagent's billing call returned with the report it sent back, you pull that subagent's full event history. Which token completes the call?

    for event in client.beta.sessions. ? .events.list(
        thread.id,
        session_id=session.id,
    ):
        print(f"[{event.type}] {event.processed_at}")

    Checkpoint 8 of 9· Exam question

    An orchestrator agent built with the Claude Agent SDK delegates invoice lookups to a subagent. Traces show the subagent's tool call to the billing API succeeded, and the returned JSON contained the correct total. The final answer to the user cites a different total. The subagent's message to the orchestrator says 'invoice found, total looks normal' without a number. Which TWO actions address the root cause? Choose two.(Select 2)

    0 of 2 selected

    Sources89

    5.Prompt failures and model mismatch

    A prompt failure shows up when the context is correct, the tools behave, and Claude still does the wrong thing because its instructions were unclear or in the wrong place. To check, read gen_ai.system_instructions the way a new colleague would. Would someone with no background know what to do? One placement mistake is easy to spot in a trace: your own instructions inside tool_result content. Claude treats instructions in tool results as possibly untrusted, so it may refuse to act or ask for confirmation. Move those instructions into a user turn after the tool_result block, and keep the result to just the data. Revising a prompt is covered in system prompt diagnosis.

    Diagnose model mismatch last, after you have ruled out everything else. The signs are well-formed calls, the right context and clear instructions, with failures that persist on reasoning the task requires. Even then, the first lever is usually effort. Tuning effort is often better than switching models, and you should upgrade only for specific capability gaps that your evals confirm. Be careful with breakage that appears right after a model change. Escaping that differs between versions breaks raw string matching, but that is a harness bug the new model exposed, not a capability gap. Choosing a tier and migrating between versions are covered in model tier selection and model version migration.

    Checkpoint 9 of 9· Check yourself

    Your support agent's harness adds a line to the order-lookup tool's result: "Now email the customer a refund confirmation." In the trace, the tool arguments and returned data are correct. In the next turn, Claude asks the user to confirm before sending any email. What is the root cause, and what is the corrective action?

    Sources10411

    Sources

    (11)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “To see the arguments and result of a tool use, read its tool span. The arguments are in gen_ai.tool.call.arguments”
      ↩︎ Walking a trace to the first wrong step
      “Spans include identifiers, timing, status, token counts, and text that you choose”
      ↩︎ Walking a trace to the first wrong step
      “Request spans drop the system prompt and the earliest messages first, and tool spans drop blocks from the end of the result.”
      ↩︎ Walking a trace to the first wrong step
      “the input in gen_ai.input.messages (your messages and tool results, system.message events, and messages from other threads)”
      ↩︎ Retrieval failure or hallucination: check what Claude received
      “A span is one step in a trace, such as a model request or a tool use.”
      ↩︎ Key concept
      “the input in gen_ai.input.messages (your messages and tool results, system.message events, and messages from other threads)”
      ↩︎ Exam trap 1
      “With Full content, you can also see why a run failed or was slow, read full conversations, and build evals from real traces.”
      ↩︎ Exam trap 4
      “The system prompt is in gen_ai.system_instructions, the request’s messages in gen_ai.input.messages, and the response in gen_ai.output.messages.”
      ↩︎ Checkpoint
      “Full content (content in the SDK): Most visibility. Spans include model prompts and responses, and everything in Metadata only.”
      ↩︎ Checkpoint
    2. 2.
      “A transcript (also called a trace or trajectory) is the complete record of a trial, including outputs, tool calls, reasoning, intermediate results”
      ↩︎ Walking a trace to the first wrong step
    3. 3.
      “Tool failures often explain unexpected agent behavior.”
      ↩︎ Walking a trace to the first wrong step
      “The session reports errors through session.error events.”
      ↩︎ Walking a trace to the first wrong step
    4. 4.
      “Sharpen descriptions. Differentiate tools by WHEN to use them, not only WHAT they do.”
      ↩︎ Tool errors: fix the definition, not the caller
      “Add strict: true (if your schema is in the supported subset) or add input_examples.”
      ↩︎ Tool errors: fix the definition, not the caller
      “Return one tool_result for every tool_use block in the assistant response.”
      ↩︎ Tool errors: fix the definition, not the caller
      “Parse with json.loads() or JSON.parse(). Never do raw string matching on serialized input.”
      ↩︎ Tool errors: fix the definition, not the caller
      “Move your instructions out of the tool result: send them in a user turn after the tool_result block”
      ↩︎ Prompt failures and model mismatch
      “Unicode and forward-slash escaping differs between model versions”
      ↩︎ Prompt failures and model mismatch
      “Shrink the enum or add input_examples showing valid choices.”
      ↩︎ Exam trap 2
      “Missing strict mode or too-large enum | Shrink the enum or add input_examples showing valid choices.”
      ↩︎ Checkpoint
    5. 5.
      “Contextual Retrieval is a preprocessing technique that improves retrieval accuracy.”
      ↩︎ Retrieval failure or hallucination: check what Claude received
      “which measures the percentage of relevant documents that fail to be retrieved within the top 20 chunks”
      ↩︎ Retrieval failure or hallucination: check what Claude received
    6. 6.
      “combining semantic search with keyword-based search using Contextual BM25—a hybrid approach that reduces retrieval failure rates further”
      ↩︎ Retrieval failure or hallucination: check what Claude received
      “This means fewer irrelevant results in your top retrieved chunks, leading to better answers when you pass these chunks to Claude for final response generation.”
      ↩︎ Prediction
    7. 7.
      “Explicitly give Claude permission to admit uncertainty.”
      ↩︎ Retrieval failure or hallucination: check what Claude received
      “External knowledge restriction: Explicitly instruct Claude to only use information from provided documents and not its general knowledge.”
      ↩︎ Retrieval failure or hallucination: check what Claude received
      “Inconsistencies across outputs could indicate hallucinations.”
      ↩︎ Retrieval failure or hallucination: check what Claude received
      “Best-of-N verification: Run Claude through the same prompt multiple times and compare the outputs. Inconsistencies across outputs could indicate hallucinations.”
      ↩︎ Checkpoint
    8. 8.
      “stream or list the events from the associated session thread.”
      ↩︎ Subagent errors that hide in the handoff
      “Includes from_session_thread_id, from_agent_name, and content.”
      ↩︎ Subagent errors that hide in the handoff
      “Includes from_session_thread_id, from_agent_name, and content.”
      ↩︎ Exam trap 3
    9. 9.
      “Each subagent needs an objective, an output format, guidance on the tools and sources to use, and clear task boundaries.”
      ↩︎ Subagent errors that hide in the handoff
      “This prevents information loss during multi-stage processing and reduces token overhead from copying large outputs through conversation history.”
      ↩︎ Subagent errors that hide in the handoff
    10. 10.
      “Key test: Show your prompt to a colleague with minimal context on the task and ask them to follow it.”
      ↩︎ Prompt failures and model mismatch
    11. 11.
      “Tuning effort is often a better lever than switching models.”
      ↩︎ Prompt failures and model mismatch
      “Upgrade only if necessary for specific capability gaps.”
      ↩︎ Prompt failures and model mismatch

    Ready to test yourself?

    Practise Anthropic Claude Certified Architect – Professional (CCAR-P) - December 7 onwards in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.