CertSafari
    Anthropic Claude Certified Developer – Professional (CCDV-P)· Lessons

    Domain 4 · Lesson 12/25

    Diagnosing complex failures in Claude agents

    Complex Failure Diagnosis

    18 min read
    5% of exam
    10 sources
    Published 19 Nov 2026
    Docs as of 18 Nov 2026

    What you will be able to do

    • Use transcripts and repeated trials to find whether a production failure came from model output, the agent harness, a tool, or the data
    • Use stop_reason to tell a refusal, a truncation, and a harness-caused empty reply apart
    • Recognise refusals as their own failure class, including refusals caused by your own prompt wording
    • Find where facts get lost between subagents and an orchestrator, and fix it at that point
    • Trace a wrong answer back to retrieval rather than generation when the model never received the needed text

    Key concept

    Failure origin — The layer where a failure begins: the model's output, the harness that builds requests and handles responses, a tool, or the data the system retrieves. A fix only works if it is applied at that layer, because a symptom that appears in the final answer may have started several steps earlier.

    1.Four places a failure can start

    A wrong agent answer is a symptom. The cause can sit in any of four layers. Model output covers what Claude generated. The agent harness is the code that processes inputs, orchestrates tool calls, and returns results. Tools are what the harness runs on the model's behalf. Data is what the system retrieves and puts into context. The evaluation of an agent always tests the harness and the model together, so the final answer cannot tell you which of the four broke.

    The evidence is the transcript: the full record of a run, with outputs, tool calls, reasoning, and intermediate results. In a Managed Agents session, the Console session viewer shows that record grouped by model request, with thinking, tool inputs, and tool results side by side. A separate Tools tab lists each tool's call count and failures. Start there, because tool failures often explain behaviour that looks like the model's fault. Walk the transcript forward from the user's request and find the first step where the record diverges from what should have happened. The layer that produced that step is the origin. Tooling for searching traces across many sessions belongs to trace analysis at scale.

    Non-deterministic failures need one more step: run the same input several times. One failing transcript may just be the unlucky sample. Running the same prompt several times and comparing the outputs shows whether the failure is consistent or intermittent. Inconsistent answers across runs can point to hallucination. Check the outcome as well as the text. An agent that says a booking was made has not booked anything unless the reservation exists in the environment.

    Checkpoint 1 of 9· Check yourself

    A support agent tells a customer that their refund has been issued, but no refund exists. In the session transcript, Claude called issue_refund with the correct order ID and amount. The tool result for that call is an error saying the payment service timed out. Claude's next message tells the customer that the refund went through. The Tools tab shows failures for issue_refund. Where should the investigation put the failure origin first?

    Sources123

    2.Reading stop_reason before reading the text

    Every successful Messages API response carries a stop_reason, and it is the first field to check when output looks wrong. It is separate from errors. An error means the request failed. A stop_reason says why a completed generation ended. Several failures that look alike in the final text show up as different values here.

    Stop reasons that indicate a failure, and what each one points to
    stop_reasonWhat happenedDiagnostic meaning
    max_tokensThe response reached your max_tokens limitOutput is truncated; a trailing tool_use block may be incomplete
    refusalClaude declined to respondA classifier decision; read stop_details
    model_context_window_exceededThe response filled the model's context windowTreat the response as truncated
    pause_turnA server-tool loop reached its iteration limitNot a failure; send the assistant content back to continue
    end_turnClaude completed its response naturallyNormal, unless content is empty, which points at the harness

    An empty reply with end_turn usually comes from the harness. The most common cause is adding text blocks immediately after tool results. The history then shows user text after every tool result, so Claude learns to end its turn at that point. The other cause is sending Claude's completed response back with nothing new added. The fix goes in the harness: send only the tool_result blocks, and if a turn really needs continuing, add a new user message instead of resending the empty one.

    With max_tokens, the problem is a parameter you chose, not a model mistake. If the truncated response ends in an incomplete tool use block, the tool input is malformed because it was cut off. Retry with a higher max_tokens to get the complete call. Parsing the partial input will not recover it.

    Checkpoint 2 of 9· Match them up

    You are triaging agent responses that all came back with HTTP 200 but looked wrong. Match each observed stop_reason to what it tells you about where to look next.

    Tap a term, then the definition that fits it.

    Checkpoint 3 of 9· Exam question

    Orchid Retail runs a customer-support agent in production that wraps the Claude API in its own orchestration harness. Over the past week, about 2% of conversations ended with the agent returning no useful text to the shopper. Engineers captured several affected requests and noticed that every one of them came back from the API as a normal completed response with a successful HTTP status, and that none hit the output token cap. Tool calls in these sessions all succeeded. The team's habit has been to replay an affected conversation and ask the model why it said nothing, but the answers vary between runs and nobody trusts them. They now need to decide whether the model declined or the harness lost content. Which statement correctly describes how to separate a model refusal from a harness-induced empty response when an agent returns no useful text?

    Sources4

    3.Refusals as a failure class

    A refusal is a normal response, not an error. Safety classifiers on recent Claude models can decline a request. When they do, content is empty, stop_reason is "refusal", and stop_details names the policy area. That puts refusals outside most monitoring: a dashboard built on error rates or 5xx counts never sees them. Emit a dedicated event for each refusal, and branch on stop_reason rather than on content or the inner stop_details fields, because category and explanation can be null.

    A refusal: HTTP 200, empty content, and stop_details naming the categoryjson
    {
      "id": "msg_01XFUDYJgAACzvnptvVoYEL",
      "type": "message",
      "role": "assistant",
      "model": "claude-fable-5",
      "content": [],
      "stop_reason": "refusal",
      "stop_details": {
        "type": "refusal",
        "category": "cyber",
        "explanation": "This request was declined because it could enable cyber harm."
      },
      "usage": {
        "input_tokens": 412,
        "output_tokens": 0
      }
    }

    A refusal does not always mean the user asked for something risky. The reasoning_extraction category fires on any request that asks the model to reproduce its internal reasoning in the response text, whether verbatim or in a fixed format. It usually comes from your own prompt: a <thinking> or scratchpad section the model must fill in, a reasoning or trace field in JSON output or a tool input, or a request for the model's private notes or a running log of its reasoning. Any other wording that asks for the same thing can trigger it too. That wording can sit in a system prompt, a skill, or a tool description, so the origin is the harness configuration, not the end user. This category has no recommended fallback model. Change the prompt rather than retrying. If you want to see the reasoning, read summarized thinking blocks.

    For the other categories, the remedy is to retry on a different model. The fallbacks request setting automates that: set to "default" (in beta), it makes the API retry a declined request on the fallback model Anthropic recommends for its refusal category. Re-sending to the same model usually earns another refusal. Refusals can also arrive mid-stream after partial output, and any partial output must be discarded. In multi-agent systems, check each path separately. The fallbacks parameter does not carry over to model calls made from inside tool execution, so a subagent with no fallback of its own can keep failing while the top-level agent looks protected. Choosing between rollback, fallback models, and kill switches during an incident belongs to incident mitigation.

    Checkpoint 4 of 9· Check yourself

    After a release, a document-summarizing subagent starts returning HTTP 200 responses with empty content. They have stop_reason "refusal" and stop_details.category "reasoning_extraction". The release added a required reasoning field to the input schema of the subagent's submit_summary tool. The user inputs have not changed. Which fix addresses where the failure starts?

    Sources5

    4.Separating tool and schema faults from model behaviour

    Many tool failures look like the model chose badly when the real cause is how the tools were described to it. If Claude calls tool A when you meant tool B, the usual cause is ambiguous descriptions. Differentiate the tools by when to use each one, not only by what it does, then check the next set of traces to confirm the fix. If inputs include parameters that are not in your schema, the cause is missing strict mode. Strict mode (strict: true on a tool) guarantees schema validation on tool names and inputs, so Claude's tool calls conform to the declared schema instead of the model over-generating extra parameters. It works for schemas inside the supported subset. In both cases the harness supplied the definitions, and the fix belongs there.

    Tool-use symptoms and the layer they originate in
    SymptomLikely causeOrigin layer
    Claude calls tool A when you wanted tool BDescription ambiguityHarness (tool definitions)
    Parameter that doesn't exist in your schemaModel over-generation without strict modeHarness (schema not strict)
    Claude refuses to act on a tool resultYour own instructions are being delivered inside the tool_result contentHarness (message construction)
    String comparison on tool inputs fails with newer modelsUnicode and forward-slash escaping differs between model versionsHarness code exposed by a model version change
    Every request is a cache misstool_choice, the thinking configuration, or output_config.effort varying between requestsHarness (request construction)

    Two rows deserve a closer look. Claude treats instructions inside tool results as potentially untrusted third-party content. If your harness puts its own directions into a tool result, Claude may hesitate or ask for confirmation, which looks like reluctance from the model. Keep the tool result to data only, and put your instructions in a user turn. The escaping row shows a model version change exposing a latent harness bug: code that string-matches serialized tool input breaks when a newer model escapes characters differently. Parse the JSON instead.

    Tool-call loops are a multi-turn failure in which each step looks reasonable on its own, and the problem only shows across the whole run. The Agent SDK caps a run with maxTurns. When the cap is reached, the run ends with the result subtype error_max_turns. If that subtype keeps appearing on a task that should take a few steps, read the transcript for the repeated call and its result. Usually a tool error or ambiguous result is being re-requested instead of handled.

    Checkpoint 5 of 9· Check yourself

    Your harness returns CRM search results from a lookup_accounts tool. To save a turn, it appends this line to the end of the tool_result content: "Next, update each of these accounts' renewal dates." In many runs, Claude stops and asks the user to confirm before it makes the updates. The team blames model reluctance. Which change fixes this at the layer where it starts?

    Sources678

    5.Information loss between subagents and the orchestrator

    In an orchestrator–worker system, every handoff is a point where a fact can be lost. A subagent reads tool results, then writes a free-text reply that the orchestrator synthesizes. If the subagent leaves a detail out of that reply, the orchestrator never sees it, whatever model it runs and however carefully it reads.

    The fix belongs at the handoff. Define a findings schema that worker replies must follow, with the fields the report needs and a source quote for each one. Have the orchestrator reject any reply that fails validation before it writes the synthesis. Structured outputs constrain a response to a schema, with guaranteed field types and required fields, so a missing field becomes a validation failure instead of a silent gap. A related pattern lets subagents write their work to external storage and pass lightweight references back, so large results are not re-copied through each conversation. A system-prompt rule telling the orchestrator to include dates does nothing when no date arrives.

    Checkpoint 6 of 9· Check yourself

    Your team will add a required findings schema at the worker-to-orchestrator handoff. Which design lets the orchestrator check, before it writes the synthesis, that each worker finding is backed by what the worker's tools actually returned?

    Checkpoint 7 of 9· Exam question

    Vireo Freight runs a production research assistant that helps its procurement team review carrier agreements. An orchestrator fans out to three subagents, one per document batch, and synthesizes their replies into a single report for each review. Users report reports missing a contract end date, which the team needs for renewal planning. The orchestrator's own prompt already tells it to include dates, and the synthesis is never truncated. Engineers pulled worker traces and found the end date in each subagent's tool results, but it is absent from that subagent's final message back to the orchestrator. Free-form summaries vary in what they keep from run to run. What is the best fix?

    Sources79

    6.Retrieval-grounding failures

    A grounded answer can be wrong because the model never received the right text. Retrieval systems split documents into chunks, and that causes problems when an individual chunk lacks the context it needs. A chunk that says revenue grew 3% but does not name the company or the quarter is hard to retrieve correctly and hard to use. A spec table split across two chunks fails the same way: the half that matches the query carries the rows, and the other half carries the column headers.

    To localise the failure, compare the retrieved context with the answer. If the needed fact is missing from what was retrieved, the origin is data or retrieval, and the fix belongs in the pipeline that builds context. Either keep the context with the chunk when you index it, or hand the model the surrounding text along with the match. Contextual Retrieval does the first by prepending chunk-specific explanatory context to each chunk before embedding. Raising temperature, adding a larger model, or telling the model to guess missing headers all act on generation. None of them can restore text that was never retrieved.

    If the fact was retrieved and the answer still contradicts it, the origin moves to model output, and grounding techniques apply. For long documents, ask Claude to extract word-for-word quotes before doing its task. Require a supporting quote for every claim, and retract any claim without one. Tell Claude to use only the provided documents, not its general knowledge. Give it explicit permission to say it lacks enough information.

    Checkpoint 8 of 9· Check yourself

    Trace review of a RAG assistant shows a wrong answer: it says a part's maximum operating temperature is 105°C. The retrieved context for that request includes the whole spec table, with column headers, and the table lists 85°C. Which fix acts at the layer where this failure starts?

    Checkpoint 9 of 9· Exam question

    Nettle Pharma runs a production assistant that answers pharmacists' questions about product specifications by retrieving catalog chunks and passing them to a generation model. Users report wrong answers about product specs, such as storage ranges attributed to the wrong column. Trace review shows that spec tables are split across two chunks, and retrieval returns only the half without the column headers. The team already tried telling the model to infer unlabeled columns, raising the temperature, and moving to a larger model. Answers stayed wrong, and the wrong ones varied more from run to run or sounded more fluent. The team controls both the nightly indexing pipeline and the query-time retrieval step. Which TWO changes address this origin? Select TWO.(Select 2)

    0 of 2 selected

    Sources102

    Sources

    (10)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “Tool failures often explain unexpected agent behavior.”
      ↩︎ Four places a failure can start
      “The conversation grouped by model request, including thinking, tool calls with their inputs and results, and message text as it streams.”
      ↩︎ Four places a failure can start
      “Review tool results: Tool failures often explain unexpected agent behavior. The Inspector’s Tools tab shows each tool’s failures.”
      ↩︎ Checkpoint
    2. 2.
      “Best-of-N verification: Run Claude through the same prompt multiple times and compare the outputs.”
      ↩︎ Four places a failure can start
      “ask Claude to extract word-for-word quotes first before performing its task.”
      ↩︎ Retrieval-grounding failures
      “External knowledge restriction: Explicitly instruct Claude to only use information from provided documents and not its general knowledge.”
      ↩︎ Retrieval-grounding failures
      “Make Claude’s response auditable by having it cite quotes and sources for each of its claims.”
      ↩︎ Checkpoint
      “Claude verify each claim by finding a supporting quote after it generates a response. If it can’t find a quote, it must retract the claim.”
      ↩︎ Checkpoint
    3. 3.
      “The outcome is the final state in the environment at the end of the trial.”
      ↩︎ Four places a failure can start
      “Because model outputs vary between runs, we run multiple trials to produce more consistent results.”
      ↩︎ Key concept
    4. 4.
      “Unlike errors, which indicate failures in processing your request, stop_reason tells you why Claude completed its response generation.”
      ↩︎ Reading stop_reason before reading the text
      “Adding text blocks immediately after tool results”
      ↩︎ Reading stop_reason before reading the text
      “the truncated response contains an incomplete tool use block”
      ↩︎ Reading stop_reason before reading the text
      “pause_turn | A server-tool loop reached its iteration limit. | Send the assistant content back to continue.”
      ↩︎ Checkpoint
    5. 5.
      “The wording can sit in a system prompt, a skill, or a tool description, so check those too.”
      ↩︎ Refusals as a failure class
      “The request asks the model to reproduce its internal reasoning in the response text.”
      ↩︎ Refusals as a failure class
      “the API retries a declined request on the fallback model Anthropic recommends for its refusal category”
      ↩︎ Refusals as a failure class
      “Re-sending a refused request to the same model usually earns another refusal.”
      ↩︎ Refusals as a failure class
      “A refusal can arrive before any output, or mid-stream after partial output.”
      ↩︎ Refusals as a failure class
      “A refusal is an HTTP 200, so monitoring built on error rates or 5xx responses never sees it.”
      ↩︎ Exam trap 1
      “This category has no recommended fallback model, so change the prompt rather than retry the request.”
      ↩︎ Exam trap 2
      “The fallbacks parameter does not carry over to model calls made from inside tool execution.”
      ↩︎ Exam trap 3
      “A refusal is a successful HTTP 200 response with stop_reason”
      ↩︎ Prediction
      “This category has no recommended fallback model, so change the prompt rather than retry the request.”
      ↩︎ Checkpoint
    6. 6.
      “Differentiate tools by WHEN to use them, not only WHAT they do.”
      ↩︎ Separating tool and schema faults from model behaviour
      “Claude is trained to treat instructions inside tool results as potentially untrusted third-party content.”
      ↩︎ Separating tool and schema faults from model behaviour
      “Move your instructions out of the tool result: send them in a user turn after the tool_result block”
      ↩︎ Checkpoint
    7. 7.
      “Strict tool use (strict: true): Guarantee schema validation on tool names and inputs.”
      ↩︎ Separating tool and schema faults from model behaviour
      “Type safe: Guaranteed field types and required fields”
      ↩︎ Information loss between subagents and the orchestrator
    8. 9.
      “Subagents call tools to store their work in external systems, then pass lightweight references back to the coordinator.”
      ↩︎ Information loss between subagents and the orchestrator
      “This prevents information loss during multi-stage processing and reduces token overhead from copying large outputs through conversation history.”
      ↩︎ Prediction
    9. 10.
      “prepending chunk-specific explanatory context to each chunk before embedding”
      ↩︎ Retrieval-grounding failures
      “it can lead to problems when individual chunks lack sufficient context.”
      ↩︎ Exam trap 4

    Ready to test yourself?

    Practise Anthropic Claude Certified Developer – Professional (CCDV-P) in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.