What you will be able to do
- Use transcripts and repeated trials to find whether a production failure came from model output, the agent harness, a tool, or the data
- Use stop_reason to tell a refusal, a truncation, and a harness-caused empty reply apart
- Recognise refusals as their own failure class, including refusals caused by your own prompt wording
- Find where facts get lost between subagents and an orchestrator, and fix it at that point
- Trace a wrong answer back to retrieval rather than generation when the model never received the needed text
Key concept
Failure origin — The layer where a failure begins: the model's output, the harness that builds requests and handles responses, a tool, or the data the system retrieves. A fix only works if it is applied at that layer, because a symptom that appears in the final answer may have started several steps earlier.
1.Four places a failure can start
A wrong agent answer is a symptom. The cause can sit in any of four layers. Model output covers what Claude generated. The agent harness is the code that processes inputs, orchestrates tool calls, and returns results. Tools are what the harness runs on the model's behalf. Data is what the system retrieves and puts into context. The evaluation of an agent always tests the harness and the model together, so the final answer cannot tell you which of the four broke.
The evidence is the transcript: the full record of a run, with outputs, tool calls, reasoning, and intermediate results. In a Managed Agents session, the Console session viewer shows that record grouped by model request, with thinking, tool inputs, and tool results side by side. A separate Tools tab lists each tool's call count and failures. Start there, because tool failures often explain behaviour that looks like the model's fault. Walk the transcript forward from the user's request and find the first step where the record diverges from what should have happened. The layer that produced that step is the origin. Tooling for searching traces across many sessions belongs to trace analysis at scale.
Non-deterministic failures need one more step: run the same input several times. One failing transcript may just be the unlucky sample. Running the same prompt several times and comparing the outputs shows whether the failure is consistent or intermittent. Inconsistent answers across runs can point to hallucination. Check the outcome as well as the text. An agent that says a booking was made has not booked anything unless the reservation exists in the environment.
Checkpoint 1 of 9· Check yourself
A support agent tells a customer that their refund has been issued, but no refund exists. In the session transcript, Claude called issue_refund with the correct order ID and amount. The tool result for that call is an error saying the payment service timed out. Claude's next message tells the customer that the refund went through. The Tools tab shows failures for issue_refund. Where should the investigation put the failure origin first?
Walk the transcript forward and find the first step that diverges. Claude's request was correct, and the tool call then failed, so the origin is the tool. Claude's false confirmation is a later problem that follows from it. Fixing only the reply wording would treat the symptom.
“Review tool results: Tool failures often explain unexpected agent behavior. The Inspector’s Tools tab shows each tool’s failures.”Source: platform.claude.com
2.Reading stop_reason before reading the text
Every successful Messages API response carries a stop_reason, and it is the first field to check when output looks wrong. It is separate from errors. An error means the request failed. A stop_reason says why a completed generation ended. Several failures that look alike in the final text show up as different values here.
| stop_reason | What happened | Diagnostic meaning |
|---|---|---|
| max_tokens | The response reached your max_tokens limit | Output is truncated; a trailing tool_use block may be incomplete |
| refusal | Claude declined to respond | A classifier decision; read stop_details |
| model_context_window_exceeded | The response filled the model's context window | Treat the response as truncated |
| pause_turn | A server-tool loop reached its iteration limit | Not a failure; send the assistant content back to continue |
| end_turn | Claude completed its response naturally | Normal, unless content is empty, which points at the harness |
An empty reply with end_turn usually comes from the harness. The most common cause is adding text blocks immediately after tool results. The history then shows user text after every tool result, so Claude learns to end its turn at that point. The other cause is sending Claude's completed response back with nothing new added. The fix goes in the harness: send only the tool_result blocks, and if a turn really needs continuing, add a new user message instead of resending the empty one.
With max_tokens, the problem is a parameter you chose, not a model mistake. If the truncated response ends in an incomplete tool use block, the tool input is malformed because it was cut off. Retry with a higher max_tokens to get the complete call. Parsing the partial input will not recover it.
Checkpoint 2 of 9· Match them up
You are triaging agent responses that all came back with HTTP 200 but looked wrong. Match each observed stop_reason to what it tells you about where to look next.
Tap a term, then the definition that fits it.
Each stop_reason points to a different cause, even when the final text looks the same. max_tokens is a parameter you chose. refusal is a classifier decision. pause_turn only needs the loop continued. An empty end_turn points at the harness. model_context_window_exceeded means the context window filled and the output was cut off.
“pause_turn | A server-tool loop reached its iteration limit. | Send the assistant content back to continue.”Source: platform.claude.com
Checkpoint 3 of 9· Exam question
Orchid Retail runs a customer-support agent in production that wraps the Claude API in its own orchestration harness. Over the past week, about 2% of conversations ended with the agent returning no useful text to the shopper. Engineers captured several affected requests and noticed that every one of them came back from the API as a normal completed response with a successful HTTP status, and that none hit the output token cap. Tool calls in these sessions all succeeded. The team's habit has been to replay an affected conversation and ask the model why it said nothing, but the answers vary between runs and nobody trusts them. They now need to decide whether the model declined or the harness lost content. Which statement correctly describes how to separate a model refusal from a harness-induced empty response when an agent returns no useful text?
Correct answer: D — A refusal is reported by the API as its own stop condition in a completed response, whereas an empty reply from the harness lacks that signal and points to dropped content.
- A. Incorrect: the response metadata does distinguish them, and asking the model after the fact is unreliable evidence.
- B. Incorrect: reaching the output limit is a different stop condition from refusing, and harness-side empties have many causes besides tool failures.
- C. Incorrect: a refusal is delivered in a normal successful response, not as an HTTP error status.
- D. Correct: refusals are surfaced in a successful response through a distinct stop reason, so checking it, together with the raw content, separates model declines from harness loss.
Sources4
3.Refusals as a failure class
A refusal is a normal response, not an error. Safety classifiers on recent Claude models can decline a request. When they do, content is empty, stop_reason is "refusal", and stop_details names the policy area. That puts refusals outside most monitoring: a dashboard built on error rates or 5xx counts never sees them. Emit a dedicated event for each refusal, and branch on stop_reason rather than on content or the inner stop_details fields, because category and explanation can be null.
{
"id": "msg_01XFUDYJgAACzvnptvVoYEL",
"type": "message",
"role": "assistant",
"model": "claude-fable-5",
"content": [],
"stop_reason": "refusal",
"stop_details": {
"type": "refusal",
"category": "cyber",
"explanation": "This request was declined because it could enable cyber harm."
},
"usage": {
"input_tokens": 412,
"output_tokens": 0
}
}A refusal does not always mean the user asked for something risky. The reasoning_extraction category fires on any request that asks the model to reproduce its internal reasoning in the response text, whether verbatim or in a fixed format. It usually comes from your own prompt: a <thinking> or scratchpad section the model must fill in, a reasoning or trace field in JSON output or a tool input, or a request for the model's private notes or a running log of its reasoning. Any other wording that asks for the same thing can trigger it too. That wording can sit in a system prompt, a skill, or a tool description, so the origin is the harness configuration, not the end user. This category has no recommended fallback model. Change the prompt rather than retrying. If you want to see the reasoning, read summarized thinking blocks.
For the other categories, the remedy is to retry on a different model. The fallbacks request setting automates that: set to "default" (in beta), it makes the API retry a declined request on the fallback model Anthropic recommends for its refusal category. Re-sending to the same model usually earns another refusal. Refusals can also arrive mid-stream after partial output, and any partial output must be discarded. In multi-agent systems, check each path separately. The fallbacks parameter does not carry over to model calls made from inside tool execution, so a subagent with no fallback of its own can keep failing while the top-level agent looks protected. Choosing between rollback, fallback models, and kill switches during an incident belongs to incident mitigation.
Checkpoint 4 of 9· Check yourself
After a release, a document-summarizing subagent starts returning HTTP 200 responses with empty content. They have stop_reason "refusal" and stop_details.category "reasoning_extraction". The release added a required reasoning field to the input schema of the subagent's submit_summary tool. The user inputs have not changed. Which fix addresses where the failure starts?
A reasoning_extraction refusal usually comes from your own prompt or tool definition. Here, a reasoning field in a tool input asks the model to put its reasoning in the output, so the origin is the harness configuration. This category has no recommended fallback model, so adding a fallback or retrying won't fix it. Changing the prompt will.
“This category has no recommended fallback model, so change the prompt rather than retry the request.”Source: platform.claude.com
Sources5
4.Separating tool and schema faults from model behaviour
Many tool failures look like the model chose badly when the real cause is how the tools were described to it. If Claude calls tool A when you meant tool B, the usual cause is ambiguous descriptions. Differentiate the tools by when to use each one, not only by what it does, then check the next set of traces to confirm the fix. If inputs include parameters that are not in your schema, the cause is missing strict mode. Strict mode (strict: true on a tool) guarantees schema validation on tool names and inputs, so Claude's tool calls conform to the declared schema instead of the model over-generating extra parameters. It works for schemas inside the supported subset. In both cases the harness supplied the definitions, and the fix belongs there.
| Symptom | Likely cause | Origin layer |
|---|---|---|
| Claude calls tool A when you wanted tool B | Description ambiguity | Harness (tool definitions) |
| Parameter that doesn't exist in your schema | Model over-generation without strict mode | Harness (schema not strict) |
| Claude refuses to act on a tool result | Your own instructions are being delivered inside the tool_result content | Harness (message construction) |
| String comparison on tool inputs fails with newer models | Unicode and forward-slash escaping differs between model versions | Harness code exposed by a model version change |
| Every request is a cache miss | tool_choice, the thinking configuration, or output_config.effort varying between requests | Harness (request construction) |
Two rows deserve a closer look. Claude treats instructions inside tool results as potentially untrusted third-party content. If your harness puts its own directions into a tool result, Claude may hesitate or ask for confirmation, which looks like reluctance from the model. Keep the tool result to data only, and put your instructions in a user turn. The escaping row shows a model version change exposing a latent harness bug: code that string-matches serialized tool input breaks when a newer model escapes characters differently. Parse the JSON instead.
Tool-call loops are a multi-turn failure in which each step looks reasonable on its own, and the problem only shows across the whole run. The Agent SDK caps a run with maxTurns. When the cap is reached, the run ends with the result subtype error_max_turns. If that subtype keeps appearing on a task that should take a few steps, read the transcript for the repeated call and its result. Usually a tool error or ambiguous result is being re-requested instead of handled.
Checkpoint 5 of 9· Check yourself
Your harness returns CRM search results from a lookup_accounts tool. To save a turn, it appends this line to the end of the tool_result content: "Next, update each of these accounts' renewal dates." In many runs, Claude stops and asks the user to confirm before it makes the updates. The team blames model reluctance. Which change fixes this at the layer where it starts?
Claude treats instructions inside tool results as possibly untrusted third-party content, so the hesitation comes from how the harness builds its messages, not from the model. Moving the instruction into a user turn and leaving only data in the tool result fixes it at the origin. Strict mode deals with parameters that aren't in the schema, and clearer descriptions deal with picking the wrong tool. Neither applies here.
“Move your instructions out of the tool result: send them in a user turn after the tool_result block”Source: platform.claude.com
5.Information loss between subagents and the orchestrator
In an orchestrator–worker system, every handoff is a point where a fact can be lost. A subagent reads tool results, then writes a free-text reply that the orchestrator synthesizes. If the subagent leaves a detail out of that reply, the orchestrator never sees it, whatever model it runs and however carefully it reads.
The fix belongs at the handoff. Define a findings schema that worker replies must follow, with the fields the report needs and a source quote for each one. Have the orchestrator reject any reply that fails validation before it writes the synthesis. Structured outputs constrain a response to a schema, with guaranteed field types and required fields, so a missing field becomes a validation failure instead of a silent gap. A related pattern lets subagents write their work to external storage and pass lightweight references back, so large results are not re-copied through each conversation. A system-prompt rule telling the orchestrator to include dates does nothing when no date arrives.
Checkpoint 6 of 9· Check yourself
Your team will add a required findings schema at the worker-to-orchestrator handoff. Which design lets the orchestrator check, before it writes the synthesis, that each worker finding is backed by what the worker's tools actually returned?
Pairing each required field with a source quote makes every finding auditable against the tool results, and a missing field fails validation instead of leaving a silent gap. A single free-text summary can still drop details without failing. A storage reference avoids re-copying large results but does not check them. A required reasoning field is the kind of wording that triggers reasoning_extraction refusals.
“Make Claude’s response auditable by having it cite quotes and sources for each of its claims.”Source: platform.claude.com
Checkpoint 7 of 9· Exam question
Vireo Freight runs a production research assistant that helps its procurement team review carrier agreements. An orchestrator fans out to three subagents, one per document batch, and synthesizes their replies into a single report for each review. Users report reports missing a contract end date, which the team needs for renewal planning. The orchestrator's own prompt already tells it to include dates, and the synthesis is never truncated. Engineers pulled worker traces and found the end date in each subagent's tool results, but it is absent from that subagent's final message back to the orchestrator. Free-form summaries vary in what they keep from run to run. What is the best fix?
Correct answer: B — Define a required findings schema with source quotes for each worker reply, and have the orchestrator reject replies that fail validation before it writes the synthesis.
- A. Incorrect: a larger model with extended thinking improves how the orchestrator reasons over what it receives, but the worker replies no longer contain the date. Re-reading them cannot surface a fact that was dropped before the handoff.
- B. Correct: the date is present in each worker's tool results and lost when the worker writes its final message. A required findings schema with source quotes, checked by the orchestrator before synthesis, fixes the loss at the handoff.
- C. Incorrect: a higher output limit and a longer synthesis only help when the orchestrator holds details it cannot fit. The date never reached it, and nothing suggests the synthesis was cut off, so a longer output gains nothing.
- D. Incorrect: the rule tells the orchestrator to copy dates from worker replies, but those replies omit the date. A prompt instruction cannot supply a fact that is absent from its inputs, so the reports stay incomplete.
6.Retrieval-grounding failures
A grounded answer can be wrong because the model never received the right text. Retrieval systems split documents into chunks, and that causes problems when an individual chunk lacks the context it needs. A chunk that says revenue grew 3% but does not name the company or the quarter is hard to retrieve correctly and hard to use. A spec table split across two chunks fails the same way: the half that matches the query carries the rows, and the other half carries the column headers.
To localise the failure, compare the retrieved context with the answer. If the needed fact is missing from what was retrieved, the origin is data or retrieval, and the fix belongs in the pipeline that builds context. Either keep the context with the chunk when you index it, or hand the model the surrounding text along with the match. Contextual Retrieval does the first by prepending chunk-specific explanatory context to each chunk before embedding. Raising temperature, adding a larger model, or telling the model to guess missing headers all act on generation. None of them can restore text that was never retrieved.
If the fact was retrieved and the answer still contradicts it, the origin moves to model output, and grounding techniques apply. For long documents, ask Claude to extract word-for-word quotes before doing its task. Require a supporting quote for every claim, and retract any claim without one. Tell Claude to use only the provided documents, not its general knowledge. Give it explicit permission to say it lacks enough information.
Data. The model is faithfully using what retrieval gave it. Re-index the updated document and remove the stale chunks. No prompt change can make the model cite text it never receives.
Checkpoint 8 of 9· Check yourself
Trace review of a RAG assistant shows a wrong answer: it says a part's maximum operating temperature is 105°C. The retrieved context for that request includes the whole spec table, with column headers, and the table lists 85°C. Which fix acts at the layer where this failure starts?
The correct fact, 85°C, was in the retrieved context, so retrieval did its job. The answer contradicts that context, so the failure starts in the model output, and grounding techniques such as required supporting quotes are the fix. The other three options change the retrieval pipeline, which would help only if the fact had been missing from what was retrieved.
“Claude verify each claim by finding a supporting quote after it generates a response. If it can’t find a quote, it must retract the claim.”Source: platform.claude.com
Checkpoint 9 of 9· Exam question
Nettle Pharma runs a production assistant that answers pharmacists' questions about product specifications by retrieving catalog chunks and passing them to a generation model. Users report wrong answers about product specs, such as storage ranges attributed to the wrong column. Trace review shows that spec tables are split across two chunks, and retrieval returns only the half without the column headers. The team already tried telling the model to infer unlabeled columns, raising the temperature, and moving to a larger model. Answers stayed wrong, and the wrong ones varied more from run to run or sounded more fluent. The team controls both the nightly indexing pipeline and the query-time retrieval step. Which TWO changes address this origin? Select TWO.(Select 2)
Correct answers: A, B — Retrieve neighboring chunks or the parent section for each hit, so the model sees the table headers that sit beside the matched rows.; Chunk on document structure so each table stays whole with its headers, or prepend table and section context to every chunk before embedding.
- A. Correct: expanding to the parent section or adjacent chunks at query time restores the headers that the matched chunk lacks.
- B. Correct: structure-aware chunking or contextual prefixes preserves header meaning inside each retrieved chunk, fixing the grounding failure at the index side.
- C. Incorrect: telling the model to guess headers is asking for fabricated column meanings, not grounded answers.
- D. Incorrect: temperature adds randomness; it does not supply missing headers, and wrong guesses become less consistent rather than less wrong.
- E. Incorrect: a larger model cannot reliably reconstruct labels that are absent from context; it will guess more fluently, which can be worse.
Sources
(10)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Tool failures often explain unexpected agent behavior.”
↩︎ Four places a failure can start“The conversation grouped by model request, including thinking, tool calls with their inputs and results, and message text as it streams.”
↩︎ Four places a failure can start“Review tool results: Tool failures often explain unexpected agent behavior. The Inspector’s Tools tab shows each tool’s failures.”
↩︎ Checkpoint - 2.https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinationsOfficial docs
“Best-of-N verification: Run Claude through the same prompt multiple times and compare the outputs.”
↩︎ Four places a failure can start“ask Claude to extract word-for-word quotes first before performing its task.”
↩︎ Retrieval-grounding failures“External knowledge restriction: Explicitly instruct Claude to only use information from provided documents and not its general knowledge.”
↩︎ Retrieval-grounding failures“Make Claude’s response auditable by having it cite quotes and sources for each of its claims.”
↩︎ Checkpoint“Claude verify each claim by finding a supporting quote after it generates a response. If it can’t find a quote, it must retract the claim.”
↩︎ Checkpoint - 3.
“The outcome is the final state in the environment at the end of the trial.”
↩︎ Four places a failure can start“Because model outputs vary between runs, we run multiple trials to produce more consistent results.”
↩︎ Key concept - 4.
“Unlike errors, which indicate failures in processing your request, stop_reason tells you why Claude completed its response generation.”
↩︎ Reading stop_reason before reading the text“Adding text blocks immediately after tool results”
↩︎ Reading stop_reason before reading the text“the truncated response contains an incomplete tool use block”
↩︎ Reading stop_reason before reading the text“pause_turn | A server-tool loop reached its iteration limit. | Send the assistant content back to continue.”
↩︎ Checkpoint - 5.
“The wording can sit in a system prompt, a skill, or a tool description, so check those too.”
↩︎ Refusals as a failure class“The request asks the model to reproduce its internal reasoning in the response text.”
↩︎ Refusals as a failure class“the API retries a declined request on the fallback model Anthropic recommends for its refusal category”
↩︎ Refusals as a failure class“Re-sending a refused request to the same model usually earns another refusal.”
↩︎ Refusals as a failure class“A refusal can arrive before any output, or mid-stream after partial output.”
↩︎ Refusals as a failure class“A refusal is an HTTP 200, so monitoring built on error rates or 5xx responses never sees it.”
↩︎ Exam trap 1“This category has no recommended fallback model, so change the prompt rather than retry the request.”
↩︎ Exam trap 2“The fallbacks parameter does not carry over to model calls made from inside tool execution.”
↩︎ Exam trap 3“A refusal is a successful HTTP 200 response with stop_reason”
↩︎ Prediction“This category has no recommended fallback model, so change the prompt rather than retry the request.”
↩︎ Checkpoint - 6.
“Differentiate tools by WHEN to use them, not only WHAT they do.”
↩︎ Separating tool and schema faults from model behaviour“Claude is trained to treat instructions inside tool results as potentially untrusted third-party content.”
↩︎ Separating tool and schema faults from model behaviour“Move your instructions out of the tool result: send them in a user turn after the tool_result block”
↩︎ Checkpoint - 7.
“Strict tool use (strict: true): Guarantee schema validation on tool names and inputs.”
↩︎ Separating tool and schema faults from model behaviour“Type safe: Guaranteed field types and required fields”
↩︎ Information loss between subagents and the orchestrator - 8.https://code.claude.com/docs/en/agent-sdk/agent-loopOfficial docs
“maxTurns: 30, // Prevent runaway sessions”
↩︎ Separating tool and schema faults from model behaviour - 9.
“Subagents call tools to store their work in external systems, then pass lightweight references back to the coordinator.”
↩︎ Information loss between subagents and the orchestrator“This prevents information loss during multi-stage processing and reduces token overhead from copying large outputs through conversation history.”
↩︎ Prediction - 10.https://www.anthropic.com/engineering/contextual-retrievalSecondary source
“prepending chunk-specific explanatory context to each chunk before embedding”
↩︎ Retrieval-grounding failures“it can lead to problems when individual chunks lack sufficient context.”
↩︎ Exam trap 4