What you will be able to do
- Use span attributes to group a fleet's requests by agent, parent and workflow run
- Match fleet-level symptoms such as throttling, cold caches, stalls and runaway spawning to their causes
- Predict which agents rerun when a failed or stopped workflow is relaunched
1.Reconstructing the fleet from span attributes
When forty coding agents run in parallel, reading one agent's transcript rarely explains why the run went wrong. The failures that matter happen across the fleet: throttling that slows every worker at once, caches that never warm, sessions stalled on a prompt nobody answers, and loops that keep spawning agents. Diagnosing them requires grouping thousands of model requests by the agent that issued them. Claude Code's trace spans carry the attributes needed for that grouping. Setting up the telemetry pipeline is covered in telemetry and instrumentation, and general trace querying in trace analysis at scale.
| Attribute | What it records | Fleet question it answers |
|---|---|---|
| agent_id | The subagent or teammate that issued the request; absent on the main session | Which worker is responsible for this load or this error? |
| parent_agent_id | The agent that spawned this one | How deep does the spawn tree go, and which parent is fanning out? |
| workflow.run_id | The Workflow tool run that spawned the agent, prefixed wf_ | Which run do these agents belong to? |
| query_source_safe | Bounded subsystem name such as repl_main_thread or agent.builtin.general-purpose | Is load coming from the main thread or from a particular agent type? |
| error_class | Short error token such as api_timeout or server_overload | Are failures capacity-related or something else? |
| attempt | Total attempts made for the request | How much retrying is hidden behind successful requests? |
| cache_read_tokens / cache_creation_tokens | Tokens read from or written to the prompt cache | Did fan-out agents reuse a shared cached prefix? |
| stop_reason | Why the response ended: end_turn, tool_use, max_tokens, refusal and others | Are agents being cut off at max_tokens? |
Use query_source_safe for grouping instead of query_source. The safe form is emitted whether or not detailed beta tracing is turned on, and its values come from a bounded set, so it works as a dimension on fleet dashboards. workflow.name replaces user-authored names with custom unless OTEL_LOG_TOOL_DETAILS is set, so key per-run analysis on workflow.run_id.
Checkpoint 1 of 3· Match them up
Forty agents in a fleet run went wrong. Match each span attribute to the fleet question it lets you answer.
Tap a term, then the definition that fits it.
agent_id names the worker that sent the request. parent_agent_id links each agent to the one that spawned it, so you can rebuild the spawn tree. workflow.run_id groups agents by run, and it stays usable when workflow.name is masked. error_class sorts failures by cause. attempt shows retries that a success flag alone would hide.
“agent_id | Identifier of the subagent or teammate that issued the request. Absent on the main session”Source: code.claude.com
Sources1
2.Reading fleet-level failure patterns
Concurrency is the first setting to check when fleet throughput looks wrong. If spans show fewer concurrent agent_id values than expected, the CPU-scaled default is the likely cause. CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS accepts values from 1 to 256. The opposite symptom calls for turning the setting down instead of up: rising attempt counts with error_class values such as server_overload or api_timeout mean the fleet is wider than it can be served. Capacity planning for that case is covered in scale and throughput management.
| Symptom | Likely cause | Confirming signal |
|---|---|---|
| Every agent in a fan-out writes its own cache | Agents do not share the first agent's prompt prefix, so each processes it uncached | cache_creation_tokens high and cache_read_tokens near zero across sibling agent_id values |
| Agent count climbing without new work | A loop keeps spawning agents; the run stops at 1,000 agents total | Long parent_agent_id chains under one workflow.run_id |
| A run errors before any agent starts | A parallel() or pipeline() list over 4,096 items is rejected outright, not truncated | No agent spans for the run; the script reports the error |
| Agents produce partial edits | Responses cut off by length | stop_reason of max_tokens on agent spans |
Prompt caching across a fan-out works because agents that share the first agent's prompt-cache prefix start up to five seconds after it and read the prefix that agent cached. A brief start delay is therefore expected, but uniformly high cache creation across siblings is not. It usually means each agent's prompt differs at the start, so put the common instructions first. The 4,096-item limit fails loudly on purpose, because a silent cap would drop part of the workload without telling the script.
Some stalls never show up as errors at all. In agent view, a session waiting on a permission decision or a question sits under Needs input. A command that needs an attached terminal, such as /install-github-app, holds an unattended session there too. A workflow likewise pauses on its own only for agent permission prompts and usage-limit waits. When fleet throughput drops but error rates stay flat, check the Needs input count in agent view before looking for an API problem.
Checkpoint 2 of 3· Check yourself
Partway through a large background fan-out, fleet throughput drops sharply. Across agent spans, the error_class values and error rates are no different from earlier in the run. What should you check first?
A session waiting on a permission decision or a question stalls without producing any error, so throughput can drop while error rates stay flat. An overloaded fleet would show server_overload or api_timeout errors, and a spawn loop shows up as a rising agent count, not as a throughput drop.
“Claude is waiting on something only you can provide: an answer to a question, a permission decision, or another prompt only you can answer”Source: code.claude.com
3.What reruns when a workflow is relaunched
How much a relaunch costs depends on the first agent whose outcome changed. Agents that completed return their saved results, until the first agent whose prompt differs from the previous run. That agent runs again, and so does every agent after it, even ones that had completed. A failed agent triggers the same cascade. This is why the script runtime makes Date.now(), Math.random() and a no-argument new Date() throw: a relaunched run has to repeat the same agent() calls in order to reuse their results. Pass any timestamp in through args.
Stopping one agent on its own counts as a failure. On relaunch, that agent runs again, and so does every agent the script starts after it, including ones that had already completed. Stopping the whole run would not have counted any agent as failed: agents that were still running would start over, and completed ones would return their saved results.
Usage limits are treated more leniently in recent versions. On v2.1.318 or later, an agent that failed at a usage limit runs again on relaunch while completed agents return their saved results. The exception is when the script called another agent after the failure and got a result back: then every agent started after the failed one reruns. In the /workflows view, f filters a phase's agents by status. Use it to find the earliest failed agent, which tells you how much of a relaunch will repeat.
Checkpoint 3 of 3· Check yourself
A 300-file migration workflow finishes with one failure: agent 120 hit an api_timeout, and every other agent completed. You relaunch the script without editing it. Which agents run again?
A failed agent triggers the same cascade as a changed prompt: it reruns, and so does every agent started after it, even completed ones. Agents before it return their saved results. Only a usage-limit failure on v2.1.318 or later lets the completed agents keep their saved results.
“Failed: runs again, and so does every agent the script starts after it, even ones that completed.”Source: code.claude.com
Sources2
Sources
(3)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://code.claude.com/docs/en/monitoring-usageOfficial docs
“Identifier of the subagent or teammate that issued the request. Absent on the main session”
↩︎ Reconstructing the fleet from span attributes“Run identifier of the Workflow tool run that spawned this agent, prefixed wf_.”
↩︎ Reconstructing the fleet from span attributes“Bounded form of query_source, emitted whether or not detailed beta tracing is active”
↩︎ Reconstructing the fleet from span attributes“Short error class token when the request failed, such as api_timeout or server_overload.”
↩︎ Reading fleet-level failure patterns“agent_id | Identifier of the subagent or teammate that issued the request. Absent on the main session”
↩︎ Checkpoint - 2.https://code.claude.com/docs/en/workflowsOfficial docs
“Up to 16 concurrent agents by default, fewer when Claude Code has fewer CPUs available, including inside a CPU-limited container.”
↩︎ Reading fleet-level failure patterns“agents that share the first agent’s prompt-cache prefix start up to 5 seconds after it by default”
↩︎ Reading fleet-level failure patterns“Failed: runs again, and so does every agent the script starts after it, even ones that completed.”
↩︎ What reruns when a workflow is relaunched“Stopping one agent alone, by selecting it in /workflows and pressing x, counts as failing.”
↩︎ What reruns when a workflow is relaunched“so that a relaunched run repeats the same agent() calls”
↩︎ What reruns when a workflow is relaunched“Stopping one agent alone, by selecting it in /workflows and pressing x, counts as failing.”
↩︎ Exam trap 1“Up to 16 concurrent agents by default, fewer when Claude Code has fewer CPUs available, including inside a CPU-limited container.”
↩︎ Exam trap 2 - 3.https://code.claude.com/docs/en/agent-viewOfficial docs
“A command that needs an attached terminal, such as /install-github-app or the /mcp settings list, holds an unattended session here too”
↩︎ Reading fleet-level failure patterns“Claude is waiting on something only you can provide: an answer to a question, a permission decision, or another prompt only you can answer”
↩︎ Checkpoint