What you will be able to do
- Identify whether accuracy, latency or request throughput is the constraint a system is actually failing on
- Choose among model tier, effort, token trimming, streaming, caching, batch processing and retrieval depth for the change that relieves that constraint at least cost to the others
- Rule out changes that help a different constraint or make the binding one worse
- Confirm a chosen change against a repeated baseline and a held-out split before adopting it
Key concept
Binding constraint — The binding constraint is the one requirement (accuracy, latency or throughput) that the system is currently failing to meet. The right change relieves that constraint while spending as little as possible of the requirements that still have headroom. A change that improves a constraint the system already meets only adds cost.
1.Name the binding constraint before changing anything
A production Claude system usually meets most of its targets and fails one. A triage classifier can score well above its accuracy target and still miss its latency budget. A policy assistant can answer instantly and still give wrong answers. The target it fails is the binding constraint, and it sets the direction for every configuration change. The targets it meets are slack. Slack is a budget you can spend, but only as much as the fix needs.
The available changes fall into two kinds. Free wins lower cost without lowering quality: prompt caching, trimming tokens, and batch processing for work that can wait. Tradeoffs move along the frontier: a different model tier, a different effort level, output caps. Some free wins still carry a price on another axis. Batch processing, for example, trades latency for its discount, so it is free only when latency is slack.
| Lever | Cost effect | Latency effect | Quality effect |
|---|---|---|---|
| Prompt caching | Cut by a factor of 4.8 to 7.2 on a research agent | Faster | None |
| Batch API | 50% | Results within 24 hours | None |
| Lower effort | Medium about 30% to 52% less than high on long coding tasks | Faster | Up to 34 points lower, depending on the task and the model |
| Ask for shorter answers | About 13% less | Faster | None |
| Fast mode | Twice the standard token prices | Up to 2.5x higher output tokens per second | None |
Read the table by constraint, not by lever. If latency is binding, the useful rows are the ones marked faster, and the quality column shows what each will cost you. If accuracy is binding, a row that only saves money or time doesn't help. Batch processing shows how one lever can be right or wrong depending on which constraint binds. It has no quality cost, so it is ideal when latency is slack. When latency is binding, its 24-hour result window makes it unusable. The wider practice of trimming spend across a whole workload belongs to cost and latency optimization. Here the question is narrower: which single change clears the failing target.
Checkpoint 1 of 8· Check yourself
A synchronous triage classifier scores well above its accuracy target, but its responses miss the latency budget. Cost is acceptable. Which change relieves the binding constraint at the least cost to the other requirements?
Latency is the binding constraint and accuracy is slack. Lower effort is faster, and its cost lands on quality, which is the requirement with headroom. Batch processing returns results within 24 hours, which breaks the latency budget. A higher tier or a higher effort level spends more time to improve accuracy, which the system already meets.
“Several Claude models support an effort parameter that trades intelligence for latency and cost within a single model.”Source: platform.claude.com
Sources1
2.When latency is binding: tier, effort, tokens and calls
Total task time breaks down into parts you can act on. The latency cookbook models the time to last token like this: TTLT ≈ calls × TTFT + output tokens ÷ OTPS + additional client- or server-side latency. Every model call pays its own time to first token (TTFT). Every output token, thinking included, has to be generated at the model’s output speed (OTPS). Any change that cuts latency shrinks one of these terms: fewer calls, a shorter uncached prompt, fewer output tokens, or a faster model.
The model tier is the most direct lever. Claude Haiku 5.5 is the tier for the lowest latency and price, and its main speed control is effort. Effort trades intelligence for latency and cost within one model, so you can lower it without changing models. When a mid-tier model clears its accuracy target by a margin and misses its latency budget, that margin is the slack. Spend it by moving to a smaller, faster tier or lowering effort, then re-run the labelled eval set to confirm accuracy still clears the target. How each tier fits different workloads is covered in model tier selection.
client = anthropic.Anthropic()
# For time-sensitive applications, use Claude Haiku 5.5 at low effort
message = client.messages.create(
model="claude-haiku-5-5",
max_tokens=1024,
output_config={"effort": "low"},
messages=[
{
"role": "user",
"content": "Summarize this customer feedback in 2 sentences: [feedback text]",
}
],
)
print(next(block.text for block in message.content if block.type == "text"))Token count is the next lever, and it works in both directions. The fewer tokens the model reads and writes, the faster it responds. Asking for concise answers and setting max_tokens both cut output. This is also why some tempting fixes make a latency problem worse. Adding twenty more few-shot examples lengthens every prompt, which buys accuracy the system already has and adds latency it can't afford. Moving to the largest tier with extended thinking adds thinking tokens to the output term, which slows the response further.
Streaming relieves a latency complaint only when the complaint is about the wait before anything appears. In the cookbook’s insurer example, the phone line optimises for TTFT because a caller sits in silence until the agent speaks. The chat window can show progress, so it optimises for TTLT. A classifier whose p95 is measured on the completed label gets nothing from streaming. The Message Batches API is worse for that classifier: batch results can take up to 24 hours, which no synchronous console can wait for.
Checkpoint 2 of 8· Fill the gap
This call is for a latency-sensitive workload whose accuracy has headroom. Which effort value makes it the cheapest and fastest setting on Claude Haiku 5.5?
client = anthropic.Anthropic()
# For time-sensitive applications, use Claude Haiku 5.5 at low effort
message = client.messages.create(
model="claude-haiku-5-5",
max_tokens=1024,
output_config={"effort": " ? "},
messages=[
{
"role": "user",
"content": "Summarize this customer feedback in 2 sentences: [feedback text]",
}
],
)
print(next(block.text for block in message.content if block.type == "text"))Effort is Haiku 5.5's main control for speed and cost, and low is the cheapest and fastest level. Lower effort produces fewer output tokens, including thinking tokens, so it shrinks the output-token term of TTLT. The price is some accuracy, which is acceptable here because accuracy has headroom. Medium is Haiku 5.5's default, and high or max would add latency.
Source: platform.claude.comCheckpoint 3 of 8· Exam question
You are the architect for a support-triage classifier that calls the Messages API synchronously from an agent console. The current mid-tier model scores 97% on the labelled eval set against a 95% target, but p95 latency is 4.2 seconds against a 1.5 second budget. Which change relieves the binding constraint at the least cost to the other?
Correct answer: A — Move the classifier to a smaller, faster model tier, then re-run the labelled eval set to check that accuracy still clears the target.
- A. Correct: response time is the constraint being missed (4.2 s against 1.5 s), while accuracy has two points of headroom over 95%. A smaller, faster tier spends that headroom to cut latency, and the eval re-run confirms the accuracy cost stays acceptable.
- B. Incorrect: accuracy is already at 97% against a 95% target, so extra examples improve a number that is not the problem. They also lengthen the prompt, adding input tokens, which leaves p95 latency at 4.2 s or pushes it higher.
- C. Incorrect: the largest tier with extended thinking buys accuracy, which already exceeds the target, and it generates reasoning before each label. That makes the 4.2 s p95 worse instead of bringing it under 1.5 s.
- D. Incorrect: the Message Batches API is asynchronous and lowers cost by giving up response time, so results do not return within seconds. It cannot serve a synchronous agent console with a 1.5 s budget.
3.When volume is binding and latency is slack: move work off the synchronous path
A different failure shows up in bulk jobs. A nightly run sends hundreds of thousands of synchronous requests, keeps hitting rate limits, and has all night to finish. The binding constraint is request throughput. Per-request latency is slack. The fix that spends exactly that slack is the Message Batches API: it accepts many Messages requests together, processes them asynchronously, and lets you poll for results.
Batch processing is designed for the situation where immediate responses are not required. In return for giving up immediacy, it raises throughput and halves cost. Most batches finish in under an hour, and results are guaranteed within 24 hours, so a job due by morning loses nothing it needed. If the job were feeding a live agent console, the same change would break the latency target. Whether batch is the right answer depends entirely on which constraint binds.
| Change | Effect on the binding constraint (throughput) | Why |
|---|---|---|
| Message Batches API | Relieves it | Asynchronous processing of large volumes, with higher throughput |
| Streaming | None | Changes when output arrives, not how many requests or tokens are sent |
| cache_control on each ticket body | None | A cache hit needs a repeated prefix, and each ticket is read once |
| Larger model tier | None, and cost rises | The same number of requests, each one more expensive |
The caching row deserves a closer look, because caching is the strongest free win in most agent loops. Caching works by matching prefixes: the system checks whether the prompt prefix up to a breakpoint is already cached from a recent query. That fits repetitive tasks with consistent instructions, such as a shared system prompt sent with every ticket. It does nothing for the ticket bodies, which differ every time. Breakpoint placement is covered in cache breakpoints.
Checkpoint 4 of 8· Check yourself
Every night, hundreds of thousands of support tickets are summarised through synchronous Messages calls. The job keeps hitting rate limits, and nobody reads the summaries until the next morning. Which change fixes the requirement the job is failing?
The job is failing on throughput, and latency has room to spare because the results are only needed by morning. Batch processing uses that spare time to get higher throughput at half the price. Caching does nothing for ticket bodies that are each sent only once. Streaming changes when output arrives, not how many requests go out. A larger tier sends the same number of requests at a higher price per request.
“batch processing at 50% off for work that can wait up to 24 hours”Source: platform.claude.com
Checkpoint 5 of 8· Exam question
A team runs a nightly job that summarises 200,000 support tickets through synchronous Messages API calls and keeps hitting rate limits. Results are only needed by the next morning, and per-ticket latency does not matter. Which change relieves the binding constraint at the least cost to the other?
Correct answer: A — Move the summarisation to the Message Batches API, which processes large asynchronous volumes outside the synchronous path.
- A. Correct: nothing here needs a fast answer, so giving up latency for asynchronous processing costs the team nothing. The volume is handled outside the interactive request path.
- B. Incorrect: a larger tier consumes more of the same limits and runs slower, and summary quality was not the stated problem.
- C. Incorrect: streaming changes how tokens are delivered, not how many requests or tokens count against limits. It does not remove the rate-limit pressure.
- D. Incorrect: each ticket is unique and summarised once, so there is no repeated prefix to reuse. Caching would add write overhead without hits.
4.When accuracy is binding: fix what the model never sees
Accuracy failures need a diagnosis before a lever. If failure analysis shows Claude reasoned badly over the right evidence, a stronger tier or higher effort is the tradeoff to try. Retrieval-augmented systems often fail in a different way: the correct passage exists in the index, but it ranks below the cut-off and never reaches the prompt. No model, however capable, can reason over a passage it was never sent. A system-prompt rule to answer only from the supplied passages doesn't help either, because the passage it needs isn't among them.
When the right passage is missing from the prompt, the change belongs in retrieval, and reranking is the standard tool. It splits retrieval into two stages. The first stage casts a wide net, retrieving many more candidates than the prompt will hold, which raises recall. In the contextual-retrieval cookbook, hybrid search found the right chunk 88.86% of the time in the top 5 and 95.23% in the top 20. The second stage uses a specialised reranking model to score those candidates, and only the few best go to Claude.
The cheaper path is to rerank, not to put all 50 candidates in the prompt. Pasting 50 chunks in would fix recall, but every query would then carry ten times the input tokens. More tokens mean slower responses, and the model has to sort relevant policy text from noise on its own. Retrieving 50 and reranking down to a handful spends a little of the latency headroom on the rerank step and keeps the prompt small. For the choice between RAG, long context and agentic search, see retrieval architecture.
Checkpoint 6 of 8· Check yourself
Failure analysis on a policy assistant built on RAG finds that the passage holding the right answer is in the index but ranks around 30th. Only the top 5 chunks go into the prompt. Answers must get more accurate, and the latency budget has only a little headroom. Which change fixes the failure at the least cost to latency?
Claude never sees the correct passage, so a stronger tier or a prompt rule can't help. Retrieving widely raises recall. Reranking keeps the prompt small, so latency rises only by the cost of the rerank step. Sending all 50 chunks multiplies the input tokens on every query, and more tokens mean slower responses.
“The fewer tokens the model has to process and generate, the faster the response will be.”Source: platform.claude.com
Checkpoint 7 of 8· Exam question
A RAG assistant on the Messages API answers internal policy questions. Failure analysis shows the right passage is usually in the index but ranks outside the top 5 chunks that are sent to Claude; recall at 50 candidates is high. The latency budget has headroom. Choose two changes that relieve the accuracy constraint at the least added cost.(Select 2)
Correct answers: A, B — Retrieve 50 chunks per query in the first stage instead of 5, giving later steps a larger candidate pool to work from.; Rerank the 50 candidates with a relevance model, sending only the few best-scoring passages to Claude in the prompt.
- A. Correct: the miss happens because the right passage falls below a shallow cutoff, and recall is already high at 50 candidates. Retrieving that many puts the passage in the pool at little extra cost, and the latency headroom absorbs the larger search.
- B. Correct: a reranker reorders the wider pool by relevance, so the passage that previously ranked low reaches the top few that Claude sees. The prompt stays about as long as before, so token cost and latency barely rise.
- C. Incorrect: the missing passage is not among the 5 chunks sent, so a stronger model has nothing to reason over, and it cannot recover internal policy text it was never given. It also raises per-token cost on every request without touching the ranking problem.
- D. Incorrect: this does get the passage into the prompt, but it multiplies input tokens roughly tenfold on every query, adding far more cost and latency than reranking. The relevant text is also diluted among 49 distractors, which can lower answer quality.
- E. Incorrect: an instruction to stay within the supplied passages does not change which passages are supplied. The right one is still outside the top 5, so Claude would decline or answer from incomplete evidence, and accuracy stays constrained.
5.Confirm the change against a noise bar
Every change that spends slack needs to be measured, because the slack is real only if the eval says so. A good evaluation set built from your own prompts and data is the most important step in a model decision. Two habits make the measurement trustworthy. First, repeat the baseline before the first change: in Anthropic’s runs, identical runs of one configuration differed by several tasks, and those repeats give you a noise bar. Second, hold out a split that the tuning process never reads, so a change that only fits the tasks you looked at gets caught.
These habits decide whether a speed-up is safe to ship. After moving a classifier down a tier, the question is whether accuracy stayed above the target by more than run-to-run noise, not whether it went down at all. In one search on tau2-bench retail, lowering effort and moving down a tier produced the savings, while the prompt edits produced no durable gain once checked on held-out data. How to design such comparisons is covered in experimental design.
Checkpoint 8 of 8· Match them up
Before shipping a cheaper configuration, such as a classifier moved down a tier, a team checks its eval results. Match each element of that check to the role it plays in the decision.
Tap a term, then the definition that fits it.
Repeated baselines show how much results vary when nothing changes, so a speed-up is safe when accuracy stays above the target by more than that variation. A dip on its own does not count against it. The held-out split exposes gains that only fit the tuned tasks, like the prompt edits that gave no lasting gain on tau2-bench.
“Identical runs of one starting configuration differed by several tasks in these measurements (reference 24). The repeats give the search’s adoption gate a noise bar.”Source: platform.claude.com
Sources
(7)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligenceOfficial docs
“Free wins cut spend without touching quality”
↩︎ Name the binding constraint before changing anything“Tradeoffs exchange cost for intelligence: model choice, effort, output caps and session budgets, an elapsed-time clock, and multimodel architectures.”
↩︎ Name the binding constraint before changing anything“batch processing trades latency for its discount”
↩︎ Name the binding constraint before changing anything“batch processing at 50% off for work that can wait up to 24 hours”
↩︎ When latency is binding: tier, effort, tokens and calls“Repeat the baseline before the first change.”
↩︎ Confirm the change against a noise bar“Hold out a split the search never reads.”
↩︎ Confirm the change against a noise bar“The effort and model changes produced the savings; the prompt edits brought no durable gain.”
↩︎ Confirm the change against a noise bar“Cost and intelligence are usually pictured as a frontier where one buys the other.”
↩︎ Key concept“batch processing trades latency for its discount”
↩︎ Exam trap 2“batch processing at 50% off for work that can wait up to 24 hours”
↩︎ Checkpoint“Identical runs of one starting configuration differed by several tasks in these measurements (reference 24). The repeats give the search’s adoption gate a noise bar.”
↩︎ Checkpoint - 2.
“TTLT ≈ calls × TTFT + output tokens ÷ OTPS + additional client- or server-side latency”
↩︎ When latency is binding: tier, effort, tokens and calls“Output tokens: these include thinking tokens.”
↩︎ When latency is binding: tier, effort, tokens and calls“Here Acme optimizes for TTFT.”
↩︎ When latency is binding: tier, effort, tokens and calls - 3.https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-latencyOfficial docs
“One of the most direct ways to reduce latency is to choose the appropriate model for your use case.”
↩︎ When latency is binding: tier, effort, tokens and calls“The fewer tokens the model has to process and generate, the faster the response will be.”
↩︎ When latency is binding: tier, effort, tokens and calls“The fewer tokens the model has to process and generate, the faster the response will be.”
↩︎ When accuracy is binding: fix what the model never sees“This can significantly improve the perceived responsiveness of your application, as users can see the model’s output in real time.”
↩︎ Exam trap 1“This can significantly improve the perceived responsiveness of your application, as users can see the model’s output in real time.”
↩︎ Prediction - 4.
“The lowest latency and price”
↩︎ When latency is binding: tier, effort, tokens and calls“an effort parameter that trades intelligence for latency and cost within a single model”
↩︎ When latency is binding: tier, effort, tokens and calls“Having a good evaluation set is the most important step in the process.”
↩︎ Confirm the change against a noise bar“Several Claude models support an effort parameter that trades intelligence for latency and cost within a single model.”
↩︎ Checkpoint - 5.
“Process large volumes of Messages requests asynchronously with the Message Batches API, cutting costs by 50 percent and increasing throughput.”
↩︎ When volume is binding and latency is slack: move work off the synchronous path“Immediate responses are not required”
↩︎ When volume is binding and latency is slack: move work off the synchronous path“with most batches finishing in less than 1 hour”
↩︎ When volume is binding and latency is slack: move work off the synchronous path - 6.
“Repetitive tasks with consistent instructions”
↩︎ When volume is binding and latency is slack: move work off the synchronous path“The system checks if a prompt prefix, up to a specified cache breakpoint, is already cached from a recent query.”
↩︎ Exam trap 3 - 7.
“Reranking is a two-stage retrieval approach:”
↩︎ When accuracy is binding: fix what the model never sees“Stage 2 - Precise Selection: Use a specialized reranking model to score”
↩︎ When accuracy is binding: fix what the model never sees“Pass@20: 95.23%”
↩︎ When accuracy is binding: fix what the model never sees“Stage 1 - Broad Retrieval: Cast a wide net by retrieving more candidates than you need (e.g., retrieve 100 chunks)”
↩︎ Exam trap 4