What you will be able to do
- Explain why a suite that passed can still let a production failure through, in terms of a missing metric dimension or a wrong grading target
- Name the missing dimension (accuracy, groundedness, latency, cost, safety, security) behind a described production incident
- Tell when the final reply is the wrong thing to grade and the transcript or outcome is the right one
- Specify a replacement metric with a case set, a grader, a threshold and a release gate
Key concept
Evaluation coverage gap — A failure dimension or grading target that the pre-release suite never measured. This lets a release pass every test and still fail in production. You close the gap by adding the metric that is sensitive to that failure, not by adding more cases to the metric that already passed.
1.Why a passing suite still ships failures
An evaluation suite only detects the failures it measures. A suite can be large, automated and green on every run, and still be blind to the failure users hit first. The reason is usually not that it had too few cases. It never looked along the dimension where the failure lives. A post-incident review asks one question: which metric, run before release, would have turned red?
Most production systems need several success criteria at once. A single accuracy number can't stand in for all of them. A well-formed criterion for a sentiment classifier sets a quality floor, a safety floor and a latency target side by side, all measured on the same held-out set: an F1 score of at least 0.85, 99.5% non-toxic outputs, and 95% of responses under 200ms. Remove any one line and that dimension can regress silently while the others stay green.
Gaps come in two kinds. A dimension gap means a whole axis was never measured, such as latency, groundedness or injection resistance. A grading-target gap means the right axis was measured on the wrong artifact. The classic case is grading an agent's final reply when the failure is in what it did along the way. Either kind produces a release that passes everything it was tested on.
| Dimension | Typical production failure | Metric that detects it pre-release |
|---|---|---|
| Accuracy | Wrong label or answer on in-scope inputs | Exact match or F1 against golden answers on a held-out set |
| Groundedness | Confident statement found nowhere in the provided context | Claim-level faithfulness against the retrieved passages, including out-of-corpus questions |
| Latency | Users wait; first-response target missed | Time to first token and response-time percentiles against the product target |
| Cost | Spend per task rises after a configuration change | Token usage and cost per task on a static bank of tasks |
| Safety | Toxic or harmful output reaches users | Rate of flagged outputs over many trials, with a maximum allowed rate |
| Security | Agent follows injected instructions or takes unauthorized actions | Injection-resistance rate on a red-team set; trace check for unauthorized tool calls |
Mature evaluation programs have learned this the hard way. In one incident review, Anthropic described running benchmarks, safety evaluations and canary deployments, and still finding that the evaluations didn't capture the degradation users reported. Their fix was more sensitive evaluations, ones that reliably tell a working implementation from a broken one. More of the same evaluations would not have helped.
Checkpoint 1 of 8· Check yourself
A sentiment classifier's release suite measured only F1 on 10,000 held-out posts. It scored 0.91 and the release shipped on a larger model. Within a day, users complained that replies were too slow. Which pre-release addition would have turned the suite red?
This is a dimension gap: the suite never measured latency, so it could not detect a latency failure. Adding more cases, a stricter F1 floor or a relevancy grader only measures quality more closely. A response-time percentile target is the metric that would have failed before release.
“Most use cases need multidimensional evaluation along several success criteria.”Source: platform.claude.com
2.Accuracy against reference answers cannot see fabrication
Accuracy and groundedness answer different questions. Accuracy asks whether the output matches a known correct answer. Groundedness asks whether every statement in the output is supported by the context the model was given. A model can be accurate on every question the knowledge base answers and still invent a policy when asked about something the knowledge base never mentions. A suite made only of answerable questions never presents that situation.
def evaluate_exact_match(model_output, correct_answer):
return model_output.strip().lower() == correct_answer.lower()The missing metric has two parts, and both are needed. The case set has to include questions the corpus cannot answer. Irrelevant or nonexistent input data is a standard edge case, and out-of-corpus questions are exactly where a retrieval system is tempted to fill the gap. The grader has to break the answer into claims and check each one against the retrieved passages. That measures how well the model uses provided context, rather than how closely it matches a reference answer. A run that confidently asserts something no passage supports fails, even if the claim sounds plausible.
The tempting alternatives each measure a neighbouring property. Retrieval hit rate at k checks whether the right document came back. It grades the retriever, not the answer, and on an out-of-corpus question there is no gold document to find. Answer relevancy scores whether the reply addresses the user's request. A fabricated 60-day refund window is perfectly on-topic, so relevancy passes it. Exact match on more in-corpus questions only adds weight to a dimension that was already green.
Checkpoint 2 of 8· Check yourself
After the fabricated refund window, a team adds a grader that splits each answer into claims and checks every claim against the retrieved passages. They run it only on their existing questions, which the knowledge base answers. It passes at 99%. What change would make this suite catch the next fabrication before release?
The grader is the right one, but the case set never shows the situation where fabrication happens: a question the corpus does not answer. Adding more answerable questions only adds weight to a dimension that is already green. Relevancy passes on-topic fabrications. Hit rate has no gold document to find on out-of-corpus questions.
“Example edge cases Irrelevant or nonexistent input data”Source: platform.claude.com
Checkpoint 3 of 8· Exam question
A support assistant built on the Claude Messages API with retrieval told a customer that refunds are allowed within 60 days, a policy that appears nowhere in the help center. The pre-release suite scored answers against reference answers on questions the knowledge base covers, and all of them passed. As the architect running the post-incident review, which missing metric would have detected this failure before release?
Correct answer: C — Claim-level faithfulness, with a grader checking answer statements against retrieved passages on out-of-corpus questions
- A. Incorrect because exact match against gold answers can only be computed where the knowledge base holds an answer. A bigger sample of covered questions never probes what the assistant says when the help center is silent, so the 60-day claim would still slip through.
- B. Incorrect because hit rate at k confirms that the right document was retrieved for questions that have one. The refund question had no supporting document, so there was no gold passage to look for and the metric says nothing about the invented policy.
- C. Correct because the incident was an unsupported claim. A faithfulness grader that checks each statement against the retrieved passages, run on questions the knowledge base cannot answer, would have flagged the invented 60-day refund policy before release.
- D. Incorrect because relevancy rates how directly a reply addresses the user's request, not whether its content is supported by the retrieved passages. The invented 60-day policy answers a refund request head-on, so it would score well and the fabrication would go unnoticed.
3.Latency and cost: dimensions an accuracy gain can hide
A configuration change that raises quality often moves latency and cost in the wrong direction at the same time. Model size is one of the main factors in response time. Extended thinking adds reasoning tokens before the answer, which is why thinking budgets are tuned against both latency and quality needs. A team that moves to a larger model with thinking enabled, and re-runs only its accuracy suite, will see the improvement and none of the cost of it.
For a streaming chat product, the metric that matches user experience is time to first token (TTFT): the time from sending the prompt to receiving the first token of the response. If the product promises a first response within 2 seconds, the release gate is TTFT measured against that target, together with end-to-end response time. Both should be expressed as percentiles, the way a criterion like "95% response time < 200ms" is written. Latency also depends on the infrastructure serving the model, so the measurement has to run under conditions that mirror real-world traffic. A quiet offline run is not enough.
Two plausible substitutes fail this test. A mean wall-clock time from one sequential run hides the slow tail that users complain about, and it never sees production load. Mean output tokens per response tracks verbosity. It is a proxy that moves with latency sometimes, but it is not the target the product committed to. Neither one turns red when the 2-second promise is broken. Re-running accuracy on a bigger sample is a different dimension altogether.
Cost gaps behave the same way. Once a static bank of tasks exists, latency, token usage, cost per task and error rates can all be tracked on it at no extra design effort. A release that doubles tokens per task then shows up as a number before it shows up on an invoice. Budget is a success criterion in its own right, alongside per-call cost and frequency of usage. How to reduce latency or cost once you have measured it belongs to cost and latency optimization.
Checkpoint 4 of 8· Match them up
Your team moves a streaming chat product to a larger model with extended thinking. The product promises a first response within 2 seconds. Match each candidate release metric to what it would actually tell you about this change.
Tap a term, then the definition that fits it.
Only time to first token, written as a percentile and measured under realistic load, matches the product's first-response promise and turns red when the promise is broken. The mean and token-count metrics are proxies that can stay green while the slowest responses miss the target. Cost per task tracks spend, not speed, and accuracy measures quality, which is exactly where the upgrade looks good.
“This metric measures the time it takes for the model to generate the first token of the response, from when the prompt was sent.”Source: platform.claude.com
Checkpoint 5 of 8· Exam question
A chat product moved from a smaller Claude model to a larger one with extended thinking enabled, using the Messages API with streaming. The pre-release suite showed higher answer accuracy and the team shipped. Within a day, users complained about long waits and the 2-second first-response target was missed. Which metric would have detected this before release?
Correct answer: B — Time-to-first-token and end-to-end latency percentiles, measured at production-like concurrency and gated on the chat target
- A. Incorrect because output tokens per response is a cost proxy. It correlates loosely with latency but has no relation to the first-response target users experienced.
- B. Correct because the missing dimension was latency. Percentile time-to-first-token and total latency under realistic concurrency, gated on the target, would have failed this build.
- C. Incorrect because accuracy was already measured and had improved; sampling more of it cannot reveal a latency regression.
- D. Incorrect because a mean from a single sequential run hides tail latency and queuing under concurrent load, so it can look fine while many users wait too long.
4.Safety, security and grading the trace instead of the reply
For agents, a grader has three possible targets: the final reply, the transcript (every tool call, intermediate result and reasoning step), and the outcome (the final state of the environment). An agent that takes actions can do harm that never shows up in what it says. Code-based graders can verify tool calls directly, checking which tools were used and with what parameters. That makes "fail any run whose trace contains a delete_record call the user did not authorize" a cheap, objective, reproducible check.
Security is its own dimension because its adversary is not the ordinary user. Indirect prompt injection is the case where the user is trusted but the model processes third-party content, such as webpages, emails, documents or tool results, that carries adversarial instructions. Benign task-success cases never contain such content, however many of them there are. The suite has to add adversarial cases: retrieved pages that embed instructions to call destructive tools, plus a red-team set covering both direct and indirect injection. The defenses themselves are covered in prompt injection controls. Here the concern is measuring whether they hold.
Safety and security metrics become release gates when they are written as rates with thresholds. A safety criterion can be as concrete as fewer than 0.1% of outputs out of 10,000 trials flagged for toxicity. An injection-resistance rate over the red-team set, with a minimum pass rate, works the same way. Pair the rate, which tracks overall resistance, with a trace check that fails any run containing an unauthorized destructive call. One measures how often the agent resists; the other catches the specific harm.
Checkpoint 6 of 8· Match them up
After an injected instruction on a retrieved web page led a support agent to call delete_record, the team reviews five proposed additions to the suite. Match each addition to what it would actually tell them before the next release.
Tap a term, then the definition that fits it.
The harm happened in the agent's actions, so it only shows up when a grader looks at the transcript (which tools were called, with what parameters) or the outcome (what state the environment ends in), and when the cases contain adversarial content. A bigger benign set and reply-only rubrics stay green because they never test injected input or look at tool calls.
“Each grader evaluates some portion of either the transcript or the outcome.”Source: www.anthropic.com
Checkpoint 7 of 8· Exam question
A coding-ops agent built on the Agent SDK browses web pages and can call a delete_record tool. In production, a retrieved page contained hidden text telling the agent to delete customer records, and the agent complied. The pre-release suite only measured task success on benign requests. Choose two additions that would have detected this failure before release.(Select 2)
Correct answers: B, D — Track an injection-resistance rate over a red-team set of direct and indirect injection prompts, with a minimum rate as a release gate; Add cases where retrieved pages embed instructions to call destructive tools, and fail runs whose trace shows an unauthorized call
- A. Incorrect because a helpfulness rubric grades the final text for quality; the incident was an unauthorized action, which a text-quality score does not reveal.
- B. Correct because an injection-resistance rate over a red-team set turns the security risk into a measurable number, and a minimum value as a release gate would have blocked this build.
- C. Incorrect because latency percentiles are a performance dimension. Slow or fast tool chains tell nothing about whether the agent obeys injected instructions.
- D. Correct because it adds the missing security dimension: adversarial inputs inside tool results, graded deterministically on the trace for tool calls outside the allowlist.
- E. Incorrect because more benign paraphrases deepen accuracy coverage but contain no hostile content, so the agent would keep passing exactly as before.
5.Specifying the metric that closes the gap
Naming the missing dimension is half the review. The other half is writing the metric precisely enough that the next release cannot pass without it. Vague criteria such as "the bot should not make things up" or "it should be fast" cannot gate anything. A usable specification is specific and measurable in the same sense as any success criterion.
Build the specification in this order:
- Name the dimension the incident belongs to, and check whether the gap was the dimension itself or the grading target.
- Define the case set that exposes it: out-of-corpus questions, injected pages, production-like traffic. Source these cases from the failure you actually saw.
- Choose the grader and its target. Use a code check on tool calls, a claim-level check against context, or a timing measurement, applied to the reply, the trace or the outcome.
- Set a threshold, such as a minimum rate, a maximum rate or a percentile target, based on the product commitment.
- Make it a gate: the full suite must pass before a new version is promoted.
Run-to-run variation matters for gates. Model outputs vary between runs, so a rate measured over several trials per task is more trustworthy than a single pass. How to compare two configurations rigorously belongs to experimental design. Rerunning the expanded suite periodically also catches drift as workflows and models change. Every incident should leave behind at least one permanent case that would now fail.
The case set (for example, questions the knowledge base does not cover), the grader and its target (claims checked against the retrieved passages), a threshold (a minimum faithfulness rate) and the gate (the release cannot be promoted below it).
Checkpoint 8 of 8· Check yourself
Four drafts of a release requirement follow the incident where an injected page led the agent to call delete_record. Which draft is specified well enough to block the next version?
A usable spec names the case set (injected pages taken from the real failure), the grader and its target (a check on tool calls in the trace), a threshold measured over several trials, and a gate that stops promotion. The other drafts leave out at least one of these. The first can't be measured, the second uses benign cases and grades the reply, and the third grades the reply and has no threshold or gate.
“Require the full evaluation suite to pass before promoting new versions.”Source: platform.claude.com
Sources
(8)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Most use cases need multidimensional evaluation along several success criteria.”
↩︎ Why a passing suite still ships failures“99.5% non-toxic outputs”
↩︎ Why a passing suite still ships failures“Irrelevant or nonexistent input data”
↩︎ Accuracy against reference answers cannot see fabrication“How effectively does the model use provided context?”
↩︎ Accuracy against reference answers cannot see fabrication“95% response time < 200ms”
↩︎ Latency and cost: dimensions an accuracy gain can hide“What is your budget for running the model?”
↩︎ Latency and cost: dimensions an accuracy gain can hide“Less than 0.1% of outputs out of 10,000 trials flagged for toxicity by the content filter.”
↩︎ Safety, security and grading the trace instead of the reply“Specific: Clearly define what you want to achieve.”
↩︎ Specifying the metric that closes the gap“Most use cases need multidimensional evaluation along several success criteria.”
↩︎ Key concept“How effectively does the model use provided context?”
↩︎ Exam trap 3“Example edge cases Irrelevant or nonexistent input data”
↩︎ Checkpoint - 2.
“These issues exposed critical gaps that we should have identified earlier.”
↩︎ Why a passing suite still ships failures“we’ve developed evaluations that can more reliably differentiate between working and broken implementations”
↩︎ Why a passing suite still ships failures - 3.https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinationsOfficial docs
“can sometimes generate text that is factually incorrect or inconsistent with the given context”
↩︎ Accuracy against reference answers cannot see fabrication - 4.https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-latencyOfficial docs
“This metric measures the time it takes for the model to generate the first token of the response, from when the prompt was sent.”
↩︎ Latency and cost: dimensions an accuracy gain can hide“Latency can be influenced by various factors, such as the size of the model, the complexity of the prompt”
↩︎ Latency and cost: dimensions an accuracy gain can hide - 5.
“start with a larger budget of 16,000 tokens or more and adjust to your latency and quality needs”
↩︎ Latency and cost: dimensions an accuracy gain can hide“start with a larger budget of 16,000 tokens or more and adjust to your latency and quality needs”
↩︎ Exam trap 2 - 6.
“latency, token usage, cost per task, and error rates can be tracked on a static bank of tasks”
↩︎ Latency and cost: dimensions an accuracy gain can hide“Tool calls verification (tools used, parameters)”
↩︎ Safety, security and grading the trace instead of the reply“The outcome is the final state in the environment at the end of the trial.”
↩︎ Safety, security and grading the trace instead of the reply“Because model outputs vary between runs, we run multiple trials to produce more consistent results.”
↩︎ Specifying the metric that closes the gap“Source realistic tasks from the failures you see.”
↩︎ Specifying the metric that closes the gap“The outcome is the final state in the environment at the end of the trial.”
↩︎ Exam trap 4“A transcript (also called a trace or trajectory) is the complete record of a trial, including outputs, tool calls, reasoning, intermediate results”
↩︎ Prediction“Each grader evaluates some portion of either the transcript or the outcome.”
↩︎ Checkpoint - 7.https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/mitigate-jailbreaksOfficial docs
“Indirect prompt injection, where the user is trusted but Claude processes third-party content (webpages, emails, documents, tool results) that contains adversarial instructions.”
↩︎ Safety, security and grading the trace instead of the reply“Indirect prompt injection, where the user is trusted but Claude processes third-party content (webpages, emails, documents, tool results) that contains adversarial instructions.”
↩︎ Exam trap 1 - 8.
“Require the full evaluation suite to pass before promoting new versions.”
↩︎ Specifying the metric that closes the gap