What you will be able to do
- Design a live-traffic sample that shows regressions in small tenants and rare request types instead of averaging them away
- Combine user feedback with sampled, calibrated correctness metrics so that optimizing one signal cannot quietly degrade quality
- Set up an A/B test whose randomization unit matches the outcome being measured
- Choose online metrics that fit extraction, generation, agentic and retrieval-grounded tasks
Key concept
Online quality measurement — Grading what a Claude application actually does on real production traffic, using sampled transcripts, user signals and controlled experiments. It complements offline evals, which only test the scenarios someone thought to write down.
1.What live traffic shows that a golden set cannot
An offline suite only tells you how the system does on inputs you chose. It runs before launch, it is reproducible, and it affects no users. Its weakness is coverage: it can create false confidence when real usage looks different from the test bank. Online measurement fills that gap. Production monitoring shows real user behaviour at scale and gives ground truth on how the system actually performs. The cost is that it reacts after the fact, because a problem has already reached users by the time you see it.
The two layers do different jobs, so you run both. One proven combination uses automated evals for fast iteration, production monitoring for ground truth, and periodic human review for calibration. Offline suite design belongs to evaluation system design. This page covers the live half: which traffic you grade, which signals you trust, how you compare variants, and which metric fits which task.
The raw material is traces. The Claude API and SDK can emit OpenTelemetry spans for Messages API requests and tool uses. Teams use these traces to debug, to track tokens and latency, and to build evals from real traffic. Each request can also be linked to the user or feature that sent it, and that link is what later lets you break quality down by tenant or request type.
| Method | What it gives you | Main limitation |
|---|---|---|
| Production monitoring | Real user behaviour at scale; catches issues synthetic evals miss | Reactive, noisy, and lacks ground truth for grading |
| A/B testing | Actual user outcomes such as retention and task completion; controls for confounds | Days or weeks to reach significance; needs enough traffic |
| User feedback | Real examples of problems you did not anticipate | Sparse, self-selected, skews toward severe issues |
| Manual transcript review | Intuition for failure modes; subtle issues automated checks miss | Time-intensive, does not scale, inconsistent coverage |
Checkpoint 1 of 9· Match them up
Match each way of measuring quality to the trade-off it brings.
Tap a term, then the definition that fits it.
Each layer covers a gap that another one leaves. Offline suites only see the inputs you chose, and each online method gives up speed, coverage or scale to get ground truth from real traffic. That is why teams run several of them together.
“Reactive; problems reach users before you know about them”Source: www.anthropic.com
2.Sampling live traffic so small groups stay visible
A grader can only score content it actually receives. When you add an OpenTelemetry destination, the default content mode is Metadata only: identifiers, timing, status and token counts, with no prompts or responses. An LLM judge or human reviewer has nothing to grade until the destination or SDK client is set to Full content, which adds model prompts and responses to the spans. Privacy comes into this choice too. If your organization has a HIPAA, FERPA or K-12 data-handling restriction with Anthropic, destinations export only metadata whatever their content mode is set to.
A uniform random sample is equal per request, not per customer. A tenant sending 0.05% of traffic gets roughly 0.05% of the graded sample. Even a severe drop for that tenant moves the pooled weekly score by a rounding error. Raising the rate from 0.5% to 5% does not fix this, because the small tenant is still a sliver of a single pooled number. Swapping in a larger judge model on the same sample does not help either. That changes how accurate each grade is, but not which requests get graded.
Stratified sampling fixes the coverage problem. Group traffic by the dimensions you care about, such as tenant and request type, which the per-request user and feature links make possible. Guarantee a minimum number of graded requests in every group. Then reweight each group back to its true traffic share when you compute a global score. You get per-group scores that can show a regression in a small tenant, plus a global number that still reflects the real traffic mix.
Sampling only conversations that got a thumbs-down or an escalation is a different mistake. That sample is chosen by user behaviour, not by you. User feedback is sparse and self-selected, and it skews toward severe issues, so it misses quiet quality loss that nobody complained about. It also tells you nothing about the share of good responses.
Checkpoint 2 of 9· Check yourself
A team adds an OpenTelemetry destination in the Console and keeps the default settings. They then point an LLM judge at the exported spans to score answer quality, but the judge has nothing it can grade. What should they change?
The default content mode is Metadata only, which has no prompts or responses, so a judge has nothing to read. Full content adds that text to the spans. Under a HIPAA, FERPA or K-12 restriction, destinations still export only metadata.
“Full content (content in the SDK): Most visibility. Spans include model prompts and responses, and everything in Metadata only.”Source: platform.claude.com
Checkpoint 3 of 9· Check yourself
The weekly pooled quality score stays flat, yet a tenant that sends about 0.05% of traffic reports much worse answers. Which change to the grading pipeline would show regressions like this and still give a global score that reflects the real traffic mix?
Stratified sampling with a minimum per group makes small tenants visible, and reweighting keeps the global number true to the traffic mix. A higher uniform rate still leaves the tenant as a sliver of one pooled number. A better judge does not change which requests get graded. Feedback-selected samples are self-selected and miss quiet regressions.
“Design evals that mirror your real-world task distribution. Don’t forget to factor in edge cases!”Source: platform.claude.com
Checkpoint 4 of 9· Exam question
Fennel Robotics runs a support assistant for warehouse-automation customers, handling 2M requests a day across 300 tenants. Its quality team has an LLM judge score a uniform 0.5% random sample of traffic and publishes one pooled quality score on a weekly dashboard, which has stayed flat for two months. Two small tenants, each under 0.1% of traffic, report clear regressions after a prompt update that mainly affected their returns-handling requests, yet the dashboard never shows them. Last week's scored sample held only a handful of items from each of those tenants. The judge's grades agree closely with human labels on spot checks, the judge budget is already near its monthly limit, and these tenants' users rarely click thumbs-down. What change best fixes the measurement?
Correct answer: B — Sample by tenant and request type with a guaranteed minimum per group, then reweight the groups into a global quality score.
- A. A higher rate still draws items in proportion to traffic, so the smallest tenants stay thinly covered. The pooled score also keeps diluting their regressions among 300 tenants, and judge cost rises tenfold.
- B. Correct: a guaranteed minimum per tenant and request type gives each small tenant enough scored items to show a change. Reweighting by true traffic share keeps the global aggregate unbiased.
- C. Flagged conversations are self-selected and depend on how often users complain, with no baseline of healthy traffic. A tenant's rate of low grades among them cannot be compared fairly across tenants, and small tenants still contribute few items.
- D. A larger judge and weekly calibration raise the accuracy of each grade but leave coverage unchanged. The same traffic-proportional sample still contains very few items from the two small tenants.
3.Feedback loops and the proxy-metric trap
User signals are cheap at volume, and they are proxies for quality, not quality itself. Explicit feedback such as thumbs-down or bug reports surfaces problems you did not anticipate and often correlates with product goals. Implicit measures such as task completion rates come with no extra effort from the user. But users rarely explain why something failed, and relying mainly on users to catch problems means users are the ones who hit them.
Optimizing a single proxy invites the system to game it. Suppose a team tunes prompts to maximize thumbs-up rate and gets a 12% lift, but audits show answers got longer and more flattering while factual accuracy fell six points. The thumbs-up metric did exactly what it was told. It rewarded what users click on, and nothing in the measurement penalized being wrong. Most use cases need multidimensional evaluation along several success criteria, and a ship decision based on one signal ignores the rest.
The fix is a change to measurement: make correctness a co-primary metric next to the feedback signal. Score a sample of live responses for factual correctness with an LLM judge, and calibrate that judge against human labels, since model-based graders require calibration with human graders for accuracy. The mechanics of calibration are covered in judge calibration against human labels. Also compare variants with length held constant, so a variant cannot win just by being longer. Three tempting alternatives fail for clear reasons:
- Adding "stay accurate" to the prompt changes the system under test, not what you measure, so thumbs-up remains the only thing that decides a ship.
- A token cap limits one way of gaming the metric but still measures nothing about correctness.
- Abandoning online measurement for the golden set alone throws away ground truth from real usage.
Feedback should also flow back into the offline suite. A sustainable routine is to triage feedback constantly, sample transcripts to read weekly, and dig deeper as needed. Each confirmed failure becomes a regression case, so the loop closes: failures become test cases, test cases prevent regressions, and metrics replace guesswork.
Checkpoint 5 of 9· Check yourself
After a prompt update, the post-chat satisfaction score for a support assistant rises 9%. A manual audit of transcripts finds that answers now agree with customers more often and that more of them quote the wrong policy terms. Before the next ship decision, what should the team change?
Satisfaction is a proxy that rewards what users like, so it needs a correctness metric beside it as co-primary. A prompt instruction changes the system but not the measurement. A token cap blocks one way of gaming the metric and still measures nothing about correctness. Dropping online measurement throws away ground truth from real usage.
“Most use cases need multidimensional evaluation along several success criteria.”Source: platform.claude.com
Checkpoint 6 of 9· Exam question
Vireo Learning runs an AI homework-help assistant in production for its tutoring subscribers. Over two months, the team tuned the system prompt to maximize the thumbs-up rate and saw a 12% lift. A weekly audit of sampled conversations by trained reviewers shows that answers are now longer and more flattering, and factual accuracy fell six points. An earlier prompt revision that told the model to stay accurate did not stop the drift. A trial that hard-limited response length shortened answers, but flattering openers continued, and approval stayed high on answers the reviewers marked wrong. The pre-launch golden set still passes. It was written before launch and has none of the new question types subscribers now ask. Leadership wants live user feedback to stay in the loop. What is the best change to the measurement approach?
Correct answer: A — Add a sampled, human-calibrated correctness metric and a length-controlled comparison as co-primary metrics with thumbs-up.
- A. Correct: a proxy that can be gamed needs a counterweight that measures the real objective, and controlling for length removes the verbosity effect.
- B. The optimization pressure stays on thumbs-up, and a prompt line is not a measurement; accuracy would still go unmonitored.
- C. A cap treats only the length symptom; flattery and wrong facts can still raise approval while accuracy keeps slipping.
- D. Offline sets miss live distribution and unforeseen failures; the better fix is to add metrics, not to remove the online view.
4.A/B tests: randomize the unit you measure
An A/B test compares a candidate against a baseline model or an earlier version on real traffic. Its advantage over every other online method is that it controls for confounds: random assignment means the only systematic difference between the arms is the change. It measures actual user outcomes such as retention and task completion. The price is time. Reaching significance takes days or weeks and needs enough traffic, so A/B tests are for significant changes, and they only test changes you actually deploy.
The randomization unit has to match the measurement unit. Satisfaction with a multi-turn assistant is a property of the whole session, but per-request randomization gives one user turns from both prompts. Each arm's satisfaction score then reflects a blend of both variants, and no amount of extra traffic separates their effects. Hashing the user ID gives each user a stable, random assignment that persists across requests, so per-user outcomes belong cleanly to one variant.
The other common workarounds break randomization in their own ways. Running the old prompt one week and the new one the next mixes the prompt change with anything else that differed between those weeks, such as a release, a holiday or a change in traffic. Dropping sessions that received both prompts keeps only sessions that happened to fall entirely in one arm. Those tend to be shorter sessions, so the remaining population is no longer comparable to real usage.
Checkpoint 7 of 9· Check yourself
Two weeks into an A/B test of a new system prompt for a research assistant, the candidate arm shows a statistically significant drop in task completion. The team needs to know what the new prompt is doing differently before they revise it. What is the best next step?
An A/B test controls for confounds and shows which variant won, but it gives little signal on the cause unless you review transcripts from each arm. More traffic only makes the result more certain. Per-request randomization breaks session-level attribution. Feedback-selected samples are self-selected and skewed toward severe issues.
“Less signal on the underlying “why” for changes in metrics without being able to thoroughly review the transcripts”Source: www.anthropic.com
Checkpoint 8 of 9· Exam question
Indigo Travel runs a multi-turn coding assistant inside its internal developer portal, where engineers typically work through a task over a dozen or more requests in one session. The team A/B tests a new system prompt against the current one, with each request randomized independently between the two variants. Analysts find that many sessions contain responses from both prompts, and a single end-of-session satisfaction rating therefore cannot be tied to either one. Session-level satisfaction looks noisy and contradictory: the new prompt wins on some days and loses on others. Longer sessions are the most likely to mix variants. The portal serves a fixed pool of engineers, so traffic cannot simply be multiplied, and the team needs results attributable to the prompt rather than to timing or user mix. What should the team change?
Correct answer: C — Hash the user ID to assign one prompt per user, keep it fixed for later requests, then compare per-user satisfaction between the two prompts.
- A. A larger sample does not stop one session from receiving both prompts. Each rating still blends the effect of both, and requests from the same user remain correlated. The problem is contamination, not too few observations.
- B. Assigning by calendar week is not randomization. Prompt is confounded with week-specific factors such as the user mix, product releases and novelty effects, so a difference cannot be attributed to the prompt.
- C. Correct: hashing the user ID gives each user a stable variant, so no session blends both prompts and carryover is removed. Per-user metrics also give independent analysis units, which resolves the noisy, contradictory ratings.
- D. Mixed sessions are not a random subset. Longer sessions with more requests are more likely to receive both prompts, so dropping them leaves a biased sample. It also discards data and does not fix the assignment itself.
5.Matching the online metric to the task type
A metric only works if it fits the shape of the output. Extraction and classification have one right answer per input, so exact match is ideal: it is perfect for tasks with clear-cut, categorical answers such as sentiment analysis. The grader is two lines of code and runs on every sampled response at almost no cost.
def evaluate_exact_match(model_output, correct_answer):
return model_output.strip().lower() == correct_answer.lower()Generation tasks, such as drafting an email, have no single reference answer, so exact match is meaningless for them. Use rubric-based or Likert-scale grading by an LLM judge, for example coherence rated from 1 to 5. Use embedding cosine similarity when the question is consistency, because similar questions should yield semantically similar answers even when the wording differs.
Agentic tasks are judged on what the agent did as well as what it said. Code-based checks can verify the outcome, verify tool calls (which tools were used, with which parameters), and analyse the transcript for turns taken and tokens used. Support agents add business-level online metrics. Escalation accuracy tracks the percentage of correctly escalated conversations. CSAT is usually collected through post-interaction surveys.
Retrieval-grounded tasks need two kinds of metric, because the system can fail at retrieval or at generation. Precision, recall and mean reciprocal rank (MRR) measure retrieval, and recall matters most because the model can only use what it was given. End-to-end accuracy, graded by an LLM judge against the ground truth, evaluates the entire pipeline from retrieval to answer generation.
| Task type | Typical output | Metrics that fit |
|---|---|---|
| Extraction / classification | One correct categorical answer | Exact match, accuracy, F1 |
| Generation | Free-form text, no single reference | LLM-graded rubric or Likert scale; cosine similarity for consistency |
| Agentic | Actions, tool calls, end state | Outcome verification, tool-call verification, transcript analysis, escalation accuracy |
| Retrieval-grounded | Answer built from retrieved chunks | Precision, recall, MRR for retrieval; LLM-judged end-to-end accuracy |
Checkpoint 9 of 9· Match them up
A platform team is choosing an online metric for each production workload. Match each workload to the metric that fits the shape of its output.
Tap a term, then the definition that fits it.
The metric has to fit the output. Categorical labels have one right answer, so exact match works. Free-form drafts have no single reference, so they need rubric or Likert grading. Consistency is about meaning, not exact wording, so it calls for embedding similarity. Agents are judged on what they did. Retrieval-grounded systems can fail at retrieval or at generation, so each part needs its own metric.
“It’s ideal for evaluating consistency because similar questions should yield semantically similar answers, even if the wording varies.”Source: platform.claude.com
Sources
(5)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“build evals from real traffic”
↩︎ What live traffic shows that a golden set cannot“linked to the user or feature that sent it”
↩︎ What live traffic shows that a golden set cannot“Metadata only (metadata_only in the SDK): The default.”
↩︎ Sampling live traffic so small groups stay visible“Full content (content in the SDK): Most visibility. Spans include model prompts and responses, and everything in Metadata only.”
↩︎ Sampling live traffic so small groups stay visible“If your organization has a HIPAA, FERPA, or K-12 data-handling restriction with Anthropic, destinations export only metadata regardless of their content mode.”
↩︎ Sampling live traffic so small groups stay visible - 2.
“Can create false confidence if it doesn’t match real usage patterns”
↩︎ What live traffic shows that a golden set cannot“automated evals for fast iteration, production monitoring for ground truth, and periodic human review for calibration”
↩︎ What live traffic shows that a golden set cannot“Sparse and self-selected”
↩︎ Sampling live traffic so small groups stay visible“Users rarely explain why something failed”
↩︎ Feedback loops and the proxy-metric trap“Requires calibration with human graders for accuracy”
↩︎ Feedback loops and the proxy-metric trap“triage feedback constantly, sample transcripts to read weekly, and dig deeper as needed”
↩︎ Feedback loops and the proxy-metric trap“failures become test cases, test cases prevent regressions, and metrics replace guesswork”
↩︎ Feedback loops and the proxy-metric trap“Measures actual user outcomes (retention, task completion)”
↩︎ A/B tests: randomize the unit you measure“Slow; days or weeks to reach significance and requires sufficient traffic”
↩︎ A/B tests: randomize the unit you measure“A/B testing validates significant changes once you have sufficient traffic.”
↩︎ A/B tests: randomize the unit you measure“Tool calls verification (tools used, parameters)”
↩︎ Matching the online metric to the task type“Catches issues that synthetic evals miss”
↩︎ Key concept“Skews toward severe issues”
↩︎ Exam trap 4“Reactive; problems reach users before you know about them”
↩︎ Checkpoint“Controls for confounds”
↩︎ Prediction“Less signal on the underlying “why” for changes in metrics without being able to thoroughly review the transcripts”
↩︎ Checkpoint - 3.
“Design evals that mirror your real-world task distribution.”
↩︎ Sampling live traffic so small groups stay visible“User feedback: Implicit measures such as task completion rates.”
↩︎ Feedback loops and the proxy-metric trap“Most use cases need multidimensional evaluation along several success criteria.”
↩︎ Feedback loops and the proxy-metric trap“A/B testing: Compare performance against a baseline model or earlier version.”
↩︎ A/B tests: randomize the unit you measure“perfect for tasks with clear-cut, categorical answers such as sentiment analysis”
↩︎ Matching the online metric to the task type“similar questions should yield semantically similar answers, even if the wording varies”
↩︎ Matching the online metric to the task type“Design evals that mirror your real-world task distribution.”
↩︎ Exam trap 1“Most use cases need multidimensional evaluation along several success criteria.”
↩︎ Exam trap 2“A/B testing: Compare performance against a baseline model or earlier version.”
↩︎ Exam trap 3“Design evals that mirror your real-world task distribution. Don’t forget to factor in edge cases!”
↩︎ Checkpoint“It’s ideal for evaluating consistency because similar questions should yield semantically similar answers, even if the wording varies.”
↩︎ Checkpoint - 4.
“Track the percentage of correctly escalated conversations”
↩︎ Matching the online metric to the task type“Usually done through post-interaction surveys.”
↩︎ Matching the online metric to the task type - 5.https://platform.claude.com/cookbook/capabilities-retrieval-augmented-generation-guideSecondary source
“Recall is crucial for ensuring the LLM has access to all needed information.”
↩︎ Matching the online metric to the task type“This metric evaluates the entire pipeline, from retrieval to answer generation.”
↩︎ Matching the online metric to the task type