CertSafari
    Anthropic Claude Certified Developer – Professional (CCDV-P)· Lessons

    Domain 5 · Lesson 16/25

    Online quality measurement on live Claude traffic

    Online Quality Measurement

    17 min read
    4% of exam
    5 sources
    Published 19 Nov 2026
    Docs as of 11 Nov 2026

    What you will be able to do

    • Design a live-traffic sample that shows regressions in small tenants and rare request types instead of averaging them away
    • Combine user feedback with sampled, calibrated correctness metrics so that optimizing one signal cannot quietly degrade quality
    • Set up an A/B test whose randomization unit matches the outcome being measured
    • Choose online metrics that fit extraction, generation, agentic and retrieval-grounded tasks

    Key concept

    Online quality measurement — Grading what a Claude application actually does on real production traffic, using sampled transcripts, user signals and controlled experiments. It complements offline evals, which only test the scenarios someone thought to write down.

    1.What live traffic shows that a golden set cannot

    An offline suite only tells you how the system does on inputs you chose. It runs before launch, it is reproducible, and it affects no users. Its weakness is coverage: it can create false confidence when real usage looks different from the test bank. Online measurement fills that gap. Production monitoring shows real user behaviour at scale and gives ground truth on how the system actually performs. The cost is that it reacts after the fact, because a problem has already reached users by the time you see it.

    The two layers do different jobs, so you run both. One proven combination uses automated evals for fast iteration, production monitoring for ground truth, and periodic human review for calibration. Offline suite design belongs to evaluation system design. This page covers the live half: which traffic you grade, which signals you trust, how you compare variants, and which metric fits which task.

    The raw material is traces. The Claude API and SDK can emit OpenTelemetry spans for Messages API requests and tool uses. Teams use these traces to debug, to track tokens and latency, and to build evals from real traffic. Each request can also be linked to the user or feature that sent it, and that link is what later lets you break quality down by tenant or request type.

    Online measurement methods and what each costs you
    MethodWhat it gives youMain limitation
    Production monitoringReal user behaviour at scale; catches issues synthetic evals missReactive, noisy, and lacks ground truth for grading
    A/B testingActual user outcomes such as retention and task completion; controls for confoundsDays or weeks to reach significance; needs enough traffic
    User feedbackReal examples of problems you did not anticipateSparse, self-selected, skews toward severe issues
    Manual transcript reviewIntuition for failure modes; subtle issues automated checks missTime-intensive, does not scale, inconsistent coverage

    Checkpoint 1 of 9· Match them up

    Match each way of measuring quality to the trade-off it brings.

    Tap a term, then the definition that fits it.

    Sources12

    2.Sampling live traffic so small groups stay visible

    A grader can only score content it actually receives. When you add an OpenTelemetry destination, the default content mode is Metadata only: identifiers, timing, status and token counts, with no prompts or responses. An LLM judge or human reviewer has nothing to grade until the destination or SDK client is set to Full content, which adds model prompts and responses to the spans. Privacy comes into this choice too. If your organization has a HIPAA, FERPA or K-12 data-handling restriction with Anthropic, destinations export only metadata whatever their content mode is set to.

    A uniform random sample is equal per request, not per customer. A tenant sending 0.05% of traffic gets roughly 0.05% of the graded sample. Even a severe drop for that tenant moves the pooled weekly score by a rounding error. Raising the rate from 0.5% to 5% does not fix this, because the small tenant is still a sliver of a single pooled number. Swapping in a larger judge model on the same sample does not help either. That changes how accurate each grade is, but not which requests get graded.

    Stratified sampling fixes the coverage problem. Group traffic by the dimensions you care about, such as tenant and request type, which the per-request user and feature links make possible. Guarantee a minimum number of graded requests in every group. Then reweight each group back to its true traffic share when you compute a global score. You get per-group scores that can show a regression in a small tenant, plus a global number that still reflects the real traffic mix.

    Sampling only conversations that got a thumbs-down or an escalation is a different mistake. That sample is chosen by user behaviour, not by you. User feedback is sparse and self-selected, and it skews toward severe issues, so it misses quiet quality loss that nobody complained about. It also tells you nothing about the share of good responses.

    Checkpoint 2 of 9· Check yourself

    A team adds an OpenTelemetry destination in the Console and keeps the default settings. They then point an LLM judge at the exported spans to score answer quality, but the judge has nothing it can grade. What should they change?

    Checkpoint 3 of 9· Check yourself

    The weekly pooled quality score stays flat, yet a tenant that sends about 0.05% of traffic reports much worse answers. Which change to the grading pipeline would show regressions like this and still give a global score that reflects the real traffic mix?

    Checkpoint 4 of 9· Exam question

    Fennel Robotics runs a support assistant for warehouse-automation customers, handling 2M requests a day across 300 tenants. Its quality team has an LLM judge score a uniform 0.5% random sample of traffic and publishes one pooled quality score on a weekly dashboard, which has stayed flat for two months. Two small tenants, each under 0.1% of traffic, report clear regressions after a prompt update that mainly affected their returns-handling requests, yet the dashboard never shows them. Last week's scored sample held only a handful of items from each of those tenants. The judge's grades agree closely with human labels on spot checks, the judge budget is already near its monthly limit, and these tenants' users rarely click thumbs-down. What change best fixes the measurement?

    Sources132

    3.Feedback loops and the proxy-metric trap

    User signals are cheap at volume, and they are proxies for quality, not quality itself. Explicit feedback such as thumbs-down or bug reports surfaces problems you did not anticipate and often correlates with product goals. Implicit measures such as task completion rates come with no extra effort from the user. But users rarely explain why something failed, and relying mainly on users to catch problems means users are the ones who hit them.

    Optimizing a single proxy invites the system to game it. Suppose a team tunes prompts to maximize thumbs-up rate and gets a 12% lift, but audits show answers got longer and more flattering while factual accuracy fell six points. The thumbs-up metric did exactly what it was told. It rewarded what users click on, and nothing in the measurement penalized being wrong. Most use cases need multidimensional evaluation along several success criteria, and a ship decision based on one signal ignores the rest.

    The fix is a change to measurement: make correctness a co-primary metric next to the feedback signal. Score a sample of live responses for factual correctness with an LLM judge, and calibrate that judge against human labels, since model-based graders require calibration with human graders for accuracy. The mechanics of calibration are covered in judge calibration against human labels. Also compare variants with length held constant, so a variant cannot win just by being longer. Three tempting alternatives fail for clear reasons:

    • Adding "stay accurate" to the prompt changes the system under test, not what you measure, so thumbs-up remains the only thing that decides a ship.
    • A token cap limits one way of gaming the metric but still measures nothing about correctness.
    • Abandoning online measurement for the golden set alone throws away ground truth from real usage.

    Feedback should also flow back into the offline suite. A sustainable routine is to triage feedback constantly, sample transcripts to read weekly, and dig deeper as needed. Each confirmed failure becomes a regression case, so the loop closes: failures become test cases, test cases prevent regressions, and metrics replace guesswork.

    Checkpoint 5 of 9· Check yourself

    After a prompt update, the post-chat satisfaction score for a support assistant rises 9%. A manual audit of transcripts finds that answers now agree with customers more often and that more of them quote the wrong policy terms. Before the next ship decision, what should the team change?

    Checkpoint 6 of 9· Exam question

    Vireo Learning runs an AI homework-help assistant in production for its tutoring subscribers. Over two months, the team tuned the system prompt to maximize the thumbs-up rate and saw a 12% lift. A weekly audit of sampled conversations by trained reviewers shows that answers are now longer and more flattering, and factual accuracy fell six points. An earlier prompt revision that told the model to stay accurate did not stop the drift. A trial that hard-limited response length shortened answers, but flattering openers continued, and approval stayed high on answers the reviewers marked wrong. The pre-launch golden set still passes. It was written before launch and has none of the new question types subscribers now ask. Leadership wants live user feedback to stay in the loop. What is the best change to the measurement approach?

    Sources32

    4.A/B tests: randomize the unit you measure

    An A/B test compares a candidate against a baseline model or an earlier version on real traffic. Its advantage over every other online method is that it controls for confounds: random assignment means the only systematic difference between the arms is the change. It measures actual user outcomes such as retention and task completion. The price is time. Reaching significance takes days or weeks and needs enough traffic, so A/B tests are for significant changes, and they only test changes you actually deploy.

    The randomization unit has to match the measurement unit. Satisfaction with a multi-turn assistant is a property of the whole session, but per-request randomization gives one user turns from both prompts. Each arm's satisfaction score then reflects a blend of both variants, and no amount of extra traffic separates their effects. Hashing the user ID gives each user a stable, random assignment that persists across requests, so per-user outcomes belong cleanly to one variant.

    The other common workarounds break randomization in their own ways. Running the old prompt one week and the new one the next mixes the prompt change with anything else that differed between those weeks, such as a release, a holiday or a change in traffic. Dropping sessions that received both prompts keeps only sessions that happened to fall entirely in one arm. Those tend to be shorter sessions, so the remaining population is no longer comparable to real usage.

    Checkpoint 7 of 9· Check yourself

    Two weeks into an A/B test of a new system prompt for a research assistant, the candidate arm shows a statistically significant drop in task completion. The team needs to know what the new prompt is doing differently before they revise it. What is the best next step?

    Checkpoint 8 of 9· Exam question

    Indigo Travel runs a multi-turn coding assistant inside its internal developer portal, where engineers typically work through a task over a dozen or more requests in one session. The team A/B tests a new system prompt against the current one, with each request randomized independently between the two variants. Analysts find that many sessions contain responses from both prompts, and a single end-of-session satisfaction rating therefore cannot be tied to either one. Session-level satisfaction looks noisy and contradictory: the new prompt wins on some days and loses on others. Longer sessions are the most likely to mix variants. The portal serves a fixed pool of engineers, so traffic cannot simply be multiplied, and the team needs results attributable to the prompt rather than to timing or user mix. What should the team change?

    Sources32

    5.Matching the online metric to the task type

    A metric only works if it fits the shape of the output. Extraction and classification have one right answer per input, so exact match is ideal: it is perfect for tasks with clear-cut, categorical answers such as sentiment analysis. The grader is two lines of code and runs on every sampled response at almost no cost.

    An exact-match grader, suited to extraction and classification outputspython
    def evaluate_exact_match(model_output, correct_answer):
        return model_output.strip().lower() == correct_answer.lower()

    Generation tasks, such as drafting an email, have no single reference answer, so exact match is meaningless for them. Use rubric-based or Likert-scale grading by an LLM judge, for example coherence rated from 1 to 5. Use embedding cosine similarity when the question is consistency, because similar questions should yield semantically similar answers even when the wording differs.

    Agentic tasks are judged on what the agent did as well as what it said. Code-based checks can verify the outcome, verify tool calls (which tools were used, with which parameters), and analyse the transcript for turns taken and tokens used. Support agents add business-level online metrics. Escalation accuracy tracks the percentage of correctly escalated conversations. CSAT is usually collected through post-interaction surveys.

    Retrieval-grounded tasks need two kinds of metric, because the system can fail at retrieval or at generation. Precision, recall and mean reciprocal rank (MRR) measure retrieval, and recall matters most because the model can only use what it was given. End-to-end accuracy, graded by an LLM judge against the ground truth, evaluates the entire pipeline from retrieval to answer generation.

    Online metrics by task type
    Task typeTypical outputMetrics that fit
    Extraction / classificationOne correct categorical answerExact match, accuracy, F1
    GenerationFree-form text, no single referenceLLM-graded rubric or Likert scale; cosine similarity for consistency
    AgenticActions, tool calls, end stateOutcome verification, tool-call verification, transcript analysis, escalation accuracy
    Retrieval-groundedAnswer built from retrieved chunksPrecision, recall, MRR for retrieval; LLM-judged end-to-end accuracy

    Checkpoint 9 of 9· Match them up

    A platform team is choosing an online metric for each production workload. Match each workload to the metric that fits the shape of its output.

    Tap a term, then the definition that fits it.

    Sources3245

    Sources

    (5)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “linked to the user or feature that sent it”
      ↩︎ What live traffic shows that a golden set cannot
      “Metadata only (metadata_only in the SDK): The default.”
      ↩︎ Sampling live traffic so small groups stay visible
      “Full content (content in the SDK): Most visibility. Spans include model prompts and responses, and everything in Metadata only.”
      ↩︎ Sampling live traffic so small groups stay visible
      “If your organization has a HIPAA, FERPA, or K-12 data-handling restriction with Anthropic, destinations export only metadata regardless of their content mode.”
      ↩︎ Sampling live traffic so small groups stay visible
    2. 2.
      “Can create false confidence if it doesn’t match real usage patterns”
      ↩︎ What live traffic shows that a golden set cannot
      “automated evals for fast iteration, production monitoring for ground truth, and periodic human review for calibration”
      ↩︎ What live traffic shows that a golden set cannot
      “Users rarely explain why something failed”
      ↩︎ Feedback loops and the proxy-metric trap
      “Requires calibration with human graders for accuracy”
      ↩︎ Feedback loops and the proxy-metric trap
      “triage feedback constantly, sample transcripts to read weekly, and dig deeper as needed”
      ↩︎ Feedback loops and the proxy-metric trap
      “failures become test cases, test cases prevent regressions, and metrics replace guesswork”
      ↩︎ Feedback loops and the proxy-metric trap
      “Measures actual user outcomes (retention, task completion)”
      ↩︎ A/B tests: randomize the unit you measure
      “Slow; days or weeks to reach significance and requires sufficient traffic”
      ↩︎ A/B tests: randomize the unit you measure
      “A/B testing validates significant changes once you have sufficient traffic.”
      ↩︎ A/B tests: randomize the unit you measure
      “Tool calls verification (tools used, parameters)”
      ↩︎ Matching the online metric to the task type
      “Catches issues that synthetic evals miss”
      ↩︎ Key concept
      “Skews toward severe issues”
      ↩︎ Exam trap 4
      “Reactive; problems reach users before you know about them”
      ↩︎ Checkpoint
      “Controls for confounds”
      ↩︎ Prediction
      “Less signal on the underlying “why” for changes in metrics without being able to thoroughly review the transcripts”
      ↩︎ Checkpoint
    3. 3.
      “Design evals that mirror your real-world task distribution.”
      ↩︎ Sampling live traffic so small groups stay visible
      “User feedback: Implicit measures such as task completion rates.”
      ↩︎ Feedback loops and the proxy-metric trap
      “Most use cases need multidimensional evaluation along several success criteria.”
      ↩︎ Feedback loops and the proxy-metric trap
      “A/B testing: Compare performance against a baseline model or earlier version.”
      ↩︎ A/B tests: randomize the unit you measure
      “perfect for tasks with clear-cut, categorical answers such as sentiment analysis”
      ↩︎ Matching the online metric to the task type
      “similar questions should yield semantically similar answers, even if the wording varies”
      ↩︎ Matching the online metric to the task type
      “Design evals that mirror your real-world task distribution.”
      ↩︎ Exam trap 1
      “Most use cases need multidimensional evaluation along several success criteria.”
      ↩︎ Exam trap 2
      “A/B testing: Compare performance against a baseline model or earlier version.”
      ↩︎ Exam trap 3
      “Design evals that mirror your real-world task distribution. Don’t forget to factor in edge cases!”
      ↩︎ Checkpoint
      “It’s ideal for evaluating consistency because similar questions should yield semantically similar answers, even if the wording varies.”
      ↩︎ Checkpoint
    4. 4.
      “Track the percentage of correctly escalated conversations”
      ↩︎ Matching the online metric to the task type
      “Usually done through post-interaction surveys.”
      ↩︎ Matching the online metric to the task type
    5. 5.
      “Recall is crucial for ensuring the LLM has access to all needed information.”
      ↩︎ Matching the online metric to the task type
      “This metric evaluates the entire pipeline, from retrieval to answer generation.”
      ↩︎ Matching the online metric to the task type

    Ready to test yourself?

    Practise Anthropic Claude Certified Developer – Professional (CCDV-P) in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.