CertSafari
    Anthropic Claude Certified Developer – Professional (CCDV-P)· Lessons

    Domain 5 · Lesson 14/25

    Offline evaluation system design for Claude applications

    Evaluation System Design

    18 min read
    4% of exam
    5 sources
    Published 19 Nov 2026
    Docs as of 4 Nov 2026

    What you will be able to do

    • Turn vague quality goals into measurable success criteria and a golden set whose cases match the real task distribution and its edge cases
    • Choose code-based, model-based or human graders for each check, and write behavioural checks as golden answers
    • Find RAG failures by scoring retrieval separately, and measure groundedness by checking each claim against the retrieved passages
    • Tell regression suites from capability suites, and set CI/CD gates and release criteria for prompt, model and agent changes that hold up against run-to-run noise
    • Keep a suite valid by guarding against contamination, saturation and coverage gaps

    Key concept

    Eval suite validity — A score from an offline suite only predicts production quality if the cases match real usage, stay separate from the material used to tune the system, and can still tell a better candidate from a worse one. If any of those breaks, the suite still produces a number, but the number no longer tells you anything.

    1.From success criteria to a golden set

    An offline evaluation system tells you, before anything reaches users, whether a prompt edit, a model upgrade or an agent change made the application better or worse. It starts with success criteria. "The model should classify sentiment well" can't be tested. "F1 of at least 0.85 on a held-out set of 10,000 diverse posts, a 5% improvement over the current baseline" can: it names a metric, a threshold, the data and a target that is achievable.

    Real applications seldom pass or fail on one number. The same sentiment system might also need 99.5% non-toxic outputs, 95% of responses under 200 ms, and most errors to be inconvenient rather than egregious. Each of those becomes a separate measurement in the suite. Qualitative goals can be measured too, using a Likert scale or an expert rubric applied consistently.

    The golden set puts those criteria into practice. Each case has an input (often variables fed into a prompt template), the model's output, a golden answer and a score. The golden answer can be a mandatory exact match, such as the label negative. For open-ended tasks it can instead be a description of a correct answer that a grader compares against. Writing cases is mostly a one-time cost. Grading is paid on every run, so a design that grades cheaply pays off for as long as the suite is used.

    Checkpoint 1 of 9· Check yourself

    A team is writing success criteria for a support-ticket classifier before building its offline golden set. Which criterion can the suite actually test?

    Sources12

    2.Case coverage and behavioural checks

    A golden set is only as good as its coverage. Its cases should follow the real-world mix of questions and difficulty levels, plus deliberate edge cases. Typical edge cases are irrelevant or missing input, overly long input, poor or harmful user input, and ambiguous inputs where even human raters would disagree. The sentiment set below labels sarcasm and mixed sentiment as edge cases next to the ordinary ones.

    Golden-set cases with labelled answers, including flagged edge casespython
    tweets = [
        {"text": "This movie was a total waste of time. 👎", "sentiment": "negative"},
        {"text": "The new album is 🔥! Been on repeat all day.", "sentiment": "positive"},
        {
            "text": "I just love it when my flight gets delayed for 5 hours. #bestdayever",
            "sentiment": "negative",
        },  # Edge case: Sarcasm
        {
            "text": "The movie's plot was terrible, but the acting was phenomenal.",
            "sentiment": "mixed",
        },  # Edge case: Mixed sentiment
        # ... 996 more tweets
    ]

    Behavioural checks test what the application should *do*, not whether it gets a fact right. Take an assistant with no email tool. The prompt "Send Jane an email asking her to meet me at 9am" gets a golden answer that requires a refusal. Drafting the email is acceptable. Pretending to send it, calling a send function, or asking which address to use all fail. Refusals, privacy rules ("can it follow instructions not to share certain details?") and correct use of the provided context are behaviours. Each needs its own cases, because accuracy cases never exercise them.

    Checkpoint 2 of 9· Fill the gap

    This golden set flags sarcasm as an edge case. Which golden label must go in the blank so the case tests the tweet's real sentiment rather than its surface wording?

    tweets = [
        {"text": "This movie was a total waste of time. 👎", "sentiment": "negative"},
        {"text": "The new album is 🔥! Been on repeat all day.", "sentiment": "positive"},
        {
            "text": "I just love it when my flight gets delayed for 5 hours. #bestdayever",
            "sentiment": " ? ",
        },  # Edge case: Sarcasm
        {
            "text": "The movie's plot was terrible, but the acting was phenomenal.",
            "sentiment": "mixed",
        },  # Edge case: Mixed sentiment
        # ... 996 more tweets
    ]

    Sources12

    3.Choosing graders that can run on every change

    A suite that gates releases has to be graded quickly, cheaply and the same way every time, so automate wherever you can. Code-based grading (exact match, regex, key-phrase checks, tool-call parameter checks) is fast and reproducible, but it breaks when a valid answer doesn't match the expected pattern. Model-based grading handles open-ended output, but it is non-deterministic and has to be calibrated against human graders. Human grading is the most capable and the slowest. Use it to calibrate the other two rather than to gate every commit.

    A code-based exact-match grader: normalise case and whitespace, then comparepython
    def evaluate_exact_match(model_output, correct_answer):
        return model_output.strip().lower() == correct_answer.lower()

    Good design often turns a task that needs a judge into one that code can grade. Reformatting questions as multiple choice is the usual way. Volume matters more than polish: many automatically graded cases with slightly weaker signal beat a few hand-graded ones. When you do need model grading, give each rubric dimension its own isolated judge. Building and calibrating those judges is covered in LLM-as-judge engineering.

    Grader types and where each one fits in an offline suite
    GraderStrengthsWeaknessesBest use
    Code-basedFast, cheap, objective, reproducibleBrittle to valid variations, lacks nuanceLabels, formats, tool calls, outcome checks
    Model-basedFlexible, scalable, handles freeform outputNon-deterministic, needs calibration with humansTone, open-ended answers, rubric dimensions
    HumanGold-standard qualityExpensive, slowCalibrating model graders, subjective outputs

    Checkpoint 3 of 9· Match them up

    Match each grading need in a release-gating suite to the grader that fits it best.

    Tap a term, then the definition that fits it.

    Sources13

    4.Retrieval precision and groundedness in RAG suites

    A wrong RAG answer has two possible causes: the retriever never surfaced the right passage, or the generator mishandled a passage it did receive. End-to-end accuracy can't tell these apart. So label the relevant chunks for each question and score the retriever on its own. Precision is the share of retrieved chunks that are relevant. Recall is the share of all relevant chunks that the top-k results contain. Mean reciprocal rank shows how high the first relevant chunk ranks. If recall is low, no change to the generator will help. A bigger model, more max_tokens, extended thinking or a stricter "answer only from passages" instruction all work on evidence the model never received.

    Groundedness asks a different question: is every statement in the answer supported by the retrieved passages? Similarity metrics don't answer it. Cosine similarity between embeddings is a consistency metric: it checks that paraphrased questions get semantically similar answers. ROUGE-L measures overlap with a reference summary. An answer can score high on both and still assert a fact that no passage contains, because one invented sentence barely changes the embedding or the n-gram overlap.

    Measure groundedness at the level of individual claims instead. Split the answer into single statements, look for a supporting passage for each one, and report the fraction that are supported. This uses the same idea as the hallucination-reduction technique of finding a supporting quote for each claim and retracting any claim that has none. A single 1-to-10 faithfulness score from the generating model hides which statement lacks support. Claim-level checks are also how you grade the cases a RAG suite most often leaves out: questions the knowledge base can't answer. For those, the golden answer is an admission of uncertainty, not an answer.

    Which RAG metric answers which question
    MetricQuestion it answersBlind spot
    Retrieval precisionOf the chunks retrieved, how many were relevant?Says nothing about what the generator did
    Retrieval recallDid the needed chunks get retrieved at all?Says nothing about what the generator did
    End-to-end accuracyWas the final answer correct?Cannot separate retrieval from generation failures
    Claim-level groundednessIs each statement supported by the passages?Needs a reliable claim-splitting and support check

    Checkpoint 4 of 9· Check yourself

    Your RAG assistant's answers average 0.93 embedding cosine similarity to the passages it retrieved, and a stakeholder wants to call the answers grounded. Which measurement actually shows whether they are grounded?

    Checkpoint 5 of 9· Exam question

    Pebble Robotics runs a RAG assistant that answers employees' internal policy questions, such as expense limits and leave rules, from a vector index of about 3,000 policy documents. Its eval suite of 400 labeled questions shows wrong answers on 14% of items. Reviewing a sample of failures, the team finds some answers contradict the policy and others are fluent but cite nothing the policy says. Nobody has yet inspected which passages were returned for the failing questions. The generator already runs on the largest model the budget allows, and the team plans to rewrite the generation prompt. The eval set includes a labeled list of the policy passages that contain each correct answer. Which step should come first to localise the failure?

    Checkpoint 6 of 9· Exam question

    Juniper Games runs a RAG-based support assistant that answers player questions about game rules and account issues from retrieved help-center passages. The team measures groundedness by computing embedding similarity between each generated answer and the human-written reference answer, then averaging across the test suite. The scores are consistently high and have not moved across several releases. However, human reviewers sampling production transcripts keep finding answers that are fluent and on-topic but include a refund window, an item limit, or a policy detail that appears nowhere in the retrieved passages. Reviewers can point to the specific sentences at fault, but the aggregate score never reflects them. The team wants an automated metric that reproduces what the reviewers see and can run on every release. What is the better metric design?

    Sources415

    5.Regression suites, CI/CD gates and release criteria

    Offline suites do two different jobs, and confusing them makes a gate meaningless. A capability suite asks what the system can do well. It should start at a low pass rate so the team has room to improve. A regression suite asks whether the system still handles everything it used to, and should pass at nearly 100%. Any drop means something broke. When capability tasks reach high pass rates after launch, they graduate into the regression suite: "can we do this at all?" becomes "can we still do this reliably?"

    Capability and regression suites compared
    AspectCapability suiteRegression suite
    QuestionWhat can the system do well?Does it still handle what it used to?
    Expected pass rateStarts lowNearly 100%
    A falling score meansLess progress on hard tasksSomething is broken
    Role in CI/CDTracks improvementBlocks backsliding

    Automated suites belong in CI/CD as the first line of defence. A CI/CD gate runs them on every agent change and model upgrade. Each release criterion bundles several success criteria, each with its own threshold. Scoring can be weighted (combined scores must reach a threshold), binary (every grader must pass) or a hybrid. The trigger shapes what the gate checks:

    A prompt change has to clear the regression suite, and its claimed improvement has to show on the capability suite. A model change runs the whole bank: a team with evals can measure a new model's strengths and upgrade in days, while a team without them spends weeks testing. An agent change must be graded on the outcome as well as the transcript. A flight-booking agent that says "Your flight has been booked" passes only if the reservation exists in the environment's database. The bank also gives you latency, token usage and cost per task as baselines at no extra cost.

    Every gate has to account for nondeterminism. Outputs vary between runs, so a single run can't separate a real regression from noise. Suppose a prompt moves a 150-case suite from 82% to 84%, but reruns of the unchanged prompt already vary by about 3 points. That 2-point gain is inside the noise, so it doesn't justify shipping. Run multiple trials per task, or add cases, until the difference is larger than the variation between runs.

    Checkpoint 7 of 9· Check yourself

    After one run, a revised prompt lifts a 300-case capability suite from 71% to 73%. Reruns of the unchanged prompt already move between about 70% and 73%. What should the CI/CD gate require before the change ships?

    Sources13

    6.Keeping the suite valid: contamination, saturation, coverage gaps

    Contamination happens when the cases that grade the system leak into what builds it, such as few-shot examples, prompt wording or tuning loops. The score then rises because the system has seen the answers, not because it has improved. Release criteria are measured on a held-out set for this reason. Keep the gating split away from prompt authors, take examples from a separate pool, and rotate the split over time. Graders need the same protection: passing should require actually solving the task, not exploiting a loophole.

    Saturation affects capability suites. A suite at 100% still catches regressions but gives no signal about improvement. As it nears saturation, large real gains show up as small score changes. The code-review company Qodo underrated a new model because its one-shot evals missed gains on longer tasks, and it built an agentic eval to replace them. If a suite passes 98% on every candidate, add harder and longer tasks so it can tell candidates apart again.

    Coverage gaps appear as the product and its traffic change and the suite drifts away from real usage. A suite is a living artifact that needs a clear owner. Turn production incidents, such as an agent quoting stale pricing, into new cases, and let the people closest to users contribute tasks, even as pull requests. Read transcripts before trusting any score. A failure should be fair: the agent made a real mistake, not the grader rejecting a valid answer. One benchmark rose from 42% to 95% once its rigid grading and ambiguous tasks were fixed.

    Checkpoint 8 of 9· Match them up

    Each symptom below comes from a suite that has stopped measuring what it should. Pair each symptom with the fix that restores the suite's validity.

    Tap a term, then the definition that fits it.

    Checkpoint 9 of 9· Exam question

    Indigo Freight runs a customer support bot that answers shipment-delay and customs-hold tickets. To improve the prompt, an engineer copied the best-performing few-shot examples directly from the team's golden set, the same cases that CI scores on every prompt change. Over the next release, the golden-set pass rate rose from 81% to 96%. Meanwhile, the weekly review of live tickets by human agents shows resolution quality is flat, and the cases the bot fails in production look unlike anything in the golden set. The team lead wants a change that makes the CI score a trustworthy signal of real resolution quality again, not just a gate that is easier or harder to pass. What is the best fix to restore the suite's validity?

    Sources13

    Sources

    (5)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “The sentiment analysis model should achieve an F1 score of at least 0.85 (Measurable, Specific) on a held-out test set”
      ↩︎ From success criteria to a golden set
      “Measurable: Use quantitative metrics or well-defined qualitative scales.”
      ↩︎ From success criteria to a golden set
      “Design evals that mirror your real-world task distribution. Don’t forget to factor in edge cases!”
      ↩︎ Case coverage and behavioural checks
      “Ambiguous test cases where even humans would find it hard to reach an assessment consensus”
      ↩︎ Case coverage and behavioural checks
      “Structure questions to allow for automated grading (for example, multiple-choice, string match, code-graded, LLM-graded).”
      ↩︎ Choosing graders that can run on every change
      “More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals.”
      ↩︎ Choosing graders that can run on every change
      “It’s ideal for evaluating consistency because similar questions should yield semantically similar answers, even if the wording varies.”
      ↩︎ Retrieval precision and groundedness in RAG suites
      “Most use cases need multidimensional evaluation along several success criteria.”
      ↩︎ Regression suites, CI/CD gates and release criteria
      “On a held-out test set of 10,000 diverse Twitter posts, the sentiment analysis model should achieve”
      ↩︎ Keeping the suite valid: contamination, saturation, coverage gaps
      “should achieve an F1 score of at least 0.85 (Measurable, Specific) on a held-out test set* of 10,000 diverse Twitter posts”
      ↩︎ Checkpoint
    2. 2.
      “The golden answer could be a mandatory exact match, or it could be an example of a perfect answer”
      ↩︎ From success criteria to a golden set
      “Grading on the other hand is a cost you will incur every time you re-run your eval, in perpetuity”
      ↩︎ From success criteria to a golden set
      “A correct answer should decline to send the email since the assistant has no capabilities to send emails.”
      ↩︎ Case coverage and behavioural checks
    3. 3.
      “grade each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions”
      ↩︎ Choosing graders that can run on every change
      “Regression evals ask, “Does the agent still handle all the tasks it used to?” and should have a nearly 100% pass rate.”
      ↩︎ Regression suites, CI/CD gates and release criteria
      “capability evals with high pass rates can “graduate” to become a regression suite that is run continuously to catch any drift”
      ↩︎ Regression suites, CI/CD gates and release criteria
      “running on each agent change and model upgrade as the first line of defense against quality problems”
      ↩︎ Regression suites, CI/CD gates and release criteria
      “scoring can be weighted (combined grader scores must hit a threshold), binary (all graders must pass), or a hybrid”
      ↩︎ Regression suites, CI/CD gates and release criteria
      “the outcome is whether a reservation exists in the environment’s SQL database”
      ↩︎ Regression suites, CI/CD gates and release criteria
      “Teams can't distinguish real regressions from noise”
      ↩︎ Regression suites, CI/CD gates and release criteria
      “Make your graders resistant to bypasses or hacks.”
      ↩︎ Keeping the suite valid: contamination, saturation, coverage gaps
      “This can make results deceptive, as large capability improvements appear as small increases in scores.”
      ↩︎ Keeping the suite valid: contamination, saturation, coverage gaps
      “An eval suite is a living artifact that needs ongoing attention and clear ownership to remain useful.”
      ↩︎ Keeping the suite valid: contamination, saturation, coverage gaps
      “product managers, customer success managers, or salespeople can use Claude Code to contribute an eval task as a PR”
      ↩︎ Keeping the suite valid: contamination, saturation, coverage gaps
      “Can create false confidence if it doesn’t match real usage patterns”
      ↩︎ Key concept
      “Because model outputs vary between runs, we run multiple trials to produce more consistent results.”
      ↩︎ Exam trap 3
      “An eval at 100% tracks regressions but provides no signal for improvement.”
      ↩︎ Exam trap 4
      “Reserve systematic human studies for calibrating LLM graders or evaluating subjective outputs where human consensus serves as the reference standard.”
      ↩︎ Checkpoint
      “Because model outputs vary between runs, we run multiple trials to produce more consistent results.”
      ↩︎ Checkpoint
      “An eval at 100% tracks regressions but provides no signal for improvement.”
      ↩︎ Checkpoint
    4. 4.
      “Precision represents the proportion of retrieved chunks that are actually relevant.”
      ↩︎ Retrieval precision and groundedness in RAG suites
      “Recall measures the completeness of our retrieval system.”
      ↩︎ Exam trap 1
      “show you how to measure the retrieval pipeine & end to end performance independently”
      ↩︎ Prediction
    5. 5.
      “You can also have Claude verify each claim by finding a supporting quote after it generates a response.”
      ↩︎ Retrieval precision and groundedness in RAG suites
      “Allow Claude to say “I don’t know”: Explicitly give Claude permission to admit uncertainty.”
      ↩︎ Retrieval precision and groundedness in RAG suites
      “You can also have Claude verify each claim by finding a supporting quote after it generates a response.”
      ↩︎ Exam trap 2

    Ready to test yourself?

    Practise Anthropic Claude Certified Developer – Professional (CCDV-P) in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.