CertSafari
    Anthropic Claude Certified Architect – Professional (CCAR-P) - December 7 onwards· Lessons

    Domain 4 · Lesson 24/39

    Comparing two Claude configurations: trials, cases and run-to-run noise

    Experimental Design for Configuration Comparison

    16 min read
    2.43% of exam
    10 sources
    Published 18 Nov 2026
    Docs as of 18 Nov 2026

    What you will be able to do

    • Explain why one run per case cannot separate a configuration effect from run-to-run variance, even with temperature at 0
    • Design a paired comparison that changes one variable and runs both configurations on identical inputs
    • Split a fixed call or dollar budget between more cases and more trials per case, using a pilot to decide
    • Judge whether a measured gap between two configurations exceeds run-to-run noise in both pass rate and cost

    Key concept

    Run-to-run variance — The same configuration can give different outputs and scores when it runs the same case again. A difference between two configurations means something only when it is larger than this spread, so the design has to measure the spread as well as the difference.

    1.Why one run per case cannot separate two configurations

    Comparing two prompts or two models means attributing a score difference to the change you made. That only works if running the same configuration twice would give the same score. It doesn't. Even with temperature set to 0, identical inputs might produce different outputs across API calls, both on Anthropic's first-party service and through third-party cloud providers. A case that passes once under prompt A and fails once under prompt B might have gone the other way on a second run.

    Each attempt at a case is called a trial. Repeated trials exist to measure one thing: how much a single case's score moves when nothing has changed. With that spread in hand, you can judge whether the gap between configurations is bigger than it. Repeats don't add coverage of rare inputs, because every rerun uses the same input. Finding missing input types is a coverage gaps problem. Repeats also don't remove a judge's positional preference, because grading the same completion again with the same judge repeats the same bias. That calls for the controls taught in LLM-as-judge design.

    Turning temperature down is not the answer. It doesn't remove the variance, and current Claude models no longer accept the temperature parameter. An older recorded comparison across temperature settings has to be re-run on current models without that axis. Prompt caching doesn't remove the variance either. Caching lowers the cost of a repeated prefix, but it has no effect on output token generation. A repeat call gets a fresh completion, the same as it would without caching, so a difference between two runs never means a cache miss.

    Sampling isn't the only source of noise in an end-to-end score:

    • An LLM grader is itself a model call. Its verdict on a borderline output can change between gradings, which is why eval reports show the judge's individual votes.
    • An agent takes a multi-step path of tool calls, so an early difference in one call can send the rest of the trajectory somewhere else.
    • The infrastructure also counts. Anthropic has seen agentic pass rates fluctuate with time of day, likely because API latency varies.

    Agentic evals are end-to-end system tests, so every component adds variance that fixed sampling settings cannot reach.

    Checkpoint 1 of 9· Check yourself

    A team compares prompt A and prompt B on 40 cases, running each case once per prompt with prompt caching enabled. Prompt B passes 3 more cases. What should they do before they credit the gain to prompt B?

    Checkpoint 2 of 9· Exam question

    An engineer reports that, with sampling temperature set to 0, one run per case is enough to compare two Claude configurations. A reviewer disagrees and wants repeated trials. Which two points support the reviewer? Choose two.(Select 2)

    0 of 2 selected

    Checkpoint 3 of 9· Exam question

    A team compares two prompt configurations on the Claude Messages API, and the model's completions vary between calls. Why does a sound design run every test case several times for each configuration instead of once?

    Sources123456

    2.Change one variable and run both arms on identical inputs

    A comparison can be read only if the configuration is the one thing that differs. Anthropic's infrastructure-noise study shows the discipline. To measure the effect of container resources, it held everything else constant: same Claude model, same harness, same task set. Resource limits alone moved Terminal-Bench 2.0 by 6 percentage points between the tightest and the uncapped setup. That is more than the gap between top models on public leaderboards. If you change a prompt and the sandbox at the same time, you can't say which one moved the score.

    Confounders hide in places you don't think of as configuration. A Claude Code plugin eval excludes your user settings, hooks, CLAUDE.md files and other plugins, but managed policy still applies. Results on a managed machine can therefore differ from those on an unmanaged one. Run both arms on the same machine, in the same window, and under the same policy. For agentic evals whose results will be shared, running at multiple times and on multiple days helps average out noise that comes from time of day.

    Run both configurations on the same cases and compare them case by case. This is a paired design. Cases differ a great deal in difficulty, and pairing removes that difference from the comparison. A hard case drags down both arms equally, so its per-case difference still shows only the effect of the change. The plugin eval runner reports exactly this: each case has a delta (with-arm score minus without-arm score), and the suite reports the mean of those deltas. One fixed test set used for every variant is what makes a comparison fair, as in the Promptfoo run that scored Simple, RAG and RAG with chain-of-thought on the same 68 examples.

    Checkpoint 4 of 9· Check yourself

    An engineer reruns an agentic eval with prompt B on a cluster with uncapped container memory. The prompt A baseline ran last month under strict 1x resource limits. Both used the same model and task set, and prompt B scores 5 points higher. What is the sound conclusion?

    Sources654

    3.Splitting a fixed budget between cases and trials

    A budget fixes the product of cases × trials × configurations, so every extra trial per case costs you cases. When you have no other information, Anthropic's cost-optimization guidance gives a default: around fifty eval cases and at least five trials per configuration before making a production decision. Grading is a cost paid on every re-run, so the budget has to cover the graders as well as the model calls under test. Judge models are part of the bill.

    A CI eval run with a hard dollar ceiling; hitting it ends the run as partial (exit code 2) rather than finishing the suitebash
    claude plugin eval . \
      --trust-plugin \
      --json results.json \
      --threshold 0.8 \
      --model claude-sonnet-5 \
      --judge-model claude-haiku-4-5 \
      --no-publish \
      --max-cost-usd 20

    Size the ceiling for the whole design before you start. A run that hits it is partial: some cases or arms are missing, and partial results should be left out of trend charts. A ceiling that stops one configuration halfway leaves the two arms unpaired.

    The right number of trials depends on how much a single case actually varies, and a small pilot measures that. Run a few dozen cases several times each and look at the per-case pass counts. Mixed results like 3/5 or 2/5 mean within-case variance is large, so you need several trials per case to estimate each case's pass rate. Mostly 5/5 or 0/5 means individual cases barely vary. In that pilot most of the uncertainty is between cases, and extra trials of a case whose outcome you can already predict tell you almost nothing. Put that budget into more distinct cases, run each once per configuration on identical inputs, and compare the paired per-case differences.

    When you do run several trials, score each case by the mean across them. The plugin eval runner's per-case score is defined as the mean with-arm run score for that case. A best-of-three rule counts a case as passing if any trial passed. That rewards a configuration for being inconsistent and hides the run-to-run variance the trials were meant to expose.

    Checkpoint 5 of 9· Fill the gap

    This CI run has to stop before the comparison overspends. The ceiling covers the whole run, judge calls included, so it has to be sized for both arms. Otherwise it can stop one configuration halfway and leave the arms unpaired. Which flag completes the command?

    claude plugin eval . \
      --trust-plugin \
      --json results.json \
      --threshold 0.8 \
      --model claude-sonnet-5 \
      --judge-model claude-haiku-4-5 \
      --no-publish \
       ?  20

    Checkpoint 6 of 9· Check yourself

    Your suite runs each of 50 cases three times under prompt A and three times under prompt B. On many cases prompt B goes 1/3, while prompt A goes 3/3 or 0/3. How should each case be scored so the comparison between the prompts can be read?

    Checkpoint 7 of 9· Exam question

    You must compare two configurations with a budget of 1,200 model calls, 600 per configuration. You have 600 labelled cases. A pilot of 40 cases at 5 trials each showed most cases passing 5/5 or failing 0/5, with wide differences between cases. Which allocation gives the most interpretable comparison?

    Sources785

    4.Reading a gap against run-to-run noise

    A gap of a point or two is often noise. In that GPQA Diamond comparison, neither executor ever called the advisor, so every difference between the arms was variance. Small gaps between configurations are the ones that need a measured noise floor before anyone acts on them.

    Repeated trials go to the comparison the decision depends on. Anthropic's internal agentic-coding benchmark ran five attempts per task for the advisor pairing and its Claude Opus 5 control, and one attempt for the other points on the chart. Use the same pattern when the budget is tight. Run single passes to place most configurations, then spend repeated trials on the two candidates you will actually choose between.

    Cost varies from run to run as well, so compare it the same way you compare pass rate. In the cost-optimization cookbook, two trials of the same batched configuration both passed 9 of 10 claims and missed the same claim. One trial cost about three times as much per task as the other, because cache hits inside a batch are best effort when requests run concurrently. A single trial could have reported either cost. Measure cost per completed task, run each configuration more than once, and read cost and pass rate together.

    Two trials of one configuration: same pass rate, very different cost
    configtrialpass/n$ / task$ / 10k tasksmisses
    sonnet · medium · batch (one-shot)19/10$0.0060$60CLM-009
    sonnet · medium · batch (one-shot)29/10$0.0194$194CLM-009

    Set the bar before you compare. Decide the pass rate you need from baseline runs, treat it as a constraint, and then see which candidate meets it at lower cost. That turns an A/B test against a baseline or earlier version into a decision: a gap counts when it is larger than the run-to-run spread and moves a configuration across the bar.

    Checkpoint 8 of 9· Check yourself

    Six candidate prompts are in play, and the budget covers only one pass of every case for each of them plus a few extra rounds. The decision is which single prompt goes to production. How should the extra trials be spent?

    Checkpoint 9 of 9· Check yourself

    In one trial, a batched configuration passes 9 of 10 cases, which meets the bar, at half the cost per task of the current configuration. How should that cost figure be treated?

    Sources9710

    Sources

    (10)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “Even with temperature set to 0, the results will not be fully deterministic, and identical inputs might produce different outputs across API calls.”
      ↩︎ Why one run per case cannot separate two configurations
      “Even with temperature set to 0, the results will not be fully deterministic, and identical inputs might produce different outputs across API calls.”
      ↩︎ Key concept
      “Even with temperature set to 0, the results will not be fully deterministic, and identical inputs might produce different outputs across API calls.”
      ↩︎ Exam trap 1
    2. 2.
      “Prompt caching has no effect on output token generation.”
      ↩︎ Why one run per case cannot separate two configurations
      “Prompt caching has no effect on output token generation.”
      ↩︎ Exam trap 2
    3. 3.
      “Because model outputs vary between runs, we run multiple trials to produce more consistent results.”
      ↩︎ Why one run per case cannot separate two configurations
      “Each attempt at a task is a trial. Because model outputs vary between runs, we run multiple trials to produce more consistent results.”
      ↩︎ Checkpoint
    4. 4.
      “current models no longer accept temperature”
      ↩︎ Why one run per case cannot separate two configurations
      “The same test set (68 examples): Ensuring fair comparison across all configurations”
      ↩︎ Change one variable and run both arms on identical inputs
    5. 5.
      “an llm grader also shows the judge’s votes and the evidence it was shown”
      ↩︎ Why one run per case cannot separate two configurations
      “results on a managed machine can differ from an unmanaged one by that policy”
      ↩︎ Change one variable and run both arms on identical inputs
      “With-arm score minus without-arm score”
      ↩︎ Change one variable and run both arms on identical inputs
      “Partial run: the --max-cost-usd ceiling was hit”
      ↩︎ Splitting a fixed budget between cases and trials
      “Leave partial results out of trend charts”
      ↩︎ Splitting a fixed budget between cases and trials
      “Mean with-arm run score for the case”
      ↩︎ Splitting a fixed budget between cases and trials
      “Mean with-arm run score for the case”
      ↩︎ Exam trap 3
    6. 6.
      “any component of that system can act as a confounder”
      ↩︎ Why one run per case cannot separate two configurations
      “Everything else stayed constant: same Claude model, same harness, same task set.”
      ↩︎ Change one variable and run both arms on identical inputs
      “running at multiple times and on multiple days would help average out the noise”
      ↩︎ Change one variable and run both arms on identical inputs
    7. 7.
      “you should aim for around fifty eval cases and at least five trials per configuration to minimize nondeterministic variance”
      ↩︎ Splitting a fixed budget between cases and trials
      “cache hits inside a batch are best effort because the requests run concurrently”
      ↩︎ Reading a gap against run-to-run noise
      “Roll usage up per task, run each configuration more than once, and read cost and pass rate together.”
      ↩︎ Reading a gap against run-to-run noise
    8. 8.
      “Grading on the other hand is a cost you will incur every time you re-run your eval, in perpetuity”
      ↩︎ Splitting a fixed budget between cases and trials
    9. 9.
      “The pairings scored 0.5 points (Claude Haiku 5.5) and 1.0 points (Claude Sonnet 5.5) above the executor alone, both within run-to-run noise.”
      ↩︎ Reading a gap against run-to-run noise
      “The August runs had five attempts per task for the pairing and the Claude Opus 5 control and one for the other points.”
      ↩︎ Reading a gap against run-to-run noise
      “The pairings scored 0.5 points (Claude Haiku 5.5) and 1.0 points (Claude Sonnet 5.5) above the executor alone, both within run-to-run noise.”
      ↩︎ Exam trap 4
    10. 10.
      “A/B testing: Compare performance against a baseline model or earlier version.”
      ↩︎ Reading a gap against run-to-run noise

    Ready to test yourself?

    Practise Anthropic Claude Certified Architect – Professional (CCAR-P) - December 7 onwards in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.