What you will be able to do
- Explain why one run per case cannot separate a configuration effect from run-to-run variance, even with temperature at 0
- Design a paired comparison that changes one variable and runs both configurations on identical inputs
- Split a fixed call or dollar budget between more cases and more trials per case, using a pilot to decide
- Judge whether a measured gap between two configurations exceeds run-to-run noise in both pass rate and cost
Key concept
Run-to-run variance — The same configuration can give different outputs and scores when it runs the same case again. A difference between two configurations means something only when it is larger than this spread, so the design has to measure the spread as well as the difference.
1.Why one run per case cannot separate two configurations
Comparing two prompts or two models means attributing a score difference to the change you made. That only works if running the same configuration twice would give the same score. It doesn't. Even with temperature set to 0, identical inputs might produce different outputs across API calls, both on Anthropic's first-party service and through third-party cloud providers. A case that passes once under prompt A and fails once under prompt B might have gone the other way on a second run.
Each attempt at a case is called a trial. Repeated trials exist to measure one thing: how much a single case's score moves when nothing has changed. With that spread in hand, you can judge whether the gap between configurations is bigger than it. Repeats don't add coverage of rare inputs, because every rerun uses the same input. Finding missing input types is a coverage gaps problem. Repeats also don't remove a judge's positional preference, because grading the same completion again with the same judge repeats the same bias. That calls for the controls taught in LLM-as-judge design.
Turning temperature down is not the answer. It doesn't remove the variance, and current Claude models no longer accept the temperature parameter. An older recorded comparison across temperature settings has to be re-run on current models without that axis. Prompt caching doesn't remove the variance either. Caching lowers the cost of a repeated prefix, but it has no effect on output token generation. A repeat call gets a fresh completion, the same as it would without caching, so a difference between two runs never means a cache miss.
Sampling isn't the only source of noise in an end-to-end score:
- An LLM grader is itself a model call. Its verdict on a borderline output can change between gradings, which is why eval reports show the judge's individual votes.
- An agent takes a multi-step path of tool calls, so an early difference in one call can send the rest of the trajectory somewhere else.
- The infrastructure also counts. Anthropic has seen agentic pass rates fluctuate with time of day, likely because API latency varies.
Agentic evals are end-to-end system tests, so every component adds variance that fixed sampling settings cannot reach.
Checkpoint 1 of 9· Check yourself
A team compares prompt A and prompt B on 40 cases, running each case once per prompt with prompt caching enabled. Prompt B passes 3 more cases. What should they do before they credit the gain to prompt B?
Model outputs vary between runs, so a single pass or fail per case can't show whether the gap comes from the prompt or from run-to-run variance. Repeated trials measure that spread. Caching doesn't replay completions, and grading again with the same judge repeats the same bias.
“Each attempt at a task is a trial. Because model outputs vary between runs, we run multiple trials to produce more consistent results.”Source: www.anthropic.com
Checkpoint 2 of 9· Exam question
An engineer reports that, with sampling temperature set to 0, one run per case is enough to compare two Claude configurations. A reviewer disagrees and wants repeated trials. Which two points support the reviewer? Choose two.(Select 2)
Correct answers: A, B — Anthropic documents that even at temperature 0.0 results are not fully deterministic, so repeated calls can still differ on the same input.; Variation can also enter through an LLM grader or through multi-step agent paths, so end-to-end scores can shift even with fixed settings.
- A. Correct. Temperature 0 reduces but does not eliminate output variation, so single-run scores still contain noise that has to be quantified.
- B. Correct. Judge calls and agent trajectories are additional sources of variance, so a fixed generation setting does not make the pipeline's score stable.
- C. Incorrect. Prompt caching reuses processed prompt prefixes to save cost and latency; completions are still generated on every call.
- D. Incorrect. The Claude Messages API does not provide a seed parameter for pinning generation, so reproducibility cannot be restored that way.
- E. Incorrect. A model grader adds its own variance to the scores and does not cancel the noise from generation, so repeated trials remain relevant for it.
Checkpoint 3 of 9· Exam question
A team compares two prompt configurations on the Claude Messages API, and the model's completions vary between calls. Why does a sound design run every test case several times for each configuration instead of once?
Correct answer: A — To estimate how much one case's score varies across runs, because a gap between the two prompts has to be read against that run-to-run spread.
- A. Correct. Because outputs differ from call to call, sampling the same input several times shows how far a score moves by chance alone. That spread is the baseline for deciding whether a difference between two prompts is real.
- B. Incorrect. A repeated run of a case uses the same input, so it adds no new kind of input. Reaching rare input types takes additional, distinct cases that are written or sourced, not repeats of existing ones.
- C. Incorrect. Positional bias comes from the order in which a judge sees candidates. Grading the same completion again in the same arrangement reproduces that bias instead of removing it. Varying the order is what addresses it.
- D. Incorrect. Prompt caching reuses the processed prompt prefix to save cost and latency, and it does not store or replay completions. Repeat calls still generate fresh output, which is exactly why the results vary.
2.Change one variable and run both arms on identical inputs
A comparison can be read only if the configuration is the one thing that differs. Anthropic's infrastructure-noise study shows the discipline. To measure the effect of container resources, it held everything else constant: same Claude model, same harness, same task set. Resource limits alone moved Terminal-Bench 2.0 by 6 percentage points between the tightest and the uncapped setup. That is more than the gap between top models on public leaderboards. If you change a prompt and the sandbox at the same time, you can't say which one moved the score.
Confounders hide in places you don't think of as configuration. A Claude Code plugin eval excludes your user settings, hooks, CLAUDE.md files and other plugins, but managed policy still applies. Results on a managed machine can therefore differ from those on an unmanaged one. Run both arms on the same machine, in the same window, and under the same policy. For agentic evals whose results will be shared, running at multiple times and on multiple days helps average out noise that comes from time of day.
Run both configurations on the same cases and compare them case by case. This is a paired design. Cases differ a great deal in difficulty, and pairing removes that difference from the comparison. A hard case drags down both arms equally, so its per-case difference still shows only the effect of the change. The plugin eval runner reports exactly this: each case has a delta (with-arm score minus without-arm score), and the suite reports the mean of those deltas. One fixed test set used for every variant is what makes a comparison fair, as in the Promptfoo run that scored Simple, RAG and RAG with chain-of-thought on the same 68 examples.
Checkpoint 4 of 9· Check yourself
An engineer reruns an agentic eval with prompt B on a cluster with uncapped container memory. The prompt A baseline ran last month under strict 1x resource limits. Both used the same model and task set, and prompt B scores 5 points higher. What is the sound conclusion?
The prompt and the container resources both changed between the two arms, so there is no way to tell which change moved the score. In Anthropic's study, resource limits alone moved Terminal-Bench 2.0 by 6 points while the model, harness and task set stayed the same. Only an experiment that holds everything except the prompt constant can isolate the prompt's effect.
“Everything else stayed constant: same Claude model, same harness, same task set.”Source: www.anthropic.com
3.Splitting a fixed budget between cases and trials
A budget fixes the product of cases × trials × configurations, so every extra trial per case costs you cases. When you have no other information, Anthropic's cost-optimization guidance gives a default: around fifty eval cases and at least five trials per configuration before making a production decision. Grading is a cost paid on every re-run, so the budget has to cover the graders as well as the model calls under test. Judge models are part of the bill.
claude plugin eval . \
--trust-plugin \
--json results.json \
--threshold 0.8 \
--model claude-sonnet-5 \
--judge-model claude-haiku-4-5 \
--no-publish \
--max-cost-usd 20Size the ceiling for the whole design before you start. A run that hits it is partial: some cases or arms are missing, and partial results should be left out of trend charts. A ceiling that stops one configuration halfway leaves the two arms unpaired.
The right number of trials depends on how much a single case actually varies, and a small pilot measures that. Run a few dozen cases several times each and look at the per-case pass counts. Mixed results like 3/5 or 2/5 mean within-case variance is large, so you need several trials per case to estimate each case's pass rate. Mostly 5/5 or 0/5 means individual cases barely vary. In that pilot most of the uncertainty is between cases, and extra trials of a case whose outcome you can already predict tell you almost nothing. Put that budget into more distinct cases, run each once per configuration on identical inputs, and compare the paired per-case differences.
The pilot already showed that a single case barely varies, so fifteen trials keep confirming a near-certain outcome. Meanwhile the comparison covers only 40 inputs, which can't represent the 600 you have. Running all 600 cases once per configuration on identical inputs covers far more of the input space, and pairing still controls for differences in case difficulty.
When you do run several trials, score each case by the mean across them. The plugin eval runner's per-case score is defined as the mean with-arm run score for that case. A best-of-three rule counts a case as passing if any trial passed. That rewards a configuration for being inconsistent and hides the run-to-run variance the trials were meant to expose.
Checkpoint 5 of 9· Fill the gap
This CI run has to stop before the comparison overspends. The ceiling covers the whole run, judge calls included, so it has to be sized for both arms. Otherwise it can stop one configuration halfway and leave the arms unpaired. Which flag completes the command?
claude plugin eval . \
--trust-plugin \
--json results.json \
--threshold 0.8 \
--model claude-sonnet-5 \
--judge-model claude-haiku-4-5 \
--no-publish \
? 20--max-cost-usd sets one dollar ceiling for the whole suite. If the run hits it, the run ends as partial with exit code 2. That is why the ceiling has to be sized for every case, trial and arm before you start. --threshold sets the passing score, not a spending limit.
Checkpoint 6 of 9· Check yourself
Your suite runs each of 50 cases three times under prompt A and three times under prompt B. On many cases prompt B goes 1/3, while prompt A goes 3/3 or 0/3. How should each case be scored so the comparison between the prompts can be read?
A case's score is the mean across its trials. That keeps prompt B's inconsistency visible instead of letting it count as three passes. Best-of-three would reward that inconsistency. Using only the first trial, or dropping mixed cases, throws away the variance the trials were run to measure.
“Mean with-arm run score for the case”Source: code.claude.com
Checkpoint 7 of 9· Exam question
You must compare two configurations with a budget of 1,200 model calls, 600 per configuration. You have 600 labelled cases. A pilot of 40 cases at 5 trials each showed most cases passing 5/5 or failing 0/5, with wide differences between cases. Which allocation gives the most interpretable comparison?
Correct answer: A — Run the full set of 600 cases once per configuration on identical inputs, then compare the paired per-case differences between the two.
- A. Correct. Most cases pass or fail every time, so repeated trials add little information. Differences between cases are large, so the 600 calls are best spent covering every labelled case once, with paired per-case differences.
- B. Incorrect. The pilot showed little run-to-run variation within a case, so six trials per case buy precision that is not needed. The same budget could cover six times as many cases, and a sample of 100 leaves the comparison much looser.
- C. Incorrect. Scoring each case by its best trial measures the best attainable result, not typical performance, and it hides consistency differences between the configurations. With cases that mostly pass or fail every time, the extra trials also cost coverage of 400 labelled cases.
- D. Incorrect. Fifteen trials on 40 cases mostly repeat outcomes the pilot already showed to be stable. Because cases differ widely, a 40-case sample says little about the 600 labelled cases, so the comparison would be narrow and unrepresentative.
4.Reading a gap against run-to-run noise
A gap of a point or two is often noise. In that GPQA Diamond comparison, neither executor ever called the advisor, so every difference between the arms was variance. Small gaps between configurations are the ones that need a measured noise floor before anyone acts on them.
Repeated trials go to the comparison the decision depends on. Anthropic's internal agentic-coding benchmark ran five attempts per task for the advisor pairing and its Claude Opus 5 control, and one attempt for the other points on the chart. Use the same pattern when the budget is tight. Run single passes to place most configurations, then spend repeated trials on the two candidates you will actually choose between.
Cost varies from run to run as well, so compare it the same way you compare pass rate. In the cost-optimization cookbook, two trials of the same batched configuration both passed 9 of 10 claims and missed the same claim. One trial cost about three times as much per task as the other, because cache hits inside a batch are best effort when requests run concurrently. A single trial could have reported either cost. Measure cost per completed task, run each configuration more than once, and read cost and pass rate together.
| config | trial | pass/n | $ / task | $ / 10k tasks | misses |
|---|---|---|---|---|---|
| sonnet · medium · batch (one-shot) | 1 | 9/10 | $0.0060 | $60 | CLM-009 |
| sonnet · medium · batch (one-shot) | 2 | 9/10 | $0.0194 | $194 | CLM-009 |
Set the bar before you compare. Decide the pass rate you need from baseline runs, treat it as a constraint, and then see which candidate meets it at lower cost. That turns an A/B test against a baseline or earlier version into a decision: a gap counts when it is larger than the run-to-run spread and moves a configuration across the bar.
Checkpoint 8 of 9· Check yourself
Six candidate prompts are in play, and the budget covers only one pass of every case for each of them plus a few extra rounds. The decision is which single prompt goes to production. How should the extra trials be spent?
Repeated trials matter where the decision depends on a small gap, and that gap is between the finalists. Single passes are good enough to place the other configurations. This is how Anthropic split five attempts per task and single attempts on its internal benchmark.
“The August runs had five attempts per task for the pairing and the Claude Opus 5 control and one for the other points.”Source: platform.claude.com
Checkpoint 9 of 9· Check yourself
In one trial, a batched configuration passes 9 of 10 cases, which meets the bar, at half the cost per task of the current configuration. How should that cost figure be treated?
Cost varies from run to run, just like pass rate. In the cookbook, two trials of the same batched configuration both passed 9 of 10, but one cost about three times as much per task because cache hits in a batch are best effort. Measure cost per completed task over several trials.
“Roll usage up per task, run each configuration more than once, and read cost and pass rate together.”Source: platform.claude.com
Sources
(10)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Even with temperature set to 0, the results will not be fully deterministic, and identical inputs might produce different outputs across API calls.”
↩︎ Why one run per case cannot separate two configurations“Even with temperature set to 0, the results will not be fully deterministic, and identical inputs might produce different outputs across API calls.”
↩︎ Key concept“Even with temperature set to 0, the results will not be fully deterministic, and identical inputs might produce different outputs across API calls.”
↩︎ Exam trap 1 - 2.
“Prompt caching has no effect on output token generation.”
↩︎ Why one run per case cannot separate two configurations“Prompt caching has no effect on output token generation.”
↩︎ Exam trap 2 - 3.
“Because model outputs vary between runs, we run multiple trials to produce more consistent results.”
↩︎ Why one run per case cannot separate two configurations“Each attempt at a task is a trial. Because model outputs vary between runs, we run multiple trials to produce more consistent results.”
↩︎ Checkpoint - 4.
“current models no longer accept temperature”
↩︎ Why one run per case cannot separate two configurations“The same test set (68 examples): Ensuring fair comparison across all configurations”
↩︎ Change one variable and run both arms on identical inputs - 5.https://code.claude.com/docs/en/plugin-evalsOfficial docs
“an llm grader also shows the judge’s votes and the evidence it was shown”
↩︎ Why one run per case cannot separate two configurations“results on a managed machine can differ from an unmanaged one by that policy”
↩︎ Change one variable and run both arms on identical inputs“With-arm score minus without-arm score”
↩︎ Change one variable and run both arms on identical inputs“Partial run: the --max-cost-usd ceiling was hit”
↩︎ Splitting a fixed budget between cases and trials“Leave partial results out of trend charts”
↩︎ Splitting a fixed budget between cases and trials“Mean with-arm run score for the case”
↩︎ Splitting a fixed budget between cases and trials“Mean with-arm run score for the case”
↩︎ Exam trap 3 - 6.https://www.anthropic.com/engineering/infrastructure-noiseSecondary source
“any component of that system can act as a confounder”
↩︎ Why one run per case cannot separate two configurations“Everything else stayed constant: same Claude model, same harness, same task set.”
↩︎ Change one variable and run both arms on identical inputs“running at multiple times and on multiple days would help average out the noise”
↩︎ Change one variable and run both arms on identical inputs - 7.
“you should aim for around fifty eval cases and at least five trials per configuration to minimize nondeterministic variance”
↩︎ Splitting a fixed budget between cases and trials“cache hits inside a batch are best effort because the requests run concurrently”
↩︎ Reading a gap against run-to-run noise“Roll usage up per task, run each configuration more than once, and read cost and pass rate together.”
↩︎ Reading a gap against run-to-run noise - 8.https://platform.claude.com/cookbook/misc-building-evalsSecondary source
“Grading on the other hand is a cost you will incur every time you re-run your eval, in perpetuity”
↩︎ Splitting a fixed budget between cases and trials - 9.https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligenceOfficial docs
“The pairings scored 0.5 points (Claude Haiku 5.5) and 1.0 points (Claude Sonnet 5.5) above the executor alone, both within run-to-run noise.”
↩︎ Reading a gap against run-to-run noise“The August runs had five attempts per task for the pairing and the Claude Opus 5 control and one for the other points.”
↩︎ Reading a gap against run-to-run noise“The pairings scored 0.5 points (Claude Haiku 5.5) and 1.0 points (Claude Sonnet 5.5) above the executor alone, both within run-to-run noise.”
↩︎ Exam trap 4 - 10.
“A/B testing: Compare performance against a baseline model or earlier version.”
↩︎ Reading a gap against run-to-run noise