What you will be able to do
- Validate an LLM judge against human-labeled data before trusting its scores at scale
- Design rubrics with observable anchors and one isolated judge per dimension
- Measure inter-rater reliability among human labelers first, so the judge is compared against a stable ceiling
- Detect and correct position, length and self-preference effects by testing whether verdicts depend on things humans do not grade on
- Explain why judge-human agreement should be chance-corrected rather than reported as raw percent agreement
- Pin and re-validate a judge model so a model or infrastructure change is not mistaken for a quality change
Key concept
Judge calibration against human labels — An LLM judge is a measuring instrument. Its scores mean something only after you have measured how closely they match expert human labels on the same items, and you keep re-measuring that whenever the rubric, the judge model or the traffic changes.
1.The judge is an instrument you have to validate
An LLM judge turns free-form output into a number, and every decision downstream inherits that number's errors. Code checks such as exact match are fast and reliable, but they can't grade tone, completeness or whether an answer is faithful. Human graders can, but they are slow and expensive. Model-based grading sits in between: fast, scalable and able to make complex judgments. It comes with one condition, which is that you prove it agrees with humans before you rely on it. For how a judge fits alongside code and human graders in an overall eval, see evaluation system design.
In practice, you validate the judge before you scale it. Take a set of items that expert humans have already labeled, run the judge on the same items, and measure how far the two diverge. Once agreement is high and stable, human review drops to occasional spot checks. It never disappears. People testing outputs still find failures that a rubric never anticipated. In one Anthropic research system, human testers noticed the agents consistently preferred SEO content farms over authoritative sources, and no automated grader had flagged it.
A low score doesn't prove the system under test is bad, because the grader can be wrong too. Opus 4.5 initially scored 42% on CORE-Bench. Researchers then found rigid grading that rejected a correctly rounded number, ambiguous task specs, and tasks that couldn't be reproduced. After those were fixed, the score rose to 95%. Before you trust an aggregate, read transcripts alongside their grades and check that each failure is fair: you can see what the output got wrong, not a grader rejecting a valid answer.
Inter-rater reliability is how consistently independent raters give the same item the same label, and it sets the ceiling for any judge. Have two expert humans label an overlapping sample from the rubric alone, and compare their labels before the judge enters the picture. Where they split, the rubric is ambiguous: tighten its anchors and relabel. A judge that matches the humans at 0.6 when the humans match each other at 0.6 is performing as well as the rubric allows. Once the humans are consistent, resolve their disagreements into one label set and treat the judge as one more rater measured against it. Teams that run human grading in production call this inter-annotator agreement, and it is how human labels earn the right to calibrate a model grader.
Checkpoint 1 of 9· Check yourself
A team grades its support bot's free-form answers for tone and completeness with an LLM judge. A release candidate scores 62%, and the team wants to block releases that score below 70%. They have not yet compared the judge against human graders. What should they do before gating on this score?
A judge's score means something only after it has been checked against human labels. A low score can come from the grader rejecting valid answers, so you calibrate and read transcripts before gating. Code checks can't grade tone, and a repeatable score isn't necessarily a correct one.
“LLM-as-judge graders should be closely calibrated with human experts to gain confidence that there is little divergence between the human grading and model grading.”Source: www.anthropic.com
2.Rubrics with observable anchors, one dimension per judge
Most low judge-human agreement starts in the rubric. Score levels described with words like "good", "adequate" or "excellent" let the judge and the humans each read the scale their own way. An anchor should be something a reader can check in the text, for example "every numeric claim is sourced" or "the answer names the company in the first sentence, otherwise it is incorrect". Asking whether something is beautiful gets inconsistent answers. Asking whether it follows stated principles gives the grader a concrete standard.
Output format matters as much as wording. Restrict the judge to a fixed set of outputs, either a binary correct/incorrect or a defined 1–5 scale, because purely qualitative verdicts can't be aggregated at scale. Ask it to reason before it scores; a grader with thinking enabled performs better on complex judgments. Also give it a way out. An explicit "Unknown" option when the evidence is insufficient stops the judge from inventing a verdict.
Combining several dimensions into one number hides which dimension failed. If accuracy, tone, completeness and formatting go into a single score, a 3 could mean a factual error or a formatting slip. Engineers can't tell which, and you can't calibrate a dimension you can't see. The fix is structural: write a separate, clearly anchored rubric for each dimension and grade each one in its own isolated judge call. Each judge then gets its own agreement measurement against human labels, and a failure points to one dimension. The tempting patches don't fix this. Stating weights in one prompt still produces one combined number you can't diagnose. Scoring all four dimensions in a single call and keeping only the lowest still mixes the dimensions within one call and throws three of the four scores away. Adding more adjectives to each level makes the vague-wording problem worse.
def grade_completion(output, golden_answer):
grader_message = client.messages.create(
model="claude-opus-5-5",
max_tokens=2048,
messages=[
{"role": "user", "content": build_grader_prompt(output, golden_answer)}
],
)
grader_response = next(
block.text for block in grader_message.content if block.type == "text"
)
return (
"correct"
if "<result>correct</result>" in grader_response.lower()
else "incorrect"
)Checkpoint 2 of 9· Check yourself
Your judge prompt scores accuracy, tone, completeness and formatting as one 1–5 number. Agreement with human labels is low, and engineers can't tell why individual items fail. Which change fixes the underlying problem?
A combined score hides which dimension failed, and you can't calibrate a dimension you can't see. Giving each dimension its own isolated judge means each one can be measured against human labels and each failure points to one dimension. Weights still produce one number. Taking the lowest score still mixes the dimensions in one call. More adjectives make the vague wording worse.
“grade each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions”Source: www.anthropic.com
Checkpoint 3 of 9· Exam question
Saffron Travel runs an assistant that drafts replies to booking-change requests, and its quality team grades samples with a single judge prompt that scores accuracy, tone, completeness and formatting into one 1-5 number. Agreement with human reviewers is low. The prompt already ranks the dimensions by importance, and a previous revision that expanded the scale wording changed nothing. When reviewers dispute a score, they say the judge treats 'good' as meaning different things on different items, and a reply scored 2 might have failed on fare figures, a curt tone, or a missing refund step. Engineers cannot tell why items fail, and they have human labels for each dimension separately that they can compare against. Which two changes should the team make? Select TWO.(Select 2)
Correct answers: A, E — Define each score level with observable anchors, such as every numeric claim being sourced, rather than vague words like good.; Split the evaluation into separate judge calls that each grade one dimension, so each can be calibrated against labels independently.
- A. Observable anchors reduce interpretation variance for both judge and human raters, which raises reliability and makes disagreements traceable.
- B. Weights inside one prompt leave the dimensions entangled; the judge still blends them implicitly and failures remain undiagnosable.
- C. Subjective adjectives give no testable boundary between levels, so the judge's interpretation varies and agreement does not improve.
- D. Taking the minimum still hides which dimension failed and compounds any noise from the weakest criterion, so the combined number stays hard to diagnose.
- E. A single-dimension judge has one clear target, so disagreements with humans point to a specific criterion that can be refined.
3.Measuring judge-human agreement
Measuring agreement treats the judge as one more rater and asks how reliably it matches the humans. That requires a held-out set of items, each labeled by expert humans who worked from the same rubric the judge uses. Run the judge on those items, compare item by item, and look closely at the disagreements: each one shows either a rubric gap, a judge error or a labeling error. Labels from the people closest to the product requirements count most, because they are the ones defining success.
Raw percent agreement can mislead. If 90% of items are passes, a judge that passes everything agrees with humans 90% of the time while telling you nothing. Chance-corrected statistics such as Cohen's kappa subtract the agreement you would expect from the label distribution alone. That is why teams report them for judge-human comparisons. Run the same check on the humans themselves: if two expert graders often disagree on a dimension, the rubric is ambiguous, and no judge can do better than the humans it is measured against.
Agreement also depends on the items. A judge validated on easy, clear-cut cases can still be wrong on borderline ones. Keep the labeled set's distribution close to the real mix of questions and difficulty, and add new kinds of cases as the product grows. If the agreement set never changes while the product does, the number you check stops describing the work the judge actually grades.
Almost nothing yet. A judge that always says 'pass' would score the same 92%. You need a chance-corrected statistic, plus a look at how the judge does on the 8% of items humans failed. Those are the items a deploy gate exists to catch.
Checkpoint 4 of 9· Check yourself
Six months after launch, a judge still agrees closely with expert human labels on its original calibration set of clear-cut support questions. The product has since added multi-step troubleshooting questions, and engineers keep finding the judge misgrading them. What should the team do about the calibration set?
An agreement score describes only the items it was measured on. A set with no troubleshooting items can't show how the judge handles them, so the set has to follow the real mix of traffic as the product grows. Kappa on the same items still leaves the new cases out, and a repeatable score isn't necessarily a correct one.
“try to have the distribution in your eval represent ~ the real life distribution of questions and question difficulties”Source: platform.claude.com
4.Position, length and self-preference effects
A bias in a judge is a verdict that depends on something the human graders don't reward. Three show up often. Position: in pairwise comparison, the answer shown first wins more often than its quality warrants. Length: longer answers score higher regardless of content. Self-preference: an LLM grader is inclined to be generous toward LLM-generated output, and especially toward its own. When agents grade their own work they reliably skew positive, even when the work is mediocre. Separating the generator from the evaluator helps, and tuning a standalone evaluator to be skeptical is much easier than making a generator criticize itself.
Every one of these effects has the same test: vary the irrelevant attribute, hold quality fixed, and see whether the verdict moves. For position, present each pair in both orders. A win counts only when both orders agree, and a disagreement is a tie, so slot order can't decide the outcome. Setting temperature to 0 makes a biased verdict repeatable without making it correct. Telling the judge to ignore position is an instruction, not a measurement. Switching to a larger model while keeping the same order leaves the bias untested.
For length, compare the judge's behavior with the humans'. If judge scores track token count much more strongly than human labels do, the rubric is rewarding volume. Fix the cause: rewrite the rubric to credit the required facts and penalize padding. Then re-check agreement with human labels on held-out pairs that are matched for length, where length can't drive the verdict. Truncating responses, or subtracting points per hundred tokens, patches the score with a formula and can penalize long answers that are correct. Raising the bar for long answers and then checking pass rates measures how many items pass, not whether the judge now agrees with humans.
| Effect | How it shows up | Test that isolates it | Fix that addresses the cause |
|---|---|---|---|
| Position | First-slot answer wins too often; verdicts flip when the pair is swapped | Run every pair in both orders | Count a win only when both orders agree; disagreements score as ties |
| Length | Judge score tracks token count far more than human labels do | Compare judge and human scores on length-matched pairs | Rubric credits required facts and penalizes padding; re-check against human labels |
| Self-preference | Grader is lenient toward LLM output, especially its own | Compare a self-grading setup against human labels | Separate generator from evaluator; tune the evaluator to be skeptical |
Checkpoint 5 of 9· Check yourself
An agent builds landing-page designs and then scores each one against the team's design rubric. Its scores stay high, but human reviewers rate many of the same designs as mediocre. Which change addresses the cause of the gap?
This is self-preference: agents grading their own work skew positive. Separating the generator from the evaluator and tuning that evaluator to be skeptical works far better than asking a generator to criticize itself. Repeatable scores aren't correct scores, and a larger model grading its own output keeps the bias in place.
“tuning a standalone evaluator to be skeptical turns out to be far more tractable than making a generator critical of its own work”Source: www.anthropic.com
Checkpoint 6 of 9· Exam question
Heron Robotics runs an evaluation pipeline that ranks responses from its warehouse-assistant models, and the rankings decide which model ships. A pairwise judge compares two candidate answers per prompt. Audit logs show it picks the answer shown in the first slot 68% of the time, and 31% of verdicts flip when the same pair is presented in swapped order. Rerunning the audit with sampling made deterministic produced identical verdicts, flips included. The judge prompt already tells it to ignore answer position, and a trial with a larger judge model showed a similar first-slot preference. The team has no budget for human labeling but needs rankings it can trust and a measurable bias signal. What is the best fix?
Correct answer: C — Present each pair to the judge in both orders and credit a win only when the verdicts agree, scoring mismatches as ties.
- A. A larger judge can still prefer whichever answer comes first, and with the order unchanged the position effect remains uncorrected and cannot be measured. Model capacity does not replace counterbalancing the order.
- B. A temperature of 0 and a fixed seed make verdicts repeatable, but a repeatable verdict can still be position-biased. The judge would consistently favor the first slot, so the bias is locked in rather than removed.
- C. Judging each pair in both orders and crediting a win only when the verdicts match cancels the position effect by construction. Mismatches are scored as ties, and their rate doubles as a measurable bias signal.
- D. A prompt instruction is a request to the judge, not an enforced control, and position bias often persists despite it. Nothing verifies the change worked, so the order-dependent verdicts would still need to be measured.
Checkpoint 7 of 9· Exam question
Indigo Software runs an AI support assistant for its developer-tools customers and uses an LLM judge to score each answer for helpfulness on a 1-5 scale before weekly release decisions. The judge's scores correlate 0.7 with response token count, while human labels on the same items correlate only 0.2 with length. Reviewers who read the highest-scored long answers found repeated restatements and boilerplate. The judge's rubric simply asks for "thorough, comprehensive answers" and never says what a correct answer must contain. Meanwhile, some migration-troubleshooting answers are legitimately long because every step is required, and human raters score those highly. The team wants to reduce the length bias without discarding genuine content. What is the best change?
Correct answer: D — Rewrite the rubric to reward required facts and penalize padding; re-check agreement with human labels on length-matched held-out pairs.
- A. Truncation cuts required content out of complete answers, so the judge scores incomplete text. That removes facts humans credit and distorts quality instead of correcting a judge that over-rewards length.
- B. A fixed per-token penalty punishes length itself. Human labels barely track length, so answers that are long because the task needs it get marked down, and the judge's rubric still cannot tell substance from padding.
- C. A stricter bar for longer responses only moves the cutoff. The judge's scores stay inflated by length, so the ranking is still wrong, and longer answers that deserve a pass are rejected by a rule unrelated to quality.
- D. The scores favor length because the rubric rewards volume. Defining quality as coverage of required facts, with padding penalized, addresses that cause, and length-matched held-out pairs show whether the bias shrank.
5.Judge drift across model versions
Your agreement numbers are tied to the exact judge model that produced them. From the Claude 4.6 generation on, a model ID such as claude-opus-4-6 is the snapshot itself: Anthropic does not change the weights or configuration of an existing ID, and updates ship under new IDs. Earlier models work differently. On the Claude API, a short name such as claude-sonnet-4-5 is an alias that resolves to the most recent dated snapshot. Pin those as dated IDs, for example claude-haiku-4-5-20251001, or the judge can change underneath a calibration you already finished.
claude plugin eval . \
--trust-plugin \
--json results.json \
--threshold 0.8 \
--model claude-sonnet-5 \
--judge-model claude-haiku-4-5 \
--no-publish \
--max-cost-usd 20A pinned ID still doesn't guarantee identical behavior. The serving infrastructure around a model, including the router, safety classifiers and sampling logic, can change and cause minor differences in behavior on an ID that hasn't changed. So a judge that scored a high kappa at launch needs re-validation, not trust: re-run the human-labeled set on a schedule, re-run it every time the judge model changes, and grow it as new intents appear. Treat a judge upgrade like any other model migration. Score the same labeled set with the old and new judge, compare both against the human labels, and switch only once agreement holds. Few-shot examples with detailed score breakdowns in the judge prompt help as well; in one Anthropic harness they aligned the evaluator with the author's preferences and reduced score drift across iterations. Watching judge scores on live traffic falls under online quality measurement.
Checkpoint 8 of 9· Fill the gap
In this CI eval run, the agent under test and the grader use different models. Which flag sets the model used by the llm and baseline graders?
claude plugin eval . \
--trust-plugin \
--json results.json \
--threshold 0.8 \
--model claude-sonnet-5 \
? claude-haiku-4-5 \
--no-publish \
--max-cost-usd 20--judge-model sets the grader separately from --model, which sets the agent under test. Your agreement numbers belong to that judge model, so it needs pinning and re-validation of its own. In this sample it is the pre-4.6 alias claude-haiku-4-5, which is not a pinned snapshot.
Source: code.claude.comCheckpoint 9 of 9· Check yourself
Nobody has touched a support bot, but its CI judge scores dropped sharply overnight. The judge is configured as claude-haiku-4-5 and matched human labels closely at launch. What is the right first move?
Before the 4.6 generation, a short name like claude-haiku-4-5 is an alias that can point to a newer snapshot. So check the judge before you blame the bot: confirm what the alias resolves to, re-check agreement with the human labels, and pin a dated ID. A score that repeats exactly can still be a wrong one.
“An alias such as claude-sonnet-4-5 is a convenience pointer that resolves to the most recent dated snapshot for that minor version.”Source: platform.claude.com
Sources
(7)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“LLM-based grading: Fast and flexible, scalable and suitable for complex judgment. Test to ensure reliability first, then scale.”
↩︎ The judge is an instrument you have to validate“Have detailed, clear rubrics”
↩︎ Rubrics with observable anchors, one dimension per judge“Purely qualitative evaluations are hard to assess quickly and at scale.”
↩︎ Rubrics with observable anchors, one dimension per judge“Use a grader model with thinking on, so that it reasons before it produces an evaluation score.”
↩︎ Rubrics with observable anchors, one dimension per judge“Test to ensure reliability first, then scale.”
↩︎ Measuring judge-human agreement“Test to ensure reliability first, then scale.”
↩︎ Position, length and self-preference effects“LLM-based grading: Fast and flexible, scalable and suitable for complex judgment. Test to ensure reliability first, then scale.”
↩︎ Exam trap 3“LLM-based grading: Fast and flexible, scalable and suitable for complex judgment. Test to ensure reliability first, then scale.”
↩︎ Prediction - 2.
“Opus 4.5 initially scored 42% on CORE-Bench”
↩︎ The judge is an instrument you have to validate“we do not take eval scores at face value until someone digs into the details of the eval”
↩︎ The judge is an instrument you have to validate“Inter-annotator agreement”
↩︎ The judge is an instrument you have to validate“Used to calibrate model-based graders”
↩︎ The judge is an instrument you have to validate“To avoid hallucinations, give the LLM a way out”
↩︎ Rubrics with observable anchors, one dimension per judge“and then grade each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions.”
↩︎ Rubrics with observable anchors, one dimension per judge“LLM-as-judge graders should be closely calibrated with human experts to gain confidence that there is little divergence between the human grading and model grading.”
↩︎ Measuring judge-human agreement“The people closest to product requirements and users are best positioned to define success.”
↩︎ Measuring judge-human agreement“LLM-as-judge graders should be closely calibrated with human experts to gain confidence that there is little divergence between the human grading and model grading.”
↩︎ Key concept“Inter-annotator agreement”
↩︎ Exam trap 1“and then grade each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions.”
↩︎ Exam trap 2“LLM-as-judge graders should be closely calibrated with human experts to gain confidence that there is little divergence between the human grading and model grading.”
↩︎ Checkpoint“grade each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions”
↩︎ Checkpoint - 3.
“Human evaluation catches what automation misses.”
↩︎ The judge is an instrument you have to validate - 4.
“gives Claude something concrete to grade against”
↩︎ Rubrics with observable anchors, one dimension per judge“the evaluator is still an LLM that is inclined to be generous towards LLM-generated outputs”
↩︎ Position, length and self-preference effects“agents reliably skew positive when grading their own work”
↩︎ Position, length and self-preference effects“tuning a standalone evaluator to be skeptical turns out to be far more tractable than making a generator critical of its own work”
↩︎ Position, length and self-preference effects“I calibrated the evaluator using few-shot examples with detailed score breakdowns.”
↩︎ Judge drift across model versions - 5.https://platform.claude.com/cookbook/misc-building-evalsSecondary source
“try to have the distribution in your eval represent ~ the real life distribution of questions and question difficulties”
↩︎ Measuring judge-human agreement - 6.
“Anthropic does not update the weights or configuration of an existing model ID.”
↩︎ Judge drift across model versions“An alias such as claude-sonnet-4-5 is a convenience pointer that resolves to the most recent dated snapshot for that minor version.”
↩︎ Judge drift across model versions“Occasionally, infrastructure updates produce minor differences in observable behavior even when the model ID and weights have not changed.”
↩︎ Judge drift across model versions“An alias such as claude-sonnet-4-5 is a convenience pointer that resolves to the most recent dated snapshot for that minor version.”
↩︎ Exam trap 4“Occasionally, infrastructure updates produce minor differences in observable behavior even when the model ID and weights have not changed.”
↩︎ Exam trap 5 - 7.https://code.claude.com/docs/en/plugin-evalsOfficial docs
“Model for llm and baseline graders”
↩︎ Judge drift across model versions