Mixed-Method Eval Grading: Code, LLM and Human Graders

1.Three grader families and how to choose

Once you have a dataset, every test case needs a score. Anthropic's guidance groups graders into three families: code-based, model-based (LLM) and human. A mixed-methods framework uses each family for the cases it handles best. The selection rule in the docs is simple: use the fastest, most reliable and most scalable method that can actually judge the task.

That rule gives a clear order of preference. Code-based grading comes first wherever the task allows it. The cookbook calls it by far the best method when you can design for it. LLM-based grading is next, for judgments that rules cannot express. Human grading is the most flexible, but it is slow and expensive, and the docs advise avoiding it where possible.

For agents there is one more choice: what the grader looks at. A grader can examine the transcript, meaning the full record of tool calls and reasoning. Or it can examine the outcome, the final state of the environment. A flight-booking agent may say the flight is booked. An outcome grader checks whether the reservation actually exists in the database.

Sources123

2.Code-based graders: exact match, similarity and overlap

Code-based grading covers more than string equality. The docs use three metrics, each suited to a different task.

Exact match suits tasks with clear categorical answers, such as sentiment labels. The output is normalised for whitespace and case, then compared with the golden answer.

Cosine similarity measures *consistency*. The same question is asked several ways, including with typos and padded with irrelevant detail. The outputs are embedded, and the grader checks that they are semantically close even when the wording differs. Its dataset is built from groups of paraphrases, not single questions.

ROUGE-L grades summaries against reference summaries. It uses the longest common subsequence, so a high score means the summary captures key information in a coherent order.

The exact-match grader in the docs is only two lines long:

def evaluate_exact_match(model_output, correct_answer):
    return model_output. ? ().lower() == correct_answer.lower()

Sources1

3.LLM-based graders: rubrics, scales and binary judgments

Some qualities, such as empathy, professionalism or whether a reply leaks private health data, cannot be checked with a string comparison. For these the docs use an LLM as the grader, in two forms. A Likert grader rates tone on a 1-to-5 scale against a target such as "empathetic". A binary classification grader answers yes or no to whether a response contains personal health information (PHI). It can catch subtle or implicit PHI that rule-based checks miss.

The docs say to test an LLM grader's reliability before scaling it, and give these tips:

- Detailed, clear rubrics. State exactly what makes an answer correct or incorrect. - Empirical or specific output. Ask for only "correct"/"incorrect" or a 1–5 score. Purely qualitative evaluations are hard to assess quickly and at scale. - Encourage reasoning. Have the grader reason first, then discard the reasoning and keep only the score. - Use a different model to grade. The example code notes it is generally best practice to grade with a different model from the one that produced the output.

def build_grader_prompt(answer, rubric):
    return f"""Grade this answer based on the rubric:
    <rubric>{rubric}</rubric>
    <answer>{answer}</answer>
    Think through your reasoning in <thinking> tags, then output 'correct' or 'incorrect' in <result> tags."""

Sources1

4.Putting the methods together

A mixed-methods framework assigns each grader type its own job. Within one task, several graders can score different aspects of the output. Their scores combine in one of three ways: weighted, where the combined score must pass a threshold; binary, where every grader must pass; or a hybrid of the two. The docs also say qualitative measures add value when applied consistently alongside quantitative ones, not in place of them.

Humans still matter, mostly as calibration. Model-based graders need checking against human judgment. The guidance reserves structured human studies for two uses: calibrating LLM graders, and judging subjective outputs where human agreement is the reference standard. Descript is one example. The team moved from manual grading to LLM graders using criteria the product team defined, with periodic human calibration.

Anthropic compares this layering to the Swiss Cheese Model from safety engineering. No single evaluation layer catches everything, so problems that get through one layer are caught by another.

Sources12

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Human grading is the safest default for any eval because it gives the highest quality.Why is that wrong?

   Human grading is flexible and high quality, but the docs advise avoiding it where possible. Use the fastest, most reliable and most scalable method that works, and keep humans mainly for calibration.

   Covered in Three grader families and how to choose

2. 2.Grading with the same model that generated the output is fine, and even improves consistency.Why is that wrong?

   The docs' grader code notes it is generally best practice to grade with a different model from the one that produced the output.

   Covered in LLM-based graders: rubrics, scales and binary judgments