Evaluation Success Criteria and Accuracy Metrics

1.Start with success criteria, not with a model

Before you compare prompts or models, you need a definition of 'good'. Anthropic's testing guidance puts this first: you build an LLM application by defining success criteria, then designing evaluations that measure performance against them. Without criteria, an evaluation only produces numbers that nobody can interpret.

The guidance names four qualities a criterion needs. It must be specific: 'good performance' is not a target, but 'accurate sentiment classification' is. It must be measurable, using a quantitative metric or a well-defined qualitative scale. It must be achievable, which means the target comes from industry benchmarks, earlier experiments, AI research or expert knowledge, not from wishful thinking. And it must be relevant to what the application is for and what its users need. The guide's example: strong citation accuracy might be critical for a medical app but matter much less for a casual chatbot.

Compare the two versions of the same goal. 'The model should classify sentiments well' gives an evaluator nothing to check. The improved version reads: the model should achieve an F1 score of at least 0.85 on a held-out test set of 10,000 diverse Twitter posts, a 5% improvement over the current baseline. One sentence now names the metric, the threshold, the data and the reason the target is realistic.

Sources1

2.Kinds of metrics and why you need several

After you decide a criterion must be measurable, you choose how to measure it. The guidance sorts quantitative metrics into three families and adds qualitative scales for things numbers don't capture well.

Numbers are preferred because they're clear and scale well. Qualitative measures still have a place, as long as they are applied consistently and alongside quantitative ones. The guidance also lists methods that produce numbers: A/B testing against a baseline or earlier version, implicit user feedback such as task completion rates, and edge case analysis (the percentage of edge cases handled without errors).

No single family covers a real application. The guide lists dimensions you may need to consider: task fidelity, including rare or hard inputs; consistency (whether the same question asked twice gets a semantically similar answer); relevance and coherence; tone and style; privacy preservation; how well the model uses the context it's given; latency; and price. This is why its fully worked criterion bundles several targets on one held-out set of 10,000 Twitter posts: an F1 score of at least 0.85, 99.5% of outputs non-toxic, 90% of errors causing inconvenience rather than egregious harm, and 95% of responses in under 200ms.

Sources1

3.Accuracy and task success

Accuracy is the dimension most applications define first. For tasks with clear, categorical answers, the simplest metric is exact match: does the output equal a predefined correct answer after you normalize whitespace and case? The guide's sentiment example labels each tweet by hand, classifies it with Claude, and reports the share that match.

def evaluate_exact_match(model_output, correct_answer):
    return model_output.strip().lower() == correct_answer.lower()


outputs = [
    get_completion(
        f"Classify this as 'positive', 'negative', 'neutral', or 'mixed': {tweet['text']}"
    )
    for tweet in tweets
]
accuracy =  ? (
    evaluate_exact_match(output, tweet["sentiment"])
    for output, tweet in zip(outputs, tweets)
) / len(tweets)
print(f"Sentiment Analysis Accuracy: {accuracy * 100}%")

An accuracy figure only becomes a criterion when it has a threshold. In the ticket-routing guide, the evaluation script returns accuracy, response time and cost per classification, and the guide then sets targets, such as 95% accuracy over 100 tests. The customer-support guide sets a target for each kind of accuracy: 100% for introductory company and product information, 100% for generated quotes, and an escalation accuracy of 95% or higher.

For agents that work over many turns, task success needs a sharper definition. Anthropic's agent-evaluation post separates the transcript, which is the full record of a trial including outputs and tool calls, from the outcome, which is the final state of the environment when the trial ends. An agent can report success in the transcript and still leave the environment unchanged. Define task success on the outcome. Because model outputs vary between runs, run several trials of each task to get a stable success rate.

Sources1234

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Success targets should be as ambitious as possible to push the model.Why is that wrong?

   Targets must be achievable. Base them on benchmarks, prior experiments, research or expert knowledge, and keep them within what current frontier models can do.

   Covered in Start with success criteria, not with a model

2. 2.If the agent's final message says the task is done, the trial counts as a success.Why is that wrong?

   For agents, success is measured on the outcome, which is the final state of the environment, not on what the transcript claims.

   Covered in Accuracy and task success