What you will be able to do
- Turn vague feedback like "it's off" into a concrete example you can act on
- Define specific, measurable criteria so you can tell whether a Claude-assisted output is actually working
- Explain why one improved result is not enough evidence to standardise a new prompt
- Make sense of mixed or contradictory feedback by sorting it against agreed criteria
Key concept
Evidence before adjustment — Before you change a prompt, a project or a workflow, decide what a good result looks like and check real outputs against that standard. Feedback tells you something may be wrong. Criteria and checks against them tell you what to change, and later whether the change helped.
1.Feedback is a signal, not a specification
Once Claude is part of real work (drafting replies, summarising reports, preparing digests), feedback starts to come in. Most of it is vague: "the summaries feel off", "these replies aren't quite right". It's tempting to rewrite the prompt straight away. But a vague complaint doesn't tell you what to change, so any rewrite is a guess. A guess can fix one problem while quietly causing another.
Anthropic's engineering team describes this trap in teams that have no structured way to check their outputs. Debugging becomes a loop of waiting for complaints, reproducing the problem by hand, patching it, and hoping nothing else broke.
So the first move with vague feedback is to make it concrete. Ask for a specific example of an output that was wrong. Ask what was wrong with it, and what a right version would have looked like. The Claude Code best-practices guide gives the same advice for bug reports: "fix the login bug" becomes a description of the symptom, where it happens, and what "fixed" means. Its principle is short: "Address root causes, not symptoms." One concrete failing example tells you more than a dozen general complaints. Later, it also becomes the test case that shows whether your change worked.
Checkpoint 1 of 5· Check yourself
A department head tells you that the Claude-drafted meeting notes "aren't great lately" but gives no examples. What is the most useful first step?
Vague feedback doesn't say what to fix. A concrete failing example shows the actual symptom, so you can fix the cause instead of guessing, and you can use it later to check the fix.
“Address root causes, not symptoms”Source: code.claude.com
Checkpoint 2 of 5· Exam question
Several colleagues tell a project coordinator that Claude's meeting summaries 'miss the decisions'. Nobody has said which decisions or which meetings. What should she do first?
Correct answer: A — Read a few criticised summaries beside their transcripts to see what kind of item is missed.
- A. Correct. 'Misses the decisions' could mean several different faults, and the fix differs for each, so the first move is to see the failures against the source before changing anything.
- B. A specific rule written before the fault is understood may address something that was never the problem.
- C. Collecting more of the same vague feedback delays the diagnosis that a few examples would already allow.
- D. Model choice is being changed before anyone knows whether the summaries were even asked for the decisions.
2.Define what "working" means before you judge it
Silence and "it's fine" are not evidence. To find out whether an output is really doing its job, you first need to say what the job is. Anthropic's guidance on success criteria says good criteria are specific, measurable, achievable and relevant. It contrasts a bad criterion, "The model should classify sentiments well", with a good one that names a metric, a test set and a target.
The same discipline works outside engineering. For a rota digest you might decide that every shift change must appear, nothing may be invented, and a manager should be able to act on it in under a minute. Each of these can be checked. The guidance also lists dimensions worth considering, including accuracy, consistency, relevance and coherence, tone and style, privacy, and how well the model uses the context you gave it.
Once criteria exist, you can collect evidence against them. Not all of it has to come from people filling in forms. The guidance names implicit user feedback such as task completion rates. Anthropic Academy's builder course suggests a more hands-on method: put the output in front of a real user, don't explain it, and "Watch where they get confused, what they ignore, and what they wanted that you never built." Its 4D framework names the skill involved: "Discernment means evaluating what AI gives back before acting on it."
Checkpoint 3 of 5· Put it in order
Put these steps for finding out whether a Claude-assisted output is working into the order Anthropic's evaluation guidance gives.
- 1.Measure real outputs with those evaluations and act on the results
- 2.Define specific, measurable success criteria for the output
- 3.Design evaluations that test outputs against those criteria
The guidance makes criteria come first. Evaluations are designed to measure against them, and only then do the results tell you what to change.
“starts with clearly defining your success criteria and then designing evaluations to measure performance against them.”Source: platform.claude.com
3.One good result is not proof, and mixed feedback needs sorting
There are two opposite mistakes here. One is reacting to a single bad output. The other is trusting a single good one. If a rewritten prompt produces one strong answer, you know it can work once, on that input. That doesn't tell you whether it does better than the old prompt across the range of work it will actually get. Anthropic's eval guidance says to "Design evals that mirror your real-world task distribution" and not to forget edge cases. It also lists A/B testing, comparing against a baseline or earlier version, as a way to measure improvement. Before a new wording becomes the team standard, run it and the old one on a representative set of real inputs, including awkward ones, and compare the results against your criteria.
| Method | What it tells you when adjusting an approach |
|---|---|
| A/B testing | Whether a new prompt or setup beats a baseline or earlier version, not just whether it worked once |
| User feedback (implicit measures like task completion rates) | Whether people can actually get their work done with the output |
| Edge case analysis | What share of rare or difficult inputs are handled without errors |
| Likert scales | A consistent qualitative score, such as coherence rated from 1 to 5 |
| Expert rubrics | Quality judged by people who know the domain, against defined criteria |
Criteria also help when feedback contradicts itself. Weeks of comments where some people praise the drafts and others complain about the same drafts aren't noise to ignore. They also aren't a vote to settle by majority. Usually the comments are about different things: one person wants a warmer tone, another wants more accuracy, a third wants shorter drafts. The guidance notes that "Most use cases need multidimensional evaluation along several success criteria." So sort the comments by the criterion each one touches. Look for patterns within each group, change one thing, and check the result against those criteria.
Some disagreement shows that the criteria were never shared. Anthropic's engineering team points out that two people reading the same spec can disagree about how edge cases should be handled. Writing the criteria down, and agreeing on them, resolves that before anyone touches the prompt.
Checkpoint 4 of 5· Check yourself
A team lead rewrites a prompt after one weak draft, gets one excellent draft back, and wants to make the new wording mandatory for everyone. What is the main weakness of this plan?
A single result can't separate real improvement from chance. Comparing the new and old wording on a representative set of inputs is how you tell whether the change actually helps.
“A/B testing: Compare performance against a baseline model or earlier version.”Source: platform.claude.com
Checkpoint 5 of 5· Exam question
Half a team says a Claude-assisted drafting pilot saves hours; the other half says the drafts need so much editing they are not worth it. What should the pilot lead do first?
Correct answer: B — Agree what an acceptable draft looks like, then gather specific examples from both groups.
- A. Rewriting to satisfy the critics assumes their cases are representative, which nobody has checked.
- B. Correct. The two groups may be applying different standards to different work, so a shared definition of an acceptable draft, plus real examples, turns opinion into something that can be acted on.
- C. More time produces more opinion without a shared standard to interpret any of it against.
- D. Splitting the group accepts the disagreement without ever discovering what caused it.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.If nobody has complained about a Claude-assisted output, it is working.Why is that wrong?
Without stated criteria and checks against them, you only find problems once someone complains. Decide what "working" means, then gather evidence against it.
2.A rewritten prompt that produced one good answer is ready to become the team standard.Why is that wrong?
One result can't separate improvement from chance. Compare the new prompt against the old one on inputs that reflect real work, including edge cases, before standardising it.
Covered in One good result is not proof, and mixed feedback needs sorting
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Absent evals, debugging is reactive: wait for complaints, reproduce manually, fix the bug, and hope nothing else regressed.”
↩︎ Feedback is a signal, not a specification“Two engineers reading the same initial spec could come away with different interpretations on how the AI should handle edge cases.”
↩︎ One good result is not proof, and mixed feedback needs sorting“Absent evals, debugging is reactive: wait for complaints, reproduce manually, fix the bug, and hope nothing else regressed.”
↩︎ Exam trap 1 - 2.https://code.claude.com/docs/en/best-practicesOfficial docs
“Address root causes, not symptoms”
↩︎ Feedback is a signal, not a specification - 3.
“Specific: Clearly define what you want to achieve.”
↩︎ Define what "working" means before you judge it“User feedback: Implicit measures like task completion rates.”
↩︎ Define what "working" means before you judge it“Most use cases need multidimensional evaluation along several success criteria.”
↩︎ One good result is not proof, and mixed feedback needs sorting“A/B testing: Compare performance against a baseline model or earlier version.”
↩︎ One good result is not proof, and mixed feedback needs sorting“Building a successful LLM-based application starts with clearly defining your success criteria and then designing evaluations to measure performance against them.”
↩︎ Key concept“Be task-specific: Design evals that mirror your real-world task distribution.”
↩︎ Exam trap 2“starts with clearly defining your success criteria and then designing evaluations to measure performance against them.”
↩︎ Checkpoint - 4.
“Watch where they get confused, what they ignore, and what they wanted that you never built.”
↩︎ Define what "working" means before you judge it - 5.
“Discernment means evaluating what AI gives back before acting on it.”
↩︎ Define what "working" means before you judge it