What you will be able to do
- Run an improvement loop that measures each prompt or model change
- Spot a per-category regression that an overall score hides
- Promote, canary and roll back prompt versions using version pinning
1.Measure every step of the loop
Iterative improvement means running controlled comparisons over and over: change one thing, re-score it on the same test set, and keep the change only if the numbers support it. Anthropic's classification cookbook shows the pattern on an insurance support-ticket classifier. Each technique was added one at a time and scored on the same 68-example test set.
| Iteration | Accuracy |
|---|---|
| Random baseline | ~10% |
| Simple classifier | ~70% |
| RAG classifier | 94% |
| RAG + Chain-of-Thought | 97% |
The authors' takeaway: "Rather than relying on intuition, we measured performance at each step," and this showed which techniques added value and by how much. When they reran the variants in Promptfoo across temperatures 0.0 to 0.8, they found that temperature barely moved the CoT variant, while the RAG step accounted for the big jump. Without measurement, it would be easy to spend iterations tuning a setting that doesn't matter.
The variable you change doesn't have to be the prompt. The same infrastructure should support "Model comparisons: Testing across different Claude models (Haiku vs Sonnet) or temperature settings." Anthropic's prompt-engineering overview notes that not every failing eval is best solved by prompt engineering, and that latency and cost can sometimes be improved more easily by selecting a different model. A good loop treats a model swap as just another variant scored against the baseline.
Checkpoint 1 of 4· Check yourself
Your eval shows quality is on target but cost and latency miss their criteria. After several prompt rewrites, nothing has changed. What does Anthropic's guidance suggest trying next?
The prompt-engineering overview says latency and cost can sometimes be improved more easily by changing the model. Scoring the alternative model on the same eval keeps the comparison controlled.
“For example, you can sometimes improve latency and cost more easily by selecting a different model.”Source: platform.claude.com
2.Read results per segment, not just overall
That scenario comes from Anthropic's Managed Agents prompt-versioning cookbook. Version 1 was scored on 20 labelled tickets, five for each of four teams. Then the PM shipped v2 with the new rule, and both versions were scored on the same tickets:
| Team | v1 | v2 |
|---|---|---|
| api-platform | 5/5 | 5/5 |
| auth | 5/5 | 5/5 |
| billing | 4/5 | 2/5 (regressed) |
| dashboard | 5/5 | 5/5 |
The team the change was meant to help didn't improve, because it was already perfect. The damage landed somewhere else: "The new rule is broad, and on a usage-billed API product the billing tickets talk about API usage too." An aggregate score would have shown only a small drop. The per-category breakdown makes the regression obvious.
That's why the success-criteria guide says most use cases need evaluation along several dimensions. A single number can hide a trade-off between segments. Regressions you find should also go back into the test set. Anthropic's agent-evals post describes the payoff as "development accelerates as failures become test cases, test cases prevent regressions, and metrics replace guesswork."
Checkpoint 2 of 4· Check yourself
In the triage example, v2 was written to improve api-platform routing. What did the per-team comparison actually show?
The broad rule pulled billing tickets that mention API usage toward the platform team. The targeted team was already at 5/5.
“The new rule is broad, and on a usage-billed API product the billing tickets talk about API usage too.”Source: platform.claude.com
Checkpoint 3 of 4· Exam question
A data science lead is designing an eval suite to compare prompt candidates before an A/B test, and a colleague suggests spending the team's limited time hand-labeling 80 carefully reviewed examples rather than building an automated grader over a larger set. Following the recommended approach to eval volume versus grading quality, how should the lead respond?
Correct answer: A — Favor a larger automatically graded set even with somewhat lower per-item signal, since more questions with automated grading generally outweigh fewer high-quality hand-graded ones
- A. Correct. The recommended practice is to prioritize volume with automated grading, since more questions with slightly lower-signal automated grading give a more reliable read on performance than a small number of high-quality hand-graded examples.
- B. Incorrect. Hand-grading quality does not automatically outweigh the statistical benefit of testing across a larger, automatically graded sample.
- C. Incorrect. Shrinking the eval set to enable full manual review sacrifices coverage and volume, working against reliable prompt comparisons.
- D. Incorrect. Relying on subjective impressions with no test set removes the empirical basis needed to compare prompt versions objectively.
3.Promote, canary and roll back by version
Iteration is only safe if a bad version is cheap to undo. In Managed Agents, every agents.update creates a new immutable version under the same agent ID, and sessions.create can pin a specific version. After the billing regression, "Version 1 is still sitting on the server." Rolling back means callers pass version 1 again. Nothing has to be redeployed: billing tickets routed through version 1 scored 4/5 again.
The cookbook recommends a release discipline built on that. Production callers always pin an explicit version, so new versions get no traffic until someone promotes them. The pinned number becomes the change-control gate: creating versions is exploration, and changing the production pin goes through review. For higher-stakes agents, send a fraction of traffic to the new version and compare before promoting fully. That is a live A/B test, with the versioning acting as a feature flag. Meanwhile v2 stays on the server so the PM can keep improving it, and a fixed v3 goes through the same evaluation before promotion.
This is the general improvement cycle written as operations. Anthropic's Skill authoring guidance describes it as observe, refine, test: watch how the agent behaves on real work, make a targeted change, test again, and "Continue this observe-refine-test cycle as you encounter new scenarios." The reason to do it this way is that each iteration then rests on observed behavior, not assumptions.
Checkpoint 4 of 4· Put it in order
Put the prompt-versioning workflow from the Managed Agents cookbook in order
- 1.Detect a regression by scoring v2 on the same set
- 2.Evaluate v1 against a labelled test set
- 3.Create the agent and get version 1
- 4.Ship a v2 prompt with agents.update
- 5.Roll back by pinning sessions to version 1
Measure the baseline, ship the candidate as a new version, compare on the same labelled set, and re-pin the old version when it regresses.
“create v1, evaluate against a labelled test set, ship v2, detect a regression, roll back by pinning sessions to version 1.”Source: platform.claude.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.If a prompt change keeps the targeted category at full accuracy and the overall score barely moves, it is safe to ship.Why is that wrong?
A broad rule can quietly damage another segment. In the triage example, billing fell from 4/5 to 2/5, which only a per-category, multidimensional comparison makes visible.
Covered in Read results per segment, not just overall
2.Creating a new prompt version is the step that needs review, and rolling back requires a redeploy.Why is that wrong?
Creating versions is exploratory and gets no traffic while callers pin explicit versions. The production pin is the review gate, and rolling back just means pinning the old version again.
Covered in Promote, canary and roll back by version
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Not every success criteria or failing eval is best solved by prompt engineering.”
↩︎ Measure every step of the loop“For example, you can sometimes improve latency and cost more easily by selecting a different model.”
↩︎ Checkpoint - 2.
“Rather than relying on intuition, we measured performance at each step”
↩︎ Measure every step of the loop“Model comparisons: Testing across different Claude models (Haiku vs Sonnet) or temperature settings”
↩︎ Measure every step of the loop - 3.
“Most use cases need multidimensional evaluation along several success criteria.”
↩︎ Read results per segment, not just overall“Most use cases need multidimensional evaluation along several success criteria.”
↩︎ Exam trap 1 - 4.
“development accelerates as failures become test cases, test cases prevent regressions, and metrics replace guesswork”
↩︎ Read results per segment, not just overall - 5.
“Continue this observe-refine-test cycle as you encounter new scenarios.”
↩︎ Promote, canary and roll back by version“Each iteration improves the Skill based on real agent behavior, not assumptions.”
↩︎ Promote, canary and roll back by version - 6.https://platform.claude.com/cookbook/managed-agents-cma-prompt-versioning-and-rollbackSecondary source
“For higher-stakes agents, route a fraction of traffic to a new version and compare (as this notebook does) before promoting fully.”
↩︎ Promote, canary and roll back by version“Have production callers pin to an explicit version, not the bare agent ID.”
↩︎ Promote, canary and roll back by version“You can effectively use this versioning as a feature flag.”
↩︎ Promote, canary and roll back by version“Treat the pinned version number as the gate for changing prompts.”
↩︎ Exam trap 2“The new rule is broad, and on a usage-billed API product the billing tickets talk about API usage too.”
↩︎ Checkpoint“create v1, evaluate against a labelled test set, ship v2, detect a regression, roll back by pinning sessions to version 1.”
↩︎ Checkpoint