What you will be able to do
- Put a feedback loop and eval baselines in place before an AI solution ships
- Iterate safely using versioned prompts, regression checks against a labelled set, and rollback
- Decide whether a live AI solution should be shipped, fixed or stopped
1.Monitoring: know whether it works once it is live
The builder course treats shipping as the point where ownership gets real. Diligence in the builder model is full ownership, from whether a product should exist to whether it is still serving users after launch. The course's pre-ship checklist includes one monitoring question in particular: is there a feedback loop? Put differently, how will you know the solution is working once it ships? The prescribed habit is small and concrete: share the deployed prototype with three real users and make one iteration based on what you learn, with tests that verify the change.
Shipping also brings its own operational vocabulary, such as migrations, versioning, rate limits and feature flags, which AI won't raise unless you ask. Your success criteria should already include operational dimensions like acceptable response time and the budget for running the model. Monitoring keeps checking those numbers after launch.
Anthropic describes the typical breaking point: users report that the agent feels worse after a change, and the team is flying blind. Without evals, debugging is reactive. You wait for complaints, reproduce manually, fix the bug and hope nothing else regressed. Teams in that position catch issues only in production, where fixing one failure creates others. Once evals exist, baselines come almost for free. Latency, token usage, cost per task and error rates can all be tracked on a static bank of tasks. Claude Code followed this progression: fast iteration on feedback first, then evals, which combined with production monitoring, A/B tests and user research gave the team the signals it needed to keep improving.
| Stage | Typical signals | Limitation |
|---|---|---|
| Early prototyping | Manual testing, dogfooding, and intuition | Works until changes outpace what people can check by hand |
| In production, scaling | Evals plus production monitoring, A/B tests, user research | Eval costs are visible upfront while benefits accumulate later |
Checkpoint 1 of 5· Check yourself
A production agent has no evals. Users say it got worse after last week's prompt change. Which description best fits the team's position?
Without evals, a team can't separate real regressions from noise or test changes at scale. All it can do is react to complaints.
“Absent evals, debugging is reactive: wait for complaints, reproduce manually, fix the bug, and hope nothing else regressed.”Source: www.anthropic.com
2.Iteration: change, measure, roll back
Tests make post-launch iteration safe. The builder course says the test-first habit is why you can keep changing things confidently. A Claude Managed Agents cookbook walks through the full loop on a support-ticket triage agent. You create the agent, which comes back as version 1, and score it against 20 labelled tickets. Then a PM ships a v2 prompt telling the agent that anything about API usage or rate limits belongs to the platform team. Re-running the same labelled set against v2 shows the cost of that change:
| Team | v1 | v2 |
|---|---|---|
| api-platform | 5/5 | 5/5 |
| auth | 5/5 | 5/5 |
| billing | 4/5 | 2/5 (regressed) |
| dashboard | 5/5 | 5/5 |
The new rule was too broad. On a usage-billed API product, billing tickets talk about API usage too. Because v1 still exists on the server, rolling back isn't a redeploy: callers go back to passing version 1. The lesson for production is about where change control sits. Production callers should pin an explicit version rather than the bare agent ID, which picks up the latest version on the next session. The pinned version number then becomes the review gate. Creating versions is cheap and exploratory, and promoting one to production goes through normal review. For higher-stakes agents, route a fraction of traffic to the new version as a canary and compare before promoting fully.
Checkpoint 2 of 5· Put it in order
Put the cookbook's iteration loop in order.
- 1.Create v1
- 2.Evaluate v1 against a labelled test set
- 3.Detect a regression on the same test set
- 4.Roll back by pinning sessions to version 1
- 5.Ship a v2 prompt
The labelled set provides the baseline. Re-running it after a change is what exposes the regression, and pinning to the old version undoes the change without a deploy.
“create v1, evaluate against a labelled test set, ship v2, detect a regression, roll back by pinning sessions to version 1.”Source: platform.claude.com
Checkpoint 3 of 5· Fill the gap
Which method returns every version of an agent, so you can see what is available to pin or roll back to?
client.beta.agents.versions. ? (AGENT_ID)versions.list returns every version of the agent. agents.update creates a new version, and pinning happens on sessions.create.
Source: platform.claude.comIteration also includes model upgrades. When more powerful models come out, teams without evals face weeks of testing. Teams with evals can find the new model's strengths, tune their prompts and upgrade in days.
Checkpoint 4 of 5· Exam question
A development team is designing an internal agent that must autonomously read repository files, run shell commands, and edit code without the team writing their own tool-execution loop. Which design choice fits this requirement?
Correct answer: B — Build on the Claude Agent SDK, which provides the built-in tools, agent loop, and context management out of the box
- A. The Client SDK gives direct API access, but the team would still need to implement the tool loop themselves, which the requirement explicitly avoids.
- B. Correct - the Agent SDK ships with built-in tools, the agent loop, and context management so the team does not implement tool execution from scratch.
- C. Calling the Messages API once per file without a tool loop cannot autonomously read files, run commands, or edit code as required.
- D. The Usage and Cost Admin API reports token usage and spend; it has no role in executing tools or orchestrating an agent.
3.Ship it, fix it, or stop it
Every iteration ends in a decision, and continuing isn't the only option. The builder course frames the call as ship it, fix it or stop it, and names deprecating your own work as an underrated skill. Its reasoning is that AI makes code cheap: "Prototype freely, ship selectively. Cheap code creates value only when paired with honest evaluation." A prototype that never meets its success criteria should be stopped, not shipped because it already exists.
The same evidence-first rule applies to growing a live system. Building Effective Agents says the key to success is measuring performance and iterating, and that you should consider adding complexity only when it demonstrably improves outcomes. Whether you are adding a step, retiring a workflow or keeping things as they are, the answer comes from the measurements set up in the monitoring phase, not from habit. The course adds one more pre-ship check that belongs in this decision: access is a design decision, so check who your assumptions exclude before calling something shipped. The sources give the decision criteria, but they don't describe a step-by-step sunset procedure.
Checkpoint 5 of 5· Check yourself
A team has a working AI prototype that took two days to build, but it misses its success criteria on evaluation. What does the builder course's guidance favour?
Cheap code only creates value when honest evaluation backs it. A prototype that fails its criteria gets fixed or stopped, and added complexity has to prove it improves outcomes.
“Cheap code creates value only when paired with honest evaluation.”Source: academy.claude.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Production callers can safely use the bare agent ID, and rolling back a bad prompt needs a code deploy.Why is that wrong?
A bare agent ID picks up the latest version on the next session. Pinning an explicit version keeps new versions out of production until they are promoted, and rolling back just means pinning the old version again.
Covered in Iteration: change, measure, roll back
2.Once an AI solution passes its pre-launch tests, user complaints are enough of a monitoring signal.Why is that wrong?
Relying on complaints leaves the team catching problems only in production, where each fix can break something else. Evals and production monitoring make regressions visible before users run into them.
Covered in Monitoring: know whether it works once it is live
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Diligence in the builder model is full ownership”
↩︎ Monitoring: know whether it works once it is live“share it with three real users, and make one iteration based on what you learn”
↩︎ Monitoring: know whether it works once it is live“Shipping has its own technical vocabulary (migrations, versioning, rate limits, feature flags) that AI will not surface unless you ask.”
↩︎ Monitoring: know whether it works once it is live“Tests make post-launch iteration safe.”
↩︎ Iteration: change, measure, roll back“The test-first habit is why you can keep changing things confidently.”
↩︎ Iteration: change, measure, roll back“Make the call: ship it, fix it, or stop it”
↩︎ Ship it, fix it, or stop it“the underrated skill of deprecating your own work”
↩︎ Ship it, fix it, or stop it“Access is a design decision. Check who your assumptions exclude before you call something shipped.”
↩︎ Ship it, fix it, or stop it“Cheap code creates value only when paired with honest evaluation.”
↩︎ Checkpoint - 2.
“Is there a feedback loop?”
↩︎ Monitoring: know whether it works once it is live - 3.
“What is the acceptable response time for the model?”
↩︎ Monitoring: know whether it works once it is live - 4.
“latency, token usage, cost per task, and error rates can be tracked on a static bank of tasks.”
↩︎ Monitoring: know whether it works once it is live“When more powerful models come out, teams without evals face weeks of testing”
↩︎ Iteration: change, measure, roll back“catching issues only in production, where fixing one failure creates others.”
↩︎ Exam trap 2“once an agent is in production and has started scaling, building without evals starts to break down.”
↩︎ Prediction“Absent evals, debugging is reactive: wait for complaints, reproduce manually, fix the bug, and hope nothing else regressed.”
↩︎ Checkpoint - 5.https://platform.claude.com/cookbook/managed-agents-cma-prompt-versioning-and-rollbackSecondary source
“Treat the pinned version number as the gate for changing prompts.”
↩︎ Iteration: change, measure, roll back“route a fraction of traffic to a new version and compare”
↩︎ Iteration: change, measure, roll back“Have production callers pin to an explicit version, not the bare agent ID.”
↩︎ Exam trap 1“create v1, evaluate against a labelled test set, ship v2, detect a regression, roll back by pinning sessions to version 1.”
↩︎ Checkpoint - 6.https://www.anthropic.com/engineering/building-effective-agentsSecondary source
“you should consider adding complexity only when it demonstrably improves outcomes.”
↩︎ Ship it, fix it, or stop it