What you will be able to do
- Triage a Claude-system outage or quality regression into a scoped claim tied to a specific change
- Pick the mitigation that restores known-good behaviour fastest: revert, fallback model, batch path or capped retries
- Explain why retries amplify an overload incident and how to shed load instead
- Capture the evidence (request IDs, transcripts, versions) that communication and the postmortem depend on
- Write an early incident status update that states scope, known facts and mitigation, after checking the provider's status page
- Turn an incident into regression evals, monitors and ranked postmortem findings, and re-enable a fixed system safely
Key concept
Mitigate before you diagnose — During an LLM incident, first restore known-good behaviour by reverting or isolating the change that lines up with the regression, even before the root cause is proven. Diagnosis, the permanent fix and the regression tests come afterwards, and the incident only closes once the gap that let the failure through has been closed.
1.Triage when the signal looks like noise
Two real Anthropic incidents anchor this lesson. Between early August and mid-September, three infrastructure bugs intermittently degraded Claude's responses. Some Sonnet 4 requests were misrouted to servers set up for the 1M-token context window, starting August 5 with 0.8% of requests. A misconfiguration on the TPU servers, deployed August 25, made the model occasionally emit improbable tokens, such as Thai characters in an English answer. A sampling change deployed the same day triggered a latent bug in the XLA:TPU compiler's approximate top-k operation. In spring, three separate changes to Claude Code hurt its quality: a default reasoning effort lowered from high to medium on March 4, a bug from March 26 that cleared Claude's earlier thinking on every turn of a session, and a system prompt instruction added on April 16 to reduce verbosity. All three were resolved by April 20.
An LLM outage is easy to spot. A wall of 529 overloaded_error responses can only mean one thing. A quality regression is harder, because it shows up as complaints that sound like ordinary variation. In August, Anthropic's first reports of degraded Claude responses were difficult to tell apart from normal variation in user feedback. The investigation opened only when the reports became more frequent and kept coming. The Claude Code reports of March and April went the same way. Three separate changes each hit a different slice of traffic on a different schedule, so the combined effect looked like broad, inconsistent degradation.
Triage turns vague reports into a scoped claim: which traffic is affected, since when, and which change it lines up with. That depends on two kinds of instrumentation. Operational metrics such as response time and uptime catch outages. They say nothing about whether the agent still escalates correctly or writes valid code, so a regression in behaviour needs a quality score on sampled production traffic. The alerting lesson covers how to wire that up. You also need a change log you can line up against the metric. Anthropic's postmortem names this gap directly. When negative reports spiked on August 29, nobody linked them to a load-balancing change made that same day.
Expect impact to be uneven, and size severity with that in mind. Anthropic's routing was sticky: once a request reached a misconfigured server, follow-up requests were likely to go to the same server. Some users therefore saw failure after failure while most saw nothing. At the worst hour, 16% of Sonnet 4 requests were affected. Over the whole period, about 30% of Claude Code users had at least one message misrouted. An average across all requests understates what the worst-affected users went through. Triage needs only enough evidence to choose a mitigation. Replaying traces and separating prompt, tool and model causes belong to complex failure diagnosis.
Communication starts during triage, not after the fix. First rule out the provider: Claude Code's error guidance sends you to status.claude.com, or the provider status page named in the error message, to look for active incidents before you suspect your own system. Then post a short status update for your team and your users. It should say what users are seeing, which traffic is affected, what is known and what is still unknown, which mitigation is under way, and when the next update will come. Claim only what the evidence supports, and rule causes out in plain words when you can. Anthropic's postmortem stated that quality is never reduced because of demand, time of day or server load, which spared customers from guessing at a cause. The April report also paired its explanation with a concrete remedy, resetting usage limits for subscribers. Details of what to log for those updates are in the evidence section below.
Checkpoint 1 of 9· Check yourself
Your customer-support agent's dashboards show normal response times and 99.99% uptime. Over two weeks, though, more and more complaints say it has stopped escalating cases properly. Which triage step best turns these reports into a scoped claim that you can act on?
Operational metrics catch outages, not changes in behaviour. To scope a quality regression, you need a quality score on sampled production traffic and a change log to compare it with. Averages understate uneven impact, and replaying traces is diagnosis work that comes after mitigation, not part of triage.
“Although we were aware of an increase in reports online, we lacked a clear way to connect these to each of our recent changes.”Source: www.anthropic.com
2.Restore known-good behaviour before finding the cause
Restore the last known-good state first, and find out why afterwards. Every minute spent analysing a live regression is a minute users spend inside it. The failing traces are already recorded, so reverting does not destroy the evidence you need. Anthropic rolled back a TPU sampling change on Sonnet 4 even though it could not reproduce the bug there. Claude Code's system prompt had gained a line on April 16 telling Claude to keep text between tool calls to 25 words or fewer and final responses to 100 words or fewer unless the task needed more. When a broader evaluation linked a 3% quality drop to that one instruction, the prompt was reverted immediately in the April 20 release.
Mitigations work best in this order, starting with whatever removes the change that lines up with the regression:
- Revert the change. Flip the prompt or config flag back, or redeploy the previous version. The old version has already been validated, so this is the safest move.
- Turn the feature off. Use a feature flag or kill switch when there is no clean previous version to go back to.
- Route traffic to a fallback model, but only one you have already tested on your own tasks.
Patching forward and upgrading the model look like action, but each one stacks a new, unvalidated change on top of an incident. Take a prompt that gets firmer rules bolted on mid-incident. It has run against no evals at all. The April verbosity instruction passed weeks of internal evals and still hurt coding quality. A larger model tier is a different model, with different behaviour on your prompts. The deprecations guidance is to test replacement models well before you need them, and an outage is not that time.
Checkpoint 2 of 9· Check yourself
On Thursday, one line was added to your support agent's system prompt. Since then, the sampled escalation score has dropped. The previous prompt is still in version control. What should the on-call engineer do first?
Reverting to the prompt that was already validated is the lowest-risk way to mitigate, and the failing traces stay recorded for offline analysis. A prompt patch made mid-incident and a different model tier each add an untested change, and keeping the suspect prompt live leaves users stuck in the regression.
“We immediately reverted the prompt as part of the April 20 release.”Source: www.anthropic.com
Checkpoint 3 of 9· Exam question
Yarrow Travel runs a customer support agent in production that handles booking changes and refund disputes, deciding when a case must be escalated to a human. Its system prompt was updated at 09:00 as part of a routine release. By 11:00, sampled-trace scores for correct escalation have dropped from 94% to 71%, and complaints from customers are rising. Every sampled trace is stored with its full inputs and outputs, and every earlier prompt version is retained in the config service. The previous prompt version ran for weeks at the old score, and it can be restored by a flag flip that takes effect in seconds. Nobody has yet reviewed the failing traces, so the cause of the drop is unknown. The on-call engineer must act while complaints continue to arrive. What should the on-call engineer do FIRST?
Correct answer: A — Flip the config flag back to the earlier prompt version, confirm sampled escalation scores recover, then analyze the failing traces offline.
- A. Correct: the earlier prompt version is a known-good state that one flag flip restores, so it is the quickest way to stop the harm to customers. The failing traces are already recorded, so they can be analyzed after the impact is contained.
- B. Incorrect: reverting the prompt does not delete the recorded traces or the stored prompt versions, so nothing is lost by acting first. Waiting for a confirmed cause leaves the regressed prompt serving customers while complaints keep growing.
- C. Incorrect: the cause has not been established, so assuming the prompt lacks escalation guidance is a guess. An untested edit pushed during an incident can add a second regression, when a verified earlier version is one flag flip away.
- D. Incorrect: the drop began with the prompt update, so a different model tier does not address its cause. It changes a second variable, adds cost, and leaves the faulty prompt live, whereas the earlier version can be restored directly.
3.Overload incidents: shed load instead of amplifying it
During a capacity event, retries are part of the load. The API signals overload with 529 overloaded_error. The official SDK already retries transient failures, including 5xx errors, with exponential backoff, twice by default, and honours retry-after when the response includes it. Every retry is another request to a service that is already overloaded. When 35% of calls fail and each failure is retried, the client fleet sends more traffic than real demand, and every failed call also spends time in backoff, which pushes p95 latency up. Raising per-request retry counts or timeouts so that "every request eventually succeeds" makes both problems worse.
Two mitigations actually help. The first caps retries. The SDK client accepts max_retries to tune or disable retrying. Above that per-client setting, a retry budget shared across the fleet stops retries when the failure rate is high, and randomised backoff keeps clients from retrying in synchronised waves. The second splits traffic by criticality. Keep the interactive path, such as checkout, on the primary model. Move non-critical work to a fallback model you have already validated, or to the Message Batches API, which suits work that does not need an immediate answer; most batches finish within an hour.
The tempting non-answers fail for specific reasons. Turning off logging of failed requests saves almost nothing on your application servers, does nothing about the provider-side bottleneck, and throws away the evidence the postmortem needs. Opening a ticket and waiting is escalation, not mitigation, because users stay affected the whole time. Padding prompts with "be brief" instructions adds input tokens to every request, which is the wrong direction during an overload.
| Mitigation | Wins when | Watch out for |
|---|---|---|
| Revert prompt, config or deploy | A recent change lines up with the regression | Confirm recovery on the metric that detected the problem |
| Cap retries (max_retries, shared budget) | 529 overloaded_error rate is high and retries add load | Some requests fail fast instead of succeeding late |
| Fallback model | The primary model is overloaded or regressed | Only safe if already tested on your tasks |
| Message Batches path | Work does not need an immediate response | Results arrive asynchronously, usually within an hour |
| Feature flag or kill switch | No clean previous version exists | Users lose the feature until it is fixed |
Checkpoint 4 of 9· Check yourself
Checkout is pinned to the primary model and retries are already capped, but the API still returns a high rate of 529 overloaded_error. An in-app summary feature is not critical, but users still expect its answers within seconds. What should you do with the summary traffic?
Moving non-critical traffic to a fallback model you have already validated sheds load without adding risk. Extra retries send more requests to a service that is already overloaded, and 'be brief' instructions add input tokens. An untested tier is a new, unvalidated change in the middle of an incident, because each model behaves differently on your prompts.
“Each model behaves slightly differently, and we spend time before each release optimizing the harness and product for it.”Source: www.anthropic.com
Checkpoint 5 of 9· Exam question
Quarry Software runs a Claude-backed checkout assistant for its retail customers. During a regional capacity event, about 35% of calls return overloaded errors. The SDK already retries with backoff, yet p95 latency has tripled, application workers are saturated by requests waiting on retries, and dashboards show retry traffic arriving in synchronized waves that is itself raising load. Roughly half of all calls come from non-critical features such as product-description generation and order-status chat, which tolerate delay; a secondary model passed the team's evaluation for those tasks last quarter. The provider's status page acknowledges the event but gives no recovery estimate, and the postmortem will depend on failed-request logs. Which TWO actions best mitigate the incident? Select TWO.(Select 2)
Correct answers: A, C — Cap total retry attempts with a shared retry budget and jitter, so clients shed load instead of amplifying the overload.; Route non-critical traffic to a pre-validated fallback model or a queued batch path, keeping the interactive path for checkout flows.
- A. Correct: a bounded retry budget with jitter stops synchronized retries from multiplying load, which is a recognised cause of prolonged overload incidents.
- B. Incorrect: more retries and longer timeouts increase concurrent load and tie up workers. The failures become slower rather than rarer.
- C. Correct: shifting low-priority work to a validated fallback or deferred path preserves capacity and success rate for the revenue-critical flow. The fallback must have been evaluated beforehand.
- D. Incorrect: failed-request logs are the evidence needed for the postmortem and their overhead is negligible. Removing them reduces observability without fixing the overload.
- E. Incorrect: passively waiting leaves customers failing and ignores mitigations fully under your control. Contacting the provider is useful in parallel but is not itself a mitigation.
- F. Incorrect: lengthening the prompt increases input tokens and does not reduce overload errors. Output brevity instructions are not a capacity control.
4.Communicating and preserving evidence
An incident has two audiences: the provider you escalate to, and the users and stakeholders who were affected. Both need specific, per-request facts. Every Claude API response carries a unique request-id header, and error bodies repeat it as request_id. When you contact support about a specific request, include that ID. Log it with your own user and session identifiers on every call. Only that pairing lets you later answer questions like "how many users were affected, and what did each one see?".
{
"type": "error",
"error": {
"type": "not_found_error",
"message": "The requested resource could not be found."
},
"request_id": "req_011CSHoEeqs5C35K2UUqR7Fy"
}Capture evidence before you roll back, because access to it may not last. Anthropic's privacy and security controls limit when engineers can see user interactions that were not submitted as feedback. That protected users but kept engineers from examining the conversations they needed to reproduce the bugs. Your own retention rules can do the same to you. Before you flip the flag back, save the request IDs of failing calls, their full transcripts, and the exact prompt, model and config versions that were live.
Tell users what happened in concrete terms. Anthropic's public account gave dates, models and percentages for each platform, such as 0.8% of Sonnet 4 requests at first and 0.18% at the peak on Amazon Bedrock. It said which platforms were unaffected and when each fix finished rolling out. It also stated plainly what was not the cause: quality is never reduced because of demand or load. That precision lets each customer work out whether they were affected, which a vague "some users saw degraded quality" does not.
Checkpoint 6 of 9· Check yourself
Your logging pipeline deletes full conversation transcripts after 72 hours, and engineers cannot open user conversations without a ticket. A config flag lines up with a quality regression, and the postmortem is booked for next week. You are about to flip the flag back. What belongs in the minutes before the revert?
Reverting quickly is still right, but retention rules and access controls can put the evidence out of reach before the postmortem. Capture request IDs, transcripts and the live versions first, then roll back. Do not wait for a reproduction, do not count on the logs still being there next week, and do not settle for vague aggregates.
“This protects user privacy but prevents engineers from examining the problematic interactions needed to identify or reproduce bugs.”Source: www.anthropic.com
5.Turning the incident into regression evals and monitors
An incident is a free, realistic test case, so turn it into one. Anthropic's eval guidance describes the payoff: failures become test cases, and test cases prevent regressions. Take the transcripts you saved, make each failing input a task, and write a grader for the specific failure. Some failures suit a code-based check. After the token-corruption bug, Anthropic added detection tests for unexpected character output, such as Thai text in an English reply, to its deployment process. That eval is a gate on every future deploy. For grader and suite design, see evaluation system design.
Give the regression eval a measurable success criterion. "Escalates correctly" is a wish. "At least 94% correct escalation on the incident trace set" is a gate. Include the edge cases the incident exposed: stale sessions, a particular platform, long contexts. Run several trials per task as well. Model outputs vary between runs, and a nondeterministic failure can pass a single trial and still be live.
The evals did not measure what users experienced. Claude often recovers well from isolated mistakes, so occasional corrupted tokens barely moved benchmark scores, and the April prompt change passed every eval in the narrower set that was run. Fix both the coverage of the eval and the production monitor.
Add monitoring alongside the eval, because an eval guards only against failures you already know about. Production monitoring catches distribution drift and real-world failures nobody anticipated. In both postmortems, detection was the slowest step. In August, Anthropic relied too heavily on noisy evaluations, and its own evals did not reproduce the problem. Every incident should add a production signal for the behaviour that broke, so the next variant is caught by a metric rather than by complaints. No single layer catches everything. Evals, monitors and transcript review each catch failures the others miss.
Checkpoint 7 of 9· Match them up
After the incident is mitigated, you add four safeguards. Match each one to the gap it closes.
Tap a term, then the definition that fits it.
A regression eval only guards against failures you already know about, so production monitoring is added to catch drift and unforeseen failures. Running several trials handles output that varies between runs, and a numeric threshold makes the eval a real gate rather than a wish.
“Production monitoring kicks in post-launch to detect distribution drift and unanticipated real-world failures.”Source: www.anthropic.com
6.Re-enabling safely and ranking postmortem findings
A fix that passes its new regression eval has proven only that one failure is gone. It has not shown that the fix broke nothing else. The April prompt change shows the risk: weeks of internal testing found no regressions in the evaluations that were run, and a broader set of ablations later found a 3% drop. Before re-enabling a disabled agent for everyone:
- Run the full regression suite and compare the results against the last known-good baseline.
- Replay the saved incident traces through the fixed version, with several trials per trace.
- Enable a small canary group first, watch the same quality monitor that detected the incident, and keep the rollback flag ready.
- Widen the rollout only when the canary's metric holds.
Two other re-enable paths are traps. Bundling the fix into a large quarterly deploy leaves users without the agent for no reason, and it mixes many changes together, which is the very condition that made attribution hard in both postmortems. Having the author try five prompts by hand is a spot check on a nondeterministic system and is no better than the noisy signal that caused the trouble.
Rank postmortem findings by what will prevent the next incident, not by what felt most dramatic. A failure class that already appeared in an earlier review comes first, because it means the previous learning loop never closed. A long detection gap, such as three hours before anyone noticed, usually outranks a slow mitigation once detected, because it shows that monitoring cannot see the failure at all. The strongest action items change a mechanism rather than ask people to be more careful. Anthropic added character-output tests to its deploy gate. It also looked at why the March 26 bug, which cleared Claude's earlier thinking on every turn instead of once, got past multiple human and automated code reviews. Back-testing showed that Opus 4.7 found the bug when given the repositories needed for complete context, while Opus 4.6 did not. So code review now gets additional repositories as context. That changes what the reviewer can see, rather than asking reviewers to look harder. Anthropic also ran a broader set of evaluations on system-prompt changes.
Checkpoint 8 of 9· Match them up
Your fix passes its new regression eval, and the support agent is still disabled for everyone. Match each re-enable plan to what it actually gives you.
Tap a term, then the definition that fits it.
Passing one new eval doesn't rule out regressions elsewhere. Anthropic's April prompt change passed weeks of evals and still lowered coding quality. A big bundled deploy hides which change caused a problem, and a few manual prompts can't tell a real fix from luck. A monitored canary with rollback ready is the safe way to re-enable.
“After multiple weeks of internal testing and no regressions in the set of evaluations we ran, we felt confident about the change”Source: www.anthropic.com
Checkpoint 9 of 9· Exam question
Russet Telecom runs a customer-support agent that resolves billing disputes and plan-change requests. After a prompt update, it began misquoting early-termination fees, and the team disabled it in production; human agents have handled the queue since, and leadership wants it back. The engineer who wrote the fix added a regression eval for the fee-quoting failure, and it passes in staging. That eval was built from only the three tickets that first surfaced the problem, and nobody has checked whether the fix changed behavior on plan-change flows or other earlier-passing cases. The team has a monitoring dashboard that tracks answer quality and can disable the agent within minutes. What should happen BEFORE you re-enable it for all users?
Correct answer: A — Run the full regression suite plus incident-trace replay, then enable for a small canary with the quality monitor watching and rollback ready.
- A. Correct: broad regression coverage, incident replay and a monitored canary with a rollback ready confirm the fix works and has not caused new problems before full exposure.
- B. Incorrect: a single new eval can be overfit, and full exposure without a canary risks a second incident.
- C. Incorrect: delaying to a quarterly window prolongs the outage with no safety benefit. Controlled staged rollout is the safer route.
- D. Incorrect: five manual prompts by the author is a tiny, biased sample and is not a systematic gate.
Sources
(8)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Operational: Response time (ms), uptime (%)”
↩︎ Triage when the signal looks like noise“Edge case analysis: Percentage of edge cases handled without errors.”
↩︎ Turning the incident into regression evals and monitors“A/B testing: Compare performance against a baseline model or earlier version.”
↩︎ Re-enabling safely and ranking postmortem findings - 2.
“Between August and early September, three infrastructure bugs intermittently degraded Claude's response quality.”
↩︎ Triage when the signal looks like noise“On August 5, some Sonnet 4 requests were misrouted to servers configured for the upcoming 1M token context window.”
↩︎ Triage when the signal looks like noise“These initial reports were difficult to distinguish from normal variation in user feedback.”
↩︎ Triage when the signal looks like noise“When negative reports spiked on August 29, we didn't immediately make the connection to an otherwise standard load balancing change.”
↩︎ Triage when the signal looks like noise“once a request was served by the incorrect server, subsequent follow-ups were likely to be served by the same incorrect server.”
↩︎ Triage when the signal looks like noise“We never reduce model quality due to demand, time of day, or server load.”
↩︎ Triage when the signal looks like noise“we were unable to reproduce this bug on Sonnet 4 but decided to also roll it back out of an abundance of caution”
↩︎ Restore known-good behaviour before finding the cause“rolled out a fix to use exact top-k with enhanced precision”
↩︎ Restore known-good behaviour before finding the cause“Our internal privacy and security controls limit how and when engineers can access user interactions with Claude”
↩︎ Communicating and preserving evidence“We've added detection tests for unexpected character outputs to our deployment process.”
↩︎ Turning the incident into regression evals and monitors“More fundamentally, we relied too heavily on noisy evaluations.”
↩︎ Turning the incident into regression evals and monitors“Engineering teams perform spot checks and deploy to small "canary" groups first.”
↩︎ Re-enabling safely and ranking postmortem findings“we were unable to reproduce this bug on Sonnet 4 but decided to also roll it back out of an abundance of caution”
↩︎ Key concept“we were unable to reproduce this bug on Sonnet 4 but decided to also roll it back out of an abundance of caution”
↩︎ Exam trap 1“The evaluations we ran simply didn't capture the degradation users were reporting, in part because Claude often recovers well from isolated mistakes.”
↩︎ Exam trap 4“Although we were aware of an increase in reports online, we lacked a clear way to connect these to each of our recent changes.”
↩︎ Checkpoint“This protects user privacy but prevents engineers from examining the problematic interactions needed to identify or reproduce bugs.”
↩︎ Checkpoint - 3.https://www.anthropic.com/engineering/april-23-postmortemSecondary source
“On April 16, we added a system prompt instruction to reduce verbosity.”
↩︎ Triage when the signal looks like noise“All three issues have now been resolved as of April 20 (v2.1.116).”
↩︎ Triage when the signal looks like noise“Because each change affected a different slice of traffic on a different schedule, the aggregate effect looked like broad, inconsistent degradation.”
↩︎ Triage when the signal looks like noise“As of April 23, we’re resetting usage limits for all subscribers.”
↩︎ Triage when the signal looks like noise“We immediately reverted the prompt as part of the April 20 release.”
↩︎ Restore known-good behaviour before finding the cause“Length limits: keep text between tool calls to ≤25 words. Keep final responses to ≤100 words unless the task requires more detail.”
↩︎ Restore known-good behaviour before finding the cause“One of these evaluations showed a 3% drop for both Opus 4.6 and 4.7.”
↩︎ Restore known-good behaviour before finding the cause“After multiple weeks of internal testing and no regressions in the set of evaluations we ran, we felt confident about the change”
↩︎ Re-enabling safely and ranking postmortem findings“The changes it introduced made it past multiple human and automated code reviews”
↩︎ Re-enabling safely and ranking postmortem findings“Instead of clearing thinking history once, it cleared it on every turn for the rest of the session.”
↩︎ Re-enabling safely and ranking postmortem findings“we are now landing support for additional repositories as context for code reviews.”
↩︎ Re-enabling safely and ranking postmortem findings“After multiple weeks of internal testing and no regressions in the set of evaluations we ran, we felt confident about the change”
↩︎ Exam trap 3“Each model behaves slightly differently, and we spend time before each release optimizing the harness and product for it.”
↩︎ Checkpoint - 4.https://code.claude.com/docs/en/errorsOfficial docs
“Check status.claude.com, or the provider status page named in the message, for active incidents”
↩︎ Triage when the signal looks like noise - 5.
“consider thorough testing of your applications with the new models well before the retirement date”
↩︎ Restore known-good behaviour before finding the cause - 6.https://platform.claude.com/docs/en/api/errorsOfficial docs
“529 - overloaded_error: The API is temporarily overloaded.”
↩︎ Overload incidents: shed load instead of amplifying it“The SDK client accepts max_retries to configure or disable this behavior.”
↩︎ Overload incidents: shed load instead of amplifying it“When contacting support about a specific request, include this ID to help quickly resolve your issue.”
↩︎ Communicating and preserving evidence“Every API response includes a unique request-id header.”
↩︎ Communicating and preserving evidence“The official SDK automatically retries transient failures (such as connection errors, rate limits, and 5xx server errors) with exponential backoff, twice by default”
↩︎ Exam trap 2 - 7.
“This approach is well-suited to tasks that do not require immediate responses, with most batches finishing in less than 1 hour”
↩︎ Overload incidents: shed load instead of amplifying it - 8.
“development accelerates as failures become test cases, test cases prevent regressions, and metrics replace guesswork.”
↩︎ Turning the incident into regression evals and monitors“Because model outputs vary between runs, we run multiple trials to produce more consistent results.”
↩︎ Turning the incident into regression evals and monitors“Production monitoring kicks in post-launch to detect distribution drift and unanticipated real-world failures.”
↩︎ Turning the incident into regression evals and monitors