What you will be able to do
- Define SLIs and SLOs for an LLM-backed service from stop_reason, time to first token and completion time rather than HTTP status alone
- Build token and cost dashboards that count cache tokens correctly and attribute spend by model, workspace, API key and actor
- Choose alert signals for error rate, truncation, cache efficiency, spend and anomalous usage
- Detect and prevent behaviour drift after a model update by recording the served model and gating snapshot changes on regression evals
Key concept
User-visible success SLI — An indicator that counts a request as good only when the user got a usable answer: a successful response that finished for the right reason within its latency target. HTTP 200 on its own does not count, because a truncated answer also returns 200.
1.Defining SLIs that match what users see
An LLM-backed service can pass every infrastructure check and still fail its users. The request returns HTTP 200 inside its latency target, and the answer stops halfway through a sentence. A service level indicator (SLI) helps only if it counts that request as bad. So the first job in monitoring a Claude service is deciding what a good response is.
The field that separates a complete answer from a cut-off one is stop_reason. It is on every successful Messages API response and records why Claude stopped generating, which is a different question from whether the HTTP call worked. Two values mean the user got a partial answer even though the transport reported success: max_tokens, where the response reached your max_tokens limit, and model_context_window_exceeded, where the response filled the model's context window and should be treated as truncated. When tracing is in place (see telemetry and instrumentation), the value reaches your metrics as the anthropic.message.stop_reason span attribute.
| stop_reason | When it occurs | Good for the user? |
|---|---|---|
| end_turn | Claude completed its response naturally. | Yes: the answer is complete |
| max_tokens | The response reached your max_tokens limit. | No: the answer is cut off |
| model_context_window_exceeded | The response filled the model’s context window. | No: treat it as truncated |
| refusal | Claude declined to respond. | No: the user got no answer |
Take a summarization service that reports 99.9% availability from status codes while about 6% of its responses end in max_tokens. The SLI that matches the user's experience is the fraction of requests that return successfully and finish with end_turn, counted per request after the SDK has applied its retries. A request that failed once and succeeded on retry did not fail for the user, so counting attempts would overstate errors. Each weaker candidate passes the broken case. A status-page metric measures the provider, not your traffic. An HTTP-200-within-latency SLI accepts truncated answers. A non-empty-body check accepts half a summary.
An SLO is the SLI plus a target, written the way a measurable success criterion is. "95% response time < 200ms" is a target you can alert on. "Fast" is not. Most services need several of these at once (completion, latency, quality), because no single number covers every way a model-backed feature can let a user down.
Checkpoint 1 of 9· Check yourself
Your team needs to turn the summarizer's latency expectations into an SLO. It must be measured on your own traffic, and you must be able to alert on it. Which proposal meets both requirements?
An SLO is an SLI plus a measurable target, so you can alert when you miss it. "95% under 200 ms" meets that. "Feel fast" and "stay low" give no number. A status-page uptime figure measures the provider, not your traffic.
“95% response time < 200ms”Source: platform.claude.com
Checkpoint 2 of 9· Exam question
Vireo Retail runs a product-description summarization service in production that calls Claude through the SDK, with retries enabled, behind an internal gateway. The dashboard reports 99.9% availability because every request returns HTTP 200, and the provider status page has shown no incident all quarter. Even so, merchandisers complain that summaries stop mid-sentence, and sampled traces show about 6% of responses end with stop_reason of max_tokens. Every truncated response still carries text in its body and arrives well inside the latency target. The reliability lead wants a service level indicator that reflects what users actually experience, so that error budgets and alerts track these complaints. Which SLI best captures user-visible reliability?
Correct answer: C — Fraction of requests returning a successful response that finishes with stop_reason end_turn, counted per request after SDK retries
- A. The provider status page describes the vendor's platform health, not what your service returns. Responses are cut off at the token limit while the page shows no incident, so this ratio stays high and misses the 6% of truncated summaries.
- B. A 200 status with a truncated body still counts as good here, so the 99.9% figure would stay green while users see cut-off summaries. It measures transport health, not output usability.
- C. A response only counts as good when the call succeeds and the model stops on its own (end_turn). Responses ending in max_tokens count as bad, so the roughly 6% of truncated summaries lower the ratio and can consume error budget.
- D. A summary that stops mid-sentence still has text in it, so every truncated response counts as good here. The ratio stays near the 99.9% already reported and never shows the cut-off summaries users complain about.
2.Latency SLIs for streaming responses
Streaming splits latency into two experiences. Total latency, from prompt sent to last token, tells you when the answer is finished. Time to first token (TTFT) measures from when the prompt was sent to when the first token of the response is generated. It matters most for streaming, where it decides whether the interface looks alive or frozen.
The two can diverge sharply. A chat product's p95 total latency can sit comfortably inside its SLO while users stare at a blank pane for several seconds, because a quick token rate later in the stream makes up the time. The fix is a second SLI, not a stricter first one: a time-to-first-token SLI with its own p95 target, alongside a separate completion-time SLI for the full stream. The alternatives each measure the wrong interval. Moving total latency from p95 to p99 still times the whole stream. Tokens per second measures the stream after it has started. Splitting total latency into response-length classes never isolates the wait before the first token.
Measure TTFT where the user experiences it, in the client, because the blank screen includes everything between the user's action and the first rendered text. Spans give you a server-side counterpart. On a streaming response, gen_ai.response.time_to_first_chunk records the seconds to the stream's first content_block_delta event. When the client SLI breaches, compare the two to see whether the wait is in the model call or in your own path around it.
Checkpoint 3 of 9· Check yourself
Users of a streaming chat assistant say the reply pane stays blank for several seconds. The latency SLI is p95 duration of the SDK's anthropic.messages.create span, and it is within target. What should the team change so monitoring catches this problem?
The stream's span covers the whole stream, so a fast token rate later on can hide a long wait before the first token. Only a separate TTFT SLI, measured where the user sees the blank pane, isolates that wait. A p99 target, tokens per second and length classes all still time the wrong interval.
“A stream’s span ends when the stream ends.”Source: platform.claude.com
Checkpoint 4 of 9· Exam question
Willow Robotics runs a customer-support chat product that streams its answers to a web client. The team's only latency SLI is total request duration, and its p95 sits comfortably inside the SLO. Support tickets nonetheless say the interface stays blank for several seconds before any text appears. Engineers checked traces and found that once text starts, it flows at a steady, normal pace. The slow starts show up on short and long answers alike, and the team's dashboards show no breach. The on-call rotation is small and wants every alert to map to something users actually feel. Which change to the SLIs best closes this blind spot?
Correct answer: B — Add a client-measured time-to-first-token SLI with its own p95 target, alongside a separate completion-time SLI for the full stream
- A. A stricter percentile on total request duration still measures only when the response finishes. The wait before the first visible text is not captured, and a gateway cannot see what the user's screen shows.
- B. The blank period is a separate user-perceived delay from total duration. A client-measured time-to-first-token SLI with its own target exposes slow starts that a completion-time SLO hides, and the two SLIs cover both delays.
- C. Throughput describes how fast tokens arrive after streaming has begun. It says nothing about the delay before the first token, so the blank-screen symptom would stay invisible.
- D. Splitting by response length gives finer targets for total duration, but each class still measures completion time. A slow start inside a long generation can still meet its class target, so the blank period stays unmeasured.
3.Token and cost telemetry that adds up
Cost dashboards usually go wrong in the token arithmetic, before any pricing is applied. Each telemetry source has its own convention for prompt-cache tokens, and mixing them up either drops cache tokens or counts them twice.
The conventions differ like this:
- On OpenTelemetry spans from the Claude API and SDK,
gen_ai.usage.input_tokensalready includes tokens read from and written to the prompt cache. Addinggen_ai.usage.cache_read.input_tokenson top counts them twice. - On Claude Code's
claude_code.llm_requestspan andapi_requestevent,input_tokensexcludes the cache. Total input isinput_tokenspluscache_read_tokenspluscache_creation_tokens. - The Usage and Cost Analytics API reports
uncached_input_tokens,cache_creationandcache_read_input_tokensas separate fields, so total input is their sum.
A finance view that prices only input plus output, using a source where input excludes the cache, leaves out every cache token. On a cache-heavy service, that undercounts by a lot.
Two more rules keep span-based totals honest. On the API spans for a Messages API request, the token counts are what your organization is billed for, even when the request fails or the application cancels it. The SDK span holds what the response reported, which can be lower or missing. Spans can also arrive more than once when a delivery is retried, so count each one once by trace ID and span ID.
For organization-wide totals, use the Usage and Cost Analytics API. It is recommended for new integrations over the legacy Admin usage and cost report endpoints. GET /v1/organizations/analytics/usage reports token and request counts, and GET /v1/organizations/analytics/cost reports cost in USD. Results can be grouped by up to five dimensions, such as model, workspace, API key or actor, and split into 1m, 1h or 1d UTC buckets for dashboards. Buckets with no usage produce no rows, so fill gaps with zeros before plotting. Cost figures are decimal strings in cents. For how spend shapes architecture decisions, see cost architecture.
client = anthropic.Anthropic()
rows = client.beta.organization.analytics.usage.list(
starting_at="2026-09-01T00:00:00Z",
ending_at="2026-09-08T00:00:00Z",
time_bucket_width="1d",
group_by=["model"],
limit=10,
)
for row in rows:
print(f"{row.starting_at} {row.group.model}: {row.usage.total_tokens}")Total input is 1,284,500 + 215,300 + 3,200,000 = 4,699,800 tokens. The cache-read share is 3,200,000 / 4,699,800, about 68%. A sudden drop in this share usually means the prompt prefix changed, and the bill will rise along with it.
Checkpoint 5 of 9· Match them up
A cost dashboard pulls input-token counts from three telemetry sources. Match each source's input field to the way you get total input tokens from it without dropping or double-counting cache tokens.
Tap a term, then the definition that fits it.
Each source handles prompt-cache tokens its own way. API and SDK spans already count the cache inside input_tokens, so adding the cache attributes counts those tokens twice. Claude Code's input_tokens and the Analytics API's uncached_input_tokens leave the cache out, so you add the cache fields yourself.
“gen_ai.usage.input_tokens already includes the tokens read from and written to the prompt cache, so don’t add the cache attributes to it.”Source: platform.claude.com
Checkpoint 6 of 9· Fill the gap
This job feeds a dashboard panel with daily token totals for each model across the organization. Which endpoint name completes the call?
client = anthropic.Anthropic()
rows = client.beta.organization.analytics. ? .list(
starting_at="2026-09-01T00:00:00Z",
ending_at="2026-09-08T00:00:00Z",
time_bucket_width="1d",
group_by=["model"],
limit=10,
)
for row in rows:
print(f"{row.starting_at} {row.group.model}: {row.usage.total_tokens}")GET /v1/organizations/analytics/usage reports token and request counts, which is what row.usage.total_tokens reads. The cost endpoint reports spend in USD instead, and usage_report belongs to the legacy Admin endpoints, which the lesson says not to use for new integrations.
Source: platform.claude.com4.Alerts and anomalous usage
Good alerts fire on rates and shifts, not raw counts, and each one comes from a span attribute you already collect. A slowly rising share of max_tokens stops warns that outputs are growing, whether from a prompt change or a model change, before users start complaining about cut-off answers. A falling cache-read share warns of a cost jump. An error rate split by error.type separates throttling, such as rate_limit_error, from requests that will never succeed.
| Signal | Span attribute | Alert when |
|---|---|---|
| Error rate | error.type | Share of failed requests rises above baseline, split by type |
| Truncation | anthropic.message.stop_reason | Share of max_tokens stops climbs |
| Streaming responsiveness | gen_ai.response.time_to_first_chunk | p95 breaches the TTFT target |
| Cache efficiency | gen_ai.usage.cache_read.input_tokens | Cache-read share of input falls |
| Spend | gen_ai.usage.input_tokens, gen_ai.usage.output_tokens | Burn departs from its usual pattern |
| Model version | gen_ai.response.model | The served model changes |
Anomalous usage usually looks like cost spikes, unusual token consumption, or high session volume from specific users. Finding the source means attributing spend to an identity. In the Analytics API, the actor dimension records who a request is attributed to: a user, a service account, a federated cloud identity, or the API key itself when no identity is linked. Group by actor or api_key_id with 1m buckets to catch a runaway script within minutes. Use the same query ranked by cost to answer "which keys spent the most this month".
To tie API spans to your own end users, set user.id in the W3C baggage header. Use an opaque key, such as a database ID, because the API drops any user.id that contains an email address to protect user privacy. You can then alert on per-user spend without putting personal data in your telemetry.
Checkpoint 7 of 9· Check yourself
To alert on per-user spend, a service sends the W3C baggage header user.id=dana.kim@example.com with every Messages API request. On the API spans, user.id never appears. What explains this, and what should the team change?
To protect privacy, the API drops any user.id that contains an email address. An opaque key such as a database ID is kept, and it still lets the team alert on per-user spend without putting personal data in telemetry.
“Do not put an email address in this value. To protect user privacy, the API drops any user.id that contains one.”Source: platform.claude.com
5.Detecting quality drift after model updates
Quality drift is the regression that error and latency alerts can't see. Requests succeed, latency stays flat, and the answers get worse. The most common trigger is a change in which model serves the traffic, so start by knowing exactly when that can happen.
Aliases are different. For models before the 4.6 generation, the Claude API accepts aliases such as claude-sonnet-4-5 that resolve to the most recent dated snapshot for that minor version. A service that calls through an alias can change models with no code change, which is exactly how a router's accuracy falls from 94% to 88% between two weekly audits. Even a fixed ID is not completely static. The serving infrastructure around it (router, safety classifiers, sampling logic) can change. When behaviour shifts on a previously stable ID, an infrastructure update is the most likely cause.
Two controls prevent a repeat. First, record the served model on every response as a dimension of your quality, token and cost metrics. gen_ai.response.model differs from the requested model when the request used an alias or a fallback model served it, so a step change in a metric lines up with a version change instead of turning into a mystery. Second, pin requests to a fixed snapshot and promote a new one only after it passes a regression eval on replayed production traffic, comparing it against the current version as the baseline. For how to build that eval, see evaluation system design.
The tempting shortcuts don't work. A system prompt can't instruct a new model to behave like the previous one, and no sampling setting turns one set of weights into another. Round-the-clock human review of every decision is expensive, and it detects the regression after users have already hit it rather than stopping it. At low volume, where sampled statistics are too noisy, run a fixed automated eval set on a schedule against the production model and alert on its score. Continuous scoring of live traffic is covered in online quality measurement.
Checkpoint 8 of 9· Check yourself
Between two weekly audits, a ticket router's accuracy fell from 94% to 88%. It calls Claude through the alias claude-sonnet-4-5, and no code was deployed in that time. Which control stops a model change from causing this kind of regression again?
The alias moved to a newer snapshot without any code change. Pinning a fixed snapshot stops that, and gating each promotion on a regression eval against the current version catches a regression before users hit it. A prompt instruction or a sampling setting can't make one model's weights behave like another's. Sampled human review is slow and only finds the regression after users are affected.
“A/B testing: Compare performance against a baseline model or earlier version.”Source: platform.claude.com
Checkpoint 9 of 9· Exam question
Lantern Mobility runs a ride-dispatch routing service that sends each rider request to a support queue by calling a model through a provider-managed alias. The provider moved the alias to a new snapshot without notice. A weekly audit then shows routing accuracy fell from 94% to 88% with no code change, and the prompt and sampling settings were untouched. Engineers cannot say which day the drop began, because dashboards show only aggregate quality, token and cost figures, with nothing tying a response to the model that produced it. Company policy says any change to production routing behavior must be deliberate and measured before it ships, and the on-call team is too small to staff continuous manual review. Select TWO actions that best prevent a repeat.(Select 2)
Correct answers: A, D — Record the served model id on each response as a dimension of quality, token and cost metrics to line shifts up with versions; Pin requests to a dated snapshot and promote a new one only after it passes a regression eval on replayed production traffic
- A. Slicing metrics by the served model id lets you see exactly when behavior moved and attribute a drift to a version rather than to traffic or prompts.
- B. A prompt instruction cannot recreate another snapshot's behavior, and it does nothing to detect or gate future changes.
- C. Low temperature reduces sampling variance within one snapshot but different snapshots can still answer differently.
- D. Pinning removes silent behavior changes, and a replay-based gate makes every upgrade a deliberate, measured decision.
- E. Faster detection alone still lets bad snapshots reach production, and reviewing everything by hand does not scale. It detects rather than prevents.
Sources
(9)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“The stop_reason field is part of every successful Messages API response.”
↩︎ Defining SLIs that match what users see“The response reached your max_tokens limit.”
↩︎ Defining SLIs that match what users see“The stop_reason field is part of every successful Messages API response.”
↩︎ Key concept“The stop_reason field is part of every successful Messages API response.”
↩︎ Exam trap 1 - 2.
“95% response time < 200ms”
↩︎ Defining SLIs that match what users see“Most use cases need multidimensional evaluation along several success criteria.”
↩︎ Defining SLIs that match what users see“A/B testing: Compare performance against a baseline model or earlier version.”
↩︎ Detecting quality drift after model updates - 3.https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-latencyOfficial docs
“It’s particularly relevant when you’re using streaming”
↩︎ Latency SLIs for streaming responses“This metric measures the time it takes for the model to generate the first token of the response, from when the prompt was sent.”
↩︎ Exam trap 2 - 4.
“Only on a streaming response’s spans. The seconds to the stream’s first content_block_delta event, if it has one.”
↩︎ Latency SLIs for streaming responses“the token counts are the tokens that your organization is billed for, even when the request fails or the application cancels it.”
↩︎ Token and cost telemetry that adds up“A failed request’s error type, for example rate_limit_error.”
↩︎ Alerts and anomalous usage“To protect user privacy, the API drops any user.id that contains one.”
↩︎ Alerts and anomalous usage“It differs from the requested model when the request uses a model alias, or when a fallback model served the request.”
↩︎ Detecting quality drift after model updates“A stream’s span ends when the stream ends.”
↩︎ Checkpoint“Do not put an email address in this value. To protect user privacy, the API drops any user.id that contains one.”
↩︎ Checkpoint - 5.
“gen_ai.usage.input_tokens already includes the tokens read from and written to the prompt cache, so don’t add the cache attributes to it.”
↩︎ Token and cost telemetry that adds up“Spans can arrive more than once, for example when a delivery is retried.”
↩︎ Token and cost telemetry that adds up“Use them to debug failed or slow requests, track token spend and latency, and build evals from real traffic.”
↩︎ Detecting quality drift after model updates“gen_ai.usage.input_tokens already includes the tokens read from and written to the prompt cache, so don’t add the cache attributes to it.”
↩︎ Exam trap 3 - 6.https://code.claude.com/docs/en/monitoring-usageOfficial docs
“From the span or event: add input_tokens, cache_read_tokens, and cache_creation_tokens”
↩︎ Token and cost telemetry that adds up“High session volume from specific users”
↩︎ Alerts and anomalous usage - 7.
“For new integrations, use the Usage and Cost Analytics API.”
↩︎ Token and cost telemetry that adds up“Currency: All costs are in USD, reported as decimal strings in lowest units (cents).”
↩︎ Token and cost telemetry that adds up - 8.
“Set time_bucket_width to 1m, 1h, or 1d to split each combination by UTC bucket”
↩︎ Token and cost telemetry that adds up“The actor dimension is who a request is attributed to”
↩︎ Alerts and anomalous usage - 9.
“An alias such as claude-sonnet-4-5 is a convenience pointer that resolves to the most recent dated snapshot for that minor version.”
↩︎ Detecting quality drift after model updates“If you notice unexpected behavioral differences on a previously stable model ID, an infrastructure update is the most likely cause.”
↩︎ Detecting quality drift after model updates“An alias such as claude-sonnet-4-5 is a convenience pointer that resolves to the most recent dated snapshot for that minor version.”
↩︎ Exam trap 4“Anthropic does not update the weights or configuration of an existing model ID.”
↩︎ Prediction