What you will be able to do
- Create a serving endpoint for a Unity Catalog model with the REST API, MLflow Deployments SDK or Workspace Client
- Shift traffic between served entities with a config update and confirm the update succeeded
- Query a challenger directly and monitor it before increasing its traffic
- Automate a staged 1% → 10% → 100% rollout, with a rollback path, using an MLflow deployment job
1.Create the endpoint under a durable identity
Model Serving gives you a REST API and the MLflow Deployments API for creating, updating, deleting and querying endpoints, plus the Serving UI. The Databricks Workspace Client SDK can also create endpoints. Each served entity names a Unity Catalog model by its full catalog.schema.model path and an entity_version. A rollout is planned around those versions.
mlflow.set_registry_uri("databricks-uc")
client = get_deploy_client("databricks")
endpoint = client.create_endpoint(
name="unity-catalog-model-endpoint",
config={
"served_entities": [
{
"name": "ads-entity",
"entity_name": "catalog.schema.my-ads-model",
"entity_version": "3",
"min_provisioned_concurrency": 4,
"max_provisioned_concurrency": 12,
"scale_to_zero_enabled": False
}
]
}
)Pay attention to who creates the endpoint, because it affects every later rollout step. Databricks records the calling identity as the endpoint's creator, and that cannot be changed. Each configuration update checks again that the creator is still a workspace member and still has USE CATALOG, USE SCHEMA and EXECUTE on the model. If the creator was a person who has since left the workspace, the next traffic shift fails with PERMISSION_DENIED, even when the person submitting the update has the right permissions. The fix is to delete the endpoint and recreate it. Databricks therefore recommends creating endpoints with a long-lived service principal owned by your team.
Checkpoint 1 of 8· Fill the gap
Which MLflow Deployments client method completes this sample, which brings a new endpoint up for the first time?
mlflow.set_registry_uri("databricks-uc")
client = get_deploy_client("databricks")
endpoint = client. ? (
name="unity-catalog-model-endpoint",
config={
"served_entities": [
{
"name": "ads-entity",
"entity_name": "catalog.schema.my-ads-model",
"entity_version": "3",
"min_provisioned_concurrency": 4,
"max_provisioned_concurrency": 12,
"scale_to_zero_enabled": False
}
]
}
)create_endpoint creates a new endpoint. update_endpoint_config is what later rollout steps use to change served entities and traffic on an endpoint that already exists.
Source: docs.databricks.comCheckpoint 2 of 8· Check yourself
An endpoint was created by an engineer who has since left the company. A traffic-shift update submitted by a current admin fails with PERMISSION_DENIED. What is the documented fix?
The recorded creator cannot be changed, and updates check that creator's membership and grants. If the creator is gone, the endpoint has to be recreated.
“is used to access Unity Catalog resources on behalf of the endpoint and cannot be changed after creation”Source: docs.databricks.com
2.Shift traffic with a config update and confirm it succeeded
Each rollout step is a PUT /api/2.0/serving-endpoints/{name}/config request, or the equivalent update_endpoint_config call, or Edit configuration in the Serving tab. The request lists the served entities and a traffic_config. A canary step changes the percentages, for example from 90/10 to 50/50. A full cutover lists only the new version with a single route at 100%.
"traffic_config":
{
"routes": [
{
"served_model_name": "my-ads-model-5",
"traffic_percentage": 100
}
]
}Never assume a step has taken effect. You can read the endpoint's status in the UI or programmatically. The possible states are Ready, Ready (Update failed), Not ready (Updating), Not ready (Update failed) and Not ready (Stopped). "Ready" means the endpoint can be queried. "Update failed" means the latest change did not apply. So Ready (Update failed) is an endpoint still serving its old traffic split, not the one you just requested. Until an update finishes, you cannot submit another one. You can cancel an update in progress from the Serving UI.
from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
endpoint = w.serving_endpoints.get(name="my-endpoint")
print(f"Endpoint state: {endpoint.state.ready}")
print(f"Update state: {endpoint.state.config_update}")Checkpoint 3 of 8· Check yourself
After you submit a 10% → 50% traffic shift, the endpoint shows "Ready (Update failed)". What does that mean?
"Ready" means the endpoint can be queried. "Update failed" means the latest change did not apply, so the previous split is still serving.
“Updated failed indicates the latest change to the endpoint was unsuccessful.”Source: docs.databricks.com
Checkpoint 4 of 8· Exam question
A retailer's recommendation model runs on a Model Serving endpoint handling live web traffic around the clock, so downtime directly costs revenue. The team compares blue-green and canary strategies for rolling out a retrained model. Which statement correctly compares them for this use case?
Correct answer: A — Canary exposes the new model to a small, growing slice of real traffic first, limiting blast radius, while blue-green cuts every request over to the new version at once.
- A. This captures the real tradeoff: canary limits blast radius through gradual exposure, while blue-green replaces the serving version for all traffic in one cutover.
- B. A well-implemented blue-green switch on Model Serving routes traffic to the new served entity without taking the endpoint offline; the strategies differ in blast radius, not in guaranteed downtime.
- C. The two strategies differ in how much traffic reaches the new version and for how long, which is a meaningful operational distinction, not merely a naming difference.
- D. Both strategies apply to how a serving endpoint routes requests regardless of batch or real-time mode, so inference mode alone does not dictate which one is usable here.
3.Validate the challenger before giving it more traffic
This per-entity path, POST /serving-endpoints/{endpoint-name}/served-models/{served-model-name}/invocations, takes the same request format as the endpoint itself. You can use it to smoke-test the challenger with known inputs before any users reach it. Once it is taking real traffic, decide whether to move to the next step using these tools. Endpoint health metrics show latency, request rate, error rate, CPU and memory, and the Serving UI keeps 14 days of them. AI Gateway-enabled inference tables log requests and responses to Delta tables in Unity Catalog, so you can check the quality of the challenger's predictions. Service logs and build logs help when a new version will not deploy.
Checkpoint 5 of 8· Check yourself
You need to compare the challenger's actual predictions against the current version's over a week of canary traffic. Which tool records the request and response data you need?
Inference tables store online requests and responses in Delta tables, which makes them the tool for judging model quality. Service logs and build logs are for debugging deployments.
“Automatically logs online prediction requests and responses into Delta tables managed by Unity Catalog”Source: docs.databricks.com
4.Automate the staged rollout with a deployment job
MLflow deployment jobs (Public Preview) automate the rollout. A Lakeflow Job is connected to a Unity Catalog model and starts automatically whenever a new model version is created. The simple template has three tasks: evaluation, approval and deployment. The documentation also describes a staged rollout: after approval, deploy to 1%, then to 10%, collect and check metrics, decide whether to proceed, and then either deploy to 100% or roll back. Each deploy task is a traffic_config update like the ones in the previous section.
Checkpoint 6 of 8· Put it in order
Put the tasks of the documented staged-rollout deployment job in order, starting after approval
- 1.Deploy the model to 100% or roll it back
- 2.Decide whether to proceed or roll back
- 3.Deploy the model to 10%
- 4.Deploy the model to 1%
- 5.Collect and check metrics
The documented staged rollout increases exposure in steps and puts a metrics check before the final go or rollback decision.
“deploy the model to 1%, deploy the model to 10%, collect and check metrics, decide whether to proceed or rollback”Source: docs.databricks.com
A few settings stop the job from working against the endpoint. The job must have job-level parameters named model_name and model_version. Keep the maximum concurrent runs at 1, which is the default, so two new versions cannot start overlapping rollouts. This also matches the endpoint rule that only one update can run at a time. Turn off retries on the approval task, because it is expected to fail until someone approves. Set Run As to a service principal with minimal permissions: the job runs with the model owner's credentials, so anyone who can create model versions could otherwise run arbitrary code through it.
Checkpoint 7 of 8· Check yourself
Why does Databricks recommend keeping a deployment job's max concurrent run limit at 1?
If two versions are registered close together, concurrent runs could overwrite each other's traffic changes. A limit of one run makes the rollouts happen one after another.
“Databricks also recommends setting the max concurrent run limit to 1 (the default) to prevent deployment race conditions.”Source: docs.databricks.com
Checkpoint 8 of 8· Exam question
A team wants rollback safety on a Model Serving endpoint: if a newly promoted challenger starts returning degraded predictions, on-call engineers must revert traffic to the previous version within minutes, without redeploying the endpoint from scratch. Which design achieves this?
Correct answer: A — Keep both the champion and challenger registered as served entities on the endpoint and shift `traffic_config` percentages back to the champion when needed.
- A. Keeping both versions as served entities means rollback is just a traffic-config percentage change on the existing endpoint, which is fast and needs no redeploy.
- B. Relying on client applications to switch endpoint URLs during an incident is slower and more error-prone than an in-place traffic shift, and duplicates infrastructure needlessly.
- C. With only the challenger registered, reverting requires re-registering and redeploying the champion version, which does not meet a within-minutes rollback requirement.
- D. Scale-to-zero stops serving entirely rather than routing traffic back to a known-good version, so it removes capacity during an incident instead of providing a rollback path.
Sources6
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Requests sent to /served-models/challenger/invocations still follow the endpoint's 90/10 traffic split.Why is that wrong?
Querying a served model directly ignores the traffic settings, so all of those requests go to the challenger.
Covered in Validate the challenger before giving it more traffic
2.If the person who created an endpoint leaves, a current admin can keep updating its traffic as long as the admin has permissions.Why is that wrong?
Updates check the recorded creator's workspace membership and grants again, so they fail if the creator is gone. Use a team-owned service principal as the creator.
Covered in Create the endpoint under a durable identity
Practise it for real
Run a canary rollout of a second Unity Catalog model version on one Model Serving endpoint, from 90/10 to a full cutover
1.Create an endpoint with POST /api/2.0/serving-endpoints that has two served entities, current (your existing version) and challenger (the new version), with traffic_config routes of 90 and 10, and scale_to_zero_enabled set to false.
Why: This puts the new version next to the old one on the same endpoint, with limited exposure.
You should see: The endpoint first shows Not Ready, then reaches READY with both served entities in DEPLOYMENT_READY.
2.Check readiness with w.serving_endpoints.get(name=...) and print state.ready and state.config_update.
Why: A config change only counts once the endpoint reports that the update has finished.
You should see: state.ready is READY and the update state is NOT_UPDATING.
3.Send a few known inputs to /serving-endpoints/{name}/served-models/challenger/invocations.
Why: This tests the new version on its own, regardless of the 10% split.
You should see: Every request is answered by the challenger.
4.Submit PUT /api/2.0/serving-endpoints/{name}/config with the same served entities and routes changed to 50/50, then check the status again.
Why: Each traffic step is a full config update, and only one can run at a time.
You should see: The endpoint keeps serving 90/10 while the update runs, then reports READY with the 50/50 routes.
5.Submit a final update that lists only the new version with traffic_percentage 100.
Why: This is the cutover. The old configuration keeps serving until the new one is ready.
You should see: The endpoint reports READY with a single route at 100%. If the update fails, the status shows Update failed and the 50/50 split stays live.
Stuck? Get a nudge
If an update returns PERMISSION_DENIED, check whether the endpoint's recorded creator is still a workspace member and still has EXECUTE on both model versions.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Model Serving offers a unified REST API and MLflow Deployment API for CRUD and querying tasks.”
↩︎ Create the endpoint under a durable identity - 2.https://docs.databricks.com/aws/en/machine-learning/model-serving/create-manage-serving-endpointsOfficial docs
“Updates fail with PERMISSION_DENIED if the recorded creator is no longer a workspace member, even when the caller has valid permissions.”
↩︎ Create the endpoint under a durable identity“you can cancel an in progress update from the Serving UI.”
↩︎ Shift traffic with a config update and confirm it succeeded“Use a long-lived service principal owned by your team as the endpoint creator.”
↩︎ Exam trap 2“is used to access Unity Catalog resources on behalf of the endpoint and cannot be changed after creation”
↩︎ Checkpoint - 3.https://docs.databricks.com/aws/en/machine-learning/model-serving/serve-multiple-models-to-serving-endpointOfficial docs
“You can also update the traffic split between served models.”
↩︎ Shift traffic with a config update and confirm it succeeded“While querying the individual served model, the traffic settings are ignored.”
↩︎ Validate the challenger before giving it more traffic“if all requests are sent to /serving-endpoints/multi-model/served-models/challenger/invocations, then all requests are served by the challenger served model.”
↩︎ Exam trap 1 - 4.https://docs.databricks.com/aws/en/machine-learning/model-serving/manage-serving-endpointsOfficial docs
“Readiness refers to whether or not an endpoint can be queried.”
↩︎ Shift traffic with a config update and confirm it succeeded“Updated failed indicates the latest change to the endpoint was unsuccessful.”
↩︎ Checkpoint - 5.https://docs.databricks.com/aws/en/machine-learning/model-serving/monitor-diagnose-endpointsOfficial docs
“Provides insights into infrastructure metrics like latency, request rate, error rate, CPU usage, and memory usage.”
↩︎ Validate the challenger before giving it more traffic“Automatically logs online prediction requests and responses into Delta tables managed by Unity Catalog”
↩︎ Checkpoint - 6.
“Deployment jobs allow you to automate tasks like evaluation, approval, and deployment whenever a new model version is created”
↩︎ Automate the staged rollout with a deployment job“Databricks recommends that you set up the deployment job using a service principal with minimal permissions to prevent privilege escalation.”
↩︎ Automate the staged rollout with a deployment job“deploy the model to 1%, deploy the model to 10%, collect and check metrics, decide whether to proceed or rollback”
↩︎ Checkpoint“Databricks also recommends setting the max concurrent run limit to 1 (the default) to prevent deployment race conditions.”
↩︎ Checkpoint