What you will be able to do
- Limit production tuning to proven options and register only the best MLflow run
- Gate a retrained model with validation checks before giving it the Challenger alias
- Compare Challenger with Champion offline or online and promote it by updating the alias
- Automate validation and promotion with model update triggers and deployment jobs
1.Narrowing candidates to one registered version
A retraining run often produces several candidates, for example one per hyperparameter setting. During development, data scientists may try many algorithms and hyperparameters. Production code should be narrower: it's common to consider only the top-performing options. Limiting tuning this way saves time and can reduce the variance that tuning adds to automated retraining. If data scientists have already chosen good hyperparameters, the production pipeline can run with that set, typically supplied as a configuration file. Evaluation then uses the metrics the data scientists specified during development, which may include custom code.
When several runs are logged, select the winner by the evaluation metric and register only that run. The getting-started example sorts runs by test AUC and breaks ties by recency.
# Sort runs by their test auc. In case of ties, use the most recent run.
best_run = mlflow.search_runs(
order_by=['metrics.test_auc DESC', 'start_time DESC'],
max_results=10,
).iloc[0]Checkpoint 1 of 6· Fill the gap
Which call turns the best run into a new model version in Unity Catalog?
model_uri = 'runs:/{run_id}/model'.format(
run_id=best_run.run_id
)
mlflow. ? (model_uri, f"{CATALOG_NAME}.{SCHEMA_NAME}.wine_quality_model")mlflow.register_model takes the run's model URI and a three-level Unity Catalog name, and creates a registered model version. Only the selected best run is registered.
Source: docs.databricks.com2.Validation gate and the Challenger alias
The training task yields a model URI, and task values pass that URI to the validation task. The validation pipeline loads the version from Unity Catalog and runs checks that depend on your organization: basic format and metadata validation, performance on selected data slices, and compliance with organizational requirements such as tags or documentation. Its main job is to decide whether the model moves on to deployment.
If every check passes, the version gets the Challenger alias in Unity Catalog. If any check fails, the process exits and users can be notified automatically. Tags record the outcome. For example, a model_validation_status tag can be set to PENDING while checks run and then updated to PASSED or FAILED. Either way, results are recorded on the version in the production catalog, so data scientists can load a failed version and investigate.
Checkpoint 2 of 6· Put it in order
Put the automated retraining stages in order, from new data to a model that is ready for comparison.
- 1.Run validation checks on the registered version
- 2.Assign the Challenger alias if all checks pass
- 3.Evaluate on held-out production data and log results to MLflow Tracking
- 4.Train and tune on production catalog tables
- 5.Register the model version in the production catalog in Unity Catalog
Training, evaluation and registration come first. Validation loads the registered version through its URI, and only a version that passes every check becomes the Challenger.
“If the model does not pass all validation checks, the process exits and users can be automatically notified.”Source: docs.databricks.com
Sources1
3.Challenger versus Champion
Before the new model is deployed, confirm that it performs at least as well as the current production model, which holds the Champion alias. The comparison can be offline or online. An offline comparison scores both models on a held-out dataset and tracks the results in MLflow. For real-time serving you might run longer online comparisons, such as A/B tests or a gradual rollout. One Model Serving endpoint can host several models with a traffic split between them, and Model Serving and data profiling can collect and monitor the endpoint's inference tables. If the Challenger wins, it takes the Champion alias. If there is no Champion yet, compare the Challenger with a business heuristic or another threshold. The flow can run fully automated. If you need manual approval, add workflow notifications or CI/CD callbacks.
| Aspect | Offline comparison | Online comparison |
|---|---|---|
| Data used | A held-out dataset | Live request and response data captured in endpoint inference tables |
| Typical form | Both models evaluated, results tracked in MLflow Tracking | A/B test or gradual rollout, using an endpoint traffic split |
| Duration | Single evaluation run | Longer running |
Promotion is just an alias update. Inference pipelines load whichever version currently holds the Champion alias, so when the alias moves, they automatically use the new version on their next run. This separates deployment from inference. For real-time serving, Model Serving updates an existing endpoint with zero downtime and keeps the old configuration running until the new one is ready.
Checkpoint 3 of 6· Exam question
An engineering team maintains a Delta table of cleaned transaction features that a nightly ETL job refreshes with new rows. A separate retraining job should start automatically only after that feature table has actually been updated, and should not run at all on days when the ETL job produces no new commits. Which trigger configuration satisfies this?
Correct answer: A — A Table update trigger on the feature table, so the retraining job runs only when a new commit lands on that table.
- A. A Table update trigger fires specifically when the referenced table receives a new commit, which is exactly the condition described and avoids running on days when the ETL job produces nothing new.
- B. A fixed schedule assumes the ETL job always finishes at the same time, so it would still fire on days with no new commits and could also run before a delayed ETL job has actually written its update.
- C. Watching the raw input directory reacts to upstream files arriving, not to the feature table itself being committed, so it does not guarantee the table update has actually completed before retraining starts.
- D. The Champion alias only changes after a model is promoted, which happens after retraining, so tying the trigger to that alias would create a circular dependency rather than reacting to the feature table refresh.
Checkpoint 4 of 6· Check yourself
An automated pipeline produces the first validated version of a brand-new model. There is no Champion yet. What should the promotion step compare it against?
With nothing to beat, the comparison still needs a bar. The documented choice is a business heuristic or other threshold.
“to a business heuristic or other threshold as a baseline”Source: docs.databricks.com
Sources1
4.Automating evaluation, approval and rollout
Two features chain these steps without polling. Model update triggers (Beta) start a job when a Unity Catalog model changes. They can be scoped to one model, a schema, or a metastore, and their conditions are *Model is created*, *Model version is ready*, or *Model alias is set* (up to 10 aliases). The changes reach the run through {{job.trigger.model.updates}}.
MLflow 3 deployment jobs (Public Preview) start automatically whenever a new model version is created. A simple deployment job has three tasks: evaluation, which calls mlflow.evaluate; approval; and deployment to a Model Serving endpoint. The job needs job-level parameters model_name and model_version. Databricks recommends a maximum of one concurrent run, which is the default, to prevent deployment race conditions. It also recommends a minimal-permission service principal as Run As, because the job runs with the model owner's credentials. A more cautious version adds tasks after approval: deploy to 1%, deploy to 10%, collect and check metrics, decide whether to proceed or roll back, then go to 100% or roll back.
Checkpoint 5 of 6· Match them up
Match each model update trigger condition to the automation it suits best.
Tap a term, then the definition that fits it.
Version-ready and alias-set triggers drive validation and promotion for the models you own. Schema- or metastore-scoped model-created triggers are for auditing.
“you can run validation, testing, or promotion jobs automatically whenever a model they own changes”Source: docs.databricks.com
Checkpoint 6 of 6· Exam question
A platform team wants a separate evaluation Workflow to kick off automatically the moment any new version of a specific registered model becomes ready in Unity Catalog, so results are available before anyone manually assigns an alias. Which Databricks Jobs trigger type is designed for this?
Correct answer: A — The Model update trigger, which fires when a Unity Catalog model gets a new version or has an alias assigned to it.
- A. The Model update trigger is the Jobs trigger type built specifically to react to Unity Catalog model events, including a new version becoming ready, which lets an evaluation job start right when that happens.
- B. A Continuous trigger reacts to the evaluation job's own prior run completing, not to registry events, so it would loop evaluations regardless of whether any new model version actually exists.
- C. Unity Catalog model metadata is not exposed as a queryable Delta table that a Table update trigger can watch, so this trigger type is not the mechanism used to react to registry changes.
- D. An hourly schedule introduces up to an hour of delay before evaluation starts and still runs even when no new version has appeared, which does not match the requirement to react immediately to a new version.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Production retraining should repeat the full hyperparameter search from development so every run finds the global best.Why is that wrong?
Production training code usually considers only the top-performing options. That saves time and can reduce the variance tuning adds to automated retraining.
2.After the Champion alias moves to a new version, every batch inference pipeline has to be edited and redeployed to use it.Why is that wrong?
Inference pipelines load the model by alias. They pick up the new Champion version on their next execution, which decouples deployment from inference.
Covered in Challenger versus Champion
3.Allowing several concurrent deployment job runs speeds up promotion when retraining registers versions quickly.Why is that wrong?
Databricks recommends keeping the maximum concurrent runs at 1, the default, to avoid deployment race conditions.
Covered in Automating evaluation, approval and rollout
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“In the production training code, it's common to consider only the top-performing options.”
↩︎ Narrowing candidates to one registered version“typically included in the pipeline as a configuration file”
↩︎ Narrowing candidates to one registered version“basic format and metadata validations, performance evaluations on selected data slices, and compliance with organizational requirements”
↩︎ Validation gate and the Challenger alias“You can use tags to add key-value attributes depending on the outcome of these validation checks.”
↩︎ Validation gate and the Challenger alias“confirm that it performs at least as well as the current production model”
↩︎ Challenger versus Champion“An offline comparison evaluates both models against a held-out data set and tracks results using the MLflow Tracking server.”
↩︎ Challenger versus Champion“longer running online comparisons, such as A/B tests or a gradual rollout of the new model”
↩︎ Challenger versus Champion“Model Serving executes a zero-downtime update by keeping the existing configuration running until the new one is ready.”
↩︎ Challenger versus Champion“Limiting tuning in this way saves time and can reduce the variance from tuning in automated retraining.”
↩︎ Exam trap 1“the inference pipeline automatically uses the new version for its next execution”
↩︎ Exam trap 2“If the model does not pass all validation checks, the process exits and users can be automatically notified.”
↩︎ Checkpoint“to a business heuristic or other threshold as a baseline”
↩︎ Checkpoint - 2.
“Sort runs by their test auc. In case of ties, use the most recent run.”
↩︎ Narrowing candidates to one registered version - 3.
“they trigger automatically whenever a new model version is created”
↩︎ Automating evaluation, approval and rollout“deploy the model to 1%, deploy the model to 10%, collect and check metrics, decide whether to proceed or rollback”
↩︎ Automating evaluation, approval and rollout“Databricks also recommends setting the max concurrent run limit to 1 (the default) to prevent deployment race conditions.”
↩︎ Exam trap 3 - 4.
“Model update triggers are in Beta.”
↩︎ Automating evaluation, approval and rollout“you can run validation, testing, or promotion jobs automatically whenever a model they own changes”
↩︎ Checkpoint