What you will be able to do
- Identify which tests run in the development, staging, and production stages
- Order the staging CI flow from pull request to release branch
- Design an integration test that covers feature engineering, training, evaluation, deployment, and inference
- Explain how production validation checks decide whether a model version gets the Challenger alias
1.Separate environments, and why code is promoted instead of models
Databricks recommends a separate execution environment for each stage of ML development, with clearly defined transitions between them. The usual names are development, staging, and production. Each environment has its own catalog in Unity Catalog. Data scientists read and write the dev catalog, and models built during development are registered there. Staging has its own catalog for testing ML pipelines and registering models, and the assets written there are generally kept only until testing finishes. Production has its own catalog that ML engineers control. Data scientists usually have no write or compute access in production, but they can be given read-only access to its assets.
This rule determines where testing happens. Code moves through Git, from a development branch to the staging branch and then to a release branch. Each move is a point where tests can be run. MLOps Stacks builds this process into a project template. The entire model development process is stored as code in a source-controlled repository, so you deploy the code instead of the model. The ML code templates are designed to enforce modularized code ready for testing.
Checkpoint 1 of 6· Check yourself
Why does promoting code rather than models matter for testing?
When code is what moves between environments, every change passes through the same review and integration tests, and the production model is trained on production code.
“Moving project assets this way ensures that all code in the ML development process goes through the same code review and integration testing processes.”Source: docs.databricks.com
2.Which tests run in which stage
Unit tests are not limited to one stage. They are written and added to the codebase continuously while any code is developed. The staging stage exists to test all of the ML pipeline code before it reaches production. ML engineers build a CI pipeline that runs both the unit tests and the integration tests. Unit tests come first because they are fast and catch broken functions. Integration tests run next, against an environment that should match production as closely as is reasonable.
Checkpoint 2 of 6· Put it in order
Put the staging CI/CD flow in the correct order
- 1.The ML engineer creates a release branch
- 2.The CI process runs integration tests across all pipelines
- 3.A pull request is created against the main branch
- 4.The new code is merged into the main branch
- 5.The pull request builds source code and triggers unit tests
- 6.The CI/CD system updates production jobs
A failed unit test rejects the pull request before integration tests run. Only after all tests pass is the code merged, and a release branch then triggers the production update.
“After CI tests have passed and the dev branch is merged into the main branch, the ML engineer creates a release branch”Source: docs.databricks.com
If any test fails, the CI/CD system should notify users and post the results on the pull request. When many people merge pull requests into main at once, you can also schedule periodic integration tests on the main branch. The default MLOps Stacks project follows the same pattern. Pull requests trigger unit and integration tests in an isolated staging workspace. Once a pull request is merged, the staging training and batch inference jobs immediately switch to the latest code.
| Stage | Trigger | Testing performed |
|---|---|---|
| Development | Ongoing coding in the dev branch | Unit tests written and run continuously alongside the code |
| Staging | Pull request against main | Unit tests first (a failure rejects the PR), then integration tests of all pipelines; optional scheduled integration tests on main |
| Production | Code changes on the release branch or automated retraining jobs | Validation checks on each newly registered model version, then a Challenger-versus-Champion comparison |
Checkpoint 3 of 6· Exam question
An ML engineering team is deciding how to organize functions and their unit tests for a Scala-based feature transformation pipeline running on Databricks. Databricks documentation notes that Scala notebooks cannot use the same pattern recommended for Python and R. Which organizing approach should the team use instead?
Correct answer: B — Keep the transformation functions in one notebook and place the corresponding unit tests in a separate notebook.
- A. SQL user-defined functions are the recommended organization approach for SQL logic, not for Scala transformation functions, so this does not address a Scala notebook's constraints.
- B. For Scala, where functions cannot be moved into external modules the way Python and R functions can, keeping functions in one notebook and their tests in a separate notebook is the recommended pattern, since it still lets the functions be called from multiple other notebooks.
- C. Keeping functions and tests in the same notebook is the simplest approach but limits reusability, since the functions cannot be called from other notebooks without copying the cells.
- D. Scala code cannot be extracted into `.py` files for import, since `.py` files hold Python source, not Scala, so this does not solve the language-specific limitation the team is facing.
3.Designing an integration test for an ML system
Unit tests show that each function returns the right output for the inputs it is given. Integration tests check something unit tests cannot: that the pipelines work correctly together when run end to end. One example is a handoff between tasks. The training task produces a model URI that the validation task uses, passed along as a task value. A unit test of either task alone never exercises that connection. In staging, the integration test runs all pipelines: feature engineering, model training, inference, and monitoring. Evaluation is part of the training pipeline. It tests the model on held-out data, logs the results to MLflow, and compares the new model with the current production model. If the application serves predictions in real time, the integration test also triggers the model deployment pipeline. That pipeline creates a serving endpoint in staging and loads a model.
| Pipeline stage | What the integration test exercises | Acceptable shortcut |
|---|---|---|
| Feature engineering | Runs the feature pipeline against the staging catalog | Smaller subsets of data |
| Training | Trains the model, logs to MLflow, and registers it to the staging catalog | Small data subsets or fewer training iterations |
| Evaluation | Tests the model on held-out data and logs the results to MLflow | Same reduced data as training |
| Deployment | Triggers the model deployment pipeline, which creates a staging serving endpoint and loads a model | A temporary endpoint |
| Inference | Runs inference code end to end with the newly built model | Small batch jobs or requests instead of full-scale load testing |
The shortcuts in the last column are intentional. Integration tests can take a long time, so some steps trade testing fidelity against speed or cost. When a model is expensive to train, the test can use a small slice of data or fewer iterations. Depending on production requirements, the serving check can be a full-scale load test or just a few requests to a temporary endpoint.
Checkpoint 4 of 6· Check yourself
Model training takes six hours on full production data. How does Databricks suggest keeping the staging integration test practical?
Integration tests can trade fidelity for speed or cost. Running the whole pipeline on reduced data still checks that the pipelines work together.
“you might use small subsets of data or run fewer training iterations.”Source: docs.databricks.com
Checkpoint 5 of 6· Exam question
A team keeps every helper function and its pytest cases inside a single Databricks notebook so that one file tracks both the logic and its tests together. Which consequence follows directly from this organizing choice?
Correct answer: B — The helper functions cannot be called from any other notebook without duplicating their code.
- A. There is no built-in behavior that skips tests based on a function's argument type; test discovery in pytest is based on file and function naming conventions, not argument signatures.
- B. Keeping functions and tests together in one notebook is easy to track but limits reuse: other notebooks cannot import functions bound only to that notebook's local scope, so any other notebook needing the same logic has to copy it.
- C. Pytest can be installed and run inside a Databricks notebook via `%pip install pytest` and `pytest.main`, so tests are not refused outright; the limitation is reusability outside the notebook, not test execution inside it.
- D. Unity Catalog governs data and model objects, not the internal organization of a notebook's functions and tests, so it imposes no such scheduling restriction.
Sources1
4.Validation checks that gate deployment in production
Testing continues after code is released. In production, the training pipeline registers each new model version to the production catalog. A validation pipeline then loads that version from Unity Catalog and runs checks that depend on the organization and use case. Typical checks are basic format and metadata validation, performance evaluation on selected data slices, and compliance checks such as required tags or documentation. The pipeline's job is to decide whether the model should go on to deployment. If the model passes, it can be assigned the Challenger alias. If it fails, the process stops and users can be notified automatically. A tag such as model_validation_status can show PENDING while the checks run and change to PASSED or FAILED when they finish.
No. Passing validation only earns the Challenger alias. The deployment pipeline first confirms the new model performs at least as well as the current production model. The comparison can be offline, on a held-out dataset, or online, such as an A/B test or gradual rollout. Only a Challenger that performs better takes over the Champion alias.
Checkpoint 6 of 6· Check yourself
A newly registered production model version fails its compliance check for required documentation. What happens next?
Validation decides whether a model goes on to deployment. A failed check ends the process, and the workflow can send a notification.
“If the model does not pass all validation checks, the process exits and users can be automatically notified.”Source: docs.databricks.com
Sources1
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.A model that performs well in development should be copied straight into the production catalog.Why is that wrong?
In most situations Databricks recommends promoting code, not models, so the production model is trained by production code that has passed review and integration testing.
Covered in Separate environments, and why code is promoted instead of models
2.An integration test only counts if it trains on full production-scale data and load-tests serving at full scale.Why is that wrong?
Integration tests may trade fidelity for speed or cost, for example by using data subsets, fewer iterations, or requests to a temporary endpoint. They still check that the pipelines work together.
3.A model version that passes validation checks is immediately promoted to Champion.Why is that wrong?
Passing validation only qualifies it as the Challenger. The deployment pipeline must first confirm it performs at least as well as the current production model.
Covered in Validation checks that gate deployment in production
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Databricks recommends creating separate environments for the different stages of ML code and model development with clearly defined transitions between stages.”
↩︎ Separate environments, and why code is promoted instead of models“Assets written to this catalog are generally temporary and only retained until testing is complete.”
↩︎ Separate environments, and why code is promoted instead of models“ML engineers create a CI pipeline to implement the unit and integration tests run in this stage.”
↩︎ Which tests run in which stage“The pull request automatically builds source code and triggers unit tests. If unit tests fail, the pull request is rejected.”
↩︎ Which tests run in which stage“You can schedule periodic integration tests on the main branch.”
↩︎ Which tests run in which stage“Integration tests run all pipelines (including feature engineering, model training, inference, and monitoring) to ensure that they function correctly together.”
↩︎ Designing an integration test for an ML system“This involves triggering the model deployment pipeline, which creates a serving endpoint in the staging environment and loads a model.”
↩︎ Designing an integration test for an ML system“The model training task yields a model URI that the model validation task can use.”
↩︎ Designing an integration test for an ML system“The purpose of evaluation is to determine if the newly developed model performs better than the current production model.”
↩︎ Designing an integration test for an ML system“basic format and metadata validations, performance evaluations on selected data slices”
↩︎ Validation checks that gate deployment in production“The primary function of the model validation pipeline is to determine whether a model should proceed to the deployment step.”
↩︎ Validation checks that gate deployment in production“You can perform this comparison offline or online.”
↩︎ Validation checks that gate deployment in production“It also ensures that the production version of the model is trained on production code.”
↩︎ Exam trap 1“To reduce the time required to run integration tests, some steps can trade off between fidelity of testing and speed or cost.”
↩︎ Exam trap 2“The first step before deploying the new model is to confirm that it performs at least as well as the current production model.”
↩︎ Exam trap 3“In most situations, Databricks recommends that during the ML development process, you promote code, rather than models, from one environment to the next.”
↩︎ Prediction“Moving project assets this way ensures that all code in the ML development process goes through the same code review and integration testing processes.”
↩︎ Checkpoint“After CI tests have passed and the dev branch is merged into the main branch, the ML engineer creates a release branch”
↩︎ Checkpoint“you might use small subsets of data or run fewer training iterations.”
↩︎ Checkpoint“If the model does not pass all validation checks, the process exits and users can be automatically notified.”
↩︎ Checkpoint - 2.
“Representing the model development process as code enables you to deploy the code instead of deploying the model.”
↩︎ Separate environments, and why code is promoted instead of models“enforces modularized code ready for testing.”
↩︎ Separate environments, and why code is promoted instead of models“PRs trigger unit tests and integration tests in an isolated staging Databricks workspace.”
↩︎ Which tests run in which stage