What you will be able to do
- Contrast the deploy-code and deploy-models patterns and say where the model is trained in each
- Describe how development, staging and production map to separate workspaces and Unity Catalog catalogs
- Trace a code change through the MLOps Stacks CI/CD flow from pull request to production release
Key concept
Deploy code (not models) — You promote the training code, not a trained model artifact, through development, staging and production. Each environment runs that reviewed code against its own data, so the production model is trained in production by production code.
1.Two ways to reach production: move the code or move the model
Code creates a model, but the two change on different schedules. A fraud model might be retrained every week on new data while its code hardly changes. A large deep network might be retrained rarely while the code that serves and monitors it changes often. Because the two drift apart, Databricks describes two ways to move ML work toward production. In one you promote the model artifact. In the other you promote the training code that produces the artifact.
Databricks recommends deploy code in most situations, and its recommended MLOps workflow is built on it. Training code is written in development and then moves unchanged: "The same code moves to staging and then production." Training runs at every step. In dev it runs as part of model development. In staging it runs on a limited subset of data as part of integration tests. In production it runs on the full production data to produce the final model. This is why automated retraining is safer under this pattern: the code that retrains the model has already been reviewed, tested and approved. Supporting code, such as feature engineering, inference and monitoring pipelines, follows the same route through staging integration tests. The cost is a steeper hand-off for data scientists, and they need to be able to review training results from production.
The deploy-models pattern trains the artifact once in development, tests it in staging, then deploys it to production. It suits cases where training is very expensive or hard to reproduce, where all work happens in one workspace, or where there is no CI/CD process. It breaks down when development cannot reach production data. If the model must be trained on the full production dataset but in staging, a hybrid is possible: deploy code to staging, train there, then deploy the model to production.
| Aspect | Deploy code (recommended) | Deploy models |
|---|---|---|
| What moves toward production | Training code | The trained model artifact |
| Where training runs | Dev, staging (data subset), and production (full data) | Development environment |
| Automated retraining | Safer: training code is reviewed, tested, approved | Tricky: production team may not accept a dev-trained model |
| Supporting code (features, inference, monitoring) | Goes through staging integration tests like training code | Must be deployed to production separately |
| When to consider | Most situations, especially with restricted production data | Very expensive training, one workspace, no CI/CD |
Checkpoint 1 of 5· Exam question
A Databricks ML team's data scientists have no direct write access to production data, and the company wants every retraining run to reuse the same reviewed pipeline automatically whenever new production data lands. Which deployment architecture should the team adopt for moving ML work from development to production?
Correct answer: B — Deploy the same training code through dev, staging, and production workspaces, retraining the model on each environment's own data.
- A. This describes the deploy-models pattern, which does not solve the team's access problem: a model built without production data still cannot see it, and copying a fixed artifact does not retrain automatically as new data lands.
- B. Promoting the training code and rerunning it in each environment lets the pipeline train on production data inside the production workspace itself, without the data scientists ever needing direct access, and the same automated run repeats whenever new data arrives.
- C. Skipping staging removes the integration-testing gate the pipeline depends on for catching defects before they reach production, which is exactly what a deploy-code validation stage exists to prevent.
- D. Letting each downstream team decide independently defeats the goal of one automated, centrally reviewed pipeline and reintroduces manual, per-team artifact handling instead of consistent retraining.
Checkpoint 2 of 5· Check yourself
Production data is locked down and cannot be read from the development workspace. Which pattern still lets the final model be trained on production data?
Under deploy code, training runs in production, so restricted data never has to leave it. Deploy models depends on development having access to production data.
“In organizations where access to production data is restricted, this pattern allows the model to be trained on production data in the production environment.”Source: docs.databricks.com
Sources1
2.Execution environments: workspaces, catalogs and access
Deploy code only works if the environments are really separate. Databricks defines an execution environment as the place where code creates or consumes models and data. Each one is made up of compute instances, their runtimes and libraries, and automated jobs. The recommendation is separate environments for each stage, with clearly defined transitions between them. The common stage names are development, staging and production.
Unity Catalog provides the data-side boundary: "Typically an environment (development, staging, or production) corresponds to a catalog in Unity Catalog." The three-level namespace <catalog>.<schema>.<model> therefore shows which environment an asset belongs to. For example, staging.ml_team.fraud_model is a model registered while its code is tested in staging, and prod.ml_team.fraud_model is the version produced by the production pipeline. Pipeline code should register each model to the catalog of the environment it ran in.
Development: data scientists have read-write access to the dev catalog for temporary data and feature tables, and models trained there are registered to dev. Ideally they also get read-only access to the prod catalog, so they can analyze production predictions, inference tables and metric tables, and load production models for comparison. If that is not possible, a snapshot of production data can be written to the dev catalog. Staging: it has its own catalog for testing pipelines and registering models. Assets there are generally temporary and kept only until testing is complete. Production: models produced by the production pipeline are registered in the prod catalog.
Code moves along the same three stages through Git. Databricks recommends keeping pipelines and code in Git and treating a promotion as a move from the development branch to the staging branch to the release branch. Databricks Git folders keep notebooks and source code in sync between the workspace and your Git provider.
Checkpoint 3 of 5· Check yourself
Data scientists in the development workspace cannot be given read-only access to the prod catalog. What does Databricks suggest instead?
The fallback is a snapshot of production data in the dev catalog. This lets project code be developed and evaluated without direct prod access.
“a snapshot of production data can be written to the dev catalog”Source: docs.databricks.com
3.MLOps Stacks: the transition machinery as code
MLOps Stacks packages the transitions between environments as a project in source control. In the docs' words, "Representing the model development process as code enables you to deploy the code instead of deploying the model." You start a project with databricks bundle init mlops-stacks. You can set up the ML code components, typically used by data scientists, the CI/CD components, typically used by ML engineers, or both. The default stack has three parts:
- ML code: template notebooks for training, batch inference and so on, giving every team the same modular structure that is ready for testing. - ML resources as code: workspaces and pipelines defined in Declarative Automation Bundles, so changes to the ML environment are version-controlled and testable. - CI/CD: GitHub Actions or Azure DevOps test and deploy the code and resources, so that all production changes go through automation and only tested code reaches prod.
ML engineers set up Databricks service principal secrets for CI/CD. The CI/CD runner then runs notebooks and creates jobs in the staging and production workspaces.
The default flow works like this. Data scientists iterate in the development workspace and open pull requests. Each PR triggers unit and integration tests in an isolated staging workspace, and the PR is rejected if unit tests fail. When the PR is merged to main, the training and batch inference jobs in staging update to run the latest code. The output of staging is a release branch. As part of your scheduled release process you cut that branch, and it deploys the code changes to production, where the production jobs train and serve the new model.
Checkpoint 4 of 5· Put it in order
Put the default MLOps Stacks deploy-code flow in order.
- 1.The pull request triggers unit and integration tests in an isolated staging workspace
- 2.The PR is merged to main and staging training and inference jobs update to the latest code
- 3.The code changes are deployed to production
- 4.A release branch is cut as part of the scheduled release process
- 5.Data scientists iterate on ML code in the development workspace and file a pull request
Testing happens in staging before the merge. The merge updates the staging jobs, and the release branch carries the code into production.
“you can cut a new release branch as part of your scheduled release process and deploy the code changes to production.”Source: docs.databricks.com
Checkpoint 5 of 5· Exam question
Training a large recommendation model on a Databricks cluster takes four days and a substantial GPU budget, and only the development workspace has the specialized hardware pool needed to run it. Staging and production workspaces cannot reproduce that training job. Which lifecycle pattern fits this constraint?
Correct answer: C — Train the model once in development, then promote the resulting model artifact through staging and production for validation.
- A. Provisioning matching GPU pools everywhere defeats the stated hardware constraint and adds the infrastructure cost the scenario is trying to avoid by not reproducing training in every environment.
- B. Re-running the four-day GPU job in staging and production is exactly what the hardware limitation rules out, and repeating it in every environment abandons the reason for choosing a pattern that avoids retraining downstream.
- C. This is the deploy-models pattern, appropriate when training is expensive or hardware-constrained to one environment; the trained artifact, not the code, is what gets promoted and validated as it moves toward production.
- D. Scheduling a fresh four-day training run nightly in every environment is infeasible under the stated hardware limits and describes deploy-code retraining rather than artifact promotion.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Under the recommended workflow, a model trained and approved in development is copied to staging and then to production.Why is that wrong?
That is the deploy-models pattern. Databricks recommends deploy code in most situations: the training code is promoted and the model is retrained in each environment, with the production model trained on production data.
Covered in Two ways to reach production: move the code or move the model
2.In MLOps Stacks, pull-request tests run in the development workspace where the code was written.Why is that wrong?
Pull requests trigger unit and integration tests in a separate, isolated staging workspace, not in development.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“In most situations, Databricks recommends the “deploy code” approach.”
↩︎ Two ways to reach production: move the code or move the model“Automated model retraining is safer, since the training code is reviewed, tested, and approved for production.”
↩︎ Two ways to reach production: move the code or move the model“you can use a hybrid approach by deploying code to staging, training the model, and then deploying the model to production.”
↩︎ Two ways to reach production: move the code or move the model“Model training is very expensive or hard to reproduce.”
↩︎ Two ways to reach production: move the code or move the model“Typically an environment (development, staging, or production) corresponds to a catalog in Unity Catalog.”
↩︎ Execution environments: workspaces, catalogs and access“The same code moves to staging and then production.”
↩︎ Exam trap 1“and in the production environment (on the full production data) to produce the final model.”
↩︎ Prediction“In organizations where access to production data is restricted, this pattern allows the model to be trained on production data in the production environment.”
↩︎ Checkpoint - 2.
“Databricks recommends creating separate environments for the different stages of ML code and model development with clearly defined transitions between stages.”
↩︎ Execution environments: workspaces, catalogs and access“Ideally, data scientists working in the development workspace also have read-only access to production data in the prod catalog.”
↩︎ Execution environments: workspaces, catalogs and access“Assets written to this catalog are generally temporary and only retained until testing is complete.”
↩︎ Execution environments: workspaces, catalogs and access“Moving ML logic between stages can then be interpreted as moving code from the development branch, to the staging branch, to the release branch.”
↩︎ Execution environments: workspaces, catalogs and access“The output of the staging process is a release branch that triggers the CI/CD system to start the production stage.”
↩︎ MLOps Stacks: the transition machinery as code“If unit tests fail, the pull request is rejected.”
↩︎ MLOps Stacks: the transition machinery as code“In most situations, Databricks recommends that during the ML development process, you promote code, rather than models, from one environment to the next.”
↩︎ Key concept“a snapshot of production data can be written to the dev catalog”
↩︎ Checkpoint - 3.
“Representing the model development process as code enables you to deploy the code instead of deploying the model.”
↩︎ MLOps Stacks: the transition machinery as code“Resources are defined in Declarative Automation Bundles to facilitate testing, optimization, and version control for the ML environment.”
↩︎ MLOps Stacks: the transition machinery as code“When a PR is merged to main, model training and batch inference jobs that run in staging immediately update to run the latest code.”
↩︎ MLOps Stacks: the transition machinery as code“PRs trigger unit tests and integration tests in an isolated staging Databricks workspace.”
↩︎ Exam trap 2“you can cut a new release branch as part of your scheduled release process and deploy the code changes to production.”
↩︎ Checkpoint