What you will be able to do
- Explain how serverless environment versions replace Databricks Runtime versions
- Add PyPI, wheel and project dependencies to a serverless notebook or job task
- Avoid the PySpark, CPU-architecture and private-repository pitfalls that break serverless dependency installs
Key concept
Serverless environment (environment version) — On serverless compute you don't choose a Databricks Runtime or install libraries on a cluster. You choose a base environment version, which gives your code a stable client API, and you declare dependencies on top of it. Databricks resolves and caches those dependencies, and it upgrades the servers underneath without you having to do anything.
1.Environment versions instead of runtime versions
On classic compute, you pick a Databricks Runtime version and handle upgrades yourself. Serverless works differently: you write code against an environment version, and Databricks upgrades the underlying servers on its own schedule. The environment version gives you a stable client API, so performance improvements, security fixes and bug fixes reach your workload without code changes. Databricks supports each environment version for three years after its release, which gives you a predictable timeline for planning upgrades.
In a notebook, you choose the version with the Base environment selector in the Environment side pane. Standard is the default and includes Databricks-provided libraries. ML pre-installs the Python and system packages from Databricks Runtime for Machine Learning, which makes it the option for moving classic ML workloads to serverless. AI appears only after you select a GPU accelerator. Under More you'll find earlier versions and Custom, which loads a YAML specification. Workspace environments lists base environments that an administrator has set up. Databricks recommends the latest version.
| Option | What it gives you |
|---|---|
| Standard | Default serverless base environment with Databricks-provided libraries |
| ML | Python and system packages from Databricks Runtime for Machine Learning pre-installed |
| AI | AI-optimized environment with ML libraries; appears only when a GPU accelerator is selected |
| More → Custom | Environment specified by a YAML file |
| Workspace environments | Compatible base environments configured by a workspace administrator |
Checkpoint 1 of 4· Check yourself
A team is moving a classic Databricks Runtime ML notebook to serverless. They want the same pre-installed ML packages without listing each one. Which base environment should they choose?
The ML base environment pre-installs the packages from Databricks Runtime for Machine Learning, and it exists to support this kind of migration. AI appears only when a GPU accelerator is selected, and serverless has no runtime drop-down.
“Use this environment to migrate classic Databricks Runtime for Machine Learning workloads to serverless compute.”Source: docs.databricks.com
2.Adding dependencies without init scripts
Serverless has no init scripts and no compute policies, so you declare dependencies in the Environment side pane. In the Dependencies section, enter any line that would be valid in a requirements.txt file and click Apply. Databricks installs the dependencies and restarts the Python process. A wheel or a Python project (a directory containing pyproject.toml or setup.py) can come from workspace files, using an absolute path that starts with /Workspace/, or from a Unity Catalog volume, using a path in the form /Volumes/<catalog>/<schema>/<volume>/<path>.whl. For example, after you add a project folder at /Workspace/helper_utils that contains a helpers package, you can import it directly:
from helpers import greet
print(greet('world'))Databricks caches the notebook's virtual environment, so dependencies don't reinstall each time you reopen the notebook or resume after inactivity. Job tasks with the same dependency set share this cache within a run. To keep environments reproducible, pin exact versions. An unpinned package can resolve to a different version depending on the environment version, and installing that new version adds latency. Once an environment works, you can export it from the pane as a YAML specification and reuse it through Custom.
numpy==2.2.2
pandas==2.2.3Checkpoint 2 of 4· Exam question
A data engineering team runs a nightly Lakeflow Job on serverless compute that ingests and transforms roughly 2 TB of data. The job has no interactive users, can tolerate a startup delay of several minutes, and the team wants to minimize DBU spend without rewriting the job's logic. Which change should they make to the job's compute configuration?
Correct answer: B — Configure the job to run in Standard performance mode, accepting a startup delay of roughly four to six minutes in exchange for lower DBU consumption suited to unattended batch runs.
- A. Performance Optimized mode keeps warm compute ready so tasks start in seconds, which is the right trade-off for latency-sensitive interactive work, but it costs more per run than the team needs for an unattended nightly batch job.
- B. Standard performance mode is built for automated batch jobs and pipelines that can absorb a longer startup window, and it can cut DBU cost substantially compared to keeping compute warm, matching exactly what this nightly job needs.
- C. Serverless compute already offers a cost-optimized batch profile through Standard mode, so migrating to a classic autoscaling cluster adds operational overhead without giving the team a capability serverless doesn't already provide.
- D. The memory tier setting controls how much RAM a notebook or notebook task gets and does not select between performance modes, so raising it would not reduce DBU consumption for this ingestion job.
3.CPU architecture and private repositories
Two serverless properties catch teams out. First, serverless doesn't guarantee a CPU architecture. A run can land on aarch64 or x86_64, and that can change between runs. A wheel with native C extensions only works on the architecture it was built for, so installing it on the other one fails. Use pure-Python wheels tagged py3-none-any where you can. Otherwise, provide both aarch64 and x86_64 wheels and restrict each one to its architecture with a platform_machine environment marker.
Second, Databricks resolves and installs dependencies before any of your code runs. Nothing in a notebook cell can change how packages are fetched. Serverless also doesn't support compute-scoped environment variables, so setting a proxy at runtime has no effect. To install from a private or authenticated repository, put index-url or extra-index-url in the dependency specification, or ask a workspace admin to make that repository the default package source.
The wheel probably contains native extensions built for a single CPU architecture, and serverless can run on either aarch64 or x86_64. Rebuild it as a pure-Python py3-none-any wheel, or ship both architecture variants with platform_machine markers.
Checkpoint 3 of 4· Check yourself
A data engineer adds os.environ['HTTPS_PROXY'] = ... as the first cell of a serverless notebook so a package can be fetched from a private index. Installation still fails. What should they do?
Dependencies are installed before notebook code runs, and serverless has no init scripts or compute-scoped environment variables. The private repository has to be named in the dependency specification itself, or configured by an admin as the default repository.
“To install from a private or authenticated package repository, specify index-url or extra-index-url in the dependency specification.”Source: docs.databricks.com
Sources2
4.Environments for serverless job tasks
Job tasks use the same environment model, but you configure it on the task. When you create or edit a task, open Jobs Environment and either edit an existing environment or add a new one. In the Configure environment dialog, choose a base environment: a Databricks option such as Standard or ML, a workspace environment, or a Custom YAML file. Under Dependencies, add any extra libraries using requirements.txt syntax, an absolute workspace-file path or a Unity Catalog volume path. If the workspace base environment preview isn't enabled, the dialog shows an Environment version drop-down instead. In that case you can also reference a requirements file, for example -r /Workspace/path/to/requirements.txt.
Checkpoint 4 of 4· Fill the gap
After adding /Workspace/helper_utils as a dependency, which name completes the import?
from helpers import ?
print(greet('world'))The project's package is helpers, and it defines the function greet. common_utils is the distribution name in pyproject.toml, not the module you import from.
“In the Dependencies section, click Add Dependency then enter the path of your util file.”Source: docs.databricks.com
Sources2
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.You can install a specific PySpark version on serverless to match a library's requirements.Why is that wrong?
Installing PySpark, or a library that depends on it, stops the serverless session with an error. You must remove the library and reset the environment.
Covered in Adding dependencies without init scripts
2.Setting proxy environment variables in a notebook cell lets serverless install packages from a private repository.Why is that wrong?
Dependencies are resolved before your code runs, and runtime proxy settings have no effect. Use index-url or extra-index-url, or an admin-configured default repository.
Covered in CPU architecture and private repositories
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“You write code against an environment version, and Databricks independently upgrades the underlying server.”
↩︎ Environment versions instead of runtime versions“Databricks supports each environment version for three years from its release date”
↩︎ Environment versions instead of runtime versions“pin your Python packages to specific versions to ensure reproducible environments.”
↩︎ Adding dependencies without init scripts“Serverless compute uses environment versions instead of traditional Databricks Runtime versions.”
↩︎ Key concept - 2.
“A base environment determines the pre-installed libraries and environment version available for your serverless notebook.”
↩︎ Environment versions instead of runtime versions“You can specify a dependency in any format that is valid in a requirements.txt file.”
↩︎ Adding dependencies without init scripts“Job tasks that share the same dependency set also benefit from this cache within a run.”
↩︎ Adding dependencies without init scripts“Serverless compute does not guarantee a specific CPU architecture.”
↩︎ CPU architecture and private repositories“proxy settings applied at runtime have no effect on dependency installation.”
↩︎ CPU architecture and private repositories“For job tasks, configure the environment when you create or edit a task.”
↩︎ Environments for serverless job tasks“Under Dependencies, add any additional libraries.”
↩︎ Environments for serverless job tasks“Do not install PySpark or any library that installs PySpark as a dependency on your serverless notebooks.”
↩︎ Exam trap 1“Databricks resolves and installs dependencies before your notebook or task code runs.”
↩︎ Exam trap 2“Use this environment to migrate classic Databricks Runtime for Machine Learning workloads to serverless compute.”
↩︎ Checkpoint“Because serverless does not support compute policies or init scripts, you must install custom dependencies using the Environment side pane.”
↩︎ Prediction“To install from a private or authenticated package repository, specify index-url or extra-index-url in the dependency specification.”
↩︎ Checkpoint“In the Dependencies section, click Add Dependency then enter the path of your util file.”
↩︎ Checkpoint