What you will be able to do
- Explain why serverless dependencies must be declared in the Environment side pane and not fixed from inside notebook code
- Diagnose wheel, path, CPU-architecture, private-repository and PySpark installation failures on serverless compute
- Use pip logs, %pip and %uv pip to investigate dependency conflicts, and know where each one stops working
Key concept
Dependencies resolve before code runs — On Databricks, the environment you declare decides which external libraries get installed, and it is resolved before any of your code executes. Changing things at runtime, such as proxy variables in a cell or restarting Python in a pipeline, does not change what was installed. You fix the declaration instead.
1.Where serverless dependencies come from
On classic clusters you might reach for an init script or a compute policy when a library will not install. Serverless compute supports neither, so the Environment side pane is the only place to declare custom dependencies. It has two layers. The base environment (Standard, ML, AI, a previous version, a Custom YAML file, or an admin-configured workspace environment) sets the preinstalled libraries and environment version. Your Dependencies list sits on top of it. For a job task, you configure the same environment when you create or edit the task.
This changes how you troubleshoot. Databricks resolves the environment before your code starts, so nothing you write in a cell can rescue a failed install. The fix is always to change the declaration: a different version, path, index or base environment. Then click Apply, which installs the dependencies and restarts the Python process. Databricks also caches the notebook's virtual environment, and job tasks that share the same dependency set reuse that cache within a run. Reopening the notebook does not reinstall anything, so a broken declaration stays broken until you edit it.
Checkpoint 1 of 7· Check yourself
A serverless notebook dependency fails to install. Which approach can actually fix it?
Dependencies are resolved before your code runs, so cell code cannot change how they are fetched. The fix is to edit the declaration and apply it.
“Code in a notebook cell cannot change how dependencies are fetched.”Source: docs.databricks.com
Sources1
2.PyPI specs, local wheels and project directories: getting the path right
Each entry in the Dependencies list accepts any format that is valid in a requirements.txt file. That covers three source types:
- PyPI packages such as simplejson==3.19.*.
- Local wheel files stored in workspace files or Unity Catalog volumes.
- Python project directories, meaning a folder containing a pyproject.toml or a setup.py. Databricks builds and installs the project from source.
Many "file not found" failures come down to path syntax. A workspace path must be absolute and start with /Workspace/. A volume path follows /Volumes/<catalog>/<schema>/<volume>/<path>.whl. The documented helper-utils example adds the folder /Workspace/helper_utils, which holds a pyproject.toml and a helpers package, and then runs from helpers import greet. That example also requires other users to have read access to the folder, so a missing permission is one more thing to check when an install works for its author and fails for everyone else.
A note on scope: these sources do not describe installing packaged source archives (for example .tar.gz sdists) directly. For source-based installs, they document project directories that have a pyproject.toml/setup.py, and git+https:// VCS URLs installed with %pip.
Checkpoint 2 of 7· Check yourself
A dependency entry reads Shared/libs/etl_utils-1.0-py3-none-any.whl and fails to install on a serverless notebook. What is the most likely fix?
Workspace-file dependencies have to use an absolute path that starts with /Workspace/. A relative path like this one does not meet that rule.
“If using a workspace file, the path should be absolute and start with /Workspace/.”Source: docs.databricks.com
Sources1
3.The wheel that installed yesterday: CPU architecture
Serverless compute does not guarantee a CPU architecture. A notebook or job can land on aarch64 or x86_64, and that can change between runs. A wheel with native (C) extensions ships a prebuilt binary for one architecture, so it installs on matching hardware and fails with an error on the other. That produces intermittent failures that look unrelated to any change you made.
The documentation gives two fixes. Use pure-Python wheels tagged py3-none-any where possible, since they have no architecture-specific binary. Otherwise, supply both an aarch64 and an x86_64 variant and limit each to its own architecture with a platform_machine environment marker.
Checkpoint 3 of 7· Check yourself
Which change stops a native-extension wheel from failing intermittently on serverless?
Serverless lets you pick neither architecture, so the dependency set has to work on both. Pure-Python wheels or per-architecture variants with markers do that. Where the file is stored makes no difference to the binary.
“use pure-Python wheels (tagged py3-none-any) where possible, or include both aarch64 and x86_64 wheel variants”Source: docs.databricks.com
Sources1
4.Private indexes, PySpark collisions and reading the pip logs
Private repositories. Engineers behind a corporate proxy often set proxy variables in a cell and retry. On serverless this fails twice over: installation happens before cell code runs, and serverless has no compute-scoped environment variables. Put index-url or extra-index-url in the dependency specification, or ask a workspace admin to configure the repository as a default package repository for serverless notebooks and jobs.
PySpark. Never install PySpark, or any library that brings PySpark in as a dependency. Doing so stops the session with an error. To recover, remove the library and reset the environment.
Unpinned versions. If a version is not fully specified, Databricks installs the latest matching version. Two runs of the same job can therefore get different library versions, so a job that breaks with no code change may be picking up a new upstream release. Pin versions for jobs.
Evidence. The Installed tab lists what actually got installed. pip logs at the bottom of the Environment pane show the resolver output, which is where a conflict or a missing wheel shows up.
| Symptom | Cause | Fix |
|---|---|---|
| Wheel install fails on some runs only | Native wheel built for one architecture; compute may be aarch64 or x86_64 | py3-none-any wheel, or both variants with a platform_machine marker |
| Private package not found despite proxy set in a cell | Dependencies resolve before code runs; no compute-scoped environment variables | index-url / extra-index-url in the spec, or an admin-configured default package repository |
| Session stops with an error after adding a library | Library installs PySpark as a dependency | Remove the library and reset the environment |
| Workspace wheel or project not found | Path not absolute | Use /Workspace/... or /Volumes/<catalog>/<schema>/<volume>/... |
Checkpoint 4 of 7· Match them up
Match each serverless symptom to the correct remedy
Tap a term, then the definition that fits it.
Runtime proxy settings can't affect installation. A PySpark collision means removing the library and resetting. The pip logs record the resolver output.
“proxy settings applied at runtime have no effect on dependency installation.”Source: docs.databricks.com
Checkpoint 5 of 7· Exam question
A data engineer configures a serverless notebook and runs `%pip install some-ml-library`, where the library pulls in `pyspark` as a transitive dependency. Immediately after the cell finishes, the notebook session terminates and must be reattached. What is the most likely cause, and how should the engineer avoid it?
Correct answer: A — Installing any library that includes PySpark as a dependency on serverless compute stops the notebook session; exclude packages that bundle PySpark and rely on the pre-installed Spark runtime instead.
- A. Installing PySpark, or any package that installs PySpark as a dependency, on serverless notebooks stops the session with an error because the runtime already provides a managed Spark environment that a second PySpark install collides with. Excluding such packages and relying on the built-in Spark runtime avoids the crash entirely.
- B. A crashed session immediately after a successful-looking install is not the signature of a network timeout, and serverless compute does not require a custom mirror endpoint for ordinary PyPI installs. This explanation does not match the described symptom.
- C. Restarting the Python interpreter is a pattern associated with classic all-purpose clusters using `dbutils.library.installPyPI`, not the cause of a serverless session stopping right after a `%pip install`. Omitting that call does not produce this failure.
- D. There is no documented per-session package count limit on serverless compute, and splitting the install across cells would not address a dependency that installs PySpark. This distractor invents a limit that does not exist.
5.Interactive debugging with %pip and %uv pip, and where they fall short
For quick experiments, notebook-scoped installs let you try versions without editing the environment. They come with caveats you need to know when troubleshooting:
- %pip does not restart Python on its own. After an install or upgrade, you may need dbutils.library.restartPython() before the new version is visible.
- Upgrading, modifying or uninstalling core packages such as IPython with %pip can break notebook features. Reset by restarting the cluster or starting a new session.
- %pip uninstall -y cannot remove a library bundled with the runtime. It only reverts an override back to the bundled version.
- %uv pip (serverless environment version 5 or above) is faster and shares the same virtual environment as %pip. It does not check for dependency conflicts against preinstalled libraries, and its uninstall removes only the metadata of a preinstalled package, which stays importable.
For private repositories, %pip accepts --index-url, with the token pulled from Databricks Secrets:
%pip install --index-url https://<user>:$token@<your-package-repository>.com/<path/to/repo> <package>==<version> --extra-index-url https://pypi.org/simple/Once an experiment works, move the dependency into the Environment side panel. That panel is the reproducible definition, and %uv pip is meant only for fast iteration.
Checkpoint 6 of 7· Check yourself
You run %uv pip install to upgrade a package, it reports success, and imports then fail with a version clash against a preinstalled library. Why didn't the install catch it?
%uv pip shares the notebook environment with %pip but skips conflict checks against preinstalled packages. Clashes therefore only appear when you import.
“%uv pip does not check for dependency conflicts against preinstalled libraries.”Source: docs.databricks.com
Checkpoint 7 of 7· Exam question
A team deploys a job via a Declarative Automation Bundle with a `python_wheel_task` targeting serverless compute. The task references a `requirements.txt` file listing the project's third-party dependencies, but every run fails with an error stating the library file cannot be located or read. Which of the following changes would correctly resolve the failure? (Select 3.)(Select 3)
Correct answers: A, C, D — Reference the `requirements.txt` file in the task's `libraries` section and set the task's source to `WORKSPACE` so the bundle-synced copy in `/Workspace` is resolved at run time.; Confirm the bundle deployment step actually syncs the `requirements.txt` file to the workspace path referenced by the job, since a task cannot install from a file that deploy never uploaded.; Replace the `requirements.txt` reference with a `whl` library entry that points at a wheel built from the same project, avoiding a per-run pip install of loose requirement entries.
- A. Wiring the requirements file through the task's `libraries` section with a `WORKSPACE` source ensures the runtime looks for the file at the path the bundle actually synced it to, which is the documented pattern for serverless job tasks reading a requirements file. This directly fixes a file-not-found error caused by a mismatched source or path.
- B. Storing library files in the DBFS root is deprecated and disabled by default on current Databricks Runtime versions, so pointing a task at a `dbfs:/` path is more likely to cause install failures than fix them. DBFS root is not the recommended or default location for bundle-managed dependency files.
- C. A task can only install from a file that the deployment step actually uploaded, so verifying that the sync step includes the requirements file is a necessary check when the runtime reports the file as missing. Skipping this check is a common reason the fix in another option appears not to work.
- D. Packaging the project as a wheel and installing it via a `whl` library entry avoids the per-run resolution of a loose `requirements.txt` file and is the pattern the bundle documentation favors for reproducible task-level installs. This sidesteps the requirements-file lookup issue altogether.
- E. The `environment` block is a Lakeflow pipeline construct for layering dependencies onto a pipeline's execution environment, not a job task setting; job task libraries are declared under the task's own `libraries` section, not the pipeline environment block.
- F. A missing or unreadable library file is a path or sync problem, not evidence of a metastore-wide permission gap, and granting broad metastore ownership does not address where the file physically lives. This overreaches into an unrelated and unnecessary permission change.
Sources3
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Setting HTTP proxy variables in a serverless notebook cell will let dependencies install from a private repository.Why is that wrong?
Dependencies resolve before code runs and serverless has no compute-scoped environment variables. Use index-url/extra-index-url in the spec, or an admin-configured default repository.
Covered in Private indexes, PySpark collisions and reading the pip logs
2.%uv pip uninstall fully removes a preinstalled serverless library so you can replace it cleanly.Why is that wrong?
Only the metadata is removed. The files stay on disk and the package can still be imported.
Covered in Interactive debugging with %pip and %uv pip, and where they fall short
3.If a native wheel installed successfully once on serverless, it will keep installing on later runs.Why is that wrong?
Serverless can run on aarch64 or x86_64, and that can change between runs, so an architecture-specific wheel can fail on the next run.
Covered in The wheel that installed yesterday: CPU architecture
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Code in a notebook cell cannot change how dependencies are fetched.”
↩︎ Where serverless dependencies come from“Python projects (for example, the directory containing a pyproject.toml or a setup.py) can be located in workspace files or Unity Catalog volumes.”
↩︎ PyPI specs, local wheels and project directories: getting the path right“You can specify a dependency in any format that is valid in a requirements.txt file.”
↩︎ PyPI specs, local wheels and project directories: getting the path right“Serverless compute does not guarantee a specific CPU architecture.”
↩︎ The wheel that installed yesterday: CPU architecture“To install from a private or authenticated package repository, specify index-url or extra-index-url in the dependency specification.”
↩︎ Private indexes, PySpark collisions and reading the pip logs“Do not install PySpark or any library that installs PySpark as a dependency on your serverless notebooks.”
↩︎ Private indexes, PySpark collisions and reading the pip logs“Databricks resolves and installs dependencies before your notebook or task code runs.”
↩︎ Key concept“proxy settings applied at runtime have no effect on dependency installation.”
↩︎ Exam trap 1“A wheel built for one architecture fails to install on another, returning an error.”
↩︎ Exam trap 3“Because serverless does not support compute policies or init scripts, you must install custom dependencies using the Environment side pane.”
↩︎ Prediction“If using a workspace file, the path should be absolute and start with /Workspace/.”
↩︎ Checkpoint“use pure-Python wheels (tagged py3-none-any) where possible, or include both aarch64 and x86_64 wheel variants”
↩︎ Checkpoint“proxy settings applied at runtime have no effect on dependency installation.”
↩︎ Checkpoint - 2.
“different runs of the same job might use different library versions as new versions are published.”
↩︎ Private indexes, PySpark collisions and reading the pip logs - 3.
“%pip commands do not automatically restart the Python process.”
↩︎ Interactive debugging with %pip and %uv pip, and where they fall short“To set up a reproducible environment for the notebook, add dependencies in the notebook's Environment side panel.”
↩︎ Interactive debugging with %pip and %uv pip, and where they fall short“Only the package metadata is removed. The underlying files remain on the file system, and the package is still importable.”
↩︎ Exam trap 2“%uv pip does not check for dependency conflicts against preinstalled libraries.”
↩︎ Checkpoint