Fixing library dependencies in Lakeflow pipelines and bundle deployments

1.Pipelines: declare everything in the Environment settings

A Lakeflow pipeline gets its external Python packages from the Environment settings: Settings → Pipeline environment → Edit environment → Add dependency. Packages declared there apply to every source file in the pipeline. The other supported pattern is importing modules from workspace files or Git folders.

Notebook habits cause most pipeline failures here. You cannot call dbutils.library.restartPython() in a pipeline, so the install-then-restart pattern used in notebooks does not work. Declare every dependency up front. As a bonus, the pipeline can then reuse cached libraries between runs instead of reinstalling them each time.

Init scripts are not a fallback on serverless pipelines, which do not support them. Classic-compute pipelines do, but Databricks still recommends the Environment settings, because init scripts make breakage on runtime upgrades more likely. Pin versions, for example simplejson==3.19.*, and reference local wheels by volume path, for example /Volumes/my_catalog/my_schema/my_ldp_volume/ldpfns-1.0-py3-none-any.whl.

Also watch the language boundary. Pipelines support only SQL and Python, and JVM libraries are not supported. Installing a JAR, even indirectly from an init script, leads to unpredictable behaviour.

Sources1

2.When a runtime upgrade breaks a pipeline

Pinning your own packages covers only half the environment. The Python version and the preinstalled packages beneath them come from the runtime channel, so an upgrade can introduce a conflict without any change on your side. The fix is to set an environment version on the pipeline, a Beta feature that workspace admins enable from the Previews page. With an environment version set, runtime upgrades leave the Python version and preinstalled library versions alone, and the dependencies from your Environment settings are layered on top of that fixed base. If you must keep init scripts on classic compute, Databricks recommends automated tests that run more often, so upgrade breakage is caught early.

Sources1

3.Bundles: declaring whl, pypi and requirements dependencies

In a Declarative Automation Bundle, dependencies are declared in the bundle configuration, under a task's libraries mapping. A whl entry can point to a local file (relative to the bundle config, uploaded or synced), a workspace file, a Unity Catalog volume or cloud object storage. Support depends on the cluster configuration and the library source, so a declaration that is valid in YAML can still fail on the target compute.

resources:
  jobs:
    my_job:
      # ...
      tasks:
        - task_key: my_task
          # ...
          libraries:
            - whl: ./my-wheel-0.1.0.whl
            - whl: /Workspace/Shared/Libraries/my-wheel-0.0.1-py3-none-any.whl
            - whl: /Volumes/main/default/my-volume/my-wheel-0.1.0.whl

A common failure in older bundles is a wheel path that points into the DBFS root. That location is deprecated and disabled by default on Databricks Runtime 15.1 and above, because any workspace user can modify files there. Move the libraries to workspace files or Unity Catalog volumes. For PyPI, a pypi mapping takes a package, optionally with an exact version, and an optional repo. Without repo, the default index https://pypi.org/simple/ is used, which is the wrong place to look for an internal package. A requirements entry can point to a requirements*.txt file at a local, workspace or volume path.

resources:
  jobs:
    my_job:
      # ...
      tasks:
        - task_key: my_task
          # ...
          libraries:
            - pypi:
                package: wheel==0.41.2
            - pypi:
                package: numpy==1.25.2
                 ? : https://pypi.org/simple/

Sources2

4.uv, pyproject.toml and catching errors before they run

Databricks recommends uv for managing Python dependencies in bundles. You list dependencies in the bundle's pyproject.toml, and a resource references them through an editable environment that points to the deployed project folder. The bundle template's pyproject.toml includes a comment warning that for pipelines, dependencies are cached during development. If a pipeline keeps using an old dependency version after you edit pyproject.toml, put the dependency in the environment section of the pipeline's YAML.

resources:
  pipelines:
    test_uv_etl:
      name: test_uv_etl
      catalog: ${var.catalog}
      schema: ${var.schema}
      serverless: true
      root_path: '../src/test_uv_etl'

      libraries:
        - glob:
            include: ../src/test_uv_etl/transformations/**

      environment:
        dependencies:
          - --editable ${workspace.file_path}

For wheel-based jobs, the bundle builds the wheel itself during deploy. Run databricks bundle validate first. If it returns errors, fix them and validate again. Then run databricks bundle deploy --target dev, check that the wheel appears under .bundle/<project-name>/dev/artifacts/.internal, and run the job. One legacy case to know: installing the wheel on Databricks Runtime 12.2 LTS or below requires experimental: python_wheel_wrapper: true at the top level of databricks.yml.

Sources23

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.You can install a package in a pipeline source file and call dbutils.library.restartPython() to load it, as in a notebook.Why is that wrong?

   Pipelines do not support manual Python restarts. Every dependency has to be declared in the pipeline's Environment settings.

   Covered in Pipelines: declare everything in the Environment settings

2. 2.Uploading a bundle's wheel to the DBFS root is a safe default on current runtimes.Why is that wrong?

   Library storage in the DBFS root is deprecated and disabled by default from Databricks Runtime 15.1. Use workspace files or Unity Catalog volumes.

   Covered in Bundles: declaring whl, pypi and requirements dependencies

3. 3.Adding a dependency to the bundle's pyproject.toml is always enough for a pipeline to pick it up immediately.Why is that wrong?

   For pipelines, pyproject.toml dependencies are cached during development. The template says to add them to the pipeline's environment section instead.

   Covered in uv, pyproject.toml and catching errors before they run

Practise it for real

Deploy a Python wheel job with a bundle and verify the wheel dependency arrived before the job runs

1. 1.From the bundle root, run databricks bundle validate.

   Why: This finds configuration errors, such as bad library mappings, before anything is built or uploaded.

   You should see: A summary of the bundle configuration. If errors are returned, fix them and run validate again.

2. 2.Run databricks bundle deploy --target dev.

   Why: Deploy builds the Python wheel and creates the job in the workspace.

   You should see: The wheel appears under Workspace > Users > <your-username> > .bundle > <project-name> > dev > artifacts > .internal.

3. 3.Open Jobs & Pipelines, find [dev <your-username>] sample_job, and open the Tasks tab.

   Why: This confirms the task that consumes the wheel exists.

   You should see: One python_wheel_task.

4. 4.Run databricks bundle run --target dev sample_job and open the Run URL.

   Why: Running the job shows whether the dependency installs and imports correctly at runtime.

   You should see: The task completes and shows a green title bar.