Bundle project structure: databricks.yml, include, and targets

1.What a bundle is and why it exists

Declarative Automation Bundles were previously called Databricks Asset Bundles (DABs). They bring source control, code review, testing and CI/CD to data and AI projects. A bundle puts metadata next to your project's source files, so Databricks resources such as jobs and pipelines are described as files that live in the repository. The documentation calls this an infrastructure-as-code (IaC) approach. Bundles fit best when a project has many contributors, needs automation, and must have CI/CD.

A bundle has four kinds of content. First, the cloud infrastructure and workspace configuration it needs. Second, source files such as notebooks and Python files that hold the business logic. Third, definitions of resources: Lakeflow Jobs, Lakeflow pipelines, dashboards, Model Serving endpoints and MLflow objects. Fourth, unit and integration tests. All of this is deployed as one unit. That is why the project's structure matters: it is the structure of what gets shipped.

Sources1

2.databricks.yml: the one required root file

databricks.yml is the main configuration file. The smallest valid version contains the required bundle mapping with a name, plus one target to deploy to.

bundle:
  name: my_bundle

targets:
  dev:
    default: true

The full specification lists more top-level keys. The table shows the ones that shape a scalable project. Settings under artifacts, resources and workspace are defaults, and a target can override them.

To make the configuration modular, keep databricks.yml short and split the rest into separate files. The documentation's example keeps the bundle name and an include glob in databricks.yml. The job definition goes in its own file, and the targets go in another.

# databricks.yml

bundle:
  name: hello-bundle

include:
  - '*.yml'

bundle:
  name: hello-bundle

 ? :
  - '*.yml'

Sources2

3.The template layout: resources/, src/, tests/

Projects usually start from a template rather than an empty folder. Running databricks bundle init offers Databricks-provided templates and asks questions to fill in project variables. Teams can also write custom templates that include default permissions, service principals and CI/CD configuration, so every new project starts with the same standards.

databricks bundle init

The default-python template creates a layout that keeps configuration, code and tests apart. databricks.yml sets the bundle name, references the bundle's files, holds the whl build settings, defines catalog and schema variables, and configures the target workspaces. Each resource has its own file under resources/, for example resources/sample_job.job.yml for the job's settings. src/ holds the code that is built into the Python wheel, and tests/ holds sample unit tests.

Sources31

4.Targets and deployment modes: one bundle, many environments

The targets mapping lets the same source deploy to development and production. A target sets only what is different in its environment. Anything it does not override comes from the top-level resources mapping. In the example below, prod points to a different workspace host and cluster. It does not repeat the notebook_task.

targets:
  dev:
    default: true
  prod:
    workspace:
      host: https://<production-workspace-url>
    resources:
      jobs:
        hello-job:
          name: hello-job
          tasks:
            - task_key: hello-task
              existing_cluster_id: 2345-678901-fabcd456

Databricks recommends host over default where possible, because host makes configuration more portable. The CLI finds a matching profile in .databrickscfg. If more than one profile matches, you pass --profile. Only one target can be the default.

A target can also set a deployment mode. Modes are optional and apply a group of common settings at once. Development mode prefixes deployed resources with [dev ${workspace.current_user.short_name}], marks pipelines as development: true, allows concurrent job runs and turns off the deployment lock. Production mode checks that pipelines are development: false. It also checks that the current Git branch matches the branch set in the target, and --force overrides that check.

targets:
  dev:
    mode: development

Sources24

5.The CLI lifecycle and CI/CD

Bundles are a feature of the Databricks CLI, and all of their CI/CD use goes through it. The lifecycle is: create the bundle (usually with bundle init), develop it by defining settings and resources in databricks.yml and the resource files, then validate, deploy and run. If dev is the default target you can leave out -t dev. Any other target must be named, for example databricks bundle deploy -t prod.

These commands do not depend on an IDE, so a CI system can run them. The documentation describes starting a bundle deployment from a GitHub CI/CD workflow. For production deployments, Databricks recommends service principals, which you enforce by setting run_as to a service principal.

Sources14

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Each sub-folder or team in a large project can have its own databricks.yml that the CLI merges.Why is that wrong?

   There is exactly one databricks.yml, at the bundle root. Other YAML files are brought in through the include mapping.

   Covered in databricks.yml: the one required root file

2. 2.A target must repeat the full job definition, including notebook_task, or the deployed job will be missing it.Why is that wrong?

   A target overrides only what it declares. Undeclared settings come from the top-level resources mapping.

   Covered in Targets and deployment modes: one bundle, many environments

3. 3.Production mode's Git branch check cannot be bypassed, so a hotfix from another branch can never be deployed.Why is that wrong?

   The branch check is real, but you can override it by passing --force when you deploy.

   Covered in Targets and deployment modes: one bundle, many environments