Bundle Target Overrides: Promoting One Codebase from Dev to Prod

1.Targets: one configuration, several destinations

A bundle has exactly one databricks.yml at its root. Its top-level mappings (resources, workspace, artifacts, variables) are the defaults for every environment. The targets mapping lists the environments you deploy to. Under each target you can override workspace, resources, artifacts, variables, run_as, permissions, mode, presets and more. Only one target can be marked default: true.

The simplest promotion step is to point a prod target at a different workspace and a different cluster. Everything you don't override falls back to the top-level definition.

targets:
  dev:
    default: true
  prod:
    workspace:
      host: https://<production-workspace-url>
    resources:
      jobs:
        hello-job:
          name: hello-job
          tasks:
            - task_key: hello-task
              existing_cluster_id: 2345-678901-fabcd456

The prod target does not repeat notebook_task. It inherits it from the top-level job. The docs' first example reads its credentials from a local profile named DEFAULT. Databricks recommends the host mapping instead of the default mapping wherever possible, because it makes your bundle configuration files more portable. That advice is about how credentials are found, not about default: true, which only chooses the target used when you pass no -t. The CLI matches the host to a profile in .databrickscfg. If several profiles match, you must pass --profile.

You choose a target with -t (or --target) on validate, deploy and run. Commands without -t go to the default target.

# You must specify "-t prod", because the "dev" target
# is already set to "default: true":
databricks bundle validate
databricks bundle deploy -t prod
databricks bundle run -t prod hello_job

A note on names: the docs' YAML declares the job under the key hello-job, but its run commands say hello_job. The sources do not explain the difference, so treat it as an inconsistency in the docs. In your own bundle, pass bundle run the resource key exactly as it is written in your YAML, as the run_as example does with databricks bundle run my_test_job_1.

Sources1

2.How target overrides merge with top-level settings

A target override does not replace the whole resource. It is joined with the top-level definition, and where both set the same field, the target's value wins. For list entries such as job clusters, pipeline clusters and tasks, the bundle needs a key to know which entries to merge.

resources:
  jobs:
    my-job:
      name: my-job
      job_clusters:
        - job_cluster_key: my-cluster
          new_cluster:
            spark_version: 13.3.x-scala2.12

targets:
  development:
    resources:
      jobs:
        my-job:
          name: my-job
          job_clusters:
            - job_cluster_key: my-cluster
              new_cluster:
                node_type_id: i3.xlarge
                num_workers: 1

The validated result has all three fields. If the top level also set num_workers: 1 and the target set num_workers: 2, the result would be 2. Pipelines work the same way, matched on the cluster label. A top-level node_type_id plus a target num_workers on label default produces one cluster with both. Tasks only merge when task_key matches exactly. A target task with a different key is not joined to the top-level task.

Sources2

3.Modes and run identities across dev, staging and prod

Targets also control how a deployment behaves, not just which values it uses. The mode mapping is optional. It applies a group of commonly used settings at once. mode: development is meant for pre-production iteration, and mode: production is meant for validated deliverables.

Development mode also does the following: - prepends the prefix [dev ${workspace.current_user.short_name}] to all resources that are not deployed as files or notebooks - tags each deployed job and pipeline with a dev Databricks tag - pauses all schedules and triggers - enables concurrent job runs - disables the deployment lock

Production mode can also check that the current Git branch matches the branch set in the target. You can override that check with --force.

Databricks recommends service principals for production deployments. If you do not use one, production mode adds two more validations. It checks that artifact_path, file_path, root_path and state_path are not overridden to a specific user. It also checks that run_as and permissions are specified, so it is clear which identities hold which permissions for deployments.

To customise a mode, set presets on the target. Presets override the mode's defaults, and settings on individual resources override presets.

The identity a job runs as is also set per target. Databricks calls it best practice to configure run identities for staging and production. Production should run as a service principal, which keeps the permission to run a workflow separate from the identity that deployed it.

  # Staging deployment settings
  staging:
    workspace:
      host: https://my-host.cloud.databricks.com
      root_path: /Shared/staging-workspace/.bundle/${bundle.name}/${bundle.target}
    run_as:
      service_principal_name: '69511ed2-zb27-444c-9863-4bc8ff497637'

  # Production deployment settings
  production:
    mode: production
    workspace:
      host: https://my-host.cloud.databricks.com
      root_path: /Shared/production-workspace/.bundle/${bundle.name}/${bundle.target}
    run_as:
      service_principal_name: '68ed9cd5-8923-4851-x0c1-c7536c67ff99'

Sources34

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.A target that sets only num_workers on a job cluster replaces the whole new_cluster block, so spark_version is lost.Why is that wrong?

   Target settings are joined with the top-level settings for the same job_cluster_key. Only fields defined in both places are overridden, and the target wins those.

   Covered in How target overrides merge with top-level settings

2. 2.If a preset sets jobs_max_concurrent_runs to 20, every job uses 20, even a job whose own max_concurrent_runs is 10.Why is that wrong?

   Presets override the mode defaults, but a setting on an individual resource overrides the preset. That job keeps 10.

   Covered in Modes and run identities across dev, staging and prod

Practise it for real

Promote one hello-bundle from dev to prod by changing only variable values and target settings

1. 1.In databricks.yml, declare a variable my_cluster_id with default 1234-567890-abcde123, use existing_cluster_id: ${var.my_cluster_id} in the hello-task, and run databricks bundle validate --output json.

   Why: Confirms the variable resolves before anything is deployed.

   You should see: The variables section shows my_cluster_id with default and value both 1234-567890-abcde123.

2. 2.Add a prod target with workspace.host set to your production URL, and variables: my_cluster_id: 2345-678901-bcdef234 under it.

   Why: A value in the targets mapping is checked before the top-level default.

   You should see: Deploying to prod uses 2345-678901-bcdef234. Deploying to dev still uses the default.

3. 3.Run databricks bundle deploy -t dev, then databricks bundle run -t dev hello-job. Here hello-job is the resource key under resources.jobs in your YAML.

   Why: dev is the default target, so -t is optional, but stating it makes the CI step explicit. bundle run takes the resource key exactly as written in your YAML.

   You should see: A job prefixed [dev <your short name>] if dev uses mode: development, running on the default cluster.

4. 4.Run databricks bundle deploy -t prod, then databricks bundle run -t prod hello-job.

   Why: The same files are promoted, and only target-level values change.

   You should see: The job is deployed to the production workspace, using the prod cluster ID.