What you will be able to do
- Run a deployed job or pipeline with databricks bundle run using its resource key and a target
- Bring an existing workspace resource under bundle management with bundle deployment bind, and release it with unbind
- Recognise that bundle destroy permanently deletes deployed resources
- Read a GitHub Actions workflow that installs the CLI and deploys and runs a bundle against a target
1.Running a deployed job or pipeline with bundle run
Declarative Automation Bundles define Databricks jobs and pipelines as YAML, and the Databricks CLI uses those files to validate, deploy and run them. In Databricks' CI/CD flow the Run stage uses databricks bundle run to start a resource that has already been deployed.
You tell bundle run which resource to start by its resource key. The key is the top-level element of the resource's YAML block, not the display name in its name field. If you leave the key out, the CLI asks you to pick a resource from a list, so an unattended pipeline step should always name the key. Target selection works the same way as for deploy: -t picks a target, and without it the default target is used.
databricks bundle run -t dev hello_jobCheckpoint 1 of 4· Check yourself
A bundle declares a job under resources > jobs with the key sample_job and the name field 'Nightly Sales Load'. Which command starts it in the default target?
bundle run takes the resource key, which is the top-level YAML element, not the display name. Running it with no key opens a prompt to choose one resource; it does not run every job.
“The resource key is the top-level element of the resource's YAML block.”Source: docs.databricks.com
2.Bringing existing resources under bundle management, and removing them
Teams often adopt bundles for jobs and pipelines that already exist. databricks bundle deployment bind KEY RESOURCE_ID links a resource defined in the bundle to an existing resource in the workspace, which then becomes bundle-managed. The command shows a diff and lets you refuse the binding; --auto-approve approves it automatically instead of prompting. If you confirm, the next bundle deploy updates the existing resource to match the bundle configuration. Use bind with --target, and check the resource in the workspace before you bind it. For resources that bundle generate supports, its --bind option generates the configuration and binds it in one step.
The binding matters because a deployment tracks the resources it creates by their IDs, and resource names are not used to match a bundle resource to a workspace resource. Configuration generated for an existing job or pipeline is therefore not tied to that resource until it is bound. A resource that is in the configuration but doesn't exist in the target is created, so deploying generated configuration without binding it risks creating a new resource instead of updating the existing one.
| Resource type | RESOURCE_ID to pass |
|---|---|
| job | Numeric ID, such as 6565621249 |
| pipeline | UUID |
| schema | Full name in catalog.schema form, such as my_catalog.my_schema |
| volume | Full name in catalog.schema.volume form |
| registered_model | Full name in catalog.schema.model form |
databricks bundle deployment bind hello_job 6565621249databricks bundle deployment unbind KEY removes the link between a bundle resource and the resource in the workspace. A third subcommand, migrate, moves a bundle from the Terraform deployment engine to the direct deployment engine. It is in Public Preview.
bundle destroy is more drastic. It permanently deletes the jobs, pipelines and artifacts the bundle deployed, and it can't be undone. By default it prompts you to confirm the permanent deletion. In an unattended pipeline step there is nobody to answer, so you add --auto-approve to skip the prompt and delete automatically.
Checkpoint 2 of 4· Check yourself
You want a bundle-defined schema to take over the existing Unity Catalog schema sales.bronze. What do you pass as RESOURCE_ID to bundle deployment bind?
Unity Catalog resources are identified by their full name path. Numeric IDs are for jobs and UUIDs are for pipelines.
“schema: full name in catalog.schema form, such as my_catalog.my_schema”Source: docs.databricks.com
3.Wiring the CLI into a CI/CD pipeline
Databricks names bundles as its recommended approach to CI/CD. The bundle files live in source control, and an external system such as GitHub Actions triggers the deployments. Its recommended simple flow has four steps. Compile and test the code on a pull request or a commit to main. Upload a versioned build artifact. Run databricks bundle validate. Then run databricks bundle deploy to a staging or production environment.
In GitHub Actions, the databricks/setup-cli action installs the CLI in the workflow. The example below deploys to the dev target. It sets the target with the DATABRICKS_BUNDLE_TARGET environment variable rather than -t, and authenticates with a service principal's token stored as the SP_TOKEN secret.
deploy:
name: 'Deploy bundle'
runs-on: ubuntu-latest
steps:
# Check out this repo, so that this workflow can access it.
- uses: actions/checkout@v3
# Download the Databricks CLI.
# See https://github.com/databricks/setup-cli
- uses: databricks/setup-cli@main
# Deploy the bundle to the "dev" target as defined
# in the bundle's settings file.
- run: databricks bundle deploy
working-directory: .
env:
DATABRICKS_TOKEN: ${{ secrets.SP_TOKEN }}
DATABRICKS_BUNDLE_TARGET: devA second job in the same workflow lists the deploy job under needs, so it waits for the deploy to finish. It then starts the deployed job by its key. The workflow also sets concurrency: 1; the example's comment says this ensures that only a single job or workflow using the same concurrency group runs at a time.
On authentication, there are two levels to know. Using a service principal instead of a person's account is a security best practice: the CI platform never handles a user's token, and the service principal can be disabled or deleted without affecting users. That is what the SP_TOKEN example does. Databricks goes further and recommends workload identity federation for CI/CD authentication. It eliminates the need for Databricks secrets, which makes it the most secure way to authenticate automated flows. The Git folder example later in this section uses it.
Checkpoint 3 of 4· Fill the gap
The second job in the workflow starts the deployed job sample_job. Which bundle command completes the step?
databricks bundle ? sample_job --refresh-allbundle run starts a deployed resource by its key. deploy and validate act on the whole bundle and take no resource key.
Source: docs.databricks.comThe same CLI can also manage workspace assets outside a bundle. As an alternative to full bundle CI/CD, a production Git folder can reflect a remote Git repository, and an external tool such as GitHub Actions can pull it on merge. Databricks shows a workflow that runs databricks repos update with a branch when a push lands on that branch. It authenticates with workload identity federation (DATABRICKS_AUTH_TYPE: github-oidc) and uses a service principal's client ID.
The trade-off is scope: only code files such as notebooks and dashboard drafts are in source control. Configurations for jobs or pipelines that run assets in the Git folder are not, which is why bundles are the recommended approach.
One more piece of bundle configuration shows up in CI/CD: environment-specific values. Bundles declare variables, reference them as ${var.name}, and can give them different values under each target's variables, so the same bundle deploys to dev, staging and production. In the pipeline-update example bundle, the dev and prod targets set different values for the catalog and schema variables. Databricks' dashboard example does the same with ${var.warehouse_id} for the SQL warehouse. The CLI also accepts --var to set a variable's value from the command line.
Checkpoint 4 of 4· Exam question
A bundle defines a `warehouse_id` variable with a default value, and the `prod` target overrides it to point at a larger production SQL warehouse. Which mechanism in databricks.yml is being used here?
Correct answer: A — Setting the variable's value under the `variables` key inside the `prod` target block, which overrides the bundle-level default only for deployments made against that target
- A. This is correct: target-scoped variable overrides let each target (dev, test, prod) supply its own value for a variable declared at the bundle level, without duplicating the resource definitions themselves.
- B. `--var` is a way to override a variable ad hoc from the command line, but it is not required for a target-specific default; storing the override directly in the target block is the standard, repeatable approach.
- C. Introducing a second variable name for essentially the same value forces every resource reference to be duplicated or conditioned on the target, which defeats the purpose of a single overridable variable.
- D. Declarative Automation Bundles do not support swapping in an entirely separate top-level YAML file per environment through an environment variable; environment differences are expressed through targets in one databricks.yml.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.bundle destroy only removes the local bundle files and deployment state, so you can deploy again and get the same resources back.Why is that wrong?
Destroy permanently deletes the jobs, pipelines and artifacts the bundle deployed, and it can't be undone.
Covered in Bringing existing resources under bundle management, and removing them
2.Generating bundle configuration for an existing job and running bundle deploy updates that existing job, with no bind needed.Why is that wrong?
A deployment tracks resources by ID, not by name, so configuration that was never bound isn't tied to the existing job. Binding is what makes deploy update the existing resource.
Covered in Bringing existing resources under bundle management, and removing them
Practise it for real
Take a new bundle through validate, deploy and run against a development target using the Databricks CLI.
1.Run databricks bundle init and pick one of the default templates.
Why: Creating the bundle is the first step of its lifecycle, and a template gives you a working databricks.yml.
You should see: A project folder that contains databricks.yml and resource files.
2.From the bundle root, run databricks bundle validate.
Why: This checks the configuration's syntax and shows the bundle identity before you deploy anything.
You should see: A Name / Target / Workspace summary followed by 'Validation OK!'
3.Run databricks bundle deploy -t dev.
Why: -t chooses the target. If you leave it out, the default target is used.
You should see: The bundle's resources are created in the workspace, under the Path shown by validate.
4.Run databricks bundle run -t dev <job key>, using the key of the job in your resource YAML.
Why: bundle run starts a deployed resource by its resource key, not by its display name.
You should see: A run of the job starts in the dev target.
Stuck? Get a nudge
If bundle run asks you to choose a resource, you didn't give it a key. Look up the top-level element of the job's YAML block.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/dev-tools/ci-cdOfficial docs
“Run bundle resources using databricks bundle run.”
↩︎ Running a deployed job or pipeline with bundle run“Declarative Automation Bundles are the recommended approach to CI/CD on Databricks.”
↩︎ Wiring the CLI into a CI/CD pipeline“or use external CI/CD tools such as GitHub Actions to pull the Git folder on merge”
↩︎ Wiring the CLI into a CI/CD pipeline“Configurations for jobs or pipelines that run assets in the Git folder and configurations for publishing dashboards are not in source control.”
↩︎ Wiring the CLI into a CI/CD pipeline - 2.
“you are prompted to select a resource to run from a list of available resources”
↩︎ Running a deployed job or pipeline with bundle run“If the -t option is not specified, the default target as declared within the bundle configuration files is used.”
↩︎ Running a deployed job or pipeline with bundle run“By default, you are prompted to confirm permanent deletion of the previously-deployed jobs, pipelines, and artifacts.”
↩︎ Bringing existing resources under bundle management, and removing them“To skip these prompts and perform automatic permanent deletion, add the --auto-approve option to the bundle destroy command.”
↩︎ Bringing existing resources under bundle management, and removing them“Destroying a bundle permanently deletes a bundle's previously-deployed jobs, pipelines, and artifacts. This action can't be undone.”
↩︎ Exam trap 1“The resource key is the top-level element of the resource's YAML block.”
↩︎ Checkpoint - 3.
“is updated based on the configuration defined in the bundle it is bound to after the next bundle deploy”
↩︎ Bringing existing resources under bundle management, and removing them“The bind command should be used with the --target flag.”
↩︎ Bringing existing resources under bundle management, and removing them“Remove the link between the resource in a bundle and its remote counterpart in a workspace.”
↩︎ Bringing existing resources under bundle management, and removing them“Migrate the bundle from using the Terraform deployment engine to using the direct deployment engine.”
↩︎ Bringing existing resources under bundle management, and removing them“Resource names are not used to correlate between a bundle deployment and a resource instance”
↩︎ Bringing existing resources under bundle management, and removing them“If a resource in the bundle configuration does not exist in the target workspace, it is created.”
↩︎ Bringing existing resources under bundle management, and removing them“Automatically approve the binding, instead of prompting”
↩︎ Bringing existing resources under bundle management, and removing them“set values for variables defined in bundle config.”
↩︎ Wiring the CLI into a CI/CD pipeline“a bundle deployment tracks the resources it creates in the target workspace by their IDs”
↩︎ Exam trap 2“Bind does not recreate data.”
↩︎ Prediction“schema: full name in catalog.schema form, such as my_catalog.my_schema”
↩︎ Checkpoint - 4.
“Use databricks bundle deploy to deploy the bundle to a staging or production environment.”
↩︎ Wiring the CLI into a CI/CD pipeline“Databricks recommends workload identity federation for CI/CD authentication.”
↩︎ Wiring the CLI into a CI/CD pipeline“Use variables, for example ${var.warehouse_id}, to parameterize configurations like SQL warehouses or data sources, ensuring seamless deployment across dev, staging, and production environments.”
↩︎ Wiring the CLI into a CI/CD pipeline - 5.
“A composite action that sets up the Databricks CLI in a GitHub Actions workflow.”
↩︎ Wiring the CLI into a CI/CD pipeline“Ensure that only a single job or workflow using the same concurrency group”
↩︎ Wiring the CLI into a CI/CD pipeline - 6.
“Users can safeguard their access tokens from being accessed by CI/CD platforms.”
↩︎ Wiring the CLI into a CI/CD pipeline