What you will be able to do
- Choose between the Databricks CLI, SDKs, REST API and Declarative Automation Bundles for job automation
- Create, inspect and manage jobs and runs with the databricks jobs command group
- Distinguish scheduled runs, run-now and runs submit, and what each one means for visibility and auto-optimization
- Recognise the workspace limits that shape automated job creation
1.Choosing a tool for programmatic jobs
Anything you can build in the Jobs UI you can also create and run with developer tools. The documentation compares three of them for creating and managing jobs directly. It also points to a fourth group for jobs that live in a CI/CD pipeline.
| Tool | What it is | Typical use |
|---|---|---|
| Databricks CLI | A command-line interface that wraps the REST API | One-off tasks, experimentation, shell scripting |
| Databricks SDKs (Python, Java, Go, R) | Language libraries for calling Databricks | Applications and custom workflows, instead of sending raw curl calls |
| Databricks REST API | The underlying HTTP API, for example /api/2.2/jobs/get | Automation where no SDK exists for your language |
| Declarative Automation Bundles / Terraform provider | Infrastructure-as-code definitions of jobs | Managing jobs as part of CI/CD |
Checkpoint 1 of 7· Check yourself
A platform team wants every change to its ETL job definitions reviewed in Git and deployed through CI/CD. Which approach does Databricks point them to?
The CLI, SDK and REST examples are introductions to the tools. For managing jobs within CI/CD, the docs name bundles or Terraform.
“To programmatically manage jobs as part of CI/CD, use Declarative Automation Bundles or the Databricks Terraform provider.”Source: docs.databricks.com
Between the direct tools, Databricks recommends the CLI and an SDK. Calling the REST API directly is the fallback for languages that have no Databricks SDK. The tools are not separate systems. The CLI wraps the REST API, so most CLI commands map to a single REST request.
Sources1
2.The databricks jobs command group
The CLI is organised into command groups, and jobs is the group for jobs and job runs. Use databricks jobs -h, or -h after any command, to see its syntax. Because the CLI wraps the REST API, databricks jobs get is a GET to /api/2.2/jobs/get. To create a job, describe it in JSON and pass the file with --json @<file-path>. You don't have to write that JSON from scratch. The job page in the UI shows the JSON of any existing job.
{ "name": "My hello notebook job", "tasks": [ { "task_key": "my_hello_notebook_task", "notebook_task": { "notebook_path": "/Workspace/Users/someone@example.com/hello", "source": "WORKSPACE" } } ]}Checkpoint 2 of 7· Fill the gap
Which command saves the job definition above as a new job?
databricks jobs ? --json @<file-path>jobs create saves a new job from the JSON request body. submit runs a definition once without saving it, and run-now needs a job that already exists.
The group also covers runs. get-run RUN_ID returns a run's metadata. --include-history adds the repair history, and --include-resolved-values adds the resolved parameter values. jobs get and get-run page large arrays. Only the first 100 elements of arrays such as tasks and job_clusters come back, and you pass next_page_token as --page-token to get the rest. cancel-run and cancel-all-runs both work asynchronously. delete-run only works on runs that are no longer active.
Checkpoint 3 of 7· Check yourself
A script runs databricks jobs cancel-all-runs --job-id 123 and then immediately assumes that no run of job 123 can be active. Why is that assumption unsafe?
The runs are cancelled asynchronously, and nothing stops a trigger or caller from starting a new run right afterwards.
“The runs are canceled asynchronously, so it doesn't prevent new runs from being started.”Source: docs.databricks.com
Checkpoint 4 of 7· Exam question
An engineer builds a Lakeflow Job with a `For each` task whose input is a JSON array of ingestion source paths, for example `["s3://raw/orders/", "s3://raw/customers/", "s3://raw/products/"]`. The nested task is a notebook task that reads the source path from a task parameter. How should that parameter be configured so each loop iteration processes its own array element?
Correct answer: A — Set the nested notebook task's parameter value to `{{input}}`, which Lakeflow Jobs resolves to the current iteration's array element each time the loop invokes the nested task.
- A. The `{{input}}` reference is the mechanism Lakeflow Jobs provides specifically for a nested task to receive the current loop iteration's value from the `For each` task's array.
- B. A job-level parameter holds one fixed value for the whole run, so it cannot vary per iteration and would pass the same path to every loop execution.
- C. The `For each` task orchestrates iterations rather than executing arbitrary code, so it cannot call `taskValues.set`, and this adds an unnecessary indirection over the built-in `{{input}}` reference.
- D. Passing the entire array to every iteration and filtering inside the notebook requires the notebook to know its own iteration index, which the nested task is not given, making this approach unworkable.
3.Three ways to run: schedule, run-now and submit
The CLI and the API give you the same three ways to run a job. First, if the JSON definition includes a schedule, the job runs automatically on that Quartz cron schedule.
"schedule": { "quartz_cron_expression": "46 0 9 * * ?", "timezone_id": "America/Los_Angeles", "pause_status": "UNPAUSED"},"max_concurrent_runs": 1,Second, run-now (the CLI's databricks jobs run-now, or jobs.run_now in the API) starts a run of a job that already exists. Third, submit (databricks jobs submit, or jobs.runs.submit in the API) takes a full job definition and runs it once. No job is saved, so nothing appears in the UI, and when the run finishes the job no longer exists. This has a cost on serverless. Because the job isn't saved, a failed submitted job can't be auto-optimized for serverless compute. The docs suggest classic compute for those runs, or using create and then run-now.
Checkpoint 5 of 7· Match them up
Match each way of running a job to its behaviour
Tap a term, then the definition that fits it.
Schedules and run-now both work on saved jobs. Submit is the one-shot path that leaves no job behind.
“submitted jobs can't be auto-optimized for serverless compute in case of failure”Source: docs.databricks.com
Sources1
4.The Python SDK, raw REST calls and workspace limits
The Python SDK works from a notebook or a local machine. On Databricks Runtime 12.2 LTS and below, you have to install it first. The docs build a JobSettings object from the same dictionary shape as the CLI JSON, then pass it to jobs.create on a WorkspaceClient:
w = WorkspaceClient()
j = w.jobs.create(**test_sdk.as_shallow_dict())With no SDK, you call the REST API directly. The docs' example reads the workspace host and a personal access token from environment variables and sends a bearer-token request to the versioned Jobs endpoint:
$ curl --request GET "https://${DATABRICKS_HOST}/api/2.2/jobs/get" \
--header "Authorization: Bearer ${DATABRICKS_TOKEN}" \
--data '{ "job": "11223344" }'Automation also runs into limits that a person clicking in the UI rarely reaches. A workspace can have at most 2000 concurrent task runs. A run that can't start immediately gets 429 Too Many Requests. A workspace can create at most 10000 jobs per hour, and runs submit counts toward that limit, as do jobs created through the REST API. A workspace can hold up to 12000 saved jobs, and a job can have up to 1000 tasks.
Checkpoint 6 of 7· Check yourself
A test harness calls runs submit thousands of times an hour and occasionally receives HTTP 429. What does the 429 indicate?
A workspace is limited to 2000 concurrent task runs. A run request that can't start immediately returns 429.
“A 429 Too Many Requests response is returned when you request a run that cannot start immediately.”Source: docs.databricks.com
Checkpoint 7 of 7· Exam question
A platform team wants to automate the creation and updates of Lakeflow Jobs from a CI/CD pipeline instead of configuring jobs by hand in the Jobs UI. Which of the following are valid, currently supported ways to do this? (Choose all that apply.)(Select 3)
Correct answers: A, B, C — Run `databricks jobs create --json @job-config.json` from the Databricks CLI, passing a JSON job specification that the pipeline generates as a build artifact.; Send an authenticated `POST` request to the `/api/2.2/jobs/create` REST API endpoint, with the job definition supplied as the request body.; Define the job as a `resources.jobs` entry in a `databricks.yml` bundle configuration, then deploy and update it with `databricks bundle deploy`.
- A. This CLI subcommand wraps the Jobs REST API create call and is a documented, supported way to script job creation from a JSON specification.
- B. Calling this REST endpoint directly with a job definition payload is the underlying mechanism the CLI and SDKs use, and it is fully supported for automation.
- C. Declaring the job as a bundle resource and deploying it is the recommended pattern for CI/CD, since the bundle tracks the job's state and supports environment targets.
- D. There is no hidden sync directory that the workspace polls to auto-apply job changes; job definitions must be created or updated through the CLI, API, bundles, or UI.
- E. Placing a notebook file in workspace or DBFS storage does not register a job; a job resource must be explicitly created through the CLI, API, UI, or a bundle deployment.
- F. The newer Jobs API version, not the older deprecated one, is the version documented for current job management, and control-flow task types are not gated behind the deprecated endpoint.
Sources3
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Runs started with jobs submit / runs submit appear as saved jobs and get the same serverless auto-optimization as other jobs.Why is that wrong?
Submitted jobs are not saved and are not visible in the UI. Because they aren't saved, they can't be auto-optimized for serverless compute when they fail.
2.The CLI is a separate interface with capabilities beyond the REST API.Why is that wrong?
The CLI wraps the REST API, and most CLI commands map to a single REST request.
Covered in Choosing a tool for programmatic jobs
Practise it for real
Create, run and inspect a one-task notebook job entirely from the Databricks CLI
1.Save the minimal job JSON (name "My hello notebook job", one notebook_task) to a file, with notebook_path pointing to a notebook you own.
Why: jobs create takes its request body from a JSON file passed with --json @<file-path>.
You should see: A valid JSON file containing a name and a tasks array with one task_key.
2.Run databricks jobs create --json @<file-path>.
Why: This saves the job, so it can be run again with run-now and appears in the Jobs & Pipelines UI.
You should see: The command returns the new job_id, and the job appears in the jobs list.
3.Run databricks jobs run-now with that job ID.
Why: run-now triggers a run of a job that already exists.
You should see: A run starts and shows up in the job's run history.
4.Run databricks jobs get <job-id>.
Why: Confirms the saved definition, including defaults such as max_concurrent_runs and run_if.
You should see: JSON with job_id, creator_user_name and a settings object containing your task.
Stuck? Get a nudge
If you're unsure of a command's flags, add -h, for example databricks jobs run-now -h.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/jobs/automateOfficial docs
“Databricks recommends using the Databricks CLI and a Databricks SDK”
↩︎ Choosing a tool for programmatic jobs“For example, databricks jobs get maps to GET/api/2.2/jobs/get.”
↩︎ The databricks jobs command group“The jobs.runs.submit API takes a job definition and triggers a run for the job.”
↩︎ Three ways to run: schedule, run-now and submit“submitted jobs can't be auto-optimized for serverless compute in case of failure”
↩︎ Exam trap 1“For example, databricks jobs get maps to GET/api/2.2/jobs/get.”
↩︎ Exam trap 2“To programmatically manage jobs as part of CI/CD, use Declarative Automation Bundles or the Databricks Terraform provider.”
↩︎ Checkpoint“With submit, the job is not saved, and isn't visible in the UI.”
↩︎ Prediction“submitted jobs can't be auto-optimized for serverless compute in case of failure”
↩︎ Checkpoint - 2.
“Delete a non-active run. Returns an error if the run is active.”
↩︎ The databricks jobs command group“The runs are canceled asynchronously, so it doesn't prevent new runs from being started.”
↩︎ Checkpoint - 3.https://docs.databricks.com/aws/en/jobsOfficial docs
“A workspace is limited to 2000 concurrent task runs.”
↩︎ The Python SDK, raw REST calls and workspace limits“The number of jobs a workspace can create in an hour is limited to 10000”
↩︎ The Python SDK, raw REST calls and workspace limits“A 429 Too Many Requests response is returned when you request a run that cannot start immediately.”
↩︎ Checkpoint