What you will be able to do
- Use the run breakdown to work out whether a slow run was queued, waiting for compute, installing libraries or running
- Choose the right drill-down tool: the timeline and query profiles for serverless, the Spark UI for classic compute
- Read streaming backlog metrics on the Metrics tab and know their collection limits
- Monitor runs from the CLI and Jobs API 2.2 using list-runs, get-run, get-run-output and token-based pagination
1.Where did the time go? The run breakdown
When a run is slow, start by working out *which phase* was slow, not just the total. On the job run details page, hover over the run's Duration. The total splits into four phases: Queued, Waiting for resources, Library installation and Running. The breakdown exists for both job runs and task runs.
Two details affect how you read it. First, for a multi-task job, the job-level breakdown counts Waiting for resources only until the first task starts, so check each task's own breakdown for the full picture. Second, pipeline tasks use different phases: created, waiting for resources, initializing, setting up tables, and running.
| Phase | Possible cause | What you can do |
|---|---|---|
| Queued | The job's maximum concurrent runs reached, or the workspace-level concurrent run limit reached | Increase maximum concurrent runs; stagger overlapping schedules |
| Waiting for resources (serverless) | Standard performance mode has a startup latency of 4 to 6 minutes | Switch to performance-optimized mode |
| Waiting for resources (classic) | Slow VM launch; quota exceeded or insufficient capacity | Instance pools; group successive jobs as tasks in one multi-task job; flexible node types; migrate to serverless |
| Library installation | Uncached serverless environment or many packages; large cluster environment or init scripts on classic | Reduce the number of packages; use only the packages the run needs |
| Running | More data, changed configuration, or code changes | Review system tables; query profiles (serverless); Spark UI (classic) |
Checkpoint 1 of 7· Check yourself
A serverless job spends about five minutes in Waiting for resources on every run. Which change targets that phase directly?
Standard serverless mode has a 4 to 6 minute startup latency, and performance-optimized mode reduces it. Instance pools are the fix for classic compute.
“Switch to performance-optimized mode to reduce startup time.”Source: docs.databricks.com
Sources1
2.Inside the Running phase: timeline, query metrics and the Spark UI
If the Running phase is the problem, which tool to use depends on the compute type. Multi-task jobs have a timeline view. It shows which tasks take a long time, and how they depend on and overlap with each other. For serverless jobs, the timeline also includes queries and query profiles. Expand a task to see its SQL statements and their durations, and click a statement to open its query profile. For jobs on classic compute, use the Spark UI instead. Start with its event timeline, then drill into individual stages and tasks.
Serverless jobs can also show query performance metrics directly in the run UI. This is a Beta feature, and an admin must first enable the *Improved Lakeflow Performance Observability* preview. The metrics are aggregated from the run's queries:
- rows read and rows written per task run - total number of queries per task run - a lightbulb indicator when any of a task's queries have performance insights
The lightbulb appears in the task run sidebar, on DAG nodes, in the timeline and in an Insights column in the list view. As part of this Beta, single-task jobs also get a timeline view. The suggested workflow: open the timeline and hover over a long task. Check rows read and written to see whether more data explains the slowdown. If it doesn't, expand the task and look at the queries marked with a lightbulb.
Checkpoint 2 of 7· Check yourself
A serverless job's Running phase doubled after a code change, while row counts stayed flat. Where should you look next?
For serverless, the docs point to query history and query profiles, which are built into the Timeline view. The Spark UI is the tool for classic compute.
“For serverless jobs, use query history to find which queries took the longest, and open their query profile to identify optimization opportunities.”Source: docs.databricks.com
3.Streaming observability metrics
For a streaming task, the useful question is not how long the task took but how far behind it is. Streaming observability, in Public Preview, adds backlog metrics to the job run details page: backlog seconds, bytes, records and files. They are available for Structured Streaming sources including Kafka, Kinesis, Auto Loader, Google Pub/Sub and Delta tables. Each source supports only some of these metrics, and the UI shows only the ones the source supports.
To see them, click the task on the Job run details page, open the Metrics tab in the Task run pane, and expand a metric's chart. Each chart shows the per-minute maximum for up to the previous 48 hours. You can filter by stream_id, which is shown in the run output. You can also set thresholds on these metrics and get a notification when a stream exceeds one.
Collection has limits. Metrics refresh every minute, unless a run has more than four streams, in which case they refresh every five minutes. Metrics are gathered only for the first fifty streams in a run. They are collected at one-second intervals, so they may not appear if triggerInterval is set below one second. Most sources collect streaming metrics by default, but for others you must set spark.sql.streaming.metricsEnabled to True.
Checkpoint 3 of 7· Check yourself
A task runs eight streams, and one source shows no backlog metrics at all. Which statement is correct?
Above four streams, metrics refresh every five minutes instead of every minute. Collection is capped at the first fifty streams, not four. Some sources only collect metrics once the metricsEnabled flag is set.
“If a run has more than four streams, Databricks updates the metrics every five minutes.”Source: docs.databricks.com
Sources1
4.Monitoring from the Jobs API and CLI
Everything above can also be done in code. The Databricks CLI's jobs command group wraps the Jobs API. databricks jobs list-runs lists runs in descending order by start time, and its flags do the filtering you would otherwise do in the Runs tab:
| Flag | Effect |
|---|---|
| --job-id | Restricts the list to one job |
| --active-only / --completed-only | Returns only active runs, or only completed runs |
| --run-type | JOB_RUN, SUBMIT_RUN or WORKFLOW_RUN |
| --start-time-from / --start-time-to | Runs that started at or after / at or before a value |
| --expand-tasks | Includes task and cluster details in the response |
| --page-token | Fetches the next or previous page using next_page_token or prev_page_token |
For a single run, databricks jobs get-run RUN_ID returns the run's metadata. Add --include-history to include the repair history, or --include-resolved-values to include resolved parameter values. databricks jobs get-run-output RUN_ID returns a task run's output, including any value passed to dbutils.notebook.exit(). It returns only the first 5 MB, so larger results should be written to cloud storage. The 60-day run expiry applies here too. The response below is a run from the Jobs API 2.2 documentation. The state object holds life_cycle_state and result_state, alongside timing fields such as start_time, setup_duration, execution_duration and cleanup_duration.
{
"job_id": 53,
"run_id": 759600,
"number_in_job": 7,
"original_attempt_run_id": 759600,
"state": {
"life_cycle_state": "TERMINATED",
"result_state": "SUCCESS",
"state_message": ""
},
"cluster_spec": {},
"start_time": 1595943854860,
"setup_duration": 0,
"execution_duration": 0,
"cleanup_duration": 0,
"trigger": "ONE_TIME",
"creator_user_name": "user@databricks.com",
"run_name": "process_all_numbers",
"run_type": "JOB_RUN"Reading run state. life_cycle_state says where a run is in its life, and result_state says how a finished run ended. The docs describe a run as *active* while its life cycle state is PENDING, RUNNING or TERMINATING, which is what --active-only returns. The CLI waits for a run to reach TERMINATED or SKIPPED. In the example above, a finished run shows life_cycle_state TERMINATED with result_state SUCCESS. The in-progress run in the Jobs API 2.0 reference shows life_cycle_state RUNNING and only a state_message, with no result_state yet. So check life_cycle_state first, and read result_state once the run has finished.
Timing fields. The sources list setup_duration, execution_duration and cleanup_duration next to run_duration but do not define each one. In the Jobs API 2.0 example, run_duration (3879812) equals the sum of setup (259754), execution (3589020) and cleanup (31038). Use execution_duration as the run-time figure and compare it with the other parts to see where time went.
Account-wide queries. If you have access to the system.lakeflow schema, you can query records of job runs and tasks across your account, and join them with billing tables to monitor job cost.
curl --netrc --request GET \
'https://<databricks-instance>/api/2.0/jobs/runs/get?run_id=<run-id>' \
| jq .Databricks recommends Jobs API 2.2 for scripts, mainly because of how it paginates. Array fields such as tasks, parameters, job_clusters and environments are capped at 100 elements per response. When there are more, the response includes next_page_token. List jobs and List job runs have lost the root-level has_more field, so the presence of next_page_token is now the only sign that more pages exist. These list responses also leave out tasks and job_clusters unless you pass expand_tasks=true. Get a single run accepts only_latest=true, which drops attempts that a retry or repair has superseded. For a ForEach task run, the response has an iterations array, and pagination applies to that array instead of tasks.
Checkpoint 4 of 7· Fill the gap
Complete the request that fetches the next page of a job's tasks in Jobs API 2.2.
/api/2.2/jobs/get?job_id=11223344& ? =Z29...E=The response returns the token in a field called next_page_token. You send it back as the page_token query parameter.
Source: docs.databricks.comCheckpoint 5 of 7· Match them up
Match each CLI command or option to what it gives you
Tap a term, then the definition that fits it.
These map directly to the jobs command group reference. Use get-run for metadata and state, and get-run-output for what a task returned.
“Databricks restricts this API to returning the first 5 MB of the output.”Source: docs.databricks.com
Checkpoint 6 of 7· Exam question
An on-call alert pages only when a run's `result_state` equals `FAILED`, read from the response of `GET /api/2.2/jobs/runs/get`. After an incident, the team found a run that sat for six hours with `life_cycle_state` equal to `INTERNAL_ERROR` never paged, while a run a teammate deliberately canceled did page. Which changes would make the alert logic accurate? (Select all that apply)(Select 3)
Correct answers: A, B, E — Also page when `life_cycle_state` reaches a value such as `INTERNAL_ERROR` or `BLOCKED`, since these indicate the run stopped executing before any `result_state` was ever assigned.; Exclude runs whose `result_state` is `CANCELED`, since that value reflects a deliberate stop rather than an unexpected execution failure that needs a page.; Treat any run still reporting `RUNNING` after its configured timeout has elapsed as a likely failure and page for it in addition to checking `result_state`.
- A. Correct: non-terminal life-cycle values such as `INTERNAL_ERROR` and `BLOCKED` mean the run stopped before a `result_state` was ever set, so checking `result_state` alone misses exactly the stuck-run case described.
- B. Correct: `CANCELED` records an intentional stop rather than an execution failure, so paging on it produced the unnecessary page a teammate's cancel triggered.
- C. Incorrect: a populated `queue_details` object means a run is waiting for a concurrency slot, not that it has failed, so paging on its mere presence would generate false alarms.
- D. Incorrect: `life_cycle_state` alone does not carry success-versus-failure information for terminated runs; `result_state` is still needed to distinguish `SUCCESS` from `FAILED` once a run reaches `TERMINATED`.
- E. Correct: a run still `RUNNING` past its expected timeout is exactly the stuck-run pattern that produced the missed page, so an explicit timeout check closes that gap.
- F. Incorrect: `state_message` is documented as an unstructured, free-text field subject to change, so it cannot be relied on as a stable signal for alerting logic.
Checkpoint 7 of 7· Exam question
A multi-task Lakeflow job runs five tasks on shared job clusters. An engineer needs a report showing the average `execution_duration_seconds` and `queue_duration_seconds` for each individual task over the last 30 days, not a single aggregate figure for the whole job run. Which system table should the query target?
Correct answer: A — `system.lakeflow.job_task_run_timeline`, since it records per-task duration columns for both single-task and multi-task jobs.
- A. This table records per-task duration metrics for every job, and is the recommended source for multi-task jobs since it captures each task's own setup, queue, and execution duration.
- B. Its duration columns are documented as populated only for single-task jobs; for a multi-task job it does not give reliable per-task figures, so it does not satisfy the requirement.
- C. This table tracks task configuration such as dependencies and timeouts, not run-time duration metrics, so it cannot produce the requested average execution and queue times.
- D. This table holds job-level metadata like trigger type and schedule, not per-task or per-run duration figures, so it does not answer the reporting question.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.A Jobs API 2.2 client can keep looping on the root-level has_more field when paging through List job runs.Why is that wrong?
Jobs API 2.2 removes the root-level has_more field. Keep paging only while next_page_token is present.
Covered in Monitoring from the Jobs API and CLI
2.The Spark UI is the first place to diagnose a slow serverless job.Why is that wrong?
The Spark UI is for classic compute. For serverless, use query history and query profiles, which are built into the Timeline view.
Covered in Inside the Running phase: timeline, query metrics and the Spark UI
3.get-run-output returns a task's complete output however large it is.Why is that wrong?
The API returns only the first 5 MB. Write larger results to cloud storage.
Covered in Monitoring from the Jobs API and CLI
Practise it for real
Monitor one of your jobs from the Databricks CLI instead of the UI
1.Run
databricks jobs list --name <your job name>and note the job ID.Why: The --name filter matches the exact job name, ignoring case, so you get the ID without scrolling.
You should see: One job with its job ID.
2.Run
databricks jobs list-runs --job-id <job_id> --completed-only --limit 5.Why: This is the CLI equivalent of the job's Runs tab, newest first by start time.
You should see: Up to five completed runs, each with a run ID and a state.
3.Pick a failed or slow run and run
databricks jobs get-run <run_id> --include-history.Why: It returns the run's state and timing fields, plus any repair history.
You should see: JSON with a state object containing life_cycle_state and result_state, and start_time and duration fields.
4.Run
databricks jobs get-run-output <task_run_id>for one of the run's tasks.Why: It retrieves the task's output, such as a dbutils.notebook.exit() value.
You should see: The task output, truncated at 5 MB if larger, or an HTTP 400 if the run ID is invalid.
Stuck? Get a nudge
If the run has more than 100 tasks, look for next_page_token in the get-run response and pass it back with --page-token.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“hover over the run's Duration to see the total duration broken down into the following phases”
↩︎ Where did the time go? The run breakdown“the job-level breakdown counts Waiting for resources only until the first task starts running.”
↩︎ Where did the time go? The run breakdown“The serverless job uses standard performance mode, which has a startup latency of 4 to 6 minutes”
↩︎ Where did the time go? The run breakdown“the Improved Lakeflow Performance Observability preview must be enabled for your workspace”
↩︎ Inside the Running phase: timeline, query metrics and the Spark UI“The timeline view is now available for single-task jobs as part of this Beta.”
↩︎ Inside the Running phase: timeline, query metrics and the Spark UI“These metrics include backlog seconds, backlog bytes, backlog records, and backlog files for sources supported by Spark Structured Streaming”
↩︎ Streaming observability metrics“Each chart shows maximum values aggregated by minute, for up to the previous 48 hours.”
↩︎ Streaming observability metrics“Databricks collects metrics only for the first fifty streams in each run.”
↩︎ Streaming observability metrics“If a run has more than four streams, Databricks updates the metrics every five minutes.”
↩︎ Streaming observability metrics“For jobs running on classic compute, use the Spark UI to diagnose slow stages and tasks.”
↩︎ Exam trap 2“Your job or task is waiting to start because of concurrency limits.”
↩︎ Prediction“Switch to performance-optimized mode to reduce startup time.”
↩︎ Checkpoint“For serverless jobs, use query history to find which queries took the longest, and open their query profile to identify optimization opportunities.”
↩︎ Checkpoint - 2.https://docs.databricks.com/aws/en/jobs/monitorOfficial docs
“For serverless jobs, queries and query profiles are integrated into the timeline view.”
↩︎ Inside the Running phase: timeline, query metrics and the Spark UI“you can also view and query records of job runs and tasks from across your account.”
↩︎ Monitoring from the Jobs API and CLI - 3.
“List job runs in descending order by start time.”
↩︎ Monitoring from the Jobs API and CLI“Databricks restricts this API to returning the first 5 MB of the output.”
↩︎ Exam trap 3“Databricks restricts this API to returning the first 5 MB of the output.”
↩︎ Checkpoint - 4.
“Fields representing lists of elements such as tasks, parameters, job_clusters, or environments are limited to 100 elements per response.”
↩︎ Monitoring from the Jobs API and CLI“When the only_latest parameter is true, any runs superseded by a retry or a repair are omitted from the response.”
↩︎ Monitoring from the Jobs API and CLI“use the existence of the next_page_token to determine if more results are available”
↩︎ Exam trap 1 - 5.
“An active run is a run in the PENDING, RUNNING, or TERMINATING RunLifecycleState.”
↩︎ Monitoring from the Jobs API and CLI