What you will be able to do
- Explain how task dependencies form a DAG and how Databricks orders and parallelises tasks
- Configure task retries and predict how they interact with timeouts, serverless and continuous jobs
- Choose the correct Run if condition for success, failure and cleanup paths
- Predict when a task is marked Upstream failed versus Excluded, and how exclusion and cancellation propagate
Key concept
Run if dependency — A setting on a downstream task that decides whether it runs, based on the combined outcome of its upstream tasks. It lets the job's graph hold failure, cleanup and partial-success paths as well as the normal path where everything succeeds.
1.Dependencies turn a task list into a DAG
The simplest Lakeflow Job is a list of tasks. You control the order they run in by declaring dependencies: each task names the upstream tasks it waits for in its Depends on field. Together these dependencies form a Directed Acyclic Graph (DAG), shown in the job UI as lines between tasks. Databricks runs upstream tasks before downstream ones and runs as many tasks in parallel as it can.
The documentation gives a four-task example. Task 1 is the root and depends on nothing. Task 2 and Task 3 both depend on Task 1, so they can run in parallel after it finishes. Task 4 depends on both Task 2 and Task 3. Two practical details: Depends on only appears when the job has more than one task, and if a task is selected in the DAG when you create a new task, the new task depends on the selected one by default.
At the same time, once Task 1 finishes. Databricks runs upstream tasks first and parallelises as much as it can. Neither Task 2 nor Task 3 depends on the other, so neither has to wait.
Checkpoint 1 of 6· Check yourself
In the four-task example, Task 4 depends on Task 2 and Task 3. When can Task 4 start?
Task 4 is downstream of both Task 2 and Task 3, so it waits for both of them. Databricks runs upstream tasks before downstream tasks.
“Finally, Task 4 depends on Task 2 and Task 3 completing successfully.”Source: docs.databricks.com
Plain dependencies only cover the case where everything goes well. Real pipelines also need a way to recover from transient errors, a way to run cleanup or alerting when something breaks, and a way to branch or loop. The rest of this page covers the first two: retries and Run if conditions.
Sources1
2.Retries: re-running a failed task
A retry sets how many times a task is re-run after it fails with an error. Many errors are transient and go away on restart. Some Databricks features rely on this. Schema evolution with Structured Streaming, for example, assumes jobs run with retries so the environment can reset and the workflow can continue. With retries configured, a failing task restarts up to the number of times you specified. Not every job configuration supports task retries.
To set a policy, open the task's advanced settings and click Add next to Retries. Three details come up often:
- Interval: the retry interval is measured in milliseconds, from the start of the failed run to the start of the retry. - Timeout plus retries: if a task has both a Timeout and Retries, the timeout applies to each retry separately, not to all attempts combined. - Defaults vary by configuration: most configurations don't retry. Serverless jobs auto-optimize retries by default, and continuous jobs use an exponential backoff retry policy.
| Job configuration | Retry behaviour |
|---|---|
| Most configurations (default) | No retries on task failure unless you add a retry policy |
| Serverless jobs | Retries are auto-optimized by default |
| Continuous trigger mode | Automatic retries with exponential backoff |
| Timeout and Retries both set | The timeout applies to each retry |
Continuous jobs handle failure in their own way. When a continuous job fails more than a threshold number of times in a row, Lakeflow Jobs retries the whole job with exponential backoff. While a job is in this backoff state, the Job details pane shows the number of consecutive failures, how long the job must run without error to count as successful, and the time until the next retry. Restart run cancels the active run, resets the retry period and starts a new run.
Checkpoint 2 of 6· Exam question
Which set of comparison operators can an If/else condition task in a Lakeflow Job evaluate when comparing a task value or job parameter against a literal?
Correct answer: A — ==, !=, >, >=, <, <=
- A. The If/else task type supports equality, inequality, and the four relational operators for comparing a dynamic value or job parameter against a literal. This is the complete operator set documented for condition tasks.
- B. Pattern-matching operators like `LIKE` and set-membership operators like `IN` are not part of the If/else task's supported comparison operators. These are SQL-style operators rather than condition-task operators.
- C. String-matching helpers such as `CONTAINS`, `STARTSWITH`, and `ENDSWITH` are not operators the If/else task evaluates. The task is limited to simple equality and relational comparisons, not substring matching.
- D. `AND` and `OR` are logical operators for combining multiple conditions, not comparison operators for a single value check, and the If/else task does not expose them as selectable operators. Each If/else task evaluates one comparison at a time.
Checkpoint 3 of 6· Check yourself
A task has a Timeout and Retries configured. Its first attempt hangs and is timed out. How is the timeout applied to the retry?
The timeout is a per-attempt limit. Each retry gets its own full timeout window.
“If you configure both Timeout and Retries, the timeout applies to each retry.”Source: docs.databricks.com
3.The six Run if conditions
Retries handle a single failing task. Run if conditions decide what *downstream* tasks do once their upstream tasks have finished. To set one, select a task, adjust its Depends on list, pick an option in the Run if dependencies field, and save. There are six conditions. Notice how they differ when the condition is not met: the success-oriented conditions mark the task Upstream failed, while the failure-handling conditions mark it Excluded, which means it is skipped.
| Run if condition | Task runs when… | If unmet, task is marked |
|---|---|---|
| All succeeded (default) | All dependencies have run and succeeded | Upstream failed |
| At least one succeeded | At least one dependency has succeeded | Upstream failed |
| None failed | None of the dependencies failed, and at least one dependency was run | Upstream failed |
| All done | All dependencies have run, regardless of their status | n/a: runs regardless of outcome |
| At least one failed | At least one dependency failed | Excluded |
| All failed | All dependencies have failed | Excluded |
Here is how the conditions map to common patterns. A cleanup or teardown step that must always run uses All done. An alerting or remediation task that should only fire when something broke uses At least one failed. When several redundant upstream tasks each produce the same result and one success is enough, use At least one succeeded. None failed sounds similar to All succeeded but is more lenient: it only requires that nothing failed and that at least one dependency actually ran.
Checkpoint 4 of 6· Match them up
Match each requirement to the Run if condition that implements it
Tap a term, then the definition that fits it.
All done ignores outcomes. At least one failed is a failure handler. All succeeded is the default. None failed requires no failures plus at least one dependency that actually ran.
“All done: The task is run after all its dependencies have run, regardless of the status of the dependent runs.”Source: docs.databricks.com
Checkpoint 5 of 6· Exam question
A job fans out into three parallel notebook tasks that each write to a different table. A downstream Slack notification task should run only if at least one of the three upstream tasks fails, so the team can be alerted to partial failures. Which Run if condition satisfies this requirement while leaving the other three tasks unaffected?
Correct answer: C — At least one failed
- A. This condition requires every one of the three parallel tasks to succeed before the notification task runs, which is the opposite of what an alert-on-failure task needs. A single failure would correctly block this condition, but so would the desired alert.
- B. This condition runs the downstream task only when no dependency failed, meaning it is false whenever a failure exists. Since the notification is supposed to fire specifically when a failure exists, this condition points the wrong direction.
- C. This condition evaluates to true as soon as any one of the three parallel tasks fails, which is exactly the partial-failure alerting behavior the team wants. The other three tasks are unaffected because their own dependency configuration is untouched.
- D. This condition requires every one of the three parallel tasks to fail before the notification runs, so a single failed table write among three successful ones would not trigger the alert. That misses most of the partial-failure cases the team cares about.
- E. This condition runs the notification task after the three tasks finish regardless of outcome, including when all three succeed. That would send an alert even when nothing went wrong, which is noisier than the partial-failure alerting the team wants.
Sources1
4.Excluded, cancelled and disabled upstream tasks
How the Excluded state is evaluated decides that question. When Run if conditions are checked, an upstream task that is Excluded counts as successful. Upstream tasks marked Upstream failed or Upstream canceled count as failed, not as skipped.
There is one override. If *every* dependency of a task is excluded, that task is also excluded, whatever its own Run if condition says. This carries down a linear chain: in A → B → C, if A is excluded then B is excluded, which then excludes C, so none of them run. So in the predict above, Y has only one dependency (X), it is excluded, and Y is excluded too. Y does not run, even though Excluded normally counts as success.
Two more runtime events also go through Run if logic:
- Cancellation: cancelling a task run propagates to downstream tasks, but tasks whose Run if condition handles failure still run. This is how you make sure a cleanup task runs after a cancel. - Disabled tasks: you can disable a task to skip it at run time without deleting it. It keeps its configuration and run history. Downstream tasks are then evaluated against their own Run if conditions and may also be disabled for that run.
Checkpoint 6 of 6· Check yourself
Task D depends on Task B and Task C and uses Run if 'All succeeded'. Task B succeeded. Task C was a failure handler whose condition was not met, so it was marked Excluded. What happens to D?
Excluded upstream tasks count as successful when Run if is evaluated. D is only auto-excluded if all of its dependencies are excluded, and B succeeded.
“Upstream tasks with the Excluded result state are treated as successful when evaluating Run if conditions.”Source: docs.databricks.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Every Lakeflow Jobs task retries automatically a few times before it is marked failed.Why is that wrong?
For most configurations there are no retries unless you add a retry policy. Serverless jobs (auto-optimized) and continuous jobs (exponential backoff) are the exceptions.
Covered in Retries: re-running a failed task
2.A failure-handling task such as 'At least one failed' gets marked Upstream failed when nothing upstream failed.Why is that wrong?
Failure-handling conditions (At least one failed, All failed) mark the task Excluded when they are not met, and an excluded task is skipped. Upstream failed applies to the success-oriented conditions.
Covered in The six Run if conditions
3.Because Excluded counts as success, a task whose only dependency is excluded will still run under 'All succeeded'.Why is that wrong?
If all of a task's dependencies are excluded, the task is excluded too, whatever its Run if condition. This carries down linear chains.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/jobs/run-ifOfficial docs
“Databricks runs upstream tasks before running downstream tasks, running as many of them in parallel as possible.”
↩︎ Dependencies turn a task list into a DAG“Configuring task dependencies creates a Directed Acyclic Graph (DAG) of task execution”
↩︎ Dependencies turn a task list into a DAG“At least one succeeded: At least one dependency has succeeded. The task is marked as Upstream failed if the condition is unmet.”
↩︎ The six Run if conditions“None failed: None of the dependencies failed, and at least one dependency was run.”
↩︎ The six Run if conditions“Upstream tasks marked Upstream failed or Upstream canceled are treated as failed, not as skipped.”
↩︎ Excluded, cancelled and disabled upstream tasks“tasks with a Run if condition that handles failure are run”
↩︎ Excluded, cancelled and disabled upstream tasks“The Run if dependencies field allows you to add control flow logic to tasks based on other tasks' success, failure, or completion.”
↩︎ Key concept“Tasks configured to handle failures are marked as Excluded if their Run if condition is not met. An excluded task is skipped.”
↩︎ Exam trap 2“If all of a task's dependencies are excluded, the task itself is also excluded, regardless of its Run if condition.”
↩︎ Exam trap 3“Finally, Task 4 depends on Task 2 and Task 3 completing successfully.”
↩︎ Checkpoint“All done: The task is run after all its dependencies have run, regardless of the status of the dependent runs.”
↩︎ Checkpoint“Upstream tasks with the Excluded result state are treated as successful when evaluating Run if conditions.”
↩︎ Checkpoint - 2.https://docs.databricks.com/aws/en/jobs/control-flowOfficial docs
“If you specify retries for a task, the task restarts up to the specified number of times if it encounters an error.”
↩︎ Retries: re-running a failed task“When running in continuous trigger mode, Databricks automatically retries with exponential backoff.”
↩︎ Retries: re-running a failed task“Disable a task to skip it at run time without removing it from the job.”
↩︎ Excluded, cancelled and disabled upstream tasks - 3.
“The retry interval is calculated in milliseconds between the start of the failed run and the subsequent retry run.”
↩︎ Retries: re-running a failed task“For most configurations, the default setting does not retry any tasks on task failure.”
↩︎ Exam trap 1“For most configurations, the default setting does not retry any tasks on task failure.”
↩︎ Prediction“If you configure both Timeout and Retries, the timeout applies to each retry.”
↩︎ Checkpoint - 4.
“When consecutive failures of a continuous job exceed a threshold, Lakeflow Jobs uses exponential backoff to retry the job.”
↩︎ Retries: re-running a failed task