What you will be able to do
- Apply the Databricks production rules for Structured Streaming jobs: jobs compute, continuous job scheduling, no autoscaling
- Explain how serverless compute limits Structured Streaming triggers and when a pipeline in continuous mode is the answer
- Choose between a triggered pipeline, a continuous job and low-latency Structured Streaming based on a latency SLA and cost
- Identify where exactly-once guarantees hold and where foreachBatch needs idempotent design
1.What running Structured Streaming in production costs you
Spark Structured Streaming is the procedural streaming API. Apache Spark Declarative Pipelines (SDP), the basis of Lakeflow pipelines, is the declarative alternative. On the exam, operational constraints usually decide between them, and the first one is how much running infrastructure you are willing to own.
If you choose Structured Streaming, Databricks tells you to run production workloads as scheduled Lakeflow Jobs. It always recommends four things: remove result-returning code such as display and count from the notebook; use jobs compute instead of all-purpose compute; schedule the job in Continuous mode; and don't enable autoscaling. Autoscaling is ruled out because compute autoscaling has trouble scaling down clusters running Structured Streaming. For streaming workloads, Databricks instead recommends SDP on Lakeflow with *enhanced autoscaling*.
A job's continuous trigger does three things by default. It prevents concurrent runs, starts a new run when the previous one fails, and uses exponential backoff for retries. It also has a limit: you cannot set dependencies between tasks in a continuous job. If you run several streams, you choose between multiple tasks and multiple queries in a single task, and the two retry differently.
| Strategy | How compute is shared | How retries are handled |
|---|---|---|
| Multiple tasks | Jobs compute sized to each streaming task; sharing compute is optional | All tasks must fail before the job retries |
| Multiple queries | All queries share the same compute; optionally assign scheduler pools | The task retries if any query fails |
Checkpoint 1 of 4· Check yourself
A Structured Streaming job struggles during traffic spikes. An engineer proposes turning on cluster autoscaling. What does Databricks recommend?
Compute autoscaling has trouble scaling down Structured Streaming clusters. Databricks points elastic streaming workloads to pipelines with enhanced autoscaling.
“Databricks recommends using Spark Declarative Pipelines on Lakeflow with enhanced autoscaling for streaming workloads.”Source: docs.databricks.com
Sources1
2.Serverless, triggered and continuous execution
Serverless is a hard constraint. On serverless compute, Structured Streaming supports only Trigger.AvailableNow() and Trigger.Once(), and Databricks recommends AvailableNow(). So if you must stream continuously on serverless, the answer is a pipeline.
Pipelines themselves run in one of two modes. Triggered pipelines can cost less because the cluster runs only long enough to update the pipeline, but new data waits until the next trigger. Continuous pipelines need an always-running cluster. That costs more but reduces latency. Databricks advises starting with triggered and keeping continuous mode for a proven latency requirement, because continuously running compute is usually the largest cost factor.
How you get continuous execution matters too. Databricks recommends running continuous pipelines with a continuous *job* rather than setting the pipeline's Pipeline mode to continuous. A continuous job also unlocks serverless performance modes such as Standard mode. When a job orchestrates a pipeline, the job's execution mode takes precedence. A continuous job runs a pipeline continuously even if its Pipeline mode is triggered, and a scheduled job runs a single update even if the mode is continuous. So leave Pipeline mode on triggered (the default) when you wrap the pipeline in a continuous job. Standalone materialized views and streaming tables always run in triggered mode, whatever the job orchestration.
Checkpoint 2 of 4· Exam question
A team receives a raw change feed (inserts, updates, deletes) from an upstream OLTP database through Auto Loader and must materialize a Type 2 slowly changing dimension with full history, while minimizing hand-written merge logic and out-of-order handling. They are choosing between the AUTO CDC API inside a Lakeflow Declarative Pipeline and a custom `foreachBatch` MERGE written in Structured Streaming. Which of the following are valid reasons to prefer the Lakeflow Declarative Pipeline approach? (Select all that apply)(Select 3)
Correct answers: A, B, E — The AUTO CDC flow can be configured with `stored_as_scd_type=2` so the pipeline builds and maintains the full change history automatically, without a hand-written MERGE statement.; The AUTO CDC flow handles out-of-order and late-arriving change records using a configurable sequencing column, removing the need to write custom ordering logic before the merge.; Pipeline expectations can validate incoming change records and quarantine or drop rows that fail data quality checks before they reach the SCD table, without extra orchestration code.
- A. Configuring `stored_as_scd_type=2` is a genuine reason to prefer AUTO CDC here, because it tells the flow to build and maintain the full history of a slowly changing dimension automatically, replacing what would otherwise be hand-written MERGE and history-tracking logic.
- B. AUTO CDC's sequencing column is a real capability that lets the flow determine the correct order of change events even when they arrive out of order, which removes the need to write custom ordering logic before applying changes.
- C. AUTO CDC requires the developer to explicitly specify the key columns that identify a row; it does not infer identity from internal Delta row IDs, so this claim misstates how the flow determines which rows to update.
- D. AUTO CDC governs how change records are applied to a target Delta table and makes no guarantee about delivery semantics to an external REST API, so this claim describes a capability the flow does not provide.
- E. Pipeline expectations are a real, built-in mechanism for validating rows and dropping or quarantining ones that fail checks as part of the flow, which is exactly the kind of orchestration-free data quality enforcement the scenario is asking for.
- F. AUTO CDC applies row-level changes to a target table; it does not perform dimensional modeling or normalize a source schema into a star schema, so this claim describes work the flow does not do.
Checkpoint 3 of 4· Check yourself
A pipeline has Pipeline mode set to continuous, and it is orchestrated by a scheduled job that runs hourly. How does it run?
The job's execution mode overrides the pipeline's Pipeline mode setting, so a triggered or scheduled job runs the pipeline as a single update.
“a triggered or scheduled job runs its pipeline as a single update even if the pipeline's Pipeline mode is continuous”Source: docs.databricks.com
3.Latency SLAs and delivery guarantees
For operational workloads such as fraud detection, anomaly detection and real-time alerting, Structured Streaming offers detailed latency controls. Real-time mode reaches end-to-end latency under one second at the tail and around 300 milliseconds in common cases. When real-time mode doesn't fit, a processingTime trigger with a 0 interval gives the lowest micro-batch latency, at the cost of more cloud storage API calls. Databricks also warns about storage hops. Keep computation within a single streaming pipeline where possible, because every extra job or pipeline in the chain adds latency. This level of tuning is the "low-level performance optimizations" case that the procedural model exists for.
Delivery guarantees also affect the choice. In a pipeline, a streaming table processes each record exactly once, as long as the source is append-only. Structured Streaming also provides exactly-once guarantees through checkpoints, but some operations weaken that. foreachBatch provides at-least-once delivery, so a pipeline that uses it must be idempotent. This applies in both frameworks, because foreachBatch sinks are also one of the features Lakeflow pipelines add on top of SDP.
| Constraint | Documented guidance |
|---|---|
| Latency under one second for operational workloads | Structured Streaming real-time mode |
| Lowest micro-batch latency, if you accept more storage API calls | processingTime trigger with a 0 interval |
| Lowest cost, new data can wait for the next run | Triggered pipeline |
| Continuous streaming on serverless compute | Pipeline run continuously, preferably by a continuous job |
| Elastic compute for streaming | Spark Declarative Pipelines on Lakeflow with enhanced autoscaling |
Checkpoint 4 of 4· Check yourself
A stream writes each micro-batch to an operational database through foreachBatch. After a failure and restart, the database shows duplicate rows. What is the documented explanation?
Databricks notes that foreachBatch is at-least-once, so a restart can replay a batch. The fix is to make the sink write idempotent.
“Some operations like foreachBatch provide at-least-once rather than exactly-once guarantees.”Source: docs.databricks.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Scheduling a Structured Streaming job in Continuous mode means setting a continuous Structured Streaming trigger interval.Why is that wrong?
Continuous mode here is the Lakeflow Jobs scheduling feature, which restarts failed runs and prevents concurrent runs. It has nothing to do with the query's trigger interval.
Covered in What running Structured Streaming in production costs you
2.The recommended way to run a pipeline continuously is to set its Pipeline mode to continuous.Why is that wrong?
Databricks recommends wrapping the pipeline in a continuous job and leaving Pipeline mode on triggered. The job's mode takes precedence.
3.Because Structured Streaming is exactly-once, any sink written from foreachBatch is automatically free of duplicates.Why is that wrong?
foreachBatch is at-least-once, so the processing pipeline must be idempotent to handle replayed batches.
Covered in Latency SLAs and delivery guarantees
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Always schedule streams as Lakeflow Jobs using jobs compute.”
↩︎ What running Structured Streaming in production costs you“Compute auto-scaling has limitations scaling down cluster size for Structured Streaming workloads.”
↩︎ What running Structured Streaming in production costs you“When you configure a job using the continuous trigger, you cannot set dependencies between tasks.”
↩︎ What running Structured Streaming in production costs you“On serverless compute, only Trigger.AvailableNow() and Trigger.Once() are supported.”
↩︎ Serverless, triggered and continuous execution“which achieves end-to-end latency under one second at the tail and around 300 milliseconds in common cases”
↩︎ Latency SLAs and delivery guarantees“Splitting logic across multiple jobs or pipelines adds storage hops that increase latency.”
↩︎ Latency SLAs and delivery guarantees“This refers to the Databricks Jobs scheduling feature, not the Structured Streaming trigger interval.”
↩︎ Exam trap 1“For these operations, make sure that your processing pipeline is idempotent.”
↩︎ Exam trap 3“Databricks recommends using Spark Declarative Pipelines on Lakeflow with enhanced autoscaling for streaming workloads.”
↩︎ Checkpoint“For continuous streaming on serverless compute, use Triggered vs. continuous pipeline mode in continuous mode.”
↩︎ Prediction“Some operations like foreachBatch provide at-least-once rather than exactly-once guarantees.”
↩︎ Checkpoint - 2.
“Continuous pipelines require an always-running cluster, which is more expensive but reduces processing latency.”
↩︎ Serverless, triggered and continuous execution“Databricks recommends running continuous pipelines with a continuous job rather than setting the value of Pipeline mode to continuous.”
↩︎ Exam trap 2“a triggered or scheduled job runs its pipeline as a single update even if the pipeline's Pipeline mode is continuous”
↩︎ Checkpoint - 3.
“reserve it for a proven latency requirement”
↩︎ Serverless, triggered and continuous execution - 4.https://docs.databricks.com/aws/en/ldp/conceptsOfficial docs
“Each record is processed exactly one time, assuming an append-only source.”
↩︎ Latency SLAs and delivery guarantees