What you will be able to do
- Explain how the procedural model of Spark Structured Streaming differs from the declarative model of Apache Spark Declarative Pipelines (SDP)
- List what a declarative pipeline automates that you would otherwise hand-code and orchestrate with Lakeflow Jobs
- Tell apart the capabilities in open-source SDP from those that only Lakeflow pipelines add
- Use the documented decision points to choose a framework for a given ETL requirement
Key concept
Procedural vs. declarative processing — With Structured Streaming you write the steps yourself and orchestrate them. With Spark Declarative Pipelines you describe the datasets you want, and the framework works out the execution order, retries and optimisation. You are trading fine-grained control for less code to maintain.
1.Two ways to express the same ETL
When this exam objective asks you to choose between Spark Structured Streaming and Apache Spark Declarative Pipelines, it is really asking you to choose a programming paradigm. Databricks describes procedural programming as specifying *how* a task is done through explicit sequences of operations. Declarative programming specifies *what* result you want and lets the system decide how to produce it.
Procedural processing means step-by-step execution, where you set the order of operations. You use control structures such as loops, conditionals and functions, and you get detailed control over resources for manual performance tuning. Structured Streaming falls on this side. You write the reads, transformations, writes and triggers. You also decide where checkpoints live, and you schedule the work yourself in Lakeflow Jobs.
Declarative processing hides the execution details. The system plans and tunes execution, and you don't write explicit control structures, which makes pipelines easier to maintain. Lakeflow pipelines are built on Apache Spark Declarative Pipelines (SDP). SDP is a declarative framework that Databricks describes as designed to simplify the creation of reliable and maintainable stream processing pipelines.
Checkpoint 1 of 6· Check yourself
Which description matches declarative programming as Databricks defines it?
Declarative programming states the desired outcome and leaves the execution plan to the system. The other options describe procedural processing.
“Declarative programming focuses on what needs to be achieved, leaving the underlying system to determine the best way to execute the task.”Source: docs.databricks.com
| Aspect | Procedural processing | Declarative processing |
|---|---|---|
| Control | Full control over execution | Execution handled by system |
| Complexity | Can be complex and verbose | Generally simpler and more concise |
| Optimization | Requires manual tuning | System handles optimization |
| Flexibility | High, but requires expertise | Lower, but easier to use |
| Use cases | Custom pipelines, performance tuning | SQL queries, managed pipelines |
No. SDP is a framework for developing and running batch and streaming data pipelines in both SQL and Python. Its common use cases include incremental ingestion from message buses and incremental streaming transformations with stateless and stateful operators. Both frameworks can handle streaming. The real difference is who controls execution.
2.What a declarative pipeline takes off your hands
Databricks contrasts pipelines directly with building ETL from the Spark and Structured Streaming APIs on Databricks Runtime, *using manual orchestration via Lakeflow Jobs*. The documented benefits show what a Structured Streaming team would otherwise have to build and run itself:
- Automatic orchestration. SDP works out the dependencies between the flows in your pipeline. It runs them in the correct order with maximum parallelism, and it retries transient failures progressively: first the Spark task, then the flow, then the whole pipeline. - Less code. Declarative functions reduce hundreds of lines of manual Spark and Structured Streaming code to a few. In Lakeflow pipelines, the AUTO CDC API handles change data capture, including SCD Type 1 and Type 2, without manual code for out-of-order events or for streaming concepts like watermarks. - Incremental processing. You write materialized view logic with batch semantics. The engine reprocesses only new or changed source data when it can. - Built-in recovery. Checkpointing and retries are automatic, so a pipeline resumes from the last processed offset.
Together, these mean pipelines automate orchestration, compute management, monitoring, data quality enforcement and error handling. With Structured Streaming, each of those becomes code or a configuration decision that your team owns.
Checkpoint 2 of 6· Check yourself
A team keeps hand-writing merge logic to handle out-of-order CDC events and tuning watermarks for an SCD Type 2 dimension. Which documented pipeline benefit addresses this directly?
Databricks presents AUTO CDC as replacing hand-written CDC handling, including out-of-order events and watermarks. The other options are real benefits, but none of them addresses CDC merge logic.
“without manual code for out-of-order events or streaming concepts like watermarks”Source: docs.databricks.com
3.Open-source SDP vs Lakeflow pipelines
The exam guide names *Apache Spark* Declarative Pipelines, so you need to know which features belong to the open framework and which Databricks adds on top. Lakeflow pipelines run on the performance-optimized Databricks Runtime and work with SDP. Because they build on SDP rather than proprietary APIs, your transformation code stays portable to other SDP runtimes. Lakeflow pipelines keep the same authoring model and add production features.
| Capability | SDP | Lakeflow pipelines |
|---|---|---|
| Declarative pipelines in SQL and Python | Yes | Yes |
| Streaming tables, materialized views, temporary views | Yes | Yes |
| Append flows; sinks (Delta, Apache Kafka, Azure Event Hubs) | Yes | Yes |
| Automatic orchestration and dependency resolution | Yes | Yes |
| AUTO CDC (SCD Type 1 and SCD Type 2) and AUTO CDC from snapshot | No | Yes |
| Data quality expectations | No | Yes |
| Queryable event log | No | Yes |
| Update flows and foreachBatch sinks | No | Yes |
| Continuous mode | No | Yes |
Checkpoint 3 of 6· Match them up
Match each option to what it gives you
Tap a term, then the definition that fits it.
Lakeflow pipelines extend SDP with production features. Structured Streaming is the procedural alternative that you orchestrate yourself.
“add production features such as AUTO CDC, data quality expectations, and a queryable event log”Source: docs.databricks.com
Sources2
4.Making the call: the documented decision points
Databricks reduces the choice to three pairs of conditions. Choose procedural processing (Structured Streaming) when you need fine-grained control over execution logic, when transformations involve complex business rules that are hard to express declaratively, or when performance requires manual tuning. Choose declarative processing (pipelines) when simpler development and maintenance are priorities, when SQL-based transformations or managed workflows make procedural control unnecessary, or when the built-in optimizations of a framework like pipelines are enough.
The documentation also states a default for new work: Databricks introduced Lakeflow pipelines to reduce the complexity of managing production infrastructure for Structured Streaming workloads. Structured Streaming is still the right tool when a requirement falls on the procedural side. For example, procedural processing is recommended for low-level performance optimizations in batch and streaming workflows, and for existing imperative scripts.
Checkpoint 4 of 6· Exam question
A team ingests a continuous stream of IoT sensor events and must track custom per-device state across micro-batches using `mapGroupsWithState`, including manual timeout handling and arbitrary logic that only emits an alert when a hand-rolled state machine reaches a specific transition. They are deciding whether to implement this inside a Lakeflow Declarative Pipeline flow or as a standalone Structured Streaming job. Which approach should they choose, and why?
Correct answer: A — Run it as a standalone Structured Streaming job, because Lakeflow Declarative Pipeline flows only support declarative dataset definitions and cannot embed custom `mapGroupsWithState` state-machine logic.
- A. A standalone Structured Streaming job is correct because Lakeflow Declarative Pipeline flow functions must return a DataFrame from a declarative dataset definition, so they cannot host arbitrary stateful APIs like `mapGroupsWithState` with custom timeout logic. Plain Structured Streaming gives full imperative access to that API.
- B. Pipeline flow functions are restricted to declarative dataset definitions and cannot embed arbitrary stateful operators such as `mapGroupsWithState`; there is no equivalent manual-timeout API exposed through `@dp.table`, so this option misrepresents what flow functions can do.
- C. AUTO CDC is built for applying inserts, updates, and deletes from a change feed into a target table with `stored_as_scd_type`, not for running an arbitrary custom state machine keyed by device, so it does not replace the required `mapGroupsWithState` logic.
- D. Recomputing the state machine nightly with a materialized view abandons the requirement to emit alerts as transitions happen; a materialized view recomputes against current source state rather than replaying the streaming custom-state logic per event.
Checkpoint 5 of 6· Check yourself
For a new streaming ETL pipeline with no unusual requirements, what does Databricks recommend?
Databricks explicitly recommends Lakeflow pipelines for new Structured Streaming pipelines. Structured Streaming remains the choice when you need procedural control.
“Databricks recommends using Lakeflow pipelines for new Structured Streaming pipelines.”Source: docs.databricks.com
Checkpoint 6 of 6· Exam question
A team ingests clickstream events that arrive continuously through Auto Loader and must be queryable downstream with sub-minute freshness. Each event should be processed incrementally and exactly once, and there is no need to revisit or recompute records that already landed. Which Lakeflow Declarative Pipeline dataset type should they define for this ingestion layer, and why?
Correct answer: A — Define it as a streaming table, because streaming tables process each source record incrementally and exactly once, matching an append-only, continuously arriving clickstream with sub-minute freshness needs.
- A. A streaming table is correct because it is designed for append-only sources and processes each newly arrived record exactly once through incremental computation, which matches a continuously arriving clickstream that needs sub-minute freshness without reprocessing history.
- B. Materialized views recompute their result as needed to reflect the current state of their sources rather than guaranteeing exactly-once, record-by-record incremental processing, so equating them with a streaming table's freshness model misstates how they behave.
- C. A temporary view is scoped to the pipeline update and is not a persisted, queryable table that downstream consumers outside the pipeline can read, so it does not satisfy the requirement for a queryable ingestion layer.
- D. AUTO CDC targets are built to apply inserts, updates, and deletes from a change feed using `stored_as_scd_type`, not to ingest a plain append-only event stream, so it is the wrong tool for this clickstream ingestion layer.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.AUTO CDC, data quality expectations and continuous mode are part of open-source Apache Spark Declarative Pipelines.Why is that wrong?
These are production features that Lakeflow pipelines add on top of SDP. The capability table marks them as not available in SDP.
Covered in Open-source SDP vs Lakeflow pipelines
2.Declarative pipelines are only for SQL batch jobs, so any streaming requirement means writing Structured Streaming.Why is that wrong?
SDP develops and runs both batch and streaming pipelines, in SQL and Python, including stateful streaming transformations.
Covered in Two ways to express the same ETL
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“With procedural programming you specify how tasks should be accomplished by defining explicit sequences of operations.”
↩︎ Two ways to express the same ETL“pipelines automate key aspects of processing management, including orchestration, compute management, monitoring, data quality enforcement, and error handling.”
↩︎ What a declarative pipeline takes off your hands“Transformations involve complex business rules that are difficult to express declaratively.”
↩︎ Making the call: the documented decision points“Declarative data processing abstracts the how and focuses on defining the desired result.”
↩︎ Key concept“Apache Spark primarily follows a procedural model for data processing.”
↩︎ Prediction“Declarative programming focuses on what needs to be achieved, leaving the underlying system to determine the best way to execute the task.”
↩︎ Checkpoint - 2.
“Incremental batch and streaming transformations with stateless and stateful operators.”
↩︎ Two ways to express the same ETL“SDP automates the orchestration and organizes the dependencies between the flows in your pipeline.”
↩︎ What a declarative pipeline takes off your hands“Because pipelines build on SDP rather than proprietary APIs, the transformation code you write stays portable to other SDP runtimes.”
↩︎ Open-source SDP vs Lakeflow pipelines“add production features such as AUTO CDC, data quality expectations, and a queryable event log”
↩︎ Exam trap 1“a declarative framework for developing and running batch and streaming data pipelines in SQL and Python”
↩︎ Exam trap 2“add production features such as AUTO CDC, data quality expectations, and a queryable event log”
↩︎ Checkpoint - 3.https://docs.databricks.com/aws/en/ldp/conceptsOfficial docs
“Declarative functions reduce hundreds of lines of manual Spark and Structured Streaming code to a few.”
↩︎ What a declarative pipeline takes off your hands“without manual code for out-of-order events or streaming concepts like watermarks”
↩︎ Checkpoint - 4.
“Checkpointing and retries are automatic, so a pipeline resumes from the last processed offset rather than reprocessing everything.”
↩︎ What a declarative pipeline takes off your hands - 5.
“Databricks introduced Lakeflow pipelines to reduce the complexities of managing production infrastructure for Structured Streaming workloads.”
↩︎ Making the call: the documented decision points“Databricks recommends using Lakeflow pipelines for new Structured Streaming pipelines.”
↩︎ Checkpoint