What you will be able to do
- Explain how a streaming table processes each input row once while a materialized view returns batch-equivalent results
- Predict how each object behaves when its query changes, when data arrives late, or when a joined dimension changes
- Choose between a streaming table and a materialized view given a latency requirement
Key concept
Process-once vs. always-correct — A streaming table remembers what it has already read and only processes new input, so it is cheap and fast but never goes back to rows it has already written. A materialized view always matches what its query would return if you ran it as a batch right now, and it pays whatever recomputation that takes.
1.Two objects, two processing semantics
Lakeflow pipelines give you two persisted dataset types, and choosing between them means choosing between two processing semantics. A streaming table is a Delta table with added support for streaming or incremental processing. One or more flows read from a streaming source and append to it. The engine records which data it has already consumed, so the next update only reads what is new. This is why streaming tables are the default for ingestion: each input row is handled only once, which matches how most ingestion works (appending or upserting rows).
A materialized view works differently. Like a regular view, it is defined by a query. Unlike a regular view, it stores the result instead of recomputing it every time someone reads it. You write a query that reasons over the whole result set, not over individual rows, and Databricks keeps the stored output in sync with the sources. The example below joins two tables and keeps the join current. It contains no streaming code, no checkpoints, and no merge logic.
from pyspark import pipelines as dp
@dp.materialized_view
def regional_sales():
partners_df = spark.read.table("partners")
sales_df = spark.read.table("sales")
return (
partners_df.join(sales_df, on="partner_id", how="inner")
)Pipeline best-practice guidance turns this into a rule of thumb for each layer. Use streaming tables for ingestion and low-latency streaming transformations. Use materialized views for complex transformations and analytical queries. The rest of this page explains why that split holds.
Checkpoint 1 of 4· Check yourself
A team ingests JSON files that land in cloud storage throughout the day. Each file should be read exactly once and appended to a bronze table. Which object fits?
Append-only ingestion where each record should be read once is the case streaming tables are built for. A materialized view would treat the source with batch semantics instead.
“Each input row is handled only once, which models the vast majority of ingestion workloads”Source: docs.databricks.com
2.What "process once" costs you in correctness
Because a streaming table never revisits a row, changing its query gives you what the docs call limited evolution. After a change, the table holds rows produced by different versions of the query, and you need to know which versions have run against it. The only way to apply the new logic to older rows is a full refresh, which re-reads everything from the source.
Joins have the same weakness. In a stream-static join, a streaming table enriches each row with the dimension values that exist at the moment the row is processed. If the dimension changes later, rows already written keep the old values. The docs call this acceptable for "fast-but-wrong" scenarios. A materialized view recomputes the join when the dimension changes, so its output stays correct.
Late and out-of-order data is the third difference. In batch processing, a late record from hour one is processed together with the rest of hour one, and that hour's result is overwritten with the corrected value. In streaming processing, the late record arrives after the rest of hour one was already processed, so any stateful logic (aggregations, joins, deduplication) has to keep state to fix the earlier result. Materialized views use batch semantics and avoid this problem: they are always correct at the time of each update, even when data arrives late or out of order.
| Situation | Streaming table | Materialized view |
|---|---|---|
| Query logic changes | Only new rows get the new logic; a full refresh is needed to reprocess old rows | Results reflect the current query |
| Joined dimension changes | Joins do not recompute | Joins recompute automatically |
| Late or out-of-order data | Stateful logic must handle it, bounded by a watermark | All required data is processed |
Checkpoint 2 of 4· Check yourself
A gold table joins a stream of orders with a customer dimension. Business users require the customer region shown on every historical order to match the current dimension. Which object meets that requirement?
Streaming table joins keep the dimension values that existed when each row was processed. A materialized view recomputes the join when the dimension changes.
“Materialized views are always correct because they automatically recompute joins when dimensions change.”Source: docs.databricks.com
Checkpoint 3 of 4· Exam question
A Lakeflow Spark Declarative Pipelines pipeline uses Auto Loader (`cloudFiles`) to land append-only clickstream JSON files. Downstream consumers need every event processed exactly once, with the lowest possible latency and compute cost per micro-batch. Which table type should the engineer define for this dataset? ```python from pyspark import pipelines as dp @dp.table def clicks_raw(): return (spark.readStream .format("cloudFiles") .option("cloudFiles.format", "json") .load("/Volumes/raw/clickstream/")) ```
Correct answer: B — A streaming table, because it processes each newly arrived record exactly once through incremental append-mode computation, matching the append-only source at low latency and cost.
- A. Materialized views recompute results as needed against the current state of the source, which is useful for changing or joined data but wastes compute reprocessing an append-only stream that never needs a full recalculation.
- B. A streaming table is the correct choice here because it maintains a streaming flow over an append-only source, so Lakeflow processes only the newly arrived clickstream files each micro-batch instead of recomputing history, which minimizes both latency and compute cost.
- C. Checkpoints are what let a streaming table track which records it already processed exactly once; disabling them would break incremental processing and force reprocessing, which is the opposite of the low-cost goal.
- D. Materialized views do not use continuous refresh to force full historical joins for ordering; continuous mode only controls how often the pipeline checks for new data, and this description misrepresents how materialized views compute results.
3.Latency: where each object fits
Materialized views are not built for low-latency use cases. Even when an update is incremental, it takes seconds or minutes, not milliseconds. That is fine for reporting and analytical aggregates, but it rules out operational workloads where a few seconds is too slow.
Streaming tables are designed for low-latency streaming. They use checkpoint management to track progress, and they can reason over rows and windows of time at high volume. When a workload needs the lowest latency possible, the pipeline can run in real-time mode, which processes records with sub-second end-to-end latency.
Streaming tables do have a precondition. They expect streams that are bounded, either naturally (for example, a finite set of files that eventually stops growing) or by a watermark, which sets how long to wait for late events before a time window is treated as complete. If the stream is unbounded and has no watermark, state grows without limit and can make the pipeline fail.
Only the dashboard. Materialized view updates take seconds to minutes. Sub-second delivery calls for a streaming table, with the pipeline running in real-time mode for the lowest latency.
Checkpoint 4 of 4· Check yourself
A stateful streaming-table aggregation reads an unbounded event stream with no watermark. What risk do the docs call out?
Streaming tables need streams that are naturally bounded or bounded by a watermark. Without one, state keeps growing until memory runs out.
“An unbounded stream that does not have a watermark can cause a pipeline to fail due to memory pressure.”Source: docs.databricks.com
Sources4
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.A materialized view refreshed by a continuous pipeline can serve millisecond-latency operational reads.Why is that wrong?
Materialized view updates take seconds or minutes. Sub-second needs call for streaming tables, with the pipeline in real-time mode for the lowest latency.
Covered in Latency: where each object fits
2.Updating a dimension table makes a streaming table's earlier join results reflect the new values.Why is that wrong?
Streaming table joins are evaluated once per row and are not recomputed. Only a materialized view recomputes joins when dimensions change.
Covered in What "process once" costs you in correctness
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Unlike standard views, which recompute results on every query, materialized views cache the results and refresh them on a specified interval.”
↩︎ Two objects, two processing semantics“They are always correct at the time of their update. All required data is processed, even if it arrives late or out of order.”
↩︎ What "process once" costs you in correctness“They are not designed for low-latency use cases. The latency of updating a materialized view is in the seconds or minutes, not milliseconds.”
↩︎ Exam trap 1 - 2.
“Streaming tables are the right choice for data ingestion and low-latency streaming transformations.”
↩︎ Two objects, two processing semantics - 3.
“With streaming processing, the engine keeps track of what data is being processed and only processes new data in subsequent runs.”
↩︎ Two objects, two processing semantics“These streaming complexities are typically introduced when the processing is stateful, such as joins, aggregations, and deduplications.”
↩︎ What "process once" costs you in correctness - 4.
“To reprocess existing rows that were processed prior to the change, a full refresh is required.”
↩︎ What "process once" costs you in correctness“For operational workloads that need the lowest possible latency, you can run the pipeline in real-time mode to process records with sub-second, end-to-end latency.”
↩︎ Latency: where each object fits“Streaming tables are designed for low-latency streaming over bounded state.”
↩︎ Latency: where each object fits“Joins in streaming tables do not recompute when dimensions change.”
↩︎ Exam trap 2“Each input row is handled only once, which models the vast majority of ingestion workloads”
↩︎ Checkpoint“A row that has already been appended to a streaming table will not be re-queried with later updates to the pipeline.”
↩︎ Prediction“Materialized views are always correct because they automatically recompute joins when dimensions change.”
↩︎ Checkpoint“An unbounded stream that does not have a watermark can cause a pipeline to fail due to memory pressure.”
↩︎ Checkpoint
Also cited
“Materialized views use batch semantics and streaming tables use streaming semantics.”
↩︎ Key concept