What you will be able to do
- Compare default refresh, full refresh and checkpoint reset for streaming tables and materialized views
- Explain when a materialized view refreshes incrementally and when it fully recomputes
- Identify sources that must feed a streaming table rather than a materialized view
- Choose triggered or continuous pipeline mode independently of dataset type to meet a freshness and cost target
1.What a refresh does to each object
Every pipeline update refreshes the materialized views and streaming tables the pipeline defines. The same command does different work depending on the object type, and the cost of that work is what drives most choices between the two.
| Update type | Materialized view | Streaming table |
|---|---|---|
| Refresh (default) | Reflects current query results; incremental when Databricks judges it more efficient | Processes only records that arrived since the last update and appends them |
| Full refresh | Recomputes the entire result | Truncates the table, clears flow checkpoints, reprocesses every source record |
| Reset streaming flow checkpoints | Not applicable | Clears checkpoints for selected flows, keeps existing data, reprocesses the source |
A default refresh of a streaming table is cheap because it only touches new records. The cost is correctness: changes that would affect data already written are never applied. A materialized view never trades correctness away. A default refresh and a full refresh produce the same data, and the only difference is how much compute each uses. A full refresh of a streaming table carries an extra risk: if the source no longer holds the original data, as with a Kafka topic past its retention window, records can be lost.
Checkpoint 1 of 5· Match them up
Match each operation to its effect
Tap a term, then the definition that fits it.
Only a streaming table refresh leaves already-written rows untouched. Checkpoint reset applies only to streaming tables and keeps existing data.
“a default refresh of a streaming table trades data correctness for lower time and resource costs.”Source: docs.databricks.com
Checkpoint 2 of 5· Fill the gap
Which keyword forces a materialized view defined in Databricks SQL to clear its table and checkpoints and reprocess all source data?
REFRESH MATERIALIZED VIEW mv_name ? REFRESH MATERIALIZED VIEW ... FULL runs a full refresh. Without the keyword, Databricks decides between incremental and full refresh based on cost.
Source: docs.databricks.comSources1
2.When a materialized view is cheap and when it is not
How much a materialized view costs depends on whether its updates are incremental. On serverless, a refresh tries to process only changed data, and Databricks runs a cost analysis to pick the cheaper of an incremental refresh and a full recompute. Both produce the same output. Incremental refresh only works when the query is supported: unsupported expressions force a full recompute, and some changes to inputs always require one. Sources that support incremental refresh include Delta tables, other materialized views, streaming tables, and Unity Catalog managed Iceberg tables. Standalone materialized views always refresh on a serverless pipeline. Materialized views defined inside a pipeline only get incremental refresh if that pipeline is configured to use serverless. The aggregate below is an example of a query that can refresh incrementally.
CREATE OR REPLACE MATERIALIZED VIEW transaction_summary AS
SELECT account_id,
COUNT(txn_id) txn_count,
SUM(txn_amount) account_revenue
FROM transactions_table
GROUP BY account_idFor the best incremental performance, Databricks recommends enabling deletion vectors, row tracking, and change data feed on every source table. You can check in advance whether a query can be refreshed incrementally with EXPLAIN CREATE MATERIALIZED VIEW. If an unexpected full refresh would be a problem, you can set a refresh policy that fails instead of running a full refresh.
ALTER TABLE <table-name> SET TBLPROPERTIES (
delta.enableDeletionVectors = true,
delta.enableRowTracking = true,
delta.enableChangeDataFeed = true);Checkpoint 3 of 5· Exam question
A `daily_region_revenue` dataset aggregates `SUM(order_amount)` by region and joins against a `customers` dimension table that receives updates and deletes throughout the day. Business analysts query this dataset directly for dashboards and expect the aggregate to reflect the latest state of `customers`, but do not need sub-second freshness. Which approach correctly defines this dataset in Lakeflow Spark Declarative Pipelines? ```sql CREATE OR REFRESH ??? daily_region_revenue AS SELECT c.region, SUM(o.order_amount) AS total_revenue FROM orders o JOIN customers c ON o.customer_id = c.customer_id GROUP BY c.region; ```
Correct answer: B — MATERIALIZED VIEW, because it recomputes the aggregation as needed against the current state of both `orders` and `customers`, correctly reflecting the updates and deletes.
- A. Streaming tables assume an append-only source and process each record exactly once; they are not designed to automatically re-derive prior aggregate rows when an upstream dimension table changes, so updates to `customers` would not be reflected correctly.
- B. This is correct because results are recomputed as needed to reflect the current state of source data, which is exactly what is required when a join input like `customers` receives updates and deletes that must be reflected in the aggregate.
- C. Schema evolution controls how a pipeline reacts to changes in the structure of incoming data, such as new columns, not how it reacts to row-level updates or deletes in a joined table, so this option describes the wrong mechanism entirely.
- D. Watermarks bound how long Structured Streaming waits for late-arriving event-time data in stateful aggregations; they do not enable a streaming table to absorb update and delete operations from a changing dimension table.
3.Sources that rule out a materialized view
Any source behind a materialized view has to tolerate full-refresh semantics, even when the query supports incremental refresh, because Databricks may choose a full recompute at any time. That requirement settles several decisions outright. If a full refresh would cost too much, for example on a very large table, use a streaming table to get exactly-once processing. If records should only be processed once, a materialized view is the wrong object.
The docs name three kinds of source that belong in a streaming table: sources that do not keep history, such as Kafka; ingestion through Auto Loader from cloud object storage; and any source whose data you plan to delete or archive after processing while keeping the results downstream, such as a date-partitioned table with a retention threshold. In each case a full recompute would either re-read a large amount of data or lose records that no longer exist.
Checkpoint 4 of 5· Check yourself
Which source can safely feed a materialized view?
Delta tables support incremental refresh and keep their history, so a full refresh is safe. The other three are the docs' examples of sources that should feed a streaming table.
“Do not define a materialized view against a data source if records should only be processed once. Instead, use streaming tables.”Source: docs.databricks.com
Sources2
4.Pipeline mode sets freshness, not table type
Choosing streaming table or materialized view does not decide how often the data is refreshed. That is set by the pipeline mode, and both object types work in either mode. In triggered mode, an update processes the data available when it started and then stops, which suits refreshes every 10 minutes, hourly, or daily. Because compute only runs during the update, it is cheaper. Continuous mode processes data as it arrives and suits refresh intervals from about 10 seconds to a few minutes, at the cost of an always-running cluster. For new pipelines, Databricks recommends running continuously through a continuous job rather than through the pipeline's own continuous setting. Standalone materialized views and streaming tables always refresh in triggered mode.
| Question | Triggered | Continuous |
|---|---|---|
| When does the update stop? | Automatically when complete | Runs until manually stopped |
| What data is processed? | Data available when the update starts | All data as it arrives |
| Best-fit freshness | Every 10 minutes, hourly, or daily | Every 10 seconds to a few minutes |
| Cost profile | Compute runs only for the update | Always-running cluster, more expensive |
Checkpoint 5 of 5· Exam question
Which of the following are valid, technically accurate reasons for a data engineer to choose a materialized view instead of a streaming table for a given dataset in a Lakeflow Spark Declarative Pipelines pipeline? (Select 3)(Select 3)
Correct answers: A, B, E — Several downstream teams query the same aggregated dataset repeatedly, so precomputing and caching the result once avoids every consumer recomputing the same joins and aggregations.; The upstream source table receives in-place updates and deletes, and the dataset must reflect the current state of that source rather than only newly appended rows.; The dataset definition performs a multi-way join and a `GROUP BY` aggregation whose correctness depends on seeing the full current state of every input table.
- A. This is a valid reason: materialized views cache a precomputed result, so when many consumers repeatedly query the same aggregation, computing it once and serving it from storage is cheaper than every consumer re-deriving it.
- B. This is a valid reason: results are recomputed against the current state of the data, so an option like this correctly reflects in-place updates and deletes on the source in a way an append-only streaming table cannot.
- C. This is not a defining reason; exactly-once semantics is a property of streaming tables processing append-only data, and wanting to inspect results during development is not the technical driver behind the latency, cost, or refresh tradeoff.
- D. This describes the opposite scenario: a continuously growing append-only stream needing the lowest per-record latency is the textbook case for a streaming table, not this alternative.
- E. This is a valid reason: joins and aggregations whose correctness depends on the complete, current state of every input table are exactly the workloads this alternative is designed to recompute correctly.
- F. Schema inference timing is a property of Auto Loader configuration on the ingestion source, not a reason tied to choosing between the two table types for latency, cost, or refresh requirements.
Sources4
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.A materialized view whose query supports incremental refresh is refreshed incrementally on any compute.Why is that wrong?
Only materialized views updated by serverless pipelines can refresh incrementally. On classic compute they are always fully recomputed.
Covered in When a materialized view is cheap and when it is not
2.Materialized views only run in triggered mode, so near-real-time freshness requires converting them to streaming tables.Why is that wrong?
Pipeline mode does not depend on the dataset type. Materialized views and streaming tables in a Lakeflow pipeline can both run triggered or continuous.
Covered in Pipeline mode sets freshness, not table type
3.Kafka is a good source for a materialized view because Databricks will pick incremental refresh to keep costs down.Why is that wrong?
A materialized view may fully recompute at any time, and Kafka does not keep its history. Sources like this should feed a streaming table.
Covered in Sources that rule out a materialized view
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Because materialized views always return the same result as a batch query, a default refresh and a full refresh produce identical data.”
↩︎ What a refresh does to each object“A full refresh of a streaming table can drop records if the source no longer retains the original data”
↩︎ What a refresh does to each object“a default refresh of a streaming table trades data correctness for lower time and resource costs.”
↩︎ Checkpoint“Incremental refresh is available only on serverless pipelines. On classic compute, a materialized view is always fully recomputed.”
↩︎ Prediction - 2.
“Databricks runs a cost analysis to choose the cheaper option between an incremental refresh and a full recompute.”
↩︎ When a materialized view is cheap and when it is not“You can define a refresh policy that fails rather than run a full refresh.”
↩︎ When a materialized view is cheap and when it is not“For queries where a full refresh would be cost-prohibitive, use streaming tables to guarantee exactly-once processing.”
↩︎ Sources that rule out a materialized view“Only materialized views updated using serverless pipelines can use incremental refresh.”
↩︎ Exam trap 1“Data sources that don't retain data history, such as Kafka.”
↩︎ Exam trap 3“Do not define a materialized view against a data source if records should only be processed once. Instead, use streaming tables.”
↩︎ Checkpoint - 3.
“some changes to inputs will require a full recomputation of a materialized view, which can be expensive.”
↩︎ When a materialized view is cheap and when it is not - 4.
“Continuous pipelines require an always-running cluster, which is more expensive but reduces processing latency.”
↩︎ Pipeline mode sets freshness, not table type“Databricks recommends running continuous pipelines with a continuous job rather than setting the value of Pipeline mode to continuous.”
↩︎ Pipeline mode sets freshness, not table type“Refresh operations for standalone materialized views and streaming tables always run using triggered pipeline mode.”
↩︎ Pipeline mode sets freshness, not table type“Pipeline mode is independent of the type of table being computed.”
↩︎ Exam trap 2