What you will be able to do
- Tell batch and streaming processing apart by whether the engine tracks what it has already processed
- Pick batch or streaming for a workload using its latency, accuracy and statefulness, including the medallion-layer recommendations
- Match triggered and continuous pipeline modes to batch and streaming ingestion
- Rank Structured Streaming, Lakeflow pipelines and managed connectors from most customizable to most managed, and say which layer to start with
- Describe the types of Lakeflow Connect managed connector, their components, and how they ingest incrementally
Key concept
Incremental processing — The engine keeps track of which source data it has already handled, so each run picks up only what is new. Batch processing doesn't track this and reprocesses everything currently in the source. Streaming processing, Auto Loader and Lakeflow Connect managed connectors all work incrementally in this way.
1.Two semantics: does the engine remember what it has seen?
Databricks names two processing semantics for ingestion and transformation: batch and streaming. People often read 'streaming' as low-latency messages from Kafka, but Databricks uses a wider definition. Apache Spark and Structured Streaming handle batch and streaming in one architecture, so cloud object storage and Delta Lake can also act as streaming sources for incremental processing.
What actually separates the two is state. In batch, the engine doesn't record what it has already processed. Each run reads all the data currently in the source, so batch sources are usually partitioned, for example by day or region, to limit how much gets reprocessed. In streaming, the engine records its progress and each run reads only the data added since the last run.
Take an hourly average sales price. Run as a batch, each hour reprocesses earlier hours and overwrites the previous results. Run as a stream, each hour processes only new rows, and those results are appended to the earlier ones.
This complexity shows up when processing is stateful, as with joins, aggregations and deduplications. Stateless streaming, such as appending new source rows to a table, has little trouble with late or out-of-order data: late rows are simply appended when they arrive. That is why most raw ingestion fits streaming well.
Checkpoint 1 of 8· Check yourself
Why is late-arriving or out-of-order data less of a problem for stateless streaming, such as appending new source rows?
Appending needs no earlier state to correct, so a late row is just added when it arrives. The complexity belongs to stateful work such as joins, aggregations and deduplications.
“For stateless streaming processing, such as appending new data from the source, handling out-of-order and late arrival data is less complex”Source: docs.databricks.com
Sources1
2.Trade-offs, and where each semantic fits
Each semantic has a cost. Batch logic is simple and its results always reflect all the data in the source, but it repeats work and suits latencies of hours to minutes. Streaming processes only new data and can reach seconds or milliseconds, but stateful logic gets harder and results can be wrong when data arrives late or out of order. The products on each side matter on the exam, because Lakeflow Connect sits on the streaming side.
| Semantic | Pros | Cons | Products |
|---|---|---|---|
| Batch | Simple logic; results always accurate and reflect all available source data | Reprocesses data in a batch partition; latency of hours to minutes, not seconds | Lakeflow pipelines materialized views; spark.read.load() and spark.write.save() |
| Streaming | Only new data is processed; latency from hours down to seconds and milliseconds | Complex stateful logic (joins, aggregations, deduplications); late or out-of-order data affects accuracy | Lakeflow Connect; Lakeflow pipelines streaming tables and append flows; spark.readStream.load() and spark.writeStream.start() |
Databricks applies this to the medallion architecture. Bronze holds ingestion work: usually large volumes with no processing or only stateless appends, which is where streaming works best. Silver and Gold contain joins and aggregations, so the recommendation there is batch processing with incremental refresh in materialized views. Streaming is still an option in Silver when efficiency and latency matter more than accuracy.
Checkpoint 2 of 8· Check yourself
A team is building Bronze-layer ingestion that appends raw records from its sources with no joins or aggregations. Which processing semantic does Databricks generally recommend?
Bronze ingestion is mostly stateless appends, so it gets the efficiency of streaming without the complications of stateful streaming. Batch with materialized-view refresh is the recommendation for Silver and Gold.
“Streaming processing is generally a better choice, given users can benefit from the advantages of streaming”Source: docs.databricks.com
Sources1
3.Triggered and continuous pipeline modes
Streaming semantics doesn't mean the pipeline runs nonstop. Streaming can run in either triggered or continuous mode, and you choose based on cost and latency. Ingestion pipelines can run on a recurring schedule or continuously. Triggered mode handles new data on a schedule or when someone starts a run, which is the batch-ingestion use case. Continuous mode handles new data as it arrives, which is the streaming-ingestion use case. A triggered pipeline built on a streaming engine still reads only new data on each run, so you get incremental loading without paying for compute that is always on.
By pipeline mode, it is triggered, the mode used for batch ingestion because it runs on a schedule. By semantics, it is still streaming: the engine tracks what it has processed, so each run handles only new files. The two settings are independent, and that is how Databricks lets you trade cost against performance.
Checkpoint 3 of 8· Check yourself
Which pipeline mode does Databricks pair with batch ingestion?
Triggered mode is the batch-ingestion use case and continuous mode is the streaming-ingestion use case. Pipeline mode is separate from whether the engine tracks processed data.
“Triggered: Processes new data on a schedule or when manually triggered.”Source: docs.databricks.com
4.Three layers of ingestion products
Lakeflow Connect arranges ingestion tools as a stack, from most customizable to most managed. At the bottom is Apache Spark Structured Streaming, which gives end-to-end fault tolerance and exactly-once guarantees through Spark APIs. Lakeflow pipelines sit on top of Structured Streaming and add a declarative framework: you define the transformations, and the pipeline takes care of orchestration, monitoring, data quality and errors. Managed connectors sit on top of Lakeflow pipelines and add source-specific authentication, CDC, edge-case handling, long-term API maintenance, automatic retries and automatic schema evolution for popular sources.
The recommended approach is to start at the most managed layer and drop down a layer only when it doesn't meet your needs, for example when it doesn't support your source. Standard connectors use the lower two layers: Auto Loader and Kafka, Kinesis, Pub/Sub and Pulsar sources, run in Structured Streaming or Lakeflow pipelines.
Standard connectors differ in how many levels of customization they offer. Cloud object storage uses Auto Loader at every level: with Structured Streaming for more customization, with Lakeflow pipelines for some customization, or with Databricks SQL for more automation. Kafka, Kinesis, Pub/Sub and Pulsar each offer the same three levels, using their own source in Structured Streaming, in Lakeflow pipelines, or in Databricks SQL. SFTP servers are the exception: the table lists a single option, ingesting files from SFTP servers with Python or SQL, and N/A at the other levels. For SQL users loading from cloud object storage, CREATE STREAMING TABLE is the recommended alternative to COPY INTO.
Checkpoint 4 of 8· Put it in order
Order the three ingestion layers from most customizable to most managed.
- 1.Managed connectors
- 2.Lakeflow pipelines
- 3.Structured Streaming
Lakeflow pipelines extend Structured Streaming, and managed connectors build on Lakeflow pipelines. Each layer adds automation and gives up some customization.
“The following table describes the three layers of ingestion products, ordered from most customizable to most managed:”Source: docs.databricks.com
Sources2
5.Managed connectors: types, components, incremental runs
Managed connectors create ingestion pipelines that are governed by Unity Catalog and run on serverless compute and Lakeflow pipelines. They are grouped by source type. Database connectors use change data capture on MySQL, PostgreSQL and SQL Server. SaaS connectors cover Salesforce, HubSpot, Jira and Workday. File source connectors read from Google Drive and SharePoint. Query-based connectors query a database directly without setting up CDC. Streaming connectors read from message buses such as RabbitMQ. Community connectors ingest data using open-source, community-built connectors, and custom connectors fill gaps for sources without a built-in option.
Checkpoint 5 of 8· Match them up
Match each managed connector type to a source it handles.
Tap a term, then the definition that fits it.
Lakeflow Connect groups managed connectors by source type. SharePoint is a file source, not a SaaS connector, and SQL Server uses the CDC database connector.
“Ingest unstructured and structured files from enterprise file storage services including Google Drive and SharePoint.”Source: docs.databricks.com
SaaS and file connectors have three parts: a connection, which is a Unity Catalog securable object holding authentication details; an ingestion pipeline; and destination streaming tables. Database connectors also have an ingestion gateway and staging storage for continuous change capture, and the gateway runs in its own job as a continuous task. Each schedule you add to a pipeline gets a job created for it automatically, with the pipeline as a task in that job.
Query-based and streaming connectors have their own, smaller set of components. A query-based connector queries the source database directly on a schedule, without a gateway or staging storage. Its connection is either a direct Unity Catalog connection (foreign connection ingestion) or a Unity Catalog foreign catalog (foreign catalog ingestion using Lakehouse Federation). Its ingestion pipeline queries the source and writes the results to destination streaming tables, and runs on serverless compute by default. A streaming connector continuously reads messages from a message bus or event streaming source and writes them to streaming tables. Its connection stores the source endpoint and credentials, so no credentials sit in the pipeline configuration, and its ingestion pipeline runs continuously on serverless compute.
Ingestion is incremental. The first run loads all the selected data while change tracking starts in parallel. Later runs use that tracking to load only what changed, where the source allows it. How it works depends on the source: SQL Server supports change tracking and CDC, and Salesforce uses a cursor column. Some sources and tables don't support incremental ingestion yet.
Checkpoint 6 of 8· Check yourself
A new Lakeflow Connect pipeline for SQL Server finishes its first run. What do later runs ingest?
The first run loads everything selected and starts change tracking at the same time. Later runs use that tracking to load only changes, wherever the source supports it.
“it uses that change tracking to ingest only the data that's changed from the prior run”Source: docs.databricks.com
Sources3
6.Auto Loader: schema location, inference and evolution
Auto Loader is the standard connector for cloud object storage. It provides a Structured Streaming source called cloudFiles, which processes new files as they arrive. When it infers a schema, it needs somewhere to keep it. The option cloudFiles.schemaLocation is that place: Auto Loader stores the schema information in a _schemas directory there and tracks schema changes to the input data over time. Without it the schema can't be inferred, because the option is required to infer the schema. To infer the schema, Auto Loader samples the first 50 GB or 1000 files it discovers, whichever limit is crossed first.
By default, schema inference avoids type-mismatch problems. For JSON, CSV and XML, which don't encode data types, every column is inferred as a string, including nested JSON fields. Parquet and Avro keep the types encoded in their own schema. To make Auto Loader pick column types from sample data instead, as the Apache Spark DataFrameReader does, set cloudFiles.inferColumnTypes to true.
Schema evolution is controlled by cloudFiles.schemaEvolutionMode. In the default mode, addNewColumns, Auto Loader detects a new column, adds it to the schema location, and the stream then fails with an UnknownFieldException. Restarting the stream resumes with the updated schema, so Databricks recommends configuring Lakeflow Jobs to restart it automatically. The other modes behave differently: rescue never evolves the schema and records new columns in the rescued data column; none ignores new columns; failOnNewColumns fails and doesn't update the schema; addNewColumnsWithTypeWidening also widens supported types such as int to long. addNewColumns is the default when no schema is provided, but none is the default when you provide one.
Checkpoint 7 of 8· Exam question
A data engineer sets up an Auto Loader stream to ingest JSON files from a Unity Catalog volume into a Delta table, but the target table's schema is not explicitly specified in the code. To ensure Auto Loader can infer and evolve the schema automatically as new files arrive, which configuration is required?
Correct answer: A — Set the `cloudFiles.schemaLocation` option to a directory path where Auto Loader stores the inferred schema and tracks changes over time.
- A. Setting `cloudFiles.schemaLocation` is correct because Auto Loader requires this directory to store and track the inferred schema across microbatches, and without it schema inference and evolution cannot be enabled at all.
- B. This setting controls whether inferred columns keep their string type or are cast to inferred types, but it does not itself enable schema inference or evolution tracking, so on its own it does not satisfy this requirement.
- C. Specifying the file format tells Auto Loader how to parse each file's structure but has no bearing on where the evolving schema is persisted, so it does not enable schema evolution tracking.
- D. Tuning the batch size controls how many files are processed per trigger for throughput, but the number of files scanned in a trigger does not determine whether schema inference and evolution are tracked over time.
Checkpoint 8 of 8· Exam question
An Auto Loader stream ingesting CSV files into a Delta table has no explicit target schema defined and uses the default schema evolution mode. After several weeks, upstream source files begin including two new columns that were not present when the stream started. What happens to the running stream?
Correct answer: A — The stream stops with an `UnknownFieldException`, and the engineer must restart the stream so Auto Loader can pick up the new schema and continue processing.
- A. Halting with an `UnknownFieldException` is the default behavior when new columns appear and no schema evolution mode was chosen explicitly, requiring a stream restart so the updated schema is picked up before processing resumes.
- B. Silently rescuing new columns without interruption describes rescue-mode behavior, not the default mode, so a stream running under the default configuration would not behave this way.
- C. Automatically appending new columns without any restart describes a different, explicitly configured evolution mode, since the default behavior is to halt and require manual intervention rather than evolve automatically.
- D. Permanently failing until the table is dropped and recreated is far more destructive than the actual default behavior, which only requires a stream restart, not recreating the underlying table.
Sources4
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Streaming in Databricks only applies to message buses such as Apache Kafka.Why is that wrong?
Databricks uses a wider definition. Cloud object storage and Delta Lake can be streaming sources for incremental processing, even when the pipeline runs on a schedule.
Covered in Two semantics: does the engine remember what it has seen?
2.Managed connectors are the customizable option and standard connectors are the hands-off option.Why is that wrong?
It is the other way round. Managed connectors give the most automation, and standard connectors give up some automation in exchange for broader source support and more customization.
Covered in Three layers of ingestion products
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“With batch processing, the engine does not keep track of what data is already being processed in the source.”
↩︎ Two semantics: does the engine remember what it has seen?“Use batch processing (with incremental refresh in materialized views).”
↩︎ Trade-offs, and where each semantic fits“Streaming processing can be run in both triggered and continuous manners, giving you the flexibility to control cost and performance tradeoffs”
↩︎ Triggered and continuous pipeline modes“With streaming processing, the engine keeps track of what data is being processed and only processes new data in subsequent runs.”
↩︎ Key concept“The engine can treat sources like cloud object storage and Delta Lake as streaming sources for efficient incremental processing.”
↩︎ Exam trap 1“In streaming processing, the late-arriving data from the first hour is processed without any of the other first-hour data that has been processed.”
↩︎ Prediction“For stateless streaming processing, such as appending new data from the source, handling out-of-order and late arrival data is less complex”
↩︎ Checkpoint“Streaming processing is generally a better choice, given users can benefit from the advantages of streaming”
↩︎ Checkpoint - 2.https://docs.databricks.com/aws/en/ingestionOfficial docs
“Triggered: Processes new data on a schedule or when manually triggered.”
↩︎ Triggered and continuous pipeline modes“Databricks recommends starting with the most managed layer.”
↩︎ Three layers of ingestion products“SFTP servers | Ingest files from SFTP servers(Python, SQL) | N/A | N/A”
↩︎ Three layers of ingestion products“The following table describes the three layers of ingestion products, ordered from most customizable to most managed:”
↩︎ Checkpoint - 3.
“On the first run of your pipeline, it ingests all of the selected data from the source.”
↩︎ Managed connectors: types, components, incremental runs“A query-based connector queries the source database directly on a schedule, without a gateway or staging storage.”
↩︎ Managed connectors: types, components, incremental runs“standard connectors, which trade some automation for broader source support and customization”
↩︎ Exam trap 2“Ingest unstructured and structured files from enterprise file storage services including Google Drive and SharePoint.”
↩︎ Checkpoint“it uses that change tracking to ingest only the data that's changed from the prior run”
↩︎ Checkpoint - 4.
“Auto Loader stores the schema information in a directory _schemas at the configured cloudFiles.schemaLocation”
↩︎ Auto Loader: schema location, inference and evolution“addNewColumns (default) | Stream fails with an UnknownFieldException after Auto Loader adds the new columns to the schema.”
↩︎ Auto Loader: schema location, inference and evolution