What you will be able to do
- Configure an Auto Loader (cloudFiles) stream that ingests JSON, CSV, Parquet or binary files from cloud storage or Unity Catalog volumes
- Predict how Auto Loader infers types for each file format and what each schemaEvolutionMode does when a new column appears
- Choose between Auto Loader, COPY INTO and Lakeflow pipelines for a given file ingestion workload
- Recognise the constraints on using Delta Lake tables as streaming sources, and read binary files with the binaryFile source
Key concept
Auto Loader (the cloudFiles source) — A Structured Streaming source that watches a cloud storage directory and processes each new file once, keeping track of discovered files in its checkpoint. You choose the file format with an option, and the same stream pattern works for JSON, CSV, Parquet, binary and other formats.
1.One source for many file formats
Most file-based ingestion on Databricks starts with Auto Loader. You give it an input directory, and it processes new files as they arrive. You can also tell it to process the files already in that directory. Auto Loader works with Python and SQL in Lakeflow pipelines, and with Python and Scala when you use Structured Streaming directly. It scales from backfills of billions of files to near real-time ingestion of millions of files per hour.
The format is something you set, not a different tool for each format. Auto Loader can ingest JSON, CSV, XML, PARQUET, AVRO, ORC, TEXT and BINARYFILE, including pre-compressed files in these formats. Delta Lake and Iceberg are missing from that list because they are table formats, not loose files. Reading them is covered later on this page.
| Storage | Path prefix |
|---|---|
| Amazon S3 | s3:// |
| Azure Data Lake Storage (ADLS) | abfss:// |
| Google Cloud Storage (GCS) | gs:// |
| Unity Catalog volumes | /Volumes/ |
| Azure Blob Storage (legacy WASB driver, deprecated) | wasbs:// |
Auto Loader can promise exactly-once processing because of how it keeps state. As it discovers files, it writes their metadata to a RocksDB key-value store inside the pipeline's checkpoint location. After a failure, it resumes from that checkpoint and keeps the exactly-once guarantee when it writes to Delta Lake. You do not manage any of that state yourself. Compared with the plain Spark file source (spark.readStream.format(fileFormat)), Auto Loader also costs less to discover files: the cost grows with the number of files ingested, not with the number of directories. File notification mode can skip directory listing altogether.
Checkpoint 1 of 6· Check yourself
An Auto Loader stream fails halfway through a batch and is restarted. What lets it resume without reprocessing files it has already ingested?
Auto Loader records discovered files in RocksDB in the checkpoint location and resumes from that checkpoint after a failure.
“their metadata is persisted in a scalable key-value store (RocksDB) in the checkpoint location of your Auto Loader pipeline”Source: docs.databricks.com
Sources1
2.How the format decides the inferred schema
To infer and evolve a schema, Auto Loader needs a place to record it. Setting cloudFiles.schemaLocation turns this on, and the directory can be the same one you use for checkpointLocation. In Lakeflow pipelines, Databricks manages this location for you. The example below reads Parquet. To read csv, avro or json instead, change only cloudFiles.format.
(spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "parquet")
# The schema location directory keeps track of your data schema over time
.option("cloudFiles.schemaLocation", "<path-to-schema>")
.load("<path-to-source-data>")
.writeStream
.option("checkpointLocation", "<path-to-checkpoint>")
.start("<path-to-target>")
)Auto Loader infers the schema from a sample: the first 50 GB or 1000 files it discovers, whichever limit it hits first. It stores the result in a _schemas directory. What it infers depends on whether the format carries its own types. If you want the DataFrameReader behaviour, where types are guessed from sample values, set cloudFiles.inferColumnTypes to true. When two Parquet files disagree on a column's type, Auto Loader picks the widest type. Schema hints override that choice, and values that don't match the hinted type go to the rescued data column.
Checkpoint 2 of 6· Check yourself
Two Parquet files in the sample give the same column different data types. What does Auto Loader do by default?
For typed formats such as Parquet, Auto Loader merges the sampled schemas and picks the widest type when a column's types differ. Schema hints can override this choice.
“When a column has different data types in two Parquet files, Auto Loader chooses the widest type.”Source: docs.databricks.com
| File format | Inference/evolution support | Default inferred type |
|---|---|---|
| JSON | All versions | String |
| CSV | All versions | String |
| XML | Databricks Runtime 14.3 LTS and above | String |
| Avro | Databricks Runtime 10.4 LTS and above | Types encoded in Avro schema |
| Parquet | Databricks Runtime 11.3 LTS and above | Types encoded in Parquet schema |
| Binaryfile | Not applicable (fixed-schema) | Not applicable (fixed-schema) |
When it infers a CSV schema, Auto Loader assumes the files have headers. For headerless files, set .option("header", "false").
Sources2
3.When a new column appears
Inference sets the starting schema. Evolution decides what happens when the data changes. In the default mode, Auto Loader detects a new column, merges it onto the end of the schema stored in the schema location, and then stops the stream with an UnknownFieldException. The existing columns keep their types. When you restart the stream, it picks up the updated schema. That is why Databricks recommends running these streams as Lakeflow Jobs that restart automatically.
You change this behaviour with cloudFiles.schemaEvolutionMode. The default depends on whether you supply a schema. Without a schema, the default is addNewColumns. With a schema, the default is none. addNewColumns is not allowed with a provided schema, but it does work if you pass the schema as a schema hint instead.
Checkpoint 3 of 6· Match them up
Match each schemaEvolutionMode to its behaviour when a new column arrives
Tap a term, then the definition that fits it.
Only the two addNewColumns modes update the stored schema. rescue and none keep the stream running without evolving it, and failOnNewColumns needs manual action before the stream can restart.
“Auto Loader never evolves the schema and the stream does not fail due to schema changes.”Source: docs.databricks.com
Sources2
4.Auto Loader, COPY INTO, or a Lakeflow pipeline
Auto Loader is not the only way to load files incrementally. SQL users can use COPY INTO, which loads data from cloud object storage into Delta tables idempotently and incrementally. It runs in Databricks SQL, notebooks and Lakeflow Jobs. The guidance for choosing depends on scale, schema drift and reprocessing needs:
- Scale: COPY INTO suits thousands of files over time. Auto Loader suits millions, because it needs fewer operations to discover files and can split the work into multiple batches. - Schema drift: Auto Loader handles frequently evolving schemas better. - Reprocessing: COPY INTO makes it easier to reload a chosen subset of re-uploaded files. You can run it alongside a running Auto Loader stream.
For production, Databricks recommends Auto Loader inside Lakeflow pipelines. There you supply neither a schema location nor a checkpoint location, because the pipeline manages both. SQL users get the same engine through streaming tables.
Checkpoint 4 of 6· Exam question
A team ingests clickstream JSON files that land continuously in a deeply nested, hour-partitioned S3 prefix, and the SLA requires new files to be picked up within seconds even as the prefix grows into tens of millions of objects. Which Auto Loader configuration change best meets this requirement?
Correct answer: A — Enable `cloudFiles.useNotifications=true` so Auto Loader discovers new files through cloud storage events delivered to a managed queue instead of repeatedly listing the prefix.
- A. Correct. Event-driven notification mode avoids repeatedly listing a prefix that has grown to tens of millions of objects, so new-file latency stays low and stable regardless of prefix size.
- B. Incorrect. Directory listing still has to enumerate the entire nested prefix on every trigger, so a larger per-trigger file cap does not fix the growing listing cost or the seconds-level latency requirement.
- C. Incorrect. Backfill is a periodic safety-net listing meant to catch files discovery missed, not the primary discovery path; running a full listing every minute against tens of millions of objects is expensive and still listing-based.
- D. Incorrect. Running only once per scheduled job trades away the continuous, low-latency discovery the SLA requires, since new files would wait for the next scheduled invocation instead of being picked up within seconds.
Checkpoint 5 of 6· Check yourself
A team expects tens of millions of JSON files a month in S3, and the schema changes often. Which approach matches Databricks guidance?
At millions of files, and with frequent schema changes, the guidance points to Auto Loader. Running it in Lakeflow pipelines also takes schema and checkpoint management off your hands.
“If you are expecting files in the order of millions or more over time, use Auto Loader.”Source: docs.databricks.com
5.Delta, Iceberg and binary sources
Delta Lake is the default protocol on Databricks for reading and writing data and tables. Plain Apache Spark defaults to Parquet. When the thing you are ingesting from is already a Delta table, you read it with readStream, not Auto Loader. Delta tables work as both sources and sinks for Structured Streaming. They give exactly-once processing even with several streams or concurrent batch jobs.
Using a Delta table as a source has one operational rule. The stream must run at least once within the source table's retention window. The defaults are 7 days for files removed by VACUUM and 30 days for the transaction log. A stream that falls further behind fails with DELTA_FILE_NOT_FOUND_DETAILED and needs a full refresh. Do not work around this by setting spark.sql.files.ignoreMissingFiles to true, because that silently gives wrong results. If the stream can't run often enough, increase the source table's retention.
Checkpoint 6 of 6· Check yourself
A weekly-paused stream reading a Delta table fails with DELTA_FILE_NOT_FOUND_DETAILED. What is the documented fix to stop this happening again?
The stream fell outside the retention window. Run it within that window or lengthen the source table's retention. ignoreMissingFiles is explicitly called out as producing incorrect results.
“the streaming query must run at least one time within the source table's retention window”Source: docs.databricks.com
Iceberg is treated as a table format too. Databricks can read and write Iceberg tables and interoperate with external Iceberg engines. Iceberg tables you create in Unity Catalog are managed Iceberg tables, and streaming tables can load data into them incrementally from Kafka and cloud object storage. A foreign Iceberg table is one managed by a catalog outside Unity Catalog. The available sources do not show a code-level pattern for streaming reads from Iceberg tables, so this lesson stops at that level.
For unstructured files such as images, audio or PDFs, use the binaryFile format. It turns each file into one record with fixed columns: path, modificationTime, length and content. Because the schema is fixed, schema inference does not apply. The same format works in Auto Loader as BINARYFILE.
df = spark.read.format("binaryFile").load("/Volumes/<catalog>/<schema>/<volume>/")
display(df)Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Auto Loader infers numeric and boolean types for JSON and CSV the same way spark.read does.Why is that wrong?
By default, Auto Loader infers every JSON, CSV and XML column as a string. To get typed inference, set cloudFiles.inferColumnTypes to true.
Covered in How the format decides the inferred schema
2.If you supply an explicit schema, Auto Loader still adds new columns by default.Why is that wrong?
With a provided schema, the default mode is none, and addNewColumns is not allowed. To keep evolution, pass the schema as a schema hint.
Covered in When a new column appears
3.Setting spark.sql.files.ignoreMissingFiles to true is an acceptable fix when a Delta streaming source falls behind retention.Why is that wrong?
This setting silently produces incorrect results. Increase the source table's retention or run the stream within the window instead.
Covered in Delta, Iceberg and binary sources
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Auto Loader can ingest JSON, CSV, XML, PARQUET, AVRO, ORC, TEXT, and BINARYFILE file formats.”
↩︎ One source for many file formats“This key-value store ensures that data is processed exactly once.”
↩︎ One source for many file formats“You do not need to provide a schema or checkpoint location because Lakeflow pipelines automatically manage these settings for your pipelines.”
↩︎ Auto Loader, COPY INTO, or a Lakeflow pipeline“It provides a Structured Streaming source called cloudFiles.”
↩︎ Key concept“their metadata is persisted in a scalable key-value store (RocksDB) in the checkpoint location of your Auto Loader pipeline”
↩︎ Checkpoint - 2.
“Specifying a target directory for the cloudFiles.schemaLocation option enables schema inference and evolution.”
↩︎ How the format decides the inferred schema“When inferring the schema for CSV data, Auto Loader assumes that the files contain headers.”
↩︎ How the format decides the inferred schema“When a column has different data types in two Parquet files, Auto Loader chooses the widest type.”
↩︎ How the format decides the inferred schema“When Auto Loader detects a new column, the stream stops with an UnknownFieldException.”
↩︎ When a new column appears“Databricks recommends configuring Auto Loader streams with Lakeflow Jobs to restart automatically after such schema changes.”
↩︎ When a new column appears“To enable this behavior with Auto Loader, set the option cloudFiles.inferColumnTypes to true.”
↩︎ Exam trap 1“addNewColumns mode is the default when a schema is not provided, but none is the default when you provide a schema.”
↩︎ Exam trap 2“For formats that don't encode data types (JSON, CSV, and XML), Auto Loader infers all columns as strings”
↩︎ Prediction“Auto Loader never evolves the schema and the stream does not fail due to schema changes.”
↩︎ Checkpoint - 3.
“Loading a subset of re-uploaded files can be a bit easier to manage with COPY INTO.”
↩︎ Auto Loader, COPY INTO, or a Lakeflow pipeline“If you are expecting files in the order of millions or more over time, use Auto Loader.”
↩︎ Checkpoint - 4.https://docs.databricks.com/aws/en/query/formatsOfficial docs
“Databricks uses Delta Lake as the default protocol for reading and writing data and tables, whereas Apache Spark uses Parquet.”
↩︎ Delta, Iceberg and binary sources - 5.
“The default retention windows are 7 days for VACUUM-removed data files and 30 days for the transaction log (logRetentionDuration).”
↩︎ Delta, Iceberg and binary sources“Do not set spark.sql.files.ignoreMissingFiles to true as a workaround because this configuration silently produces incorrect results.”
↩︎ Exam trap 3“the streaming query must run at least one time within the source table's retention window”
↩︎ Checkpoint - 6.https://docs.databricks.com/aws/en/icebergOfficial docs
“Iceberg tables that you create in Unity Catalog are managed Iceberg tables.”
↩︎ Delta, Iceberg and binary sources“A foreign Iceberg table is an Iceberg table managed by a catalog outside Unity Catalog.”
↩︎ Delta, Iceberg and binary sources - 7.
“To read binary files, specify the data source format as binaryFile.”
↩︎ Delta, Iceberg and binary sources