What you will be able to do
- Explain what cloudFiles.schemaLocation turns on and how Auto Loader infers a starting schema for each file format
- Predict what each cloudFiles.schemaEvolutionMode value does when a new column arrives
- Choose between a provided schema, schema hints, the rescued data column and failOnNewColumns to enforce a schema
Key concept
cloudFiles.schemaLocation — The directory where Auto Loader stores the schema it inferred and every later change to it. Setting this directory is what turns on schema inference and evolution, and every evolution mode works by comparing incoming files against the schema saved there.
1.Where the schema lives and how Auto Loader infers it
Auto Loader is the cloudFiles Structured Streaming source. You give it an input directory, and it processes new files as they arrive. It can read JSON, CSV, XML, PARQUET, AVRO, ORC, TEXT and BINARYFILE. That list is wider than the list of formats it can infer a schema for, which matters on the exam: ORC can be ingested but has no schema inference or evolution, and text and binaryFile have fixed schemas.
Schema inference and evolution only switch on when you set cloudFiles.schemaLocation. This can be the same directory as the stream's checkpointLocation. Inside it, Auto Loader keeps a _schemas directory that records how the input schema changes over time. If you load from more than one source location into the same target table, each Auto Loader workload needs its own streaming checkpoint.
(spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "parquet")
# The schema location directory keeps track of your data schema over time
.option("cloudFiles.schemaLocation", "<path-to-schema>")
.load("<path-to-source-data>")
.writeStream
.option("checkpointLocation", "<path-to-checkpoint>")
.start("<path-to-target>")
)On its first read, Auto Loader samples the first 50 GB or 1000 files it discovers, whichever limit it reaches first. You can change the sample size with spark.databricks.cloudFiles.schemaInference.sampleSize.numBytes and ...numFiles. What it infers from that sample depends on the format. By default it tries to avoid type-mismatch problems during evolution. For text-based formats, that means every column becomes a string, including nested JSON fields. For typed formats, it merges the schemas of the sampled files. When the same Parquet column has different types in two files, Auto Loader picks the widest type.
CSV gets one extra assumption: Auto Loader expects headers. If your files have none, set .option("header", "false").
| File format | Inference and evolution supported | Default inferred data type |
|---|---|---|
| JSON | All versions | String |
| CSV | All versions | String |
| XML | Databricks Runtime 14.3 LTS and above | String |
| Avro | Databricks Runtime 10.4 LTS and above | Types encoded in Avro schema |
| Parquet | Databricks Runtime 11.3 LTS and above | Types encoded in Parquet schema |
| ORC | Unsupported | Not applicable |
| Text / Binaryfile | Not applicable (fixed-schema) | Fixed schema |
Checkpoint 1 of 6· Check yourself
You want Auto Loader to pick column types for JSON from the sampled data, the way the Spark DataFrameReader does, rather than reading everything as strings. Which option do you set?
inferColumnTypes switches Auto Loader to sample-based type inference for JSON, CSV and XML. Type widening only changes how existing types evolve later.
“To enable this behavior with Auto Loader, set the option cloudFiles.inferColumnTypes to true.”Source: docs.databricks.com
2.What happens when a new column arrives
The failure is intentional. Before the stream throws the error, Auto Loader runs schema inference on the latest micro-batch and writes the merged schema to the schema location, with the new columns added at the end. Existing columns keep their data types. When the stream restarts, it picks up the updated schema and continues. This is why Databricks recommends running Auto Loader streams with Lakeflow Jobs that restart automatically: the stop-then-restart cycle is how evolution works in the default mode.
You choose a different behaviour with cloudFiles.schemaEvolutionMode. The five modes differ on two things: whether the schema changes, and whether the stream fails.
| Mode | What happens to the new column | Does the stream fail? |
|---|---|---|
| addNewColumns (default when no schema is provided) | Added to the schema; existing column types do not change | Yes, with UnknownFieldException; a restart resumes with the updated schema |
| addNewColumnsWithTypeWidening | Added as in addNewColumns; supported types are also widened (such as int to long); unsupported changes go to the rescued data column | Yes, same as addNewColumns |
| rescue | Schema never evolves; new columns are recorded in the rescued data column | No |
| failOnNewColumns | Schema is not updated automatically | Yes, and it does not restart until you update the provided schema or remove the offending file |
| none (default when a schema is provided) | Ignored; not rescued unless rescuedDataColumn is set | No |
addNewColumnsWithTypeWidening is in Public Preview on Databricks Runtime 16.4 and above. It widens types such as int to long or float to double without rewriting data.
Partition columns are a separate case. Auto Loader infers them from Hive-style key=value directories, but it does not include them in schema evolution. If a new level such as hour=01 starts appearing, Auto Loader ignores it unless you list the column in cloudFiles.partitionColumns.
Checkpoint 2 of 6· Match them up
Match each schema evolution mode to its behaviour when a new column appears
Tap a term, then the definition that fits it.
Only addNewColumns (and its type-widening variant) changes the schema. rescue and none keep the stream running, and failOnNewColumns blocks it until someone steps in.
“Auto Loader never evolves the schema and the stream does not fail due to schema changes.”Source: docs.databricks.com
Checkpoint 3 of 6· Exam question
A data engineer wants to process all new files in a cloud storage directory using Auto Loader. They want the job to run as a batch job that processes all available data, respects rate limits across multiple micro-batches if there is a large backlog, and then stops automatically. Which trigger configuration should the engineer use?
Correct answer: B — trigger(availableNow=True)
- A. Incorrect. The trigger(once=True) configuration processes only a single micro-batch of data and then stops. If there is a large backlog of files exceeding the defined rate limits (e.g., maxFilesPerTrigger), it will not process all available data in that single run.
- B. Correct. The trigger(availableNow=True) configuration is the evolved version of Trigger.Once. It is designed to process all available data at the time the query starts, splitting it into multiple micro-batches if necessary to respect rate limits, and then automatically stopping the query once the backlog is cleared.
- C. Incorrect. trigger(processingTime='0 seconds') is the default behavior for Structured Streaming. It runs micro-batches as quickly as possible and continues to run indefinitely, waiting for new data rather than stopping automatically.
- D. Incorrect. trigger(continuous='1 minute') is for continuous processing mode, which aims for low-latency processing and does not stop automatically. It is not suitable for batch-style processing of a backlog.
- E. Incorrect. Removing the trigger entirely defaults the streaming query to the standard micro-batch mode (trigger(processingTime='0 seconds')), which runs continuously and does not stop automatically.
Checkpoint 4 of 6· Exam question
Which of the following statements accurately describes the purpose of the rescued data column (`_rescued_data`) in Databricks Auto Loader?
Correct answer: B — It captures data that does not match the expected schema, such as type mismatches or unmapped columns.
- A. Incorrect. The rescued data column is not used for operational file access problems like network errors or missing permissions. These issues typically cause ingestion failures or retries rather than being captured in a table column.
- B. Correct. The `_rescued_data` column in Databricks Auto Loader is designed to preserve data that would otherwise be lost due to schema enforcement. It captures fields or values that cannot be parsed according to the expected schema, including type mismatches and extra unmapped columns.
- C. Incorrect. Auto Loader does not automatically 'fix' the mismatch by casting the primary column; instead, it preserves the non-conforming content in the `_rescued_data` column so that users can inspect it or adjust the schema later without data loss.
- D. Incorrect. The `_rescued_data` column is not a disaster recovery mechanism or a backup for source files; it is a schema management feature that stores unparsed record content.
- E. Incorrect. Incremental ingestion and file tracking for deduplication are handled by Auto Loader's metadata and checkpointing mechanisms, not by the `_rescued_data` column.
Sources1
3.Enforcing a schema: provided schemas, hints and rescued data
Evolution means letting the schema change. Enforcement means controlling how much it can change, and the first lever is whether you provide a schema at all. If Auto Loader infers the schema, the default mode is addNewColumns. If you provide the schema yourself, the default becomes none, and addNewColumns is not allowed with a provided schema. There is one way around that. You can supply the schema you care about as a schema hint rather than as a full schema. Inference still runs, so addNewColumns keeps working, but the hinted columns get the types you chose.
.option("cloudFiles.schemaHints", "tags map<string,string>, version int")Auto Loader applies hints only when you do not provide a schema. A hint can also add a column that is not present at the start of the stream. For Parquet, a hint does not cast the data. It tells the Parquet reader which type to read the column as, and if the data does not match, Auto Loader moves the value to the rescued data column.
The rescued data column is the safety net. When Auto Loader infers the schema, it automatically adds a _rescued_data column. You can rename it, or add one to a schema you provide, with the rescuedDataColumn option. It catches data that would otherwise be lost: columns missing from the schema, type mismatches and case mismatches. Each entry is a JSON blob holding the rescued columns and the source file path of the record. With rescuedDataColumn set, type mismatches no longer cause records to be dropped in DROPMALFORMED mode or to throw errors in FAILFAST mode. Only corrupt records (incomplete or malformed JSON or CSV) fail.
The strictest lever is failOnNewColumns. Nothing new gets through until a person updates the provided schema or removes the offending file.
Checkpoint 5 of 6· Check yourself
A stream reads JSON with an explicit schema passed to the reader, and cloudFiles.schemaEvolutionMode is not set. A file arrives with an extra column. What happens?
Providing a schema makes none the default mode. none does not fail and does not evolve, and it only rescues data if you configure the rescued data column.
“Does not evolve the schema, new columns are ignored, and data is not rescued unless the rescuedDataColumn option is set.”Source: docs.databricks.com
Checkpoint 6 of 6· Exam question
A data engineer is configuring Auto Loader to ingest millions of files from an S3 bucket. Directory listing is becoming too slow and expensive due to the massive number of files in the input directory. They want to switch to a more scalable file discovery method. Which action should the engineer take?
Correct answer: A — Set `cloudFiles.useNotifications` to `true` to leverage cloud provider event services.
- A. Correct. Setting `cloudFiles.useNotifications` to `true` enables Auto Loader to use cloud provider event services (such as AWS S3 Event Notifications and SQS) instead of directory listing. This method is much more scalable and cost-effective for large datasets because it only tracks new files as they arrive rather than scanning the entire directory structure.
- B. Incorrect. Setting `cloudFiles.useDirectoryListing` to `false` alone does not enable a scalable alternative. To use the event-driven discovery method, the `cloudFiles.useNotifications` parameter must be explicitly set to `true`.
- C. Incorrect. Increasing `cloudFiles.maxFilesPerTrigger` only adjusts how many files are processed per micro-batch once they have already been discovered. It does not solve the bottleneck and expense associated with the file discovery method itself.
- D. Incorrect. While partitioning and glob filtering can improve file organization, they still rely on the directory listing mechanism, which is the source of the performance and cost issues in this scenario.
- E. Incorrect. Standard Structured Streaming with `format("json")` is generally less efficient than Auto Loader for cloud storage ingestion. It lacks Auto Loader's specialized features for scalable file discovery and optimized metadata management.
Sources1
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.addNewColumns is always the default schema evolution mode.Why is that wrong?
addNewColumns is the default only when Auto Loader infers the schema. When you provide a schema the default is none, and addNewColumns is not allowed at all, though it still works if you supply the schema as a schema hint.
Covered in Enforcing a schema: provided schemas, hints and rescued data
2.With addNewColumns, Auto Loader adds the new column and the stream carries on without stopping.Why is that wrong?
The stream fails with an UnknownFieldException after the schema location has been updated. Processing continues with the new column only after a restart, which is why Databricks recommends jobs that restart automatically.
Covered in What happens when a new column arrives
3.The none and rescue modes both keep unexpected columns in the rescued data column.Why is that wrong?
rescue records new columns in the rescued data column. none ignores them and rescues nothing unless rescuedDataColumn is set.
Covered in What happens when a new column arrives
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Specifying a target directory for the cloudFiles.schemaLocation option enables schema inference and evolution.”
↩︎ Where the schema lives and how Auto Loader infers it“Auto Loader samples the first 50 GB or 1000 files that it discovers, whichever limit is crossed first.”
↩︎ Where the schema lives and how Auto Loader infers it“each Auto Loader ingestion workload requires a separate streaming checkpoint.”
↩︎ Where the schema lives and how Auto Loader infers it“When Auto Loader detects a new column, the stream stops with an UnknownFieldException.”
↩︎ What happens when a new column arrives“This feature is in Public Preview in Databricks Runtime 16.4 and above.”
↩︎ What happens when a new column arrives“Auto Loader does not consider partition columns for schema evolution.”
↩︎ What happens when a new column arrives“addNewColumns is not allowed when the schema of the stream is provided, but does work if you provide your schema as a schema hint.”
↩︎ Enforcing a schema: provided schemas, hints and rescued data“You can use schema hints to enforce the schema information that you know and expect on an inferred schema.”
↩︎ Enforcing a schema: provided schemas, hints and rescued data“When Auto Loader infers the schema, Auto Loader automatically adds a rescued data column to your schema as _rescued_data.”
↩︎ Enforcing a schema: provided schemas, hints and rescued data“Stream fails and does not restart unless you update the provided schema or remove the offending data file.”
↩︎ Enforcing a schema: provided schemas, hints and rescued data“Specifying a target directory for the cloudFiles.schemaLocation option enables schema inference and evolution.”
↩︎ Key concept“addNewColumns mode is the default when a schema is not provided, but none is the default when you provide a schema.”
↩︎ Exam trap 1“Databricks recommends configuring Auto Loader streams with Lakeflow Jobs to restart automatically after such schema changes.”
↩︎ Exam trap 2“Does not evolve the schema, new columns are ignored, and data is not rescued unless the rescuedDataColumn option is set.”
↩︎ Exam trap 3“For formats that don't encode data types (JSON, CSV, and XML), Auto Loader infers all columns as strings”
↩︎ Prediction“To enable this behavior with Auto Loader, set the option cloudFiles.inferColumnTypes to true.”
↩︎ Checkpoint“Auto Loader never evolves the schema and the stream does not fail due to schema changes.”
↩︎ Checkpoint“Does not evolve the schema, new columns are ignored, and data is not rescued unless the rescuedDataColumn option is set.”
↩︎ Checkpoint - 2.
“Auto Loader can ingest JSON, CSV, XML, PARQUET, AVRO, ORC, TEXT, and BINARYFILE file formats.”
↩︎ Where the schema lives and how Auto Loader infers it