What you will be able to do
- Explain why every Databricks table is a Delta table by default and where its data and metadata live
- Trace how the Delta transaction log makes commits atomic and keeps concurrent writes from corrupting a table
- Describe the Unity Catalog metastore, securable objects and the catalog.schema.object namespace
- Tell managed assets from external assets
1.Delta Lake: the default table format
Delta Lake is the optimised storage layer behind tables in a Databricks lakehouse. It is open-source software that adds a file-based transaction log to ordinary Parquet data files. The log is what gives you ACID transactions and metadata handling that scales. Delta Lake is the default format for every operation on Databricks, so with default settings you get its benefits whether you write through Spark DataFrames or SQL. It is fully compatible with Apache Spark APIs and was built to work closely with Structured Streaming, so one copy of the data can serve both batch and streaming.
Physically, a Delta table is a directory of files in cloud object storage. Databricks keeps all of a Delta table's data and metadata there, and registers the table's metadata in the metastore under a catalog and schema. Every write creates a new table version, and you can read the transaction log to review changes or query earlier versions. Delta Lake also checks the schema on write, so all data written to a table must match the requirements you've set. The log follows an open, well-defined protocol that any system can use to read it.
Checkpoint 1 of 8· Check yourself
Which statement about how Delta tables are stored on Databricks is correct?
Databricks stores all data and metadata for Delta tables in cloud object storage. Versions persist in the transaction log, and the log protocol is open.
“Databricks stores all data and metadata for Delta Lake tables in cloud object storage.”Source: docs.databricks.com
Beyond the log, Delta Lake brings a set of table features. Schema enforcement validates data quality by enforcing the schema on write, while schema evolution lets you update a table's schema manually or automatically without rewriting data. Column mapping lets you rename or drop columns without rewriting data. Because each write creates a new table version, table history lets you review modifications and query previous versions. For layout and storage, you can use liquid clustering and data skipping to cut the number of files scanned, optimize the data file layout to compact small data files, and use vacuum to remove stale data files and reduce storage costs.
Constraints work differently from what many people assume. Databricks supports enforced integrity constraints, but primary key, foreign key and unique constraints are informational. Declaring a primary key documents a relationship; it does not make the table reject duplicate key values.
Checkpoint 2 of 8· Check yourself
You declare a primary key on a Databricks table and then insert two rows with the same key value. What happens?
Primary key and foreign key relationships on Databricks are informational and not enforced, so declaring one does not stop duplicate values from being written.
“Primary key and foreign key relationships on Databricks are informational and not enforced.”Source: docs.databricks.com
2.The transaction log and ACID guarantees
ACID stands for atomicity, consistency, isolation and durability. Atomicity means every transaction either succeeds completely or fails completely. Consistency is about how simultaneous operations see the state of the data. Isolation is about how simultaneous operations can conflict. Durability means committed changes are permanent. These guarantees apply to tables backed by Delta Lake; other data formats may not give transactional guarantees.
The transaction log controls commit atomicity. During a transaction, data files are written into the table's directory. When the transaction completes, a new entry is committed to the log listing the paths of every file that was written. That commit increments the table version and makes the new files visible to readers. The table's current state is every data file the log marks as valid. A file the log never recorded is not part of the table. If a transaction fails after writing files, those files don't corrupt the table; they never become part of it, and VACUUM later deletes untracked files like these. Because the log sits alongside the data in cloud object storage, tables inherit that storage's durability.
The table as it was at the last committed version. No new log entry was committed, so the orphaned files are not tracked and stay invisible. VACUUM can remove them later.
To handle concurrent work, Delta Lake uses optimistic concurrency control. Reads and writes take no locks, so deadlock cannot happen. A write goes through three stages. First it reads the latest table version, if needed, to work out which files must change. Then it writes new data files. Finally it validates and commits: it checks for conflicts with anything committed since its snapshot. If there are none, it commits a new version; if there are, the write fails with a concurrent modification exception rather than corrupting the table. By default, reads get snapshot isolation and writes get write-serializable isolation. Each SQL statement is its own atomic transaction against one table, and BEGIN ATOMIC ... END; groups statements across tables (participating tables need catalog commits enabled).
Checkpoint 3 of 8· Put it in order
Put the stages of a Delta Lake write under optimistic concurrency control in order
- 1.Validate against concurrent commits since the snapshot, then commit a new version
- 2.Write data files to the table's directory
- 3.Read the latest available table version to identify which files need to change
The documentation fixes three stages: read, write, then validate and commit. The check for conflicting concurrent commits happens in the last stage.
“Under this mechanism, writes operate in three stages:”Source: docs.databricks.com
Checkpoint 4 of 8· Exam question
A data engineering team must connect a SQL warehouse to an on-premises database over a private network link and needs the compute to run inside their own AWS account to satisfy a networking review. Which SQL warehouse type satisfies this requirement?
Correct answer: A — A pro SQL warehouse, since its compute runs in the customer's own AWS account and supports custom networking for hybrid connectivity
- A. Pro SQL warehouses run compute in the customer's own AWS account, which allows the custom networking configurations, such as VPC peering or private connectivity, needed to reach an on-premises database.
- B. Serverless SQL warehouses run in Databricks-managed infrastructure with built-in customer isolation, so they do not expose the customer's own account networking needed for this hybrid connectivity requirement.
- C. Classic SQL warehouses do not include intelligent workload management at all, and having compute in a customer account is a property of pro warehouses, not a networking feature specific to classic.
- D. All-purpose clusters are separate compute resources for notebooks and jobs; they are not attached to SQL warehouses and do not provide hybrid networking for SQL warehouse traffic.
Sources3
3.Unity Catalog: metastore, securable objects, and the three-level namespace
Unity Catalog is the unified governance layer for data and AI built into Databricks. When a workspace has it enabled, it sits beneath every interaction with data and AI. It enforces access control when you query a table or call a model, tracks lineage as assets are used, and logs activity for auditing. You work with what it governs through Catalog Explorer, SQL, the Databricks CLI and REST APIs. It is enabled automatically for every workspace created after November 8, 2023; older workspaces must be upgraded. There is also an open-source implementation of Unity Catalog.
Unity Catalog's metastore is account-level. It records metadata about data and AI assets, and the permissions on catalogs, schemas and tables. The Hive metastore is the legacy alternative for customers who have not adopted Unity Catalog. Every asset Unity Catalog governs is a securable object: something you can grant permissions on to users, service principals or groups. Catalogs are the highest-level container for organising and isolating data. Schemas, also called databases, sit inside catalogs. Tables, views, volumes, functions and models sit inside schemas. That gives the three-level name used to address any data asset:
<catalog-name>.<schema-name>.<object-name>Some securable objects are outside that namespace: storage credentials, external locations, connections and shares sit directly under the metastore. Tables and volumes come in two kinds. For a managed table or volume, Unity Catalog handles both governance and the lifecycle of the underlying file storage. For an external one, it handles governance only. Volumes govern non-tabular data in cloud object storage, and Databricks recommends them for all access to non-tabular data.
In practice, the difference is about who takes care of the files. With a managed table or volume, Unity Catalog looks after the underlying file storage lifecycle as well as access control. With an external table or volume, Unity Catalog governs access, but the file storage lifecycle is not something it handles. Choose by asking whether you want Unity Catalog to own the storage lifecycle or only to govern data that lives elsewhere.
| Securable objects | Where they sit | What Unity Catalog handles |
|---|---|---|
| Managed tables and volumes | catalog.schema.object | Governance and the underlying file storage lifecycle |
| External tables and volumes | catalog.schema.object | Governance only |
| Views, functions, models, services | catalog.schema.object | Governance |
| Storage credentials, external locations, connections, shares | Directly under the metastore | Governance |
Checkpoint 5 of 8· Match them up
Match each Unity Catalog term to its description
Tap a term, then the definition that fits it.
The metastore sits above the namespace. Catalog and schema are its first two levels, and volumes are one kind of object inside a schema.
“Catalogs are the highest level container for organizing and isolating data on Databricks.”Source: docs.databricks.com
Checkpoint 6 of 8· Check yourself
Which object sits directly under the metastore, rather than inside the catalog.schema.object namespace?
Storage credentials, external locations, connections and shares sit directly under the metastore. Views, volumes and functions use the three-level namespace.
“Other objects, such as storage credentials, external locations, connections, and shares, sit directly under the metastore.”Source: docs.databricks.com
Checkpoint 7 of 8· Check yourself
What does Unity Catalog handle for an external table?
For an external table or volume Unity Catalog handles governance only. For a managed one it handles governance and the underlying file storage lifecycle.
“where Unity Catalog handles both governance and the underlying file storage lifecycle, or external, where Unity Catalog handles governance only”Source: docs.databricks.com
Checkpoint 8 of 8· Exam question
A nightly ETL pipeline runs a single Lakeflow Job on a schedule, and the team wants compute that spins up only for that run, is not shared with any other workload, and automatically terminates once the job finishes to avoid paying for idle time. What compute choice fits this pattern?
Correct answer: A — A job cluster created and managed by the job itself, which is provisioned for the run and torn down on completion
- A. A job cluster is created specifically for a job run and is automatically terminated when the run completes, which gives isolated, non-shared compute with no idle cost between runs.
- B. An all-purpose cluster left running continuously and shared across the team is billed for idle time and is exposed to other users' workloads, which contradicts the isolation and cost goals.
- C. SQL warehouses are designed for SQL query workloads such as dashboards and ad hoc analytics, not for running notebook or Spark-based ETL job tasks, and pro warehouses are not the mechanism for isolated per-job compute.
- D. Consolidating a scheduled ETL job onto a shared serverless SQL warehouse used by dashboards mixes unrelated workloads and does not give the dedicated, terminate-on-completion behavior the team wants.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Declaring a primary key on a Databricks table makes it reject duplicate key values.Why is that wrong?
Primary key and foreign key relationships on Databricks are informational only and are not enforced.
Covered in Delta Lake: the default table format
2.Delta Lake locks a table during a write, so concurrent writers queue behind it.Why is that wrong?
Delta Lake uses optimistic concurrency control and takes no locks. Conflicts are caught at commit time and the losing write fails with a concurrent modification exception.
Covered in The transaction log and ACID guarantees
3.Each workspace has its own Unity Catalog metastore, so catalogs can't be shared between workspaces.Why is that wrong?
The metastore is account-level, and catalogs can be shared across workspaces in the same region and account.
Covered in Unity Catalog: metastore, securable objects, and the three-level namespace
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/deltaOfficial docs
“extends Parquet data files with a file-based transaction log for ACID transactions and scalable metadata handling”
↩︎ Delta Lake: the default table format“Each write to a Delta Lake table creates a new table version.”
↩︎ Delta Lake: the default table format“Apply enforced integrity constraints and informational primary key, foreign key, and unique constraints.”
↩︎ Delta Lake: the default table format“Remove stale data files to reduce storage costs.”
↩︎ Delta Lake: the default table format“Unless otherwise specified, all tables on Databricks are Delta Lake tables.”
↩︎ Prediction“Databricks stores all data and metadata for Delta Lake tables in cloud object storage.”
↩︎ Checkpoint - 2.
“registers table metadata to the metastore within a catalog and schema”
↩︎ Delta Lake: the default table format“Unity Catalog provides an account-level metastore that registers metadata about data, AI, and permissions about catalogs, schemas, and tables.”
↩︎ Unity Catalog: metastore, securable objects, and the three-level namespace“You can share catalogs across workspaces within the same region and account.”
↩︎ Exam trap 3“Catalogs are the highest level container for organizing and isolating data on Databricks.”
↩︎ Checkpoint - 3.https://docs.databricks.com/aws/en/lakehouse/acidOfficial docs
“The transaction log controls commit atomicity.”
↩︎ The transaction log and ACID guarantees“If a transaction fails after writing data files to a table, these data files will not corrupt the table state”
↩︎ The transaction log and ACID guarantees“By default, Databricks provides snapshot isolation on reads and write-serializable isolation on writes.”
↩︎ The transaction log and ACID guarantees“Primary key and foreign key relationships on Databricks are informational and not enforced.”
↩︎ Exam trap 1“This means that there are no locks on reading or writing against a table, and deadlock is not a possibility.”
↩︎ Exam trap 2“Primary key and foreign key relationships on Databricks are informational and not enforced.”
↩︎ Checkpoint“Under this mechanism, writes operate in three stages:”
↩︎ Checkpoint - 4.
“Unity Catalog is automatically enabled for all Databricks workspaces created after November 8, 2023.”
↩︎ Unity Catalog: metastore, securable objects, and the three-level namespace“an object on which you can grant permissions to users, service principals, or groups”
↩︎ Unity Catalog: metastore, securable objects, and the three-level namespace“or external, where Unity Catalog handles governance only”
↩︎ Unity Catalog: metastore, securable objects, and the three-level namespace“Other objects, such as storage credentials, external locations, connections, and shares, sit directly under the metastore.”
↩︎ Checkpoint“where Unity Catalog handles both governance and the underlying file storage lifecycle, or external, where Unity Catalog handles governance only”
↩︎ Checkpoint