What you will be able to do
- Recognize update-heavy access patterns where deletion vectors avoid rewriting whole files
- Enable deletion vectors and weigh the protocol-upgrade tradeoff for other clients
- Make soft-deleted data physically disappear with OPTIMIZE, REORG TABLE ... APPLY (PURGE) and VACUUM
- Map a described workload to deletion vectors, liquid clustering, CLUSTER BY AUTO, or a combination
1.The write problem deletion vectors solve
Liquid clustering decides where rows live, which mainly helps reads. Deletion vectors deal with a different cost: the write amplification of DELETE, UPDATE and MERGE. Without them, changing a handful of rows rewrites every file those rows sit in. With them enabled, Databricks records the affected rows in metadata and leaves the data file untouched. Readers then apply those entries at query time to see the current state of the table.
The access pattern to look for is frequent small changes: CDC merges, GDPR-style deletes, corrections to a few records. In each case the rows touched are a small fraction of the files that contain them. Deletion vectors also power predictive I/O for updates on Photon-enabled compute. On Databricks Runtime 14.2 and above, tables with deletion vectors support row-level concurrency, which reduces write conflicts between concurrent writers.
Checkpoint 1 of 5· Check yourself
A nightly MERGE updates about 0.1% of rows in a large Delta table, but those rows are spread across most of its files. Which feature most directly cuts the write cost?
The cost comes from rewriting whole files to change a few rows. Deletion vectors record the changes in metadata instead, so the files aren't rewritten.
“Deletion vectors accelerate DELETE, UPDATE, and MERGE operations on Delta Lake and Apache Iceberg tables.”Source: docs.databricks.com
Sources1
2.Enabling deletion vectors and the compatibility cost
For Delta Lake tables you turn deletion vectors on explicitly with the delta.enableDeletionVectors table property, in either CREATE TABLE or ALTER TABLE. A workspace setting can also enable them on new tables when you use a SQL warehouse or Databricks Runtime 14.3 LTS and above, and the default varies by region. Materialized views and streaming tables are different: you can't add or remove deletion vectors with ALTER, only with a CREATE statement.
The cost is compatibility. Enabling deletion vectors upgrades the table protocol, and clients without deletion vector support can no longer read the table. Support varies by client, as the table below shows. If an outside reader breaks, Databricks Runtime 14.1 and above can drop the deletionVectors table feature to restore compatibility.
Checkpoint 2 of 5· Fill the gap
Which table property enables deletion vectors on an existing Delta Lake table?
ALTER TABLE <table-name> SET TBLPROPERTIES (' ? ' = true);delta.enableDeletionVectors is the Delta Lake property. The iceberg. prefix is for Iceberg tables, and the other two properties control different features.
| Client | Write deletion vectors | Read deletion vectors |
|---|---|---|
| Databricks Runtime with Photon | MERGE, UPDATE, and DELETE on Databricks Runtime 12.2 LTS and above | Databricks Runtime 12.2 LTS or above |
| Databricks Runtime without Photon | DELETE on 12.2 LTS+, UPDATE on 14.1+, MERGE on 14.3 LTS+ | Databricks Runtime 12.2 LTS or above |
| OSS Apache Spark with OSS Delta Lake | DELETE on OSS Delta 2.4.0+, UPDATE on OSS Delta 3.0.0+ | OSS Delta 2.3.0 or above |
Sources1
3.Soft deletes are not physical deletes
A DELETE on a table with deletion vectors is a soft delete. The rows are logically gone, but the Parquet files still hold them. Three operations rewrite files and apply the recorded changes: OPTIMIZE, REORG TABLE ... APPLY (PURGE), and a write with auto-compaction. Compaction gives no strict guarantee, though. Files that aren't compaction candidates can keep their deletion vector changes unapplied.
When the data must really be gone, for example for a regulatory erasure, use REORG with PURGE. It rewrites every file that has deletion vector changes. Then run VACUUM with the retention threshold set to the purge completion time to remove the old file versions. On large tables, setting spark.databricks.delta.reorg.purgeMode to rows can speed up the purge. It processes only files with soft-deleted rows instead of scanning every Parquet footer, which helps when the table has no dropped columns.
Checkpoint 3 of 5· Put it in order
Put these steps in order to make rows deleted on a deletion-vector table physically disappear from storage
- 1.Run REORG TABLE ... APPLY (PURGE) to rewrite files containing those rows
- 2.Run VACUUM with the retention threshold set to the purge completion timestamp
- 3.Run DELETE, which records the rows as soft-deleted in deletion vectors
The purge rewrites the data files without the deleted rows. VACUUM then removes the older file versions that still hold them.
“Run VACUUM with the retention threshold set to the purge completion timestamp to physically remove files from previous table versions.”Source: docs.databricks.com
Sources1
4.Matching the technique to the access pattern
The two techniques fix different problems, so many tables use both. Liquid clustering reduces how much data reads have to scan. Deletion vectors reduce how much data row-level writes have to rewrite. The table below maps typical exam scenarios to the documented choice.
One overlap is worth remembering. An unpartitioned table keeps ingestion time clustering only while modifications don't scramble it. If the table takes a large number of UPDATE or MERGE statements, the docs recommend liquid clustering on a column that matches ingestion order, such as an event timestamp.
| Access pattern | Technique | Why |
|---|---|---|
| Frequent DELETE, UPDATE or MERGE touching few rows per file | Deletion vectors | Changed rows are marked in metadata instead of rewriting whole Parquet files |
| Filters on known, stable, high-cardinality columns | Liquid clustering with explicit CLUSTER BY keys | Data is grouped by the keys for data skipping, with no fixed partition boundaries |
| Varied or changing query patterns on a Unity Catalog managed table | CLUSTER BY AUTO with predictive optimization | Databricks selects clustering keys and adapts them over time |
| Concurrent writers conflicting with each other | Liquid clustering and/or deletion vectors | Both enable row-level concurrency |
| Heavy UPDATE/MERGE on an ingestion-ordered table | Liquid clustering on the event timestamp or creation date | Preserves ingestion-order layout despite modifications |
Checkpoint 4 of 5· Check yourself
An unpartitioned 5 TB orders table gets constant MERGE upserts and is mostly queried by order_date. Which layout does the documentation recommend to keep the ingestion-order benefits?
A large number of UPDATE or MERGE statements breaks down ingestion time clustering. Clustering on a column that follows ingestion order keeps that layout.
“Databricks recommends using liquid clustering on a column that matches the ingestion order, such as an event timestamp or a creation date.”Source: docs.databricks.com
Checkpoint 5 of 5· Exam question
A pipeline runs frequent `MERGE INTO` statements on a Delta table with deletion vectors enabled to reconcile CDC updates. The team must also permanently purge rows removed for a GDPR right-to-erasure request before an upcoming audit. Select the two actions needed to correctly and completely remove that data from storage.(Select 2)
Correct answers: A, B — Run `REORG TABLE orders APPLY (PURGE)` to rewrite the affected files and materialize the pending deletion vector removals.; Run `VACUUM orders` after the purge completes to physically delete the now-unreferenced old data files from storage.
- A. `REORG TABLE ... APPLY (PURGE)` explicitly rewrites the data files to remove rows marked by deletion vectors, which is the step required before the underlying data can be considered physically gone.
- B. `VACUUM` physically removes old, unreferenced data files from cloud storage once they are no longer needed by any live table version, completing the erasure after the purge.
- C. Disabling deletion vectors changes behavior for future writes only; it does nothing to purge rows that are already marked as deleted in existing data files.
- D. Compaction has no strict guarantee of resolving every pending deletion vector marker, so running `OPTIMIZE` alone is not a reliable way to guarantee the rows are physically removed.
- E. Manually deleting Parquet files outside of Delta's transaction log breaks the table's consistency and history rather than safely purging the flagged rows.
- F. Lowering the retention duration only affects how long old log and file versions are kept for time travel; it does not purge or vacuum anything on its own without those commands actually running.
Sources2
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.With deletion vectors on, a DELETE removes the rows from storage right away.Why is that wrong?
Deletion vectors are soft deletes. The data stays in Parquet files until a rewrite (OPTIMIZE, or REORG ... APPLY (PURGE) for a guarantee) followed by VACUUM.
Covered in Soft deletes are not physical deletes
2.Enabling deletion vectors is a free switch with no effect on other readers.Why is that wrong?
It upgrades the table protocol, and clients without deletion vector support can no longer read the table until the feature is dropped.
Covered in Enabling deletion vectors and the compatibility cost
Practise it for real
Create a liquid-clustered Delta table, add a clustering dimension, apply it, and enable deletion vectors for update-heavy writes
1.Run: CREATE TABLE t(a int, b string) CLUSTER BY (a);
Why: Declares liquid clustering when the table is created, so it is never partitioned
You should see: The table is created with column a as its clustering key
2.Run: ALTER TABLE t CLUSTER BY (a, b);
Why: Adds a second clustering dimension without rewriting existing data
You should see: The clustering definition now lists a and b. Existing rows are unchanged
3.Run: OPTIMIZE t;
Why: Rows are clustered by altered keys only when OPTIMIZE runs
You should see: Data is incrementally clustered by the new keys
4.Run: ALTER TABLE t SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);
Why: Later DELETE, UPDATE and MERGE operations mark rows in metadata instead of rewriting files
You should see: The table protocol is upgraded and deletion vectors are on
Stuck? Get a nudge
If ALTER TABLE t CLUSTER BY (a, b) seems to have no effect on query speed, check whether you ran OPTIMIZE afterwards.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“reads apply the deletion vector entries at query time to resolve the current table state”
↩︎ The write problem deletion vectors solve“In Databricks Runtime 14.2 and above, tables with deletion vectors support row-level concurrency.”
↩︎ The write problem deletion vectors solve“For Delta Lake tables, you must explicitly enable deletion vectors.”
↩︎ Enabling deletion vectors and the compatibility cost“When you enable deletion vectors, Databricks upgrades the table protocol.”
↩︎ Enabling deletion vectors and the compatibility cost“File compaction events don't have strict guarantees for resolving changes recorded in deletion vectors.”
↩︎ Soft deletes are not physical deletes“The rows value limits the operation to handle only files with soft-deleted rows.”
↩︎ Soft deletes are not physical deletes“Deletion vectors mark changes to rows as soft-deletes that logically modify existing Parquet data files in the table.”
↩︎ Exam trap 1“When you enable deletion vectors, Databricks upgrades the table protocol.”
↩︎ Exam trap 2“Without deletion vectors, modifying a single row requires rewriting the entire Parquet file containing that record.”
↩︎ Prediction“Deletion vectors accelerate DELETE, UPDATE, and MERGE operations on Delta Lake and Apache Iceberg tables.”
↩︎ Checkpoint“Run VACUUM with the retention threshold set to the purge completion timestamp to physically remove files from previous table versions.”
↩︎ Checkpoint - 2.https://docs.databricks.com/aws/en/tables/clusteringOfficial docs
“Reduced write conflicts because tables with liquid clustering allow for row-level concurrency.”
↩︎ Matching the technique to the access pattern“Automatic performance improvements, using CLUSTER BY AUTO, for tables with frequently changing query patterns.”
↩︎ Matching the technique to the access pattern
Also cited
- https://docs.databricks.com/aws/en/tables/partitionsOfficial docs
“Databricks recommends using liquid clustering on a column that matches the ingestion order, such as an event timestamp or a creation date.”
↩︎ Checkpoint