What you will be able to do
- State the table-size thresholds Databricks gives for when partitioning helps, hurts, or should give way to liquid clustering
- Explain why an unpartitioned Delta table already has a useful layout through ingestion time clustering
- Apply the minimum-partition-size rule and the other partitioning constraints when partitioning is justified
Key concept
Liquid clustering as the default layout — Liquid clustering organizes a table's data files by clustering keys. Databricks positions it as the replacement for both Hive-style partitioning and ZORDER, so the size and query-pattern decisions in this lesson start from liquid clustering. You move away from it only when the evidence supports doing so.
1.Why partitioning stopped being the default
For years, partitioning was the main way to let a query skip data it did not need. A partitioned table splits its files into one directory per value of the partition column, so a filter on that column only reads the matching directories. The problem is that this layout is fixed. Databricks describes partitioning as static, something that "manifests itself as a filesystem hierarchy", and notes there is no easy way to change partitions when access patterns change.
The usual way it goes wrong is over-partitioning. If you choose a column with too many values, or a table that is too small to divide, you end up with many partitions, each holding a few small files. Query performance then gets worse, not better. Liquid clustering was built for this situation. It groups data by clustering keys without a directory hierarchy, and you can redefine those keys without rewriting the data already in the table.
Over-partitioning: you get too many partitions, each holding files that are too small, and query performance is poor. Partitioning a table of that size was not a good idea to begin with, as the next section shows.
Checkpoint 1 of 5· Check yourself
Your team's query patterns change and you want to change the layout keys of a table. What advantage does liquid clustering have over partitioning here?
Partitioning is static, so changing it means rewriting data. Liquid clustering lets you redefine the clustering keys without rewriting existing data.
“Unlike traditional partitioning, you can redefine clustering keys without rewriting existing data.”Source: docs.databricks.com
2.The size thresholds: 1 TB, 100 TB, and 1 GB per partition
Many people assume a big table should be partitioned. Databricks sets the bar much higher than that: most tables under 100 TB do not need partitioning at all. The guidance has three size bands, and the higher bands still say "try liquid clustering first".
Under 1 TB, do not partition. Between 1 TB and 100 TB, use liquid clustering instead, because in that range partitioning hurts performance more often than it helps. At 100 TB or more, partitioning might improve performance, but the recommendation is still to start with liquid clustering and verify whether partitioning actually does better.
There is also a second threshold that applies to each partition rather than the whole table: every partition should contain at least 1 GB of data. A large table can still be over-partitioned if you choose a column that divides it into many small pieces. Below these minimums, the docs say partitioning is likely to make queries slower.
| Table size | Partitioning guidance | Recommended approach |
|---|---|---|
| Less than 1 TB | Don't partition | Liquid clustering (or rely on ingestion time clustering) |
| More than 1 TB to 100 TB | Partitioning likely hurts more often than it helps | Liquid clustering instead of partitioning |
| 100 TB or more | Partitioning might improve performance | Liquid clustering first; verify any partitioning gain |
| Any partitioned table | Each partition should hold at least 1 GB | Prefer fewer, larger partitions |
Checkpoint 2 of 5· Check yourself
A 300 TB table is partitioned by a column that leaves most partitions with about 200 MB of data. Which statement matches the Databricks guidance?
At this table size partitioning is allowed, but each partition should still hold at least 1 GB. Partitions well below that minimum tend to slow queries down.
“For partitions, verify that each partition contains at least 1 GB of data.”Source: docs.databricks.com
Checkpoint 3 of 5· Exam question
According to current Databricks documentation, what is the maximum number of columns you can specify in a `CLUSTER BY` clause when enabling liquid clustering on a table?
Correct answer: C — Four
- A. Liquid clustering supports specifying more than two columns as clustering keys, so a two-column ceiling is not the documented limit.
- B. Three is below the documented ceiling; Databricks allows one additional column beyond this count in a `CLUSTER BY` clause.
- C. Databricks documents a maximum of four columns in a `CLUSTER BY` clause; beyond this, single-column filter performance can degrade, especially on smaller tables.
- D. Six columns exceeds the documented ceiling for clustering keys; Databricks caps this feature at a lower number of columns.
Sources3
3.What an unpartitioned table already gives you
Leaving a table unpartitioned does not leave it unorganized. Delta Lake automatically clusters unpartitioned tables by ingestion time. For queries that filter on time, the docs say this gives performance improvements similar to partitioning on a datetime field, without any manual optimizing or tuning. That is why most tables under 100 TB can skip partitioning: if the main query pattern is "recent data first", the default layout already handles it.
Ingestion time clustering can be disrupted when a table receives a lot of UPDATE or MERGE activity. For that case, Databricks recommends liquid clustering on a column that matches ingestion order, such as an event timestamp or a creation date. That keeps the time-ordered layout while the table is being modified heavily.
The cost of getting partitioning wrong is the strongest reason to start without it. An ineffective partitioning strategy can make queries slower, and fixing it requires a full rewrite of the data. For a large table, that rewrite can be very expensive and slow.
Checkpoint 4 of 5· Check yourself
An 800 GB unpartitioned Delta table is mostly queried for recent records and receives frequent MERGE operations. What do the docs recommend to keep its time-ordered layout?
Heavy UPDATE or MERGE activity can disrupt ingestion time clustering, so the docs recommend liquid clustering on an ingestion-ordered column. Partitioning is ruled out anyway, because the table is under 1 TB.
“Databricks recommends using liquid clustering on a column that matches the ingestion order”Source: docs.databricks.com
Sources3
4.If a very large table does get partitioned
Suppose a table is over 100 TB and you have measured that partitioning beats liquid clustering. Some habits from older Spark, Parquet, and Hive systems still do not apply on Databricks. You do not need partitions to make a batch atomic, because Delta Lake gets ACID guarantees from its transaction log, not from partition boundaries. Data locality is not a reason to partition either: the data lives in cloud object storage, and Databricks uses file-level statistics to decide which files to load.
There are also hard type limits. Partition columns must be top-level columns, so you cannot partition by a complex type or by a struct field. If you need to organize a table by a struct field, liquid clustering accepts struct fields as clustering keys. The docs call it the only way to skip data on a struct field without first extracting it into a top-level column.
Partitioning and ZORDER can be used together on large tables, with ZORDER applied inside each partition. Liquid clustering cannot be combined with either of them. A table uses either clustering or partitioning plus ZORDER, never both.
Checkpoint 5 of 5· Check yourself
Queries on a 150 TB table filter on device.region, a field inside a struct column. Which layout lets Databricks skip data on that field directly?
Struct fields cannot be partition columns, and Delta treats them as expressions in PARTITIONED BY. Liquid clustering accepts a struct field as a clustering key.
“Liquid clustering is the only way to data-skip on a struct field without first extracting it into a top-level column.”Source: docs.databricks.com
No. Clustering is not compatible with partitioning or ZORDER. A partitioned table has to be converted to liquid clustering, not layered with it.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Any table over 1 TB should be partitioned, and the bigger the table, the more partitioning helps.Why is that wrong?
Between 1 TB and 100 TB, Databricks says to use liquid clustering instead. Even at 100 TB or more, you start with liquid clustering and only partition if you can show it performs better.
Covered in The size thresholds: 1 TB, 100 TB, and 1 GB per partition
2.An unpartitioned Delta table has no data layout, so queries on recent data always scan everything.Why is that wrong?
Unpartitioned Delta tables are automatically clustered by ingestion time. For time-based filters this performs similarly to partitioning on a datetime field.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/lakehouse-architecture/performance-efficiency/best-practicesOfficial docs
“partitioning is static and manifests itself as a filesystem hierarchy”
↩︎ Why partitioning stopped being the default“Often, partitioning leads to over-partitioning - in other words, too many partitions with too small files, resulting in poor query performance.”
↩︎ Why partitioning stopped being the default - 2.https://docs.databricks.com/aws/en/tables/clusteringOfficial docs
“Unlike traditional partitioning, you can redefine clustering keys without rewriting existing data.”
↩︎ Why partitioning stopped being the default“Clustering is not compatible with partitioning or ZORDER.”
↩︎ If a very large table does get partitioned“Liquid clustering is a data layout optimization technique that replaces table partitioning and ZORDER.”
↩︎ Key concept - 3.https://docs.databricks.com/aws/en/tables/partitionsOfficial docs
“Most tables on Databricks with less than 100 TB of data don't need partitioning.”
↩︎ The size thresholds: 1 TB, 100 TB, and 1 GB per partition“With 100 TB or more of data, partitioning might improve performance, but Databricks recommends using liquid clustering first and verifying performance improvements.”
↩︎ The size thresholds: 1 TB, 100 TB, and 1 GB per partition“Partitioning below these minimum sizes is likely to negatively affect query performance rather than improve it.”
↩︎ The size thresholds: 1 TB, 100 TB, and 1 GB per partition“Ingestion time has query performance improvements similar to partitioning strategies with datetime fields, without any need to manually optimize or tune your data.”
↩︎ What an unpartitioned table already gives you“An ineffective partitioning strategy might negatively affect query performance and require a full rewrite of data to fix.”
↩︎ What an unpartitioned table already gives you“Transactions are not defined by partition boundaries.”
↩︎ If a very large table does get partitioned“You can use Z-order indexes alongside partitions to speed up queries on large datasets.”
↩︎ If a very large table does get partitioned“With more than 1 TB to 100 TB of data, use liquid clustering instead of partitioning.”
↩︎ Exam trap 1“By using Delta Lake, unpartitioned tables automatically use ingestion time clustering.”
↩︎ Exam trap 2“With more than 1 TB to 100 TB of data, use liquid clustering instead of partitioning.”
↩︎ Prediction“For partitions, verify that each partition contains at least 1 GB of data.”
↩︎ Checkpoint“Databricks recommends using liquid clustering on a column that matches the ingestion order”
↩︎ Checkpoint“Liquid clustering is the only way to data-skip on a struct field without first extracting it into a top-level column.”
↩︎ Checkpoint