Disk cache (formerly Delta cache): faster repeated reads on Databricks

1.Why repeated reads get faster: copying remote files to local SSD

On Databricks, Delta tables are stored in cloud object storage such as S3 or ABFS. Every scan of a table means fetching Parquet files over the network, and for a dashboard, an interactive analyst session or an iterative job that reads the same table again and again, that remote fetch is paid on every read. The disk cache removes the repeated cost. The first time a worker has to fetch a file from remote storage, Databricks also writes a copy of it to the worker's local disk in a fast intermediate format. The next query that needs the same data reads that local copy instead of going back to object storage.

Three facts in the documentation define the feature. First, it works by creating copies of remote Parquet data files in nodes' local storage using a fast intermediate data format. Second, it is automatic: data is cached whenever a file has to be fetched from a remote location, and you don't call anything to start it. Third, it covers all Parquet data files, including Delta Lake tables. The exam guide still says "Delta cache", but current documentation calls it the disk cache. Databricks says the feature was formerly referred to as the Delta cache and the DBIO cache, and that the disk cache is a proprietary Databricks feature, not part of the Delta Lake protocol. If an exam question asks about the "Delta cache", read it as the disk cache.

Sources1

2.Disk cache vs Apache Spark cache

Exam questions often mix the disk cache up with Spark's own caching (.cache(), .persist(), CACHE TABLE). They solve different problems. The disk cache works on Parquet files and needs no code. The Spark cache holds a DataFrame or RDD that you choose to persist, usually in memory, and you have to request it in code. The documentation compares them directly:

The trade-off is coverage against control. The Spark cache can hold things the disk cache cannot, such as the result of an arbitrary subquery or data in CSV, JSON or ORC. The disk cache stores raw file data, so it can't store the results of arbitrary subqueries. Databricks' performance guidance still favours the disk cache: it recommends automatic disk caching and, as a rule of thumb, avoiding Spark caching, because badly placed persists can use up memory and slow queries down. CACHE TABLE is not compatible with serverless compute. If you need to keep an intermediate result, the guidance is to write it to a Delta table instead.

Sources234

3.Staying correct: invalidation, eviction and restarts

A cache only helps if it never returns wrong data. The disk cache handles correctness for you: it automatically detects when data files are created, deleted, modified or overwritten, and updates its contents to match. This fits naturally with Delta Lake, because Delta writes new files rather than editing files in place, so a changed table shows up as changed files. That is why the comparison table lists "on any file change" as an eviction trigger. Space is managed the same way: when the cache fills, older entries are evicted in least-recently-used (LRU) order.

The cache lives on the cluster's local disks, so it lasts only as long as the cluster. The SQL caching documentation says the disk cache shares the lifecycle of the local result cache: when the cluster is stopped or restarted, the cache is cleaned and has to be filled again. So the first queries after a restart are "cold" and read from remote storage, and later repeated reads get fast again as the cache refills. Turning the cache off with the configuration flag behaves differently. It doesn't delete data already on local storage; it stops queries from adding to the cache or reading from it.

Sources14

4.Getting the disk cache: worker types, flags and CACHE SELECT

Because the cache writes to local SSDs, the hardware you choose matters most. Databricks recommends choosing a worker type with SSD volumes, ideally a cache-accelerated instance type. These workers come already enabled and configured for disk caching, so you don't need to set anything. By default the cache uses at most half of the space on the workers' local SSDs. If you need to change that, you set Spark configuration options when you create the cluster:

spark.databricks.io.cache.maxDiskUsage 50g
spark.databricks.io.cache.maxMetaDataCache 1g
spark.databricks.io.cache.compression.enabled false

spark.conf.set("spark.databricks.io.cache. ? ", "[true | false]")

Older material teaches you to warm the cache ahead of time with CACHE SELECT, which copies a chosen set of columns and rows of a Delta or Parquet table into the disk cache. Don't rely on it on current compute. On SQL warehouses and on Databricks Runtime 14.2 and above, CACHE SELECT is ignored and an improved automatic caching algorithm is used instead. On serverless compute, caching is tuned automatically. The current answer to "how do I make repeated reads fast?" is to use SSD-backed or serverless compute and let the automatic cache work.

Sources15

5.Where the disk cache sits among Databricks SQL caches

SQL warehouses have several cache layers, and exam questions often test whether you can tell which one is meant. The disk cache is the data-level layer: local SSD caching for data read from storage. It speeds up any query that reads the same files, even when the SQL text is different. The result cache works one level up. It stores the result of a whole query, per cluster and optionally in a remote cache on serverless, and returns it when the same query runs again on unchanged data. Both caches are invalidated when the underlying data changes. The difference is scope: a new aggregation over yesterday's files can't use the result cache, but it can still read those files from the disk cache.

Sources4

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.The Delta cache is part of the open-source Delta Lake protocol, so any Delta reader gets it.Why is that wrong?

   The disk cache (formerly Delta cache or DBIO cache) is a proprietary Databricks feature. It works on any Parquet files, including Delta tables, and was renamed to clear up this exact confusion.

   Covered in Why repeated reads get faster: copying remote files to local SSD

2. 2.After an UPDATE, DELETE or MERGE you have to refresh or clear the disk cache, or queries will return stale data.Why is that wrong?

   The disk cache detects created, deleted, modified or overwritten files and evicts stale entries automatically. Writers don't need to invalidate anything.

   Covered in Staying correct: invalidation, eviction and restarts

3. 3.To get fast repeated reads on a SQL warehouse, you should pre-warm the cache with CACHE SELECT.Why is that wrong?

   On SQL warehouses and Databricks Runtime 14.2 and above, CACHE SELECT is ignored and an improved automatic caching algorithm is used instead.

   Covered in Getting the disk cache: worker types, flags and CACHE SELECT

4. 4.Calling .cache() or CACHE TABLE is the recommended way to speed up repeated reads of a Delta table.Why is that wrong?

   That is the Spark cache, which is manual, uses memory and isn't compatible with serverless. Databricks recommends the automatic disk cache and, as a rule of thumb, avoiding Spark caching.

   Covered in Disk cache vs Apache Spark cache