What you will be able to do
- Find the longest stage of a Spark job and read its Input, Output, Shuffle Read and Shuffle Write columns
- Recognise disk spill from a stage's details and explain why it is most common during shuffles
- Diagnose data skew by comparing Max task duration with the 75th percentile in Summary Metrics
Key concept
Data skew (Max vs. 75th percentile) — Skew means one or a few tasks in a stage run much longer than the others, so most of the cluster sits idle while it waits for them. You look for it in the stage's Summary Metrics: when the Max task duration is well above the 75th percentile duration, the stage may be suffering from skew.
1.Start with the longest stage and its shuffle columns
Databricks' Spark UI guide sets out a fixed order for diagnosing a slow job. First, use the Jobs Timeline to find major issues. Next, go to the longest stage and check it for skew or spill. Then decide whether that stage is I/O bound. Only after that do you look for other causes. The order matters. A stage-level bottleneck only explains a slow job if it sits in the stage that takes most of the time, so you find that stage before you diagnose anything.
Checkpoint 1 of 7· Check yourself
A job has several stages. Following the Databricks guide, which stage do you open first to look for skew or spill?
The guide starts by identifying the longest stage, because a bottleneck only explains a slow job if it is in the stage that takes most of the time. You then open that stage's page to look for skew and spill.
“Start by identifying the longest stage of the job.”Source: docs.databricks.com
To find the longest stage, scroll to the list of stages at the bottom of the job's page and sort it by duration. Each stage row has four columns that summarise its data movement. Two of them, Input and Output, measure reads from and writes to storage. The other two, Shuffle Read and Shuffle Write, measure data exchanged between tasks during a shuffle. Write these numbers down, because the later checks rely on them.
A shuffle is Spark's mechanism for re-distributing data so that it is grouped differently across partitions. It typically means copying data across executors and machines. That is why a shuffle is costly: it involves disk I/O, data serialization and network I/O. Large Shuffle Read and Shuffle Write values therefore mark a stage that spends much of its time moving data, and a stage with large shuffle columns is where spill is most likely to show up.
| Column | What it measures | Examples of what it covers |
|---|---|---|
| Input | How much data the stage read from storage | Reading from Delta, Parquet, CSV |
| Output | How much data the stage wrote to storage | Writing to Delta, Parquet, CSV |
| Shuffle Read | How much shuffle data the stage read | Data pulled from other tasks after an exchange |
| Shuffle Write | How much shuffle data the stage wrote | Data written out for a later stage to read |
Also look at the stage's task count. If a long stage has only one task, that is a separate problem and is covered with the no-skew, no-spill causes. If it has more than one task, click the stage's description to open its stage page. The skew and spill evidence is on that page.
Checkpoint 2 of 7· Match them up
Match each stage-list column to what it reports.
Tap a term, then the definition that fits it.
Input and Output measure storage I/O. The two Shuffle columns measure data exchanged during a shuffle, not reads from or writes to tables.
“Shuffle Read: How much shuffle data this stage read.”Source: docs.databricks.com
2.Spill: when execution memory runs out
Spark uses execution memory for shuffles, joins, sorts and aggregations. When that memory runs low, Spark moves data from memory to disk. This is called spill, and it is expensive because disk is much slower than memory. Spill is most common during shuffles, which is why the shuffle columns you just noted point you to it.
On the stage page, spill shows up in the details at the top, next to metrics such as Peak Execution Memory. If no spill statistics appear, the stage had no spill. If you open an operator in the SQL DAG, it also reports metrics such as peak memory and spill size. Because spill is mostly a shuffle problem, the guide sends stages with spill to its guidance on dealing with spill caused by shuffle.
One cause is the number of shuffle partitions. If there are too few, each partition can be very large, and the tasks that process these large partitions may need to spill data to disk, for example when a sort or aggregate is involved. That slows the query down.
Checkpoint 3 of 7· Check yourself
A stage's details at the top of its stage page show Peak Execution Memory but no spill statistics. What does that tell you?
If spill statistics are missing, there was no spill. You can move straight on to checking for skew.
“If you don't see any stats for spill, that means the stage doesn't have any spill.”Source: docs.databricks.com
Checkpoint 4 of 7· Exam question
A data engineer is troubleshooting a slow stage that performs a large sort-merge join. In the stage detail page of the Spark UI, the Summary Metrics show non-zero values for both `Spill (Memory)` and `Spill (Disk)`, and the stage takes far longer than similar joins on smaller datasets. What is the most likely cause of this behavior?
Correct answer: A — The executors ran low on execution memory during the join, forcing Spark to write intermediate sort data to disk before continuing.
- A. This is correct. `Spill (Memory)` and `Spill (Disk)` both being non-zero is the Spark UI's direct signal that execution memory ran out during the sort-merge join, so Spark serialized intermediate data to local disk to keep processing, which adds significant I/O overhead.
- B. A data type mismatch in the join condition would typically cause a query planning error or an implicit cast, not the specific spill metrics shown here; spill is tied to memory pressure, not to comparison logic.
- C. Missing predicate pushdown increases the volume of data scanned before the join even starts, but it does not directly produce the `Spill (Memory)`/`Spill (Disk)` metrics, which reflect memory exhaustion during the join's sort step itself.
- D. A cluster stuck at a fixed size would limit available parallelism and resources, but the specific spill metrics point to memory being exhausted during processing rather than to a shortage of worker nodes.
3.Skew: reading Max against the 75th percentile
Next, check for skew. A skewed stage has one or a few tasks that run much longer than the rest. The stage can't finish until its slowest task finishes, so the other executors sit idle. The result is poor cluster utilization and longer jobs.
Skew comes from the data. Data skew occurs when data is unevenly distributed among partitions, and severe skew can significantly hurt query performance, especially in joins. A task that has to process an oversized partition runs much longer than the tasks that process small ones.
The skew evidence in the guide is in the stage page's Summary Metrics table. Compare the Max duration with the 75th percentile duration. The guide's screenshot of a healthy stage shows the 75th percentile and Max the same. Its rule of thumb: if Max is 50% more than the 75th percentile, you may be suffering from skew. For example, if the 75th percentile is 20 seconds and Max is 4 minutes, a few tasks are doing far more work than the others. If the stage shows neither skew nor spill, go back to the job page and look at other causes.
For remediation, the guide points to its skew remediation steps. Adaptive query execution (AQE) is also on by default and handles skew in sort merge joins and shuffle hash joins by splitting skewed tasks into roughly evenly sized tasks. AQE treats a partition as skewed when it is larger than the skewed-partition factor (default 5) times the median partition size and larger than the skewed-partition threshold (default 256MB).
| What you see on the stage page | Bottleneck it indicates |
|---|---|
| Spill stats at the top of the stage page | Spark ran low on execution memory and moved data to disk |
| No spill stats | The stage has no spill |
| Max duration much higher than the 75th percentile in Summary Metrics | Possible skew: one or a few tasks take much longer than the rest |
Checkpoint 5 of 7· Check yourself
In a stage's Summary Metrics, the 75th percentile task duration is 40 s. Which Max duration first meets the guide's rule of thumb for possible skew?
The rule compares Max with the 75th percentile, not with the median or average. 50% more than 40 s is 60 s.
“If the Max duration is 50% more than the 75th percentile, you may be suffering from skew.”Source: docs.databricks.com
Checkpoint 6 of 7· Exam question
A data engineer notices that one stage in a Spark job takes far longer to finish than the surrounding stages. In the Spark UI's Stages tab, the stage detail page shows a Max task duration of 12 minutes while the 75th percentile task duration is only 4 minutes, and most tasks finish within 3-5 minutes. Which condition does this metric pattern most strongly indicate?
Correct answer: A — Data skew: a small number of partitions hold disproportionately more records, so a few tasks run far longer than their peers.
- A. This is correct. A Max duration far above the 75th percentile is the classic Spark UI signature of skew: most tasks finish quickly because their partitions are small, while a handful of tasks carry much larger partitions and drag out the stage.
- B. Spilling would typically show up as non-zero `Spill (Memory)`/`Spill (Disk)` metrics affecting many or all tasks fairly evenly, not as a gap between the max and 75th-percentile task duration for a few outlier tasks.
- C. A broadcast join sends a full copy of the smaller table to every executor, which would inflate memory usage and duration roughly uniformly across tasks rather than producing a small number of unusually long-running tasks.
- D. Slow autoscaling would delay the start of the stage or reduce overall parallelism, but it would not produce the specific pattern of a few tasks running dramatically longer than the 75th-percentile task in the same stage.
Checkpoint 7 of 7· Exam question
While reviewing a stage that performs a wide `groupBy` aggregation, a data engineer sees in the Spark UI that shuffle read size varies dramatically across tasks: most tasks read 2-3 MB of shuffle data, but a handful of tasks read over 500 MB each. What does this pattern most likely indicate?
Correct answer: A — A small set of grouping keys receive a disproportionate share of records, so their shuffle partitions grow much larger than the rest.
- A. This is correct. During a shuffle, Spark assigns records to partitions by key, so when a few keys are heavily overrepresented, the partitions containing them accumulate far more shuffle data than the rest, producing exactly this kind of read-size imbalance.
- B. Setting the shuffle partition count too high spreads data across more, smaller partitions rather than causing a subset of tasks to pull hundreds of times more data than others; it does not cause redundant reads of the same blocks.
- C. An inefficient upstream file format would raise read costs fairly uniformly across all tasks reading that data, not create a sharp divide between a handful of tasks reading 500 MB and the rest reading only a few megabytes.
- D. Uneven network throttling is not tied to key distribution and would not explain why the same specific tasks consistently read far more shuffle data on repeated runs; key skew is the more direct explanation for this pattern.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.On a slow stage, check for skew first. Spill is a secondary check.Why is that wrong?
The Databricks guide checks spill first on the stage page and then moves on to skew.
Covered in Spill: when execution memory runs out
2.Skew means the Max task duration is far above the average or median duration.Why is that wrong?
The guide's Summary Metrics test compares Max with the 75th percentile duration.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Use the Jobs Timeline to identify major issues”
↩︎ Start with the longest stage and its shuffle columns - 2.
“Start by identifying the longest stage of the job.”
↩︎ Start with the longest stage and its shuffle columns“If you see one task, that could be a sign of a problem.”
↩︎ Start with the longest stage and its shuffle columns“Shuffle Read: How much shuffle data this stage read.”
↩︎ Checkpoint - 3.https://spark.apache.org/docs/latest/rdd-programming-guide.htmlSecondary source
“The Shuffle is an expensive operation since it involves disk I/O, data serialization, and network I/O.”
↩︎ Start with the longest stage and its shuffle columns - 4.
“Spill is what happens when Spark runs low on execution memory.”
↩︎ Spill: when execution memory runs out“It is most common during data shuffling.”
↩︎ Spill: when execution memory runs out“The stage details also include other metrics, such as Peak Execution Memory.”
↩︎ Spill: when execution memory runs out“Skew is when one or just a few tasks take much longer than the rest.”
↩︎ Skew: reading Max against the 75th percentile“This results in poor cluster utilization and longer jobs.”
↩︎ Skew: reading Max against the 75th percentile“If the Max duration is 50% more than the 75th percentile, you may be suffering from skew.”
↩︎ Key concept“The first thing to look for in a long-running stage is whether there's spill.”
↩︎ Exam trap 1“The main thing we're looking for is the Max duration being much higher than the 75th percentile duration.”
↩︎ Exam trap 2“The first thing to look for in a long-running stage is whether there's spill.”
↩︎ Prediction“If you don't see any stats for spill, that means the stage doesn't have any spill.”
↩︎ Checkpoint“If the Max duration is 50% more than the 75th percentile, you may be suffering from skew.”
↩︎ Checkpoint - 5.https://docs.databricks.com/aws/en/optimizations/spark-ui-guide/slow-spark-stage-low-ioOfficial docs
“When you open an operator, it shows metrics such as number of output rows, peak memory, and spill size.”
↩︎ Spill: when execution memory runs out - 6.https://www.databricks.com/blog/2020/05/29/adaptive-query-execution-speeding-up-spark-sql-at-runtime.htmlSecondary source
“the tasks to process these large partitions may need to spill data to disk”
↩︎ Spill: when execution memory runs out“Data skew occurs when data is unevenly distributed among partitions in the cluster.”
↩︎ Skew: reading Max against the 75th percentile - 7.https://docs.databricks.com/aws/en/optimizations/aqeOfficial docs
“Dynamically handles skew in sort merge join and shuffle hash join by splitting (and replicating if needed) skewed tasks into roughly evenly sized tasks.”
↩︎ Skew: reading Max against the 75th percentile