What you will be able to do
- Use the termination error code in the cluster event log to place a startup failure in a category
- Pick the right fix for cloud capacity, subnet IP, quota and spot instance failures
- Recognise bootstrap and network failures and test connectivity from the workspace subnet
- Isolate SPARK_STARTUP_FAILURE and init script problems using init script events and logs
Key concept
Termination error code in the cluster event log — A classic cluster that fails to start is terminated, and the reason is written to its event log as an error code. Each code points to one cause, so you start diagnosing from the code, not from guesses about the workload.
1.Start with the event log, not the workload
A cluster that never reaches the running state hasn't run any of your code, so a slow query or a bad join isn't the cause. Databricks terminates the cluster and records why in the cluster event log. The termination error codes fall into a few groups, and each group has a different owner. Some are cloud-provider problems: no capacity, no free IP addresses, a quota reached, or a spot instance taken back. Some are network problems that stop the VM from bootstrapping. Some are Spark startup problems caused by your own configuration or init scripts. Others are workspace configuration problems, such as an IAM role the control plane can't assume. A single code can still have more than one possible cause. A bootstrap timeout, for example, can come from network connectivity, slow artifact downloads or the cloud provider. The troubleshooting steps listed for each code are what narrow it down.
The event log is also the first place to look when a new workspace seems to work but its first test cluster never responds. The troubleshooting guide says to wait roughly 20–30 minutes and then check the compute event log. The message you find there tells you which group you're in. For the workspace-configuration group, a typical message is an IAM user that is not authorized to perform sts:AssumeRole on the workspace role. The fix is to verify IAM role trust relationships and instance profile permissions.
Because the fixes are mostly different from group to group. A capacity stockout is fixed by changing availability zone or instance type. A bootstrap timeout is fixed in firewall, DNS or routing. A Spark startup failure is fixed by removing custom Spark configs and init scripts. Some remedies do appear under more than one code, such as trying another availability zone or instance type, so the group tells you which fix to try first. Applying a fix from the wrong group wastes a restart cycle.
Checkpoint 1 of 5· Check yourself
A cluster launch fails with a message saying a user is not authorized to perform sts:AssumeRole on the databricks-workspace-stack-role. What should you check first?
This is a workspace-level misconfiguration. The recommended fix is to verify the IAM trust relationship and instance profile permissions.
“Verify IAM role trust relationships and instance profile permissions.”Source: docs.databricks.com
Sources1
2.Cloud-side failures: capacity, IPs, quotas and spot
The most common startup failures come from the cloud provider not handing over the VMs Databricks asked for. In each case the AWS error text appears in the event log, and the fix changes where or what you ask for. It doesn't touch your Spark code. If the subnet runs out of addresses, you see Not enough free addresses in subnet with AWS error code InvalidParameterValue. If AWS has no capacity for the instance type in your zone, you see InsufficientInstanceCapacity. A quota failure means the launch would push the AWS account past a Service Quotas limit.
| Symptom in the event log | Recommended fix |
|---|---|
| Not enough free addresses in subnet (InvalidParameterValue) | Use a different availability zone or the auto availability zone setting, expand the subnet CIDR range, or clean up unused network resources |
| InsufficientInstanceCapacity / no Spot capacity available | Try another availability zone or instance type, or use on-demand instead of spot; ask an admin to enable flexible node types |
| Launch would exceed the AWS account's quota | Request a quota increase in the AWS Service Quotas console, clean up unused resources, or spread workloads across regions |
| Server.SpotInstanceTermination | Use on-demand instances for production, add job retry logic, or mix on-demand and spot instances |
| CLOUD_PROVIDER_LAUNCH_FAILURE (Server.InternalError) | Try again later; most of these are transient |
| CLUSTER_OPERATION_TIMEOUT (SERVICE_FAULT) | Retry the job or cluster launch; contact Databricks support if it recurs across many runs |
Two details are easy to miss. First, flexible node types let Databricks fall back automatically to a compatible instance type when your preferred type is stocked out, but they don't apply to GPU instance types. Second, CLUSTER_OPERATION_TIMEOUT is documented as a Databricks- or cloud-side problem, not a problem with your workload, and it usually clears on retry. Spot instances are described as best suited to fault-tolerant workloads.
Not every startup deadline is a capacity problem, though. The Databricks knowledge base describes messages such as Driver failed to start in time and The Spark driver failed to start within 300 seconds. One documented cause is a cluster connected to an external Hive metastore that downloads all the Hive metastore libraries from a Maven repo. That is almost 200 JAR files, including dependencies. If the cluster manager cannot confirm the driver is ready within 5 minutes, the launch fails, and the article says this can happen because JAR downloading takes too much time. The documented fix is to store the Hive libraries in DBFS and access them locally from there. So a large set of JARs fetched from an external Maven repo during startup can make the driver miss its readiness deadline. That is a slow-download cause, separate from capacity, IP and quota failures. It is also separate from the configuration and init script causes covered in the Spark startup section.
Checkpoint 2 of 5· Match them up
Match each event-log symptom to its recommended fix
Tap a term, then the definition that fits it.
Each cloud-side failure has its own remedy. Subnet exhaustion is about IP space, capacity is about instance type and zone, quotas are raised in AWS, and spot losses are handled with on-demand instances and retries.
“Update your cluster to use a different availability zone with sufficient IP addresses, use the auto availability zone setting”Source: docs.databricks.com
Checkpoint 3 of 5· Exam question
A data engineering team creates an all-purpose cluster that references a large custom JAR hosted on an external Maven repository as a cluster library. The cluster consistently fails to launch, and the event log shows the driver never reported readiness within the initialization window before the launch was aborted. What is the most likely cause of this failure?
Correct answer: A — A large custom JAR takes too long to download from the external repository, so the driver cannot finish initializing before the startup deadline.
- A. Databricks enforces a bounded window for the driver to become ready during cluster launch, and downloading a large JAR from an external repository over the network can easily exceed it. When that download has not finished by the deadline, the launch is aborted with exactly this kind of timeout, making this the most likely explanation.
- B. A minimum worker count of zero simply means the cluster can scale down to no executors when idle; it does not prevent the driver node from launching. This scenario describes the driver itself failing to become ready, which autoscaling minimums do not affect.
- C. If outbound HTTPS were fully blocked, the JAR download and most other startup dependencies would fail immediately with a connectivity error rather than a readiness timeout. The described symptom of the driver simply never reporting ready points to a slow download, not a hard network block.
- D. An unsupported runtime version is typically rejected at cluster configuration time with a validation error before any launch attempt begins. It would not produce a driver that starts launching and then times out waiting for a large library download.
3.Bootstrap and network failures
Sometimes AWS provides the VM, but the VM can't finish bootstrapping. Bootstrap has a fixed deadline of 700 seconds. Common reasons for missing it are network connectivity problems, slow artifact downloads, or a cloud-provider issue. Related errors include failing to download the Spark container image from Databricks artifact storage, and Command_UpdateWorker failures that show a DNS probe to a storage URL. The fix is in the network: firewall rules, DNS resolution and routes to Databricks storage and the control plane. The docs also recommend VPC endpoints for S3, so that artifact downloads don't go over the public internet. One exception: if the event log says the artifact servers are under load and throttling requests, the fix is just to retry after a short delay.
On a newly deployed workspace, the classic symptom is Node daemon ping timeout in 600000 ms. The instances started but couldn't reach the control plane, so the compute manager terminated them. The 700-second bootstrap timeout comes from the termination error codes page. The ping timeout message comes from the separate troubleshooting guide. The docs don't map one message onto the other. Both describe a node that can't finish bootstrapping and connecting, and both point you at routing, firewall, DNS and connectivity. To test faster than by launching a cluster, deploy an EC2 instance into one of the workspace subnets and test each required endpoint from there.
# Verify access to the secure compute connectivity relay
nc -zv tunnel.eu-west-1.cloud.databricks.com 443
# Verify S3 global and regional access
nc -zv s3.amazonaws.com 443
nc -zv s3.eu-west-1.amazonaws.com 443
# Verify STS global and regional access
nc -zv sts.amazonaws.com 443
nc -zv sts.eu-west-1.amazonaws.com 443If every port check succeeds, the network paths may be fine and a firewall could still be the problem. A firewall with deep packet inspection or SSL inspection can make Databricks commands fail. From the same EC2 instance, call the workspace's /api/2.0/clusters/spark-versions endpoint with curl and a personal access token. If the request fails, repeat it with the -k option, which removes SSL verification. If it works with -k, the firewall is causing an issue with SSL certificates. Then inspect the certificates with openssl s_client, as below. An error means the firewall is misconfigured and must be fixed. SSL issues are not a networking-layer issue, so viewing traffic at the firewall won't show them. Source and destination requests look normal.
openssl s_client -showcerts -connect oregon.cloud.databricks.com:443STS shows up in that list for a reason. Compute nodes use STS to reach the workspace's root S3 bucket. If the logs say failed to launch spark container on instance and Could not add container, the likely cause is that STS was blocked by exfiltration protection, VPC endpoints or a new firewall. Allow the global endpoint sts.amazonaws.com, or set up a regional STS VPC endpoint. A different symptom has another network cause. The cluster starts, but the event log shows METASTORE_DOWN events. In that case, check whether traffic goes through a Web Application Firewall such as a Squid proxy, because some services that compute nodes need don't work through a WAF.
Checkpoint 4 of 5· Check yourself
After a firewall is added, clusters fail with 'failed to launch spark container on instance … Could not add container'. What is the most likely cause?
The troubleshooting guide links this exact log message to STS being unreachable after exfiltration protection, VPC endpoints or a firewall is added. Allowing the STS endpoint fixes it.
“This compute log error is likely due to the instance not being able to use STS to get into the root S3 bucket.”Source: docs.databricks.com
4.Spark startup failures and init scripts
SPARK_STARTUP_FAILURE means the driver daemon didn't start within its timeout, typically 200 seconds. The troubleshooting steps point at things you control. Check Spark configuration for mistakes such as an invalid metastore URI or conflicting settings, and check init scripts for errors that could delay or block driver startup. If removing those doesn't help, try a different instance type, because slow hardware on smaller instances can also cause driver startup timeouts. A related runtime error says the cluster could not find an internal Spark image. In that case, select a supported Databricks Runtime version, or check that the custom Spark image exists.
To see whether an init script is the problem, look in two places. The cluster event log records INIT_SCRIPTS_STARTED and INIT_SCRIPTS_FINISHED. These events show which scripts were scheduled and which have completed successfully, and INIT_SCRIPTS_FINISHED also records how long they ran. The key global marks global init scripts, and cluster marks cluster-scoped ones. The event log covers one representative node only, so you need the script output too. With cluster log delivery configured, init script logs go to /<cluster-log-path>/<cluster-id>/init_scripts, with a subdirectory for each container. Each container's folder holds a .stdout.log and a .stderr.log file per script. Without log delivery, they are written to /databricks/init_scripts on the cluster.
%sh
ls /databricks/init_scripts/
cat /databricks/init_scripts/<timestamp>_<log-id>_<init-script-name>.sh.stdout.logCheckpoint 5 of 5· Check yourself
The cluster event log shows INIT_SCRIPTS_FINISHED for a cluster-scoped script. Can you conclude that the script succeeded on every worker?
Init script events in the event log come from one representative node. Per-container stdout and stderr logs show what happened on each node.
“Cluster event logs do not log init script events for each cluster node; only one node is selected to represent them all.”Source: docs.databricks.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.SPARK_STARTUP_FAILURE means AWS couldn't supply the instances, so you should change zone or switch to on-demand.Why is that wrong?
The VMs were provisioned. The driver daemon didn't start in time, so first remove custom Spark configs and init scripts to isolate the cause.
Covered in Spark startup failures and init scripts
2.Enabling flexible node types prevents stockout errors for every instance type, including GPU clusters.Why is that wrong?
Flexible node types fall back to compatible instance types, but they don't apply to GPU instance types.
Covered in Cloud-side failures: capacity, IPs, quotas and spot
3.Init script events in the cluster event log show the result on every node.Why is that wrong?
One node is selected to represent all of them. The per-container init script logs show the result on each node.
Covered in Spark startup failures and init scripts
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Use the error code from your cluster event log to find the relevant cause and recommended fix.”
↩︎ Start with the event log, not the workload“Workspace-level misconfiguration is preventing cluster launch, including issues with IAM roles or service principal permissions.”
↩︎ Start with the event log, not the workload“Request a quota increase through the AWS Service Quotas console”
↩︎ Cloud-side failures: capacity, IPs, quotas and spot“Switch to on-demand instances for production workloads, implement job retry logic to handle interruptions, or use a mix of on-demand and spot instances.”
↩︎ Cloud-side failures: capacity, IPs, quotas and spot“Try again later, as most cloud provider launch failures are transient.”
↩︎ Cloud-side failures: capacity, IPs, quotas and spot“The bootstrap timeout is 700 seconds.”
↩︎ Bootstrap and network failures“Try a different instance type, as hardware slowness on smaller instances can cause driver startup timeouts.”
↩︎ Spark startup failures and init scripts“Select a supported Databricks Runtime version or verify the custom Spark image exists.”
↩︎ Spark startup failures and init scripts“Use the error code from your cluster event log to find the relevant cause and recommended fix.”
↩︎ Key concept“Review Spark configuration for misconfigurations (for example, invalid metastore URI or conflicting settings).”
↩︎ Exam trap 1“Flexible node types are not available for GPU instance types.”
↩︎ Exam trap 2“Verify IAM role trust relationships and instance profile permissions.”
↩︎ Checkpoint“Update your cluster to use a different availability zone with sufficient IP addresses, use the auto availability zone setting”
↩︎ Checkpoint“Remove custom Spark configs and init scripts to isolate the issue.”
↩︎ Prediction - 2.https://kb.databricks.com/clusters/cluster-failed-launchSecondary source
“If the Databricks cluster manager cannot confirm that the driver is ready within 5 minutes, then cluster launch fails.”
↩︎ Cloud-side failures: capacity, IPs, quotas and spot“This can occur because JAR downloading is taking too much time.”
↩︎ Cloud-side failures: capacity, IPs, quotas and spot - 3.
“This message indicates that the routing or the firewall is incorrect.”
↩︎ Bootstrap and network failures“If this works with the -k option, then the firewall is causing an issue with SSL certificates.”
↩︎ Bootstrap and network failures“Compute members must connect to several services that don't work over a WAF.”
↩︎ Bootstrap and network failures“This compute log error is likely due to the instance not being able to use STS to get into the root S3 bucket.”
↩︎ Checkpoint - 4.https://docs.databricks.com/aws/en/init-scripts/logsOfficial docs
“When cluster log delivery is not configured, logs are written to /databricks/init_scripts.”
↩︎ Spark startup failures and init scripts“Cluster event logs do not log init script events for each cluster node; only one node is selected to represent them all.”
↩︎ Exam trap 3“Cluster event logs do not log init script events for each cluster node; only one node is selected to represent them all.”
↩︎ Checkpoint