What you will be able to do
- Decide whether a workload calls for Spark ML or a single-node library, based on data size and the type of parallelism it needs
- Pick the Spark ML API (pyspark.ml or pyspark.ml.connect) that runs on a given compute type, and know the limits of Standard compute
- Tell transformers from estimators and chain them into a feature-and-model pipeline
Key concept
Data parallelism — Running the same operation on every row of a large dataset spread across a cluster. Spark is built for this, so Spark ML pays off when the training or scoring data is too big for one machine. It is not the best fit for many small, independent tasks.
1.When Spark ML is the right tool
Apache Spark MLlib is Spark's machine learning library. It covers the usual algorithm families, including classification, regression, clustering, collaborative filtering and dimensionality reduction, along with the optimization primitives underneath them. Having the algorithms is not a reason to use it, though. The real question is what kind of parallelism your workload needs.
Databricks describes the split this way. Spark is good at data parallelism, where one operation runs over every element of a large dataset. Ray is good at task parallelism, where many independent tasks run at the same time. Spark's MLlib and SparkML libraries are optimized for large-scale machine learning algorithms and statistical modeling. Ray is the suggested engine for reinforcement learning, simulation, hyperparameter search and deep learning training.
| Engine | Strength | Typical workloads |
|---|---|---|
| Spark | Data parallelism: the same operation on each element of a large dataset | ETL, feature engineering, data preprocessing, large-scale ML with MLlib and SparkML |
| Ray | Task parallelism: independent tasks run concurrently | Reinforcement learning, hierarchical time series forecasting, simulation modeling, hyperparameter search, deep learning training |
The Spark Connect documentation gives the most direct rule. Use Spark ML when the data does not fit in memory on a single node, or when you need distributed hyperparameter tuning. If neither is true, Spark ML has no special advantage. Single-machine libraries such as scikit-learn, which Databricks also documents as standard tools, become the natural choice.
Checkpoint 1 of 5· Exam question
A data science team must train a gradient-boosted tree classifier on a curated Unity Catalog Delta table with 800 million rows and roughly 400 feature columns. The full table cannot fit into the memory of the largest available driver node, and the team is not allowed to subsample the data. Which approach lets them train on the complete dataset?
Correct answer: A — Train a `GBTClassifier` from `pyspark.ml.classification` inside a Spark ML Pipeline, so executors process the Delta table in parallel while the model fits.
- A. Spark ML estimators such as `GBTClassifier` distribute both feature access and gradient computation across executors, so the full 800-million-row table is used without ever collecting it onto one machine. This is exactly the workload Spark MLlib is built for.
- B. Pulling the entire Delta table into a pandas DataFrame forces it into driver memory, which is the out-of-memory failure the team is trying to avoid in the first place. A single-node library cannot process data that does not fit on one machine.
- C. Sampling down to 10% of the rows before fitting discards most of the labeled data, which violates the requirement to train without subsampling even though it avoids the memory limit.
- D. `MlflowSparkStudy` parallelizes independent hyperparameter trials, but each trial still trains a separate single-node model on only its own shard rather than one model learning from the full table.
Checkpoint 2 of 5· Check yourself
Which situation does Databricks name as a reason to use Spark ML on Standard compute?
Data that won't fit on one node is the documented trigger. Standard compute has no GPU support and caps models at 1 GB, and Databricks points reinforcement learning to Ray.
“if you need Spark-level distribution for data that doesn’t fit in memory on a single node”Source: docs.databricks.com
2.Which Spark ML API runs where
There are two Python entry points. The pyspark.ml package is supported on serverless, standard and dedicated compute. Spark 3.5 added pyspark.ml.connect for Spark Connect and Databricks Connect. It has a similar interface but only a subset of the algorithms: LogisticRegression for classification, MaxAbsScaler and StandardScaler as feature transformers, Regression, BinaryClassification and MulticlassClassification evaluators, a Pipeline, and a CrossValidator. CrossValidator is the only tuning class in that list; TrainValidationSplit, which Databricks' MLlib tuning notes mention alongside it, is not listed for this module. Databricks also offers distributed training for Spark ML models through this module.
On Runtime 17.0 and above, Spark ML is enabled by default on Standard-access compute. Standard compute is shared, so it comes with limits. The trained model is cached on the driver node, and other users share that node. To answer the prediction: no. Only pyspark.ml is supported there, and the old pyspark.mllib package is not.
| Constraint | What the docs say |
|---|---|
| Language | Python only; R and Scala are not supported |
| Library | Only pyspark.ml is supported; pyspark.mllib is not |
| Model size | Maximum 1 GB; tree training stops early near that size |
| Model cache | Up to 10 GB per session, cached on the shared driver |
| Session timeout | Cached model times out after 15 minutes of inactivity |
| GPU | Not supported; use dedicated GPU clusters |
| Unsupported models | DistributedLDAModel, FPGrowthModel |
Checkpoint 3 of 5· Check yourself
A data scientist trains a tree model on Standard compute and leaves for a 30-minute meeting without saving it. What happens?
After 15 minutes of inactivity the cached model times out on Standard compute. Save it promptly or keep the session active.
“save it to disk within 15 minutes after training is completed”Source: docs.databricks.com
3.Transformers, estimators and pipelines
A Spark ML workflow is built from stages of two kinds. A transformer takes a DataFrame and returns it with a new column added, so you call transform. An estimator first has to learn from data, so you call fit. That returns a fitted model, and the model is a transformer you then apply. In a Databricks reference example, VectorAssembler is a transformer that combines feature columns into one vector column. StandardScaler and KMeans are estimators: each is fit and then used to transform.
Checkpoint 4 of 5· Fill the gap
StandardScaler has to learn statistics from the data before it can scale anything. Which method completes this line?
model = scaler. ? (assembled)StandardScaler is an estimator, so you call fit to get a fitted model, and that model does the transform. Calling transform on the unfitted estimator skips the learning step.
Source: docs.databricks.comThe Pipelines API puts these stages into one ordered object (pyspark.ml.connect.pipeline.Pipeline is its Spark Connect form). The whole chain can then be fit and applied as a unit. The Databricks example below builds a Pipeline from its stages with setStages, then calls fit on the pipeline. What fit returns is a pipeline model that can be reused across multiple DataFrames, and you call transform on it.
pipeline = Pipeline().setStages([document_assembler, sentence_detector, marian_transformer])# Create a pipeline model that can be reused across multiple data frames
model = pipeline.fit(data)
# You can use the model on any data frame that has a “text” column
result = model.transform(data)Databricks' MLlib example notebooks build binary classification, decision-tree classification and gradient-boosted-tree regression this way. Pipelines are not limited to built-in stages. One notebook shows how to write a custom transformer, and an XGBoost model can be embedded into an MLlib ML pipeline. Choosing stages for a use case means matching each one to its job: assemblers and scalers prepare features, and an algorithm estimator sits at the end. For text columns, Spark ML supports tokenization, stop-word processing, word2vec and feature hashing directly in your pipelines. These sources do not list the full feature-transformer catalog. For that, Databricks points to the MLlib Programming Guide and Apache Spark API references.
Checkpoint 5 of 5· Exam question
A team is deciding whether to build a churn model with Spark MLlib or with single-node scikit-learn. The labeled training set is 40,000 rows and 25 columns, comfortably fitting in the memory of one driver node, and the team wants access to a wide range of scikit-learn-only algorithms and diagnostic plotting libraries. Which factor most strongly favors the single-node approach here?
Correct answer: A — The dataset fits comfortably in memory on one node, and the team needs algorithms and tooling that exist only in the single-node ecosystem.
- A. When data comfortably fits on one machine and the team specifically needs single-node-only algorithms or plotting tools, the cost of distributed execution buys nothing, making single-node the pragmatic choice.
- B. Spark MLlib estimators can fit on small datasets; there is no minimum row-count floor below which MLlib refuses to train, so this claim is false.
- C. Accuracy depends on the algorithm and data, not on whether the implementation is distributed; neither approach has an inherent accuracy advantage over the other.
- D. A Spark `Pipeline` can be fit on any Spark DataFrame, including one read from CSV or Parquet in memory; Delta storage is not a prerequisite for training.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Any Spark ML code, including the old pyspark.mllib RDD API, runs on Standard-access compute.Why is that wrong?
Spark ML on Standard compute supports only the pyspark.ml package. pyspark.mllib is not supported, and the platform is Python-only.
Covered in Which Spark ML API runs where
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“including classification, regression, clustering, collaborative filtering, dimensionality reduction, and underlying optimization primitives”
↩︎ When Spark ML is the right tool“The pyspark.ml package from Apache Spark MLlib is supported on serverless, standard, and dedicated compute.”
↩︎ Which Spark ML API runs where“This notebook shows you how to build a binary classification application using the Apache Spark MLlib Pipelines API.”
↩︎ Transformers, estimators and pipelines“This notebook illustrates how to create a custom transformer.”
↩︎ Transformers, estimators and pipelines“For reference information about MLlib features, Databricks recommends the following Apache Spark API references”
↩︎ Transformers, estimators and pipelines - 2.
“Spark's MLlib and SparkML libraries are optimized for large-scale machine learning algorithms and statistical modeling.”
↩︎ When Spark ML is the right tool“Ray excels at task parallelism - Run a set of independent tasks concurrently.”
↩︎ When Spark ML is the right tool“Spark excels at data parallelism - Apply the same operation to each element of a large dataset.”
↩︎ Key concept - 3.https://docs.databricks.com/aws/en/machine-learning/train-model/distributed-training/distributed-ml-for-spark-connectOfficial docs
“if you need Spark-level distribution for data that doesn’t fit in memory on a single node”
↩︎ When Spark ML is the right tool“the pyspark.ml.connect module currently only contains a subset of the algorithms in pyspark.ml”
↩︎ Which Spark ML API runs where“Model tuning: pyspark.ml.connect.tuning.CrossValidator”
↩︎ Which Spark ML API runs where“Spark ML on Spark connect is enabled by default on compute resources with Standard access mode”
↩︎ Which Spark ML API runs where“the trained model itself is cached on the driver node, which is shared among other users”
↩︎ Which Spark ML API runs where“Only the pyspark.ml package is supported. The pyspark.mllib package is not supported.”
↩︎ Exam trap 1“save it to disk within 15 minutes after training is completed”
↩︎ Checkpoint - 4.
“when you run tuning code that uses CrossValidator or TrainValidationSplit”
↩︎ Which Spark ML API runs where - 5.
“Databricks also offers distributed training for Spark ML models with the pyspark.ml.connect module.”
↩︎ Which Spark ML API runs where - 6.https://docs.databricks.com/aws/en/machine-learning/reference-solutions/natural-language-processingOfficial docs
“Create a pipeline model that can be reused across multiple data frames”
↩︎ Transformers, estimators and pipelines“Spark ML supports a range of text processors, including tokenization, stop-word processing, word2vec, and feature hashing.”
↩︎ Transformers, estimators and pipelines - 7.
“The first example shows how to embed an XGBoost model into an MLlib ML pipeline.”
↩︎ Transformers, estimators and pipelines