What you will be able to do
- Tune a Spark ML model with CrossValidator or TrainValidationSplit, and choose the right Hyperopt trials class for MLlib
- Pick the evaluator that matches a regression, binary or multiclass problem, and log its metrics
- Score a model in batch or streaming, and pick Spark-based scoring or Model Serving based on the inference type
1.Tuning Spark ML models
Spark ML has its own tuning classes, CrossValidator and TrainValidationSplit. They wrap an estimator, which can be a whole pipeline, and search hyperparameter settings against an evaluation metric. MLlib cross validation can even tune an XGBoost model. For tracking, the old MLlib automated MLflow tracking logged hyperparameters and metrics for these tuners. It is now deprecated and disabled by default from Runtime 10.4 LTS ML onward. Use mlflow.pyspark.ml.autolog() instead, which Databricks Autologging turns on by default.
Hyperopt can also tune MLlib, but the correct setup is easy to get backwards. SparkTrials sends trials to the workers. It is meant for single-machine models such as scikit-learn. An MLlib model already uses the whole cluster for each fit. So you use the default Trials class: Hyperopt generates each trial on the driver, and the algorithm itself does the distributed training. Trials does not autolog to MLflow, so you log each trial yourself. Also note that Hyperopt is not included in Databricks Runtime ML after 16.4 LTS ML. Databricks now recommends Optuna for single-node optimization or Ray Tune for distributed tuning.
| Trials class | Use with | Where each trial runs | MLflow logging |
|---|---|---|---|
| SparkTrials | Single-machine models such as scikit-learn | Distributed to Spark workers | Automatic |
| Trials | Distributed algorithms such as MLlib or Horovod | Driver node, with full cluster resources | Manual: call MLflow yourself |
Checkpoint 1 of 5· Check yourself
On Databricks Runtime 15.4 LTS ML, a team runs CrossValidator over a pyspark.ml pipeline and wants the run tracked in MLflow. What is the documented approach?
MLlib automated tracking is deprecated and disabled from 10.4 LTS ML. The replacement is mlflow.pyspark.ml.autolog(), which Databricks Autologging enables by default.
“Instead, use MLflow PySpark ML autologging by calling mlflow.pyspark.ml.autolog(), which is enabled by default with Databricks Autologging.”Source: docs.databricks.com
2.Evaluating Spark ML models
Evaluation is what tuning optimizes, so the evaluator has to match the problem type. pyspark.ml.connect lists three: RegressionEvaluator for continuous targets, BinaryClassificationEvaluator for two-class problems, and MulticlassClassificationEvaluator for more than two classes. CrossValidator and TrainValidationSplit score every candidate with the evaluator you give them. If you tune with Hyperopt, the objective function does the training and loss calculation and returns the loss for fmin to minimize.
Record the metrics so runs can be compared. MLflow tracking can log metrics computed with any framework. With Hyperopt's Trials class that logging is manual, as covered above. These sources do not list each evaluator's metricName options, such as which ones apply to regression versus multiclass. For those details Databricks points to the Apache Spark API references.
Checkpoint 2 of 5· Exam question
A machine learning engineer is assembling a Spark ML `Pipeline` with a `StringIndexer`, a `OneHotEncoder`, a `VectorAssembler`, and a `LogisticRegression` estimator as its stages, in that order. What happens when `pipeline.fit(trainingData)` is called?
Correct answer: A — Each stage runs in order: `Transformer` stages transform immediately, `Estimator` stages fit and are replaced by fitted transformers, giving a `PipelineModel`.
- A. `Pipeline.fit()` runs each stage in sequence: transformer stages like `OneHotEncoder` and `VectorAssembler` apply their transform immediately, while estimator stages like `StringIndexer` and `LogisticRegression` are fit on the data produced by prior stages and become fitted transformers inside the returned `PipelineModel`.
- B. Every stage in the list actually executes; the earlier stages are not skipped or treated as inert configuration, since each one produces the transformed columns the next stage depends on.
- C. Pipeline stages run sequentially because each stage typically consumes the output column of the previous one, so they cannot be executed in parallel as independent, unrelated steps.
- D. Mixing indexing, encoding, assembling, and a classifier in one `Pipeline` is the standard pattern the API is designed for, so no exception is raised for combining these stage types.
Checkpoint 3 of 5· Check yourself
A product classifier predicts one of ten categories. Which evaluator should the tuning step use?
Ten classes make it a multiclass problem. BinaryClassificationEvaluator is for two classes, RegressionEvaluator is for continuous targets, and StandardScaler is a feature transformer, not an evaluator.
“Evaluator: pyspark.ml.connect.RegressionEvaluator, pyspark.ml.connect.BinaryClassificationEvaluator and MulticlassClassificationEvaluator”Source: docs.databricks.com
3.Scoring in batch and streaming
Scoring a table means applying the same model to every row, which is data parallelism and exactly what Spark is built for. Databricks describes batch and streaming scoring as high-throughput and low-cost, with latencies as low as minutes. A fitted Spark ML model or pipeline scores a DataFrame through transform. For any model logged with MLflow, Databricks recommends Spark pandas UDFs to scale batch and streaming inference across the cluster. MLflow generates the inference code that applies a logged model as a pandas UDF. Databricks' MLlib example notebooks include a Structured Streaming example.
Run a preprocessing job that ETLs the data into a Delta Lake table, then run inference from that table. The ingest and preparation cost is then shared across every read. You can also pick different hardware for each job, for example CPUs for ETL and GPUs for inference.
Checkpoint 4 of 5· Check yourself
A model logged with MLflow has to score a multi-terabyte table every night across the cluster. Which approach does Databricks recommend?
Pandas UDFs spread scoring across the cluster, and MLflow supplies the UDF inference code for logged models.
“Use Spark Pandas UDFs to scale batch and streaming inference across a cluster.”Source: docs.databricks.com
4.Spark-based scoring or Model Serving: choosing by inference type
The inference type decides where the model runs. Batch and streaming workloads take whole datasets and accept latency measured in minutes, so they fit Spark-based scoring. Real-time workloads are different: an application sends one request and waits for the answer. For that, Databricks offers Model Serving. It is a highly available, low-latency service that exposes each model as a REST API and scales automatically on serverless compute. Model Serving also supports batch work through ai_query, which runs efficient batch inference against a served endpoint.
| Inference type | Databricks option | What you get |
|---|---|---|
| Batch | Apache Spark UDFs or mlflow.pyfunc; ai_query against a Model Serving endpoint | High-throughput, low-cost scoring of whole tables |
| Streaming | Spark pandas UDFs over a Structured Streaming source | Continuous scoring at latencies as low as minutes |
| Real-time | Model Serving REST endpoint | Low-latency, high-uptime responses with serverless autoscaling |
Checkpoint 5 of 5· Check yourself
A loan-approval web app needs a prediction within the user's request. Which deployment fits?
A per-request, low-latency need is real-time inference. Model Serving exposes the model as a REST API that the app can call, while batch and streaming jobs work on latencies of minutes.
“Model Serving provides low-latency, high-uptime managed REST endpoints with serverless autoscaling.”Source: docs.databricks.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.SparkTrials is the right way to parallelize Hyperopt for any model on a cluster, including MLlib.Why is that wrong?
SparkTrials is for single-machine models. MLlib models already train in a distributed way, so use the default Trials class, which runs each trial from the driver.
Covered in Tuning Spark ML models
2.CrossValidator runs are logged to MLflow automatically by MLlib's built-in tracking on current runtimes.Why is that wrong?
MLlib automated MLflow tracking is deprecated and off by default from 10.4 LTS ML. Use mlflow.pyspark.ml.autolog() instead.
Covered in Tuning Spark ML models
3.Hyperopt ships with every current Databricks Runtime ML, so new tuning code should be built on it.Why is that wrong?
Hyperopt is not included after 16.4 LTS ML. Databricks recommends Optuna for single-node optimization or Ray Tune for distributed tuning.
Covered in Tuning Spark ML models
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“when you run tuning code that uses CrossValidator or TrainValidationSplit, hyperparameters and evaluation metrics are automatically logged in MLflow.”
↩︎ Tuning Spark ML models“MLlib automated MLflow tracking is deprecated and disabled by default on clusters that run Databricks Runtime 10.4 LTS ML and above.”
↩︎ Exam trap 2“Instead, use MLflow PySpark ML autologging by calling mlflow.pyspark.ml.autolog(), which is enabled by default with Databricks Autologging.”
↩︎ Checkpoint - 2.
“The second example shows how to use MLlib cross validation to tune an XGBoost model.”
↩︎ Tuning Spark ML models - 3.https://docs.databricks.com/aws/en/machine-learning/automl-hyperparam-tuning/hyperopt-conceptsOfficial docs
“SparkTrials is designed to parallelize computations for single-machine ML models such as scikit-learn.”
↩︎ Tuning Spark ML models“This function typically contains code for model training and loss calculation.”
↩︎ Evaluating Spark ML models“For models created with distributed ML algorithms such as MLlib or Horovod, do not use SparkTrials.”
↩︎ Exam trap 1“Use Trials when you call distributed training algorithms such as MLlib methods or Horovod in the objective function.”
↩︎ Prediction - 4.https://docs.databricks.com/aws/en/machine-learning/automl-hyperparam-tuning/hyperopt-distributed-mlOfficial docs
“Each trial is executed from the driver node, giving it access to the full cluster resources.”
↩︎ Tuning Spark ML models“Databricks does not support automatic logging to MLflow with the Trials class.”
↩︎ Tuning Spark ML models“Databricks recommends using either Optuna for single-node optimization or RayTune”
↩︎ Tuning Spark ML models“Hyperopt is not included in Databricks Runtime for Machine Learning after 16.4 LTS ML.”
↩︎ Exam trap 3 - 5.https://docs.databricks.com/aws/en/machine-learning/train-model/distributed-training/distributed-ml-for-spark-connectOfficial docs
“Evaluator: pyspark.ml.connect.RegressionEvaluator, pyspark.ml.connect.BinaryClassificationEvaluator and MulticlassClassificationEvaluator”
↩︎ Evaluating Spark ML models - 6.
“MLflow tracking can log metrics computed using your custom framework”
↩︎ Evaluating Spark ML models“You can also use custom code with Apache Spark UDFs (example) or mlflow.pyfunc for batch inference.”
↩︎ Spark-based scoring or Model Serving: choosing by inference type“Model Serving provides low-latency, high-uptime managed REST endpoints with serverless autoscaling.”
↩︎ Checkpoint - 7.
“For reference information about MLlib features, Databricks recommends the following Apache Spark API references”
↩︎ Evaluating Spark ML models - 8.
“Batch and streaming scoring supports high-throughput, low-cost scoring at latencies as low as minutes.”
↩︎ Scoring in batch and streaming“When you log a model from Databricks, MLflow automatically provides inference code to apply the model as a pandas UDF.”
↩︎ Scoring in batch and streaming“consider creating a preprocessing job to ETL the data into a Delta Lake table before running the inference job”
↩︎ Scoring in batch and streaming“Use Spark Pandas UDFs to scale batch and streaming inference across a cluster.”
↩︎ Checkpoint - 9.
“Binary classification, decision trees, GBT regression, Structured Streaming, custom transformer”
↩︎ Scoring in batch and streaming - 10.
“Model Serving provides a highly available and low-latency service for deploying models.”
↩︎ Spark-based scoring or Model Serving: choosing by inference type“Each model you serve is available as a REST API that you can integrate into your web or client application.”
↩︎ Spark-based scoring or Model Serving: choosing by inference type