Building Lakehouse Monitors: Slicing, Custom Metrics, Alerts and Endpoint Health

1.Creating and refreshing a monitor

A monitor is created with w.data_quality.create_monitor. The call takes a DataProfilingConfig naming the output schema and exactly one profile type: time_series=TimeSeriesConfig(...), inference_log=InferenceLogConfig(...) or snapshot=SnapshotConfig(). A table in a Unity Catalog metastore can have only one profile. To create one you need USE CATALOG, USE SCHEMA, SELECT on the table and MANAGE on the catalog, schema or table. An inference monitor looks like this:

config = DataProfilingConfig(
 output_schema_id=schema.schema_id,
 assets_dir=f"/Workspace/Users/{username}/databricks_quality_monitoring/{TABLE_NAME}",
 inference_log=InferenceLogConfig(
    problem_type=InferenceProblemType.INFERENCE_PROBLEM_TYPE_CLASSIFICATION,
    prediction_column="preds",
    model_id_column="model_ver",
    label_column="label", # optional
    timestamp_column="ts",
    granularities=[AggregationGranularity.AGGREGATION_GRANULARITY_1_DAY])
)

info = w.data_quality.create_monitor(
   monitor=Monitor(
     object_type="table",
     object_id=table.table_id,
     data_profiling_config=config,
   ),
)

The output goes to {output_schema}.{table_name}_profile_metrics and {output_schema}.{table_name}_drift_metrics, along with a dashboard you can customise. create_refresh recomputes the tables on serverless compute, not on your notebook's cluster. For scheduled refreshes, pass a schedule with a Quartz cron expression. For time series and inference tables, turn on change data feed so that each refresh processes only newly appended rows instead of the whole table.

Sources1

2.Granularities and slices

Granularity sets the window size for time series and inference profiles. The options run from AGGREGATION_GRANULARITY_5_MINUTES through 30 minutes, 1 hour, 1 day, 1 to 4 weeks and 1 month, up to AGGREGATION_GRANULARITY_1_YEAR. You can choose several at once, and each row in the metric tables records its granularity. Small windows catch sudden changes but contain fewer rows each. Large windows smooth out noise. A snapshot profile has no granularity; its window is the moment of the refresh.

Slices appear in the metric tables as slice_key and slice_value. The whole table is the row where both are NULL. Every slice key is a single expression, so slices are not crossed with one another. Inference profiles also slice automatically by every distinct model_id_col value. Slicing is what turns a vague whole-table signal into a specific one. If drift appears in one slice but not in the others, the cause lies in that segment.

Sources2

3.Custom metrics

When the built-in statistics don't capture what the business cares about, you add MonitorMetric definitions. Each one is a Jinja template for a single SQL column expression, with no joins or subqueries. It has a type, a name that becomes a column in the metric tables, input_columns (:table when the metric uses several columns), a definition and an output_data_type.

MonitorMetric(
    type=MonitorMetricType.CUSTOM_METRIC_TYPE_AGGREGATE,
    name="weighted_error",
    input_columns=[":table"],
    definition="""avg(CASE
      WHEN {{prediction_col}} = {{label_col}} THEN 0
      WHEN {{prediction_col}} != {{label_col}} AND critical=TRUE THEN 2
      ELSE 1 END)""",
    output_data_type=T.StructField("output", T.DoubleType()).json(),
)

Only aggregate metrics read the primary table. Derived and drift metrics are computed from values already aggregated, so where you can, build on an aggregate instead of scanning the table again. To track how weighted_error changes, reference it in the current window as {{current_df}} and in the comparison window, either the baseline or the previous window, as {{base_df}}.

MonitorMetric(
    type=MonitorMetricType. ? ,
    name="error_rate_delta",
    input_columns=[":table"],
    definition="{{current_df}}.weighted_error - {{base_df}}.weighted_error",
    output_data_type=T.StructField("output", T.DoubleType()).json(),
)

Sources3

4.Alerting when drift crosses a threshold

Metric tables are ordinary Unity Catalog Delta tables, so alerts are ordinary Databricks SQL alerts. You write a query on the profile or drift metrics table that returns, for example, the latest population_stability_index or the ks_test.pvalue for a feature. You then create an alert on that query with a threshold, a schedule for evaluating it, and destinations. Email is the default, and webhooks, Slack and PagerDuty are also supported. Typical uses: a statistic leaves its range, the distribution changes, or inputs drift from the baseline, which for an inference profile can signal that the model needs retraining.

Two details catch people out. If the alert query has parameters, the alert uses their default values, so check those defaults. And the notification_settings you pass to create_monitor are a different thing: its on_failure destination, with up to 5 email addresses, tells you when a refresh fails or times out. It does not tell you that the data has drifted.

Sources4

5.Endpoint health metrics

Data drift and model quality come from the inference table. The health of the serving infrastructure comes from the endpoint itself, on the Metrics tab of the Serving UI or through the metrics export API, which can feed Prometheus or Datadog.

Sources5

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Setting notification_settings.on_failure in create_monitor emails stakeholders when drift exceeds a threshold.Why is that wrong?

   on_failure notifies you when a monitoring refresh fails or times out. Drift thresholds need a Databricks SQL alert on a query over the metric tables.

   Covered in Alerting when drift crosses a threshold

2. 2.A custom metric definition can join the inference table to a reference table.Why is that wrong?

   The definition must be a single SQL expression in a Jinja template, with no joins or subqueries.

   Covered in Custom metrics

Practise it for real

Create an inference monitor on a model's request-log table, refresh it, and alert when a feature's drift crosses a threshold.

1. 1.Install the SDK with %pip install "databricks-sdk>=0.68.0", then get the output schema and the inference table with w.schemas.get and w.tables.get.

   Why: create_monitor needs the schema_id and table_id, not the names.

   You should see: Objects whose schema_id and table_id you can pass to the config.

2. 2.Build a DataProfilingConfig with inference_log=InferenceLogConfig(...) setting problem_type, prediction_column, model_id_column, label_column, timestamp_column and a 1-day granularity, and call w.data_quality.create_monitor.

   Why: With a label column you get model quality metrics. With a model id column you get per-version slices.

   You should see: The monitor info is returned, and a dashboard is created in assets_dir.

3. 3.Call w.data_quality.create_refresh, then poll get_refresh until the state is no longer PENDING or RUNNING.

   Why: The refresh writes the metric tables on serverless compute.

   You should see: The tables <table>_profile_metrics and <table>_drift_metrics appear in the output schema.

4. 4.Query the drift metrics table for one numeric feature with slice_key IS NULL, ordered by window, and read ks_test and population_stability_index.

   Why: These are the drift signals you will set a threshold on.

   You should see: CONSECUTIVE rows, plus BASELINE rows if you provided a baseline table.

5. 5.Save that query and create a Databricks SQL alert that triggers when population_stability_index >= 0.2, with an email or Slack destination.

   Why: 0.2 is the documented threshold for significant population change.

   You should see: The alert is evaluated on its schedule and notifies you when the threshold is crossed.