Lakeflow pipeline expectations: warn, drop, and fail

1.Expectations: where quality rules go in a pipeline

In the medallion architecture the Silver layer is where data is checked. Databricks describes the layers as bronze (raw), silver (validated), and gold (enriched), and says of the Silver layer: "Data cleanup and validation are performed in this layer." Gold datasets are aggregated from Silver, so any bad record that gets through Silver ends up in Gold totals too. In Lakeflow pipelines the main tool for this job is the expectation.

An expectation is an optional clause on a pipeline materialized view, streaming table, or view declaration. It has three parts. The first is a name, which identifies the expectation in monitoring and must be unique within a dataset. You can reuse the same expectation on other datasets. The second is a constraint: a standard SQL Boolean expression that has to evaluate to true or false for each record. The third is an action, which decides what happens to a record that fails. In SQL the constraint goes in a CONSTRAINT name EXPECT (expr) clause. In Python it is a decorator on the dataset function. You can also use the SQL clause in CREATE STREAMING TABLE and CREATE MATERIALIZED VIEW when a standalone pipeline created in Databricks SQL backs them.

CREATE OR REFRESH STREAMING TABLE customers(
  CONSTRAINT valid_customer_age EXPECT (age BETWEEN 0 AND 120)
) AS SELECT * FROM STREAM(datasets.samples.raw_customers);

@dp.table
@dp.expect("valid_customer_age", "age BETWEEN 0 AND 120")
def customers():
  return spark.readStream.table("datasets.samples.raw_customers")

A constraint can use SQL functions, CASE statements, and compound Boolean logic. For example, year(transaction_date) >= 2020 or a CASE that allows different status values depending on the record type are both fine. A constraint cannot contain custom Python functions, external service calls, or subqueries that reference other tables. Each rule looks at one record at a time.

Databricks' expectation patterns page shows how a cross-table check is built. Instead of a subquery inside the constraint, you declare a separate validation dataset that joins the two tables, and the constraint tests a column of the joined result. In this example a left outer join keeps every row of validation_copy, and r_key is null when no matching row exists in report, so r_key IS NOT NULL catches missing records. The example uses a fail expectation, but note that a validation table does not by itself gate other datasets: reading it from another dataset does not make that dataset wait for the validation result.

CREATE OR REFRESH MATERIALIZED VIEW report_compare_tests(
  CONSTRAINT no_missing_records EXPECT (r_key IS NOT NULL)
)
AS SELECT v.*, r.key as r_key FROM validation_copy v
  LEFT OUTER JOIN report r ON v.key = r.key

Sources123

2.The three actions: warn, drop, fail

The action is what turns a rule into a quality policy. Each action has a SQL form and a Python form. Warn keeps the data and records metrics. Drop filters out bad records and logs how many were removed. Fail treats any bad record as a blocking error.

Pick the action according to what the dataset can tolerate. Use warn when you need visibility, for example while you are still learning how dirty a source is. Use drop when a Silver table must contain only usable rows, such as current_page_id IS NOT NULL AND current_page_title IS NOT NULL on clickstream data. Use fail when invalid records are unacceptable, for example a count that must be positive in a Gold aggregate. If you need to keep the bad rows somewhere without failing or dropping them, there is a separate quarantine pattern for that.

CONSTRAINT valid_count EXPECT (count > 0) ON VIOLATION  ?  UPDATE

Sources2

3.Stacking several rules on one dataset

Real Silver tables need more than one rule, and the rules can differ in strictness. In Python you stack decorators. The dp module provides six of them: each of the three actions comes in a single-expectation version and an expect_all version that takes several expectations. The decorators go after the dataset decorator and before the function that defines the dataset.

from pyspark import pipelines as dp

@dp.table()
@dp.expect(description, constraint)
@dp.expect_or_drop(description, constraint)
@dp.expect_or_fail(description, constraint)
@dp.expect_all({description: constraint, ...})
@dp.expect_all_or_drop({description: constraint, ...})
@dp.expect_all_or_fail({description: constraint, ...})
def <function-name>():
    return (<query>)

The expect_all decorators take a dict that maps each description to its constraint. Each entry still reports its own metrics, so grouping rules does not hide which rule a record failed. SQL has no expect_all. You either list several comma-separated CONSTRAINT ... EXPECT clauses, or you combine conditions into one constraint with AND when they should share a single action and a single name.

Sources43

4.Reading metrics and handling a failed update

Warn and drop expectations both produce tracking metrics, which you can view in the pipeline UI. Fail does not, because the update stops before anything is recorded. To see the metrics in the UI, click Jobs & Pipelines in the workspace sidebar, click the name of your pipeline, click a dataset that has an expectation defined, and select the Data quality tab in the right sidebar. Streaming tables and materialized views backed by a standalone Databricks SQL pipeline have no Data quality tab, so you query the pipeline event log for their metrics instead. The event log holds the metrics for any Lakeflow pipeline, and the expectation name you chose is how entries are told apart: Databricks' quarantine example notes that merging conditions into one constraint gives a single name in the event log, while separate constraints give multiple entries under different names. The documentation points to its page on querying data quality or expectations metrics for the exact queries.

When a fail expectation is violated, execution stops immediately. If the operation is a table update, the transaction is rolled back atomically, so the target never shows a half-written state. How far the failure spreads depends on the run mode. In a triggered pipeline, only the failing flow fails and parallel flows continue. In a continuous pipeline, the failure stops the flow and every flow that depends on it, and the pipeline emits a message saying why it stopped. In both modes you have to fix the pipeline code so it handles the invalid data before you re-run. Fail expectations also change the query plan so that violations can be traced. The resulting EXPECTATION_VIOLATION error message names the violated expectation and can show the input record and the output record involved.

Sources2

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.An EXPECT clause with no ON VIOLATION keeps invalid rows out of the target table.Why is that wrong?

   Without ON VIOLATION the action is warn: invalid records are written to the target, and only the metrics record the violations. To keep them out you need DROP ROW or FAIL UPDATE.

   Covered in The three actions: warn, drop, fail

2. 2.After an expect_or_fail violation, you can check the Data quality tab to see how many records failed.Why is that wrong?

   Fail records no metrics, because the update stops when the first invalid record is detected. Only warn and drop produce tracking metrics.

   Covered in Reading metrics and handling a failed update

3. 3.A failed expectation always stops the entire pipeline.Why is that wrong?

   That only happens in continuous mode, where the failure stops the flow and its dependent flows. In a triggered pipeline, the other parallel flows keep running.

   Covered in Reading metrics and handling a failed update