Quarantining Invalid Records and Grouping Pipeline Expectations

1.Applying a set of rules at once with expect_all

Real datasets rarely have only one rule. SQL and Python both let you put several expectations on a dataset, but only Python lets you group them and give the whole group one action. The decorators are expect_all, expect_all_or_drop and expect_all_or_fail. Each takes a Python dictionary in which every key is an expectation name and every value is its constraint.

Grouping doesn't hide detail. Each expectation in the dictionary keeps its own description and reports its own metrics, so you can still see which rule a record broke. Because the dictionary is an ordinary Python object, you can apply one rule set to several datasets with different strictness.

valid_pages = {"valid_count": "count > 0", "valid_current_page": "current_page_id IS NOT NULL AND current_page_title IS NOT NULL"}

@dp.table
@dp.expect_all(valid_pages)
def raw_data():
  # Create a raw dataset

@dp.table
@dp.expect_all_or_drop(valid_pages)
def prepared_data():
  # Create a cleaned and prepared dataset

@dp.table
@dp.expect_all_or_fail(valid_pages)
def customer_facing_data():
  # Create cleaned and prepared to share the dataset

Sources12

2.Keeping rules outside the pipeline code

Because expect_all takes a dictionary, the rules can come from somewhere other than the pipeline source. Databricks recommends storing expectation definitions separately from pipeline logic, tagging related expectations so you can filter them, and applying the same expectations to similar datasets.

The documented pattern keeps a rules Delta table, or a Python module such as rules_module.py, with a name, a constraint and a tag for each rule. A helper get_rules(tag) returns a {name: constraint} dictionary for one tag. For example, @dp.expect_all_or_drop(get_rules('validity')) drops records from raw_farmers_market that fail any rule tagged validity. You can then audit or change the rules without editing the pipeline code. This approach is Python-only: SQL cannot load expectations dynamically from a file.

Sources3

3.Quarantine: keep bad rows without letting them through

There isn't. Quarantine is a pattern you build from datasets and flows, not a fourth action. The expectations docs describe it as advanced logic to quarantine invalid records without failing or dropping data. It solves drop's weakness: dropped rows are counted, but they are gone, so nobody can look at them or reprocess them.

The best-practices guide describes it with two flows. One flow drops invalid rows from the main table. A second flow writes only the invalid rows to a separate quarantine streaming table. You can then investigate, fix and reprocess the bad data without contaminating the clean dataset.

The advanced-patterns page shows a version built on a flag column. A rules dictionary holds the constraints, and quarantine_rules combines them into one NOT(... AND ...) expression. A temporary table applies the rules with expect_all, so you get per-rule metrics without dropping anything. It also adds an is_quarantined column calculated from the combined expression and is partitioned on that column. Views downstream split the table into valid and invalid rows.

quarantine_rules = "NOT({0})".format(" AND ".join(rules.values()))

@dp.view
def raw_trips_data():
  return spark.readStream.table("samples.nyctaxi.trips")

@dp.table(
  temporary=True,
  partition_cols=["is_quarantined"],
)
@dp.expect_all(rules)
def trips_data_quarantine():
  return (
    spark.readStream.table("raw_trips_data").withColumn("is_quarantined", expr(quarantine_rules))
  )

@dp.view
def valid_trips_data():
  return spark.read.table("trips_data_quarantine").filter("is_quarantined=false")

Sources243

4.Choosing an action per constraint, and what expectations can't do

The production-readiness checklist asks you to choose the action for each constraint on purpose, not to apply one action everywhere. Use fail for conditions that should stop everything, such as a broken primary key. Use drop for records you can safely discard, with a quarantine table capturing what was dropped. Use warn only where you actively watch the trend. The checklist also expects every dataset that can receive bad data to have at least one expectation, and you to know its current pass rate from the Data quality tab or the event log.

One limit catches people out. Expectations enforce data quality, not orchestration. They decide which records reach a target, but they don't conditionally run or skip other parts of the pipeline. You can build a validation table, such as a materialized view comparing row counts with expect_or_fail("no_rows_dropped", "a_count == b_count"). But a dataset that reads it does not wait for its result, so a failed validation does not stop downstream tables from updating. To gate downstream work, put the validation and the downstream logic in separate pipelines. Then orchestrate them with a job in which the downstream task depends on the validation task.

Sources53

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Quarantine is a built-in expectation action, like DROP ROW or FAIL UPDATE.Why is that wrong?

   There are only three actions: warn, drop and fail. Quarantine is a pattern built from extra flows or a flag column with views that route valid and invalid records separately.

   Covered in Quarantine: keep bad rows without letting them through

2. 2.A failing validation table with expect_or_fail stops downstream tables in the same pipeline from updating.Why is that wrong?

   Expectations don't orchestrate. To gate downstream work, split it into a separate pipeline that runs only after the validation pipeline's task succeeds in a job.

   Covered in Choosing an action per constraint, and what expectations can't do

3. 3.SQL can group several constraints under one collective ON VIOLATION action.Why is that wrong?

   SQL allows several constraints, but each has its own action. Only Python's expect_all, expect_all_or_drop and expect_all_or_fail apply one action to a group.

   Covered in Applying a set of rules at once with expect_all