What you will be able to do
- Choose between table-level row filters and column masks and ABAC policies for a given pipeline
- Define column masks and row filters on pipeline streaming tables and materialized views with CREATE OR REFRESH
- Predict whose identity a mask function sees during pipeline refresh and during a query
- Avoid the type-mismatch and performance pitfalls of masking UDFs
1.Two ways to attach a filter or mask
Unity Catalog row filters and column masks restrict which rows and values a user sees at query time. Users can't see base-table values before the filter or mask has been applied. There are two ways to attach these controls to a table that a pipeline writes. The first is table-level. The table owner binds a UDF to a single table with ALTER TABLE ... SET ROW FILTER or ALTER TABLE ... ALTER COLUMN ... SET MASK. Each column can have only one mask. The second is ABAC. A policy attached higher up applies to every table and column whose governed tags match. Dynamic views are a third option for publishing a curated or transformed copy.
| Approach | Applies to | Managed using | Best used for |
|---|---|---|---|
| Table-level row filters and column masks | Individual tables and columns | ALTER TABLE by the table owner or a user with MANAGE | Table-specific logic |
| ABAC policies | Tables and columns matched by tag conditions | CREATE POLICY on a catalog, schema, or table | Centralized rules applied automatically across many tables |
| Dynamic views | A view built from one or more base tables | SQL logic in the view definition | Sharing a curated or transformed version of the data |
For compliance, the important difference is who can undo the control. A table owner can change a table-level mask. An ABAC policy can be defined by higher-level admins and can't be removed by table owners. Databricks also says policy logic runs more efficiently than table-specific UDFs.
Batch pipelines have two more constraints. Runtimes below 12.2 LTS don't support these controls and fail securely, returning no data. And MERGE can't target tables whose filter or mask policies contain nesting, aggregations, windows, limits or non-deterministic functions, which matters for upsert-style silver jobs.
Checkpoint 1 of 4· Match them up
Match each mechanism to how it is applied
Tap a term, then the definition that fits it.
Table-level masks are bound to one column of one table. ABAC policies select columns by tag across a scope. Dynamic views put the logic in the view's SQL.
“Table-level column masks are bound to a column using ALTER TABLE ... ALTER COLUMN ... SET MASK and managed by the table owner.”Source: docs.databricks.com
Sources1
2.Masks on pipeline streaming tables and materialized views
Streaming tables and materialized views that pipelines publish are part of the dataset definition, so you declare a mask on a column and a row filter on the table in the same statement that defines the flow. ALTER TABLE commands are not allowed on streaming tables. Changes go through CREATE OR REFRESH or ALTER STREAMING TABLE.
CREATE OR REFRESH STREAMING TABLE customers_silver (
id int COMMENT 'This is the customer ID',
name string,
region string,
ssn string MASK catalog.schema.ssn_mask_fn COMMENT 'SSN masked for privacy'
)
WITH ROW FILTER catalog.schema.us_filter_fn ON (region)
AS SELECT * FROM STREAM(customers_bronze)Mask functions often check identity with CURRENT_USER or IS_MEMBER, and in a pipeline they are evaluated in two different contexts. Refresh as owner: when an update refreshes the streaming table or materialized view, the functions run with the pipeline owner's rights. Query: when someone queries the table, the same functions use the invoker's user context, so each reader gets their own masking. To audit which filters and masks apply to a pipeline dataset, use DESCRIBE EXTENDED, INFORMATION_SCHEMA or Catalog Explorer.
Checkpoint 2 of 4· Check yourself
A mask function on a streaming table returns the raw value when IS_MEMBER('pii_readers') is true. The pipeline owner is in pii_readers and an analyst is not. What does the analyst see when they query the table?
The owner's context applies only while the pipeline refreshes the table. At query time, identity functions are evaluated for the person running the query.
“functions that check user context (such as CURRENT_USER and IS_MEMBER) are evaluated using the invoker's user context.”Source: docs.databricks.com
Masking hides values from readers, but it doesn't remove them. Databricks says the safest option for a right-to-be-forgotten request is complete erasure, because eliminating the risk of reidentification often requires deleting the PII. Streaming tables support this. You can run DML statements, including insert, update, delete and merge, against streaming tables published to Unity Catalog, which covers use cases such as updating tables for GDPR compliance. DML that changes the table schema is not supported.
The complication is that streaming expects append-only sources. If a downstream streaming read comes from a source streaming table that has had changes applied by DML, set the skipChangeCommits flag when reading that source. Transactions that delete or modify records on the source are then ignored instead of breaking the stream. skipChangeCommits is only a read option. It does not delete anything itself. The recommended order is: (1) delete the data in the source Delta tables with DML, (2) delete it from the streaming table with DML, then (3) update the streaming read to use skipChangeCommits. Materialized views handle deletions in their sources automatically, but you must refresh them and run maintenance so the deletions are fully processed.
Checkpoint 3 of 4· Exam question
A `silver.customer_events` streaming table incrementally reads from `bronze.customer_events` using `spark.readStream.table("bronze.customer_events")`. A GDPR right-to-erasure request arrives, and an engineer runs `DELETE FROM bronze.customer_events WHERE user_id = 5`. On the next scheduled run, the streaming job fails with an error stating that a data update was detected in the source table, which is not supported by default in this kind of incremental read. What should the engineer do to satisfy the erasure request while keeping the stream running?
Correct answer: A — Also delete the matching rows already materialized in `silver.customer_events`, then add `.option("skipChangeCommits", "true")` to the streaming read so future non-append source operations are skipped, not fatal.
- A. Deleting the corresponding rows from the downstream streaming table directly is necessary because Structured Streaming does not propagate source deletes on its own, and `skipChangeCommits` tells the stream to skip non-append changes on the source going forward instead of throwing an unsupported-operation error.
- B. This is incorrect: by default, Structured Streaming's incremental read assumes the source is append-only, which is exactly why the delete triggered the failure described in the scenario rather than being silently propagated downstream.
- C. Restoring the bronze table to a version before the delete undoes the erasure entirely, reintroducing the very row that needed to be removed, so this fails the GDPR request rather than satisfying it.
- D. Recreating the table as a materialized view changes how future upstream deletes are handled going forward, but it does nothing to remove the already-deleted user's row that is still sitting in the existing silver data, so the erasure request remains unfulfilled.
3.Writing masking UDFs that don't leak or stall
A mask is only as good as its UDF, and one of the documented failures is silent. If a UDF parameter type doesn't match the column type, for example a STRING column passed to an INT parameter, Databricks casts the value implicitly. With ANSI mode off, values that can't be cast become NULL and no error is raised. The documented example is a row filter meant to return only rows where department IS NULL. Because every department string became NULL, it returned every row. Databricks recommends enabling ANSI mode so that a failed cast raises an error. The fix is to declare the right parameter type:
Checkpoint 4 of 4· Fill the gap
Which parameter type makes this row filter safe for the STRING department column?
CREATE FUNCTION salary_filter(dept ? ) RETURNS BOOLEAN
RETURN dept IS NULL;The parameter type has to match the column's type. Declared as INT, 'exec' and 'engineering' are cast to NULL and the filter lets every row through.
Source: docs.databricks.comPerformance comes second. The engine always chooses security over optimization, so a heavy UDF slows down every query on the table. The guidance is to keep UDFs simple (CASE expressions rather than mapping tables or subqueries), use few distinct masks on large tables, pass few UDF arguments, and prefer SQL UDFs to Python ones. Also use expressions that can't throw, such as try_divide, because an error like division by zero could reveal values before they're masked. Only one distinct row filter resolves per user and table, so teams often combine conditions with AND. Too many conjuncts make it more likely that one of them hits a slow pattern.
Getting the UDF right is only half of compliance. You also need to verify that the controls work. Data Classification's results show, for each classification type, the percentage of users who accessed unmasked data of that classification over the last 7 days, so you can assess how exposed the sensitive data is. For a single classification, the User Access tab lists the users who accessed those columns and whether each had masked or unmasked access. Classification results are also available in a results system table, and seeing sample values from it requires SELECT on that table.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.A table owner who dislikes an ABAC mask on their table can simply remove it.Why is that wrong?
ABAC policies can be defined by higher-level admins, and table owners can't override or remove them. Only table-level masks are under the owner's control.
Covered in Two ways to attach a filter or mask
2.Use ALTER TABLE ... ALTER COLUMN ... SET MASK to add a mask to a pipeline's streaming table.Why is that wrong?
ALTER TABLE commands are not allowed on streaming tables. Declare the mask or row filter in the CREATE OR REFRESH definition.
Covered in Masks on pipeline streaming tables and materialized views
3.During a pipeline refresh, CURRENT_USER in a mask function evaluates to whoever will eventually query the table.Why is that wrong?
During refresh, mask and filter functions run with the pipeline owner's rights. The invoker's context is used only at query time.
Covered in Masks on pipeline streaming tables and materialized views
4.If a row filter UDF's parameter type doesn't match the column, the query fails, so the mistake can't leak data.Why is that wrong?
With ANSI mode disabled, values that can't be cast silently become NULL and no error is raised. A filter can then return every row.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Each column can have one mask.”
↩︎ Two ways to attach a filter or mask“These runtimes fail securely, meaning that if you try to access tables from these runtimes, no data is returned.”
↩︎ Two ways to attach a filter or mask“MERGE statements do not support tables with row filter or column-mask policies that contain nesting, aggregations, windows, limits, or non-deterministic functions.”
↩︎ Two ways to attach a filter or mask“When the query engine must choose between optimization and protecting against information leakage from filtered or masked values, it always makes the secure choice”
↩︎ Writing masking UDFs that don't leak or stall“Use deterministic expressions that never throw errors, such as try_divide.”
↩︎ Writing masking UDFs that don't leak or stall“Only one distinct row filter can resolve at runtime for a given user and table”
↩︎ Writing masking UDFs that don't leak or stall“ABAC scales across catalogs and schemas and can be defined by higher-level admins, so table owners can't override or remove them.”
↩︎ Exam trap 1“With ANSI mode disabled (spark.sql.ansi.enabled = false), uncastable values are silently converted to NULL, no error is raised”
↩︎ Exam trap 4“Table-level column masks are bound to a column using ALTER TABLE ... ALTER COLUMN ... SET MASK and managed by the table owner.”
↩︎ Checkpoint - 2.https://docs.databricks.com/aws/en/ldp/unity-catalogOfficial docs
“Row filters and column masks on materialized views and streaming tables should be added, updated, or dropped through the CREATE OR REFRESH statement.”
↩︎ Masks on pipeline streaming tables and materialized views“Use DESCRIBE EXTENDED, INFORMATION_SCHEMA, or the Catalog Explorer to examine the existing row filters and column masks”
↩︎ Masks on pipeline streaming tables and materialized views“set the skipChangeCommits flag when reading the source streaming table.”
↩︎ Masks on pipeline streaming tables and materialized views“row filter and column mask functions run with the pipeline owner's rights.”
↩︎ Exam trap 3“functions that check user context (such as CURRENT_USER and IS_MEMBER) are evaluated using the invoker's user context.”
↩︎ Checkpoint - 3.https://docs.databricks.com/aws/en/ldp/gdprOfficial docs
“(1) delete data in the source Delta tables using DML, (2) delete data from the streaming table using DML”
↩︎ Masks on pipeline streaming tables and materialized views - 4.
“Unmasked user access (last 7d): The percentage of users who accessed unmasked data of that classification over the last 7 days.”
↩︎ Writing masking UDFs that don't leak or stall
Also cited
“ALTER TABLE commands are disallowed on streaming tables.”
↩︎ Exam trap 2