PII Detection and Tag-Driven Masking with Data Classification and ABAC

1.Detecting PII with Data Classification

A pipeline can't mask PII that nobody has found. In Unity Catalog, Databricks Data Classification does the finding. It uses an agent, helped by a large language model, to classify the tables in a catalog. Scanning is incremental, so new data is classified without manual setup. Scans also go beyond plain Delta tables. By default Data Classification covers tables, streaming tables and materialized views, so one detector handles both what batch jobs write and what streaming pipelines publish. New tables and columns are usually scanned within 24 hours of being created.

Detections are expressed as class.* system governed tags, for example class.name, class.email_address and class.us_ssn. The tags are grouped by compliance framework (PII, PCI DSS, GDPR, HIPAA and others), and there are regional tags such as class.uk_nino and class.in_aadhaar. You don't need any custom tags for column-level detection.

The results page lists every classification with its detected columns, its auto-tagging status, its compliance frameworks, and an Unmasked user access (last 7d) metric that shows how exposed the data is. Clicking Review opens the detected columns with sample values. Use the Exclude icon on any false positive, and only then enable automatic tagging. Tagging is not instant: after you enable it, tags populate in the next scan, within about 24 hours, so a detected column stays untagged (and unmasked by any tag-based policy) until then. Auto-tagging can be set at metastore level or catalog level, and the catalog setting overrides the metastore one. For a single catalog, the same panel also lets you create an ABAC policy straight from a classification.

The privileges differ for each action. One default is a common trap: only account admins hold MANAGE and ASSIGN on the data classification system governed tags until they grant them to other users, service principals or groups.

Sources12

2.Turning tags into masks with ABAC policies

A tag by itself hides nothing. Enforcement comes from an ABAC policy. The policy is attached to a securable (catalog, schema or table; metastore-level attachment is in Beta), selects columns by their tags, and runs a masking function on them. Creating a policy needs Databricks Runtime 16.4 or above or serverless compute, MANAGE on (or ownership of) the securable the policy attaches to, and EXECUTE on the UDF that holds the masking logic. Here is the documented column mask policy:

CREATE FUNCTION ssn_to_last_nr (ssn STRING, nr INT) RETURNS STRING
  RETURN right(ssn, nr);

CREATE POLICY mask_ssn
ON SCHEMA prod.customers
COLUMN MASK ssn_to_last_nr
TO us_analysts EXCEPT admins
FOR TABLES
MATCH COLUMNS has_tag_value('pii', 'ssn') AS ssn
ON COLUMN ssn
USING COLUMNS (4);

Here is what each clause does. TO names the principals the policy applies to, and EXCEPT exempts principals, who then see unmodified data. An optional WHEN clause decides which tables the policy applies to, based on table tags. MATCH COLUMNS then selects columns with has_tag / has_tag_value. A policy can have up to three of these expressions, and all of them must match for the policy to apply. That is different from combining conditions with AND, OR and NOT inside a single expression, which decides whether one column matches. ON COLUMN chooses which matched alias to mask. USING COLUMNS passes arguments to the UDF, here the constant 4.

The masking method lives in the UDF, and its return type must be castable to the column's type. The sources show two approaches. The first is partial suppression, like right(ssn, nr) above. The second is the built-in system.data_classification.mask_value, a type-aware masking function that returns 0 for integers, DATE '1970-01-01' for dates and a SHA-256 hash for strings.

Streaming and pipeline workloads add a wrinkle: there are two identity contexts. When a pipeline refreshes a materialized view or streaming table, ABAC policies are evaluated with the pipeline owner's or run-as identity. If that identity is subject to a policy, the table permanently contains masked or filtered data. To avoid this, add the pipeline owner or run-as identity to the EXCEPT clause of the policies on every table the pipeline reads, and use TO to say who receives masked data. Only exempt trusted identities such as ETL service principals, because exempted principals see unmasked data. At query time it is different: functions that check user context, such as CURRENT_USER and IS_MEMBER, are evaluated using the invoker's context, so each analyst gets their own masking.

You can also mask directly in a streaming table definition, for example when ingesting from cloud files or from another streaming table. This per-table form is managed through CREATE OR REFRESH:

CREATE OR REFRESH STREAMING TABLE customers_silver (
  id int COMMENT 'This is the customer ID',
  name string,
  region string,
  ssn string MASK catalog.schema.ssn_mask_fn COMMENT 'SSN masked for privacy'
)
WITH ROW FILTER catalog.schema.us_filter_fn ON (region)
AS SELECT * FROM STREAM(customers_bronze)

Sources3245

3.Closing the gap: secure new tables by default

There's still a window of exposure. A new table can take up to 24 hours to be scanned, and tags from auto-tagging populate in the next scan, so until it's tagged, any user with SELECT on the schema can read its raw PII. Databricks' control-tag pattern closes that window. You create a governed tag review_status with the values pending and reviewed, then set review_status = 'pending' on the schema. Every table created in that schema inherits the tag. A first policy masks every column of any table still marked pending. Remember that by default only account admins have MANAGE and ASSIGN on the classification system tags, so make sure whoever enables auto-tagging has been granted ASSIGN on them.

CREATE POLICY review_pending_policy
ON SCHEMA abac_tutorial.secure_default
COLUMN MASK system.data_classification.mask_value
TO `account users`
FOR TABLES
WHEN has_tag_value('review_status', 'pending')
MATCH COLUMNS  ?  AS m ON COLUMN m;

Data Classification scans in the background and, once auto-tagging is on, applies class.* tags. A steward reviews those tags and corrects false positives. The steward then sets review_status to reviewed on the table itself, which overrides the tag inherited from the schema. A second policy now applies and masks only the classified columns:

CREATE POLICY review_complete_policy
ON SCHEMA abac_tutorial.secure_default
COLUMN MASK system.data_classification.mask_value
TO `account users`
FOR TABLES
WHEN has_tag_value('review_status', 'reviewed')
MATCH COLUMNS (
  has_tag('class.name')
  OR has_tag('class.email_address')
  OR has_tag('class.us_ssn')
  OR has_tag('class.phone_number')
  OR has_tag('class.credit_card')
  OR has_tag('class.date_of_birth')
) AS m
ON COLUMN m;

Sources21

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.As soon as Data Classification detects PII, the column is tagged and ready for ABAC policies.Why is that wrong?

   Detection and tagging are separate. You review the detections, then enable automatic tagging for each classification. Only after that are existing and future detections tagged.

   Covered in Detecting PII with Data Classification

2. 2.Several MATCH COLUMNS expressions in one policy are ORed, so a column that matches any of them is masked.Why is that wrong?

   Every MATCH COLUMNS expression must match for the policy to apply. To mean 'either', combine the conditions with OR inside one expression, as review_complete_policy does.

   Covered in Turning tags into masks with ABAC policies

3. 3.A pipeline's service principal needs no exemption, because masking is applied per analyst at query time anyway.Why is that wrong?

   Pipeline refreshes evaluate ABAC policies with the pipeline owner's or run-as identity. If that identity is masked, the streaming table or materialized view permanently stores masked data, so exempt it with EXCEPT.

   Covered in Turning tags into masks with ABAC policies