Databricks Certified Data Engineer Professional Lessons
46 lessons, one per exam-guide subdomain, in the order the guide teaches them. Every claim is cited to the official documentation.
Domain 1: Developing Code for Data Processing using Python and SQL
12 lessons · 23% of the exam
1.1Bundle project structure: databricks.yml, include, and targets
Subdomain 1.1: Implement scalable Python project structures for Declarative Automation Bundles (formerly Databricks Asset Bundles / DABs) to support modular development and CI/CD integration.
2 pages · 21 min read
1.2Troubleshooting serverless library installs: paths, architecture, private repos and %pip
Subdomain 1.2: Apply troubleshooting techniques to dependency conflicts and installation failures for external libraries (PyPI, local wheels, source archives) across serverless, pipeline, and bundle-deployed environments.
2 pages · 21 min read
1.3Session-scoped Python and pandas UDFs in PySpark
Subdomain 1.3: Develop User-Defined Functions (UDFs) using Pandas, Python, and SQL, including Unity Catalog functions for the given constraints.
2 pages · 23 min read
1.4Auto Loader Ingestion into Lakeflow Streaming Tables
Subdomain 1.4: Create production-ready data pipelines for streaming data using Lakeflow Declarative Pipelines and Auto Loader.
2 pages · 23 min read
1.5Lakeflow Jobs in the UI: Tasks, Triggers and Retries
Subdomain 1.5: Create and automate ETL workloads using Lakeflow Jobs via the UI, API, and CLI.
2 pages · 20 min read
1.6Streaming Tables vs Materialized Views: Semantics, Correctness and Latency
Subdomain 1.6: Choose between a streaming table and a materialized view for a given latency, cost, and refresh requirement.
2 pages · 17 min read
1.7AUTO CDC Flows and SCD Type 1 in Lakeflow Pipelines
Subdomain 1.7: Implement CDC pipelines using AUTO CDC APIs (formerly APPLY CHANGES) in Lakeflow Declarative Pipelines, including SCD Type 1 and Type 2 via stored_as_scd_type.
2 pages · 20 min read
1.8Structured Streaming vs Spark Declarative Pipelines: Control or Automation
Subdomain 1.8: Choose between Spark Structured Streaming and Apache SparkTM Declarative Pipelines for scalable ETL given operational constraints.
2 pages · 17 min read
1.9If/else Condition Tasks in Lakeflow Jobs
Subdomain 1.9: Create Lakeflow Jobs that use control-flow operators such as If/Else conditions and For Each loops.
2 pages · 21 min read
1.10Serverless Environments and Dependency Management
Subdomain 1.10: Choose appropriate compute and configuration for environments and dependencies — including serverless compute (serverless environments, dependency management, performance mode), high-memory notebook tasks, and auto-optimization settings (e.g., disallowing retries).
2 pages · 16 min read
1.11PySpark Unit Tests with DataFrame.transform and assertDataFrameEqual
Subdomain 1.11: Develop unit and integration tests using assertDataFrameEqual, assertSchemaEqual, DataFrame.transform, and testing frameworks to ensure code correctness.
2 pages · 23 min read
1.12Structured Streaming Watermarks and Output Modes
Subdomain 1.12: Apply Structured Streaming stateful-processing semantics, including watermarks, output modes, foreachBatch, and checkpoints for fault-tolerant exactly-once state recovery.
2 pages · 19 min read
Domain 2: Data Ingestion & Acquisition
5 lessons · 12% of the exam
2.1Auto Loader File Ingestion: JSON, CSV, Parquet, Binary, Delta and Iceberg
Subdomain 2.1: Develop data ingestion pipelines to ingest a variety of data formats, including Delta Lake, Parquet, JSON, Iceberg, CSV, and Binary from diverse sources such as message buses (Kafka, Kinesis, Pub/Sub) and cloud storage.
2 pages · 22 min read
2.2AUTO CDC in Lakeflow Pipelines: Keys, Sequencing and SCD Types
Subdomain 2.2: Configure incremental CDC pipelines using Lakeflow Pipelines with Delta or Iceberg as the target table format.
2 pages · 23 min read
2.3Lakeflow Connect Database Connectors: Gateway Architecture and Source Setup
Subdomain 2.3: Configure CDC ingestion pipelines from relational database sources, including SQL Server, MySQL, and PostgreSQL, using Lakeflow Connect.
2 pages · 23 min read
2.4OpenSharing: Databricks-to-Databricks vs Databricks-to-Open Sharing
Subdomain 2.4: Implement OpenSharing (D2D and Databricks-to-Open) and Clean Rooms for privacy-preserving collaboration.
2 pages · 19 min read
2.5Lakehouse Federation: Connections, Foreign Catalogs and UC Permissions
Subdomain 2.5: Configure Lakehouse Federation with governance across supported source systems, applying UC permissions and connection-level credentials.
15 min read
Domain 3: Data Manipulation
4 lessons · 12% of the exam
3.1Window Functions in Spark SQL and PySpark: Partitions, Ranking and Frames
Subdomain 3.1: Apply advanced data transformations, including window functions, joins, and aggregations, using Spark SQL and PySpark to process large datasets.
2 pages · 21 min read
3.2Modeling Semi-Structured Data with the VARIANT Type
Subdomain 3.2: Develop a model and query semi-structured data using the VARIANT data type and related functions (e.g., parse_json, variant_get).
2 pages · 18 min read
3.3ai_query and AI Functions: Syntax, Endpoints and Arguments
Subdomain 3.3: Apply AI functions, including ai_query, to perform model inference within data pipelines for enrichment and classification tasks.
2 pages · 21 min read
3.4Pipeline Expectations: Warn, Drop, and Fail on Bad Records
Subdomain 3.4: Implement data quality expectations in Lakeflow Declarative Pipelines to quarantine, drop, or fail on bad records.
2 pages · 20 min read
Domain 4: Monitoring and Alerting
5 lessons · 10% of the exam
4.1Cost Analysis with system.billing.usage and Compute System Tables
Subdomain 4.1: Use Databricks system tables (billing, compute, access, lakeflow) for cost analysis, auditing, and workload monitoring.
2 pages · 20 min read
4.2Monitor Jobs and Pipelines with the Databricks CLI, REST API and SDK
Subdomain 4.2: Use Databricks REST APIs / CLI and SDK for monitoring and analyzing jobs and pipelines.
15 min read
4.3Pipeline Event Log: Location, Schema and Update Health
Subdomain 4.3: Use Apache Spark Declarative Pipelines event logs to monitor pipeline health and data quality.
2 pages · 23 min read
4.4Databricks SQL Alerts: Conditions, Metric Views, and Data Quality Signals
Subdomain 4.4: Use Databricks Lakehouse alerts to monitor governed business metrics, data quality and anomaly signals, usage and cost, SQL warehouse/query health, audit and security events, AI agent quality, and Lakeflow Job branching.
2 pages · 24 min read
4.5Monitor Job Status in the Lakeflow Jobs UI
Subdomain 4.5: Use Lakeflow Jobs UI and Jobs API to monitor job status and performance metrics.
2 pages · 24 min read
Domain 5: Cost & Performance Optimization
6 lessons · 15% of the exam
5.1Unity Catalog Managed Tables and Predictive Optimization
Subdomain 5.1: Explain how Unity Catalog managed tables, Predictive Optimization, and Liquid Clustering reduce operational and maintenance overhead for a given workload.
2 pages · 20 min read
5.2Liquid Clustering and CLUSTER BY AUTO: Choosing a Table Layout
Subdomain 5.2: Choose the appropriate Delta optimization technique (deletion vectors, Liquid Clustering, CLUSTER BY AUTO) for a given table access pattern.
2 pages · 19 min read
5.3Disk cache (formerly Delta cache): faster repeated reads on Databricks
Subdomain 5.3: Understand how caching (Delta cache) improves query performance for repeated reads of the same data.
13 min read
5.4Delta Change Data Feed: Enable It and Read Row-Level Changes
Subdomain 5.4: Apply Change Data Feed (CDF) to expose row-level changes (updates/deletes) for efficient incremental downstream processing.
2 pages · 21 min read
5.5Query Profile Basics: Opening It, Reading the DAG, and Using Performance Insights
Subdomain 5.5: Use the query profile to identify performance bottlenecks, such as data-skipping inefficiencies, join strategies, and shuffle operations.
2 pages · 19 min read
5.6Partitioning vs Liquid Clustering by Table Size
Subdomain 5.6: Compare Liquid Clustering vs partitioning/ZORDER for a given table size and query pattern.
2 pages · 19 min read
Domain 6: Ensuring Data Security and Compliance
5 lessons · 8% of the exam
6.1Unity Catalog Privileges: Least-Privilege Grants on Securable Objects
Subdomain 6.1: Apply least-privilege access control lists (ACLs) to secure Unity Catalog securable objects and workspace resources.
2 pages · 20 min read
6.2ABAC in Unity Catalog: Governed Tags and Policy Scope
Subdomain 6.2: Apply attribute-based access control (ABAC) policies with governed tags to enforce row filters and column masks at scale.
2 pages · 22 min read
6.3Unity Catalog column masks: masking UDFs, syntax and type safety
Subdomain 6.3: Apply anonymization and pseudonymization methods — such as hashing, tokenization, suppression, and generalization — to confidential data (e.g., using column masks and related Unity Catalog features).
2 pages · 18 min read
6.4PII Detection and Tag-Driven Masking with Data Classification and ABAC
Subdomain 6.4: Implement a compliant data pipeline that enforces PII detection and masking controls across batch and streaming workloads using Unity Catalog features.
2 pages · 21 min read
6.5GDPR Right-to-Erasure Deletes in Delta Lake: Bronze First, Then Propagate
Subdomain 6.5: Implement a data purging strategy that satisfies data retention and deletion policies (e.g., GDPR right-to-erasure, GDPR right-to-be-forgotten deletion) using Delta Lake and Unity Catalog features.
2 pages · 19 min read
Domain 7: Data Governance
2 lessons · 5% of the exam
7.1Unity Catalog Tags for Data Discovery
Subdomain 7.1: Demonstrate understanding of Unity Catalog tags and comments as mechanisms for adding metadata to securable objects to improve data discoverability.
2 pages · 22 min read
7.2Unity Catalog Privilege Inheritance: Catalogs, Schemas, and Objects
Subdomain 7.2: Demonstrate understanding of the Unity Catalog permission inheritance model as a mechanism for managing access control across catalogs, schemas, and objects.
2 pages · 20 min read
Domain 8: Debugging and Deploying
5 lessons · 10% of the exam
8.1Spark UI, Thread Dumps, and Cluster Logs for Troubleshooting
Subdomain 8.1: Identify pertinent diagnostic information using Spark UI, cluster logs, system tables, and query profiles to troubleshoot errors.
2 pages · 19 min read
8.2Diagnosing Failed Lakeflow Job Runs
Subdomain 8.2: Analyze the errors and remediate the failed job runs with job repairs and parameter overrides.
2 pages · 20 min read
8.3Debugging Lakeflow Pipelines with the Event Log
Subdomain 8.3: Use event logs and the Spark UI to debug pipelines and Spark workloads.
2 pages · 20 min read
8.4Declarative Automation Bundles: databricks.yml, Targets and the Bundle Lifecycle
Subdomain 8.4: Deploy Databricks resources using Declarative Automation Bundles (formerly Databricks Asset Bundles).
2 pages · 19 min read
8.5Git Folders CI/CD Flows: Admin, User and Merge
Subdomain 8.5: Configure Git-based CI/CD workflows using Databricks Git folders (formerly Repos) to deploy notebooks and code.
2 pages · 16 min read
Domain 9: Data Modeling
2 lessons · 5% of the exam
9.1Partitioning vs Liquid Clustering: Choosing a Table Layout for Large Delta and Iceberg Tables
Subdomain 9.1: Design scalable Delta/Iceberg table layouts for large data assets by mapping partitioning to data grain, aligning clustering to relationship access patterns, and maintaining balanced file sizes through compaction.
2 pages · 20 min read
9.2Star Schemas and Materialized Views for Analytical Workloads
Subdomain 9.2: Design dimensional models for analytical workloads, leveraging Materialized Views for pre-computed aggregation and Unity Catalog Metric Views for governed, reusable metric definitions, to ensure efficient querying and aggregation.
2 pages · 19 min read