Pipeline Event Log: Location, Schema and Update Health

1.Why the event log is the source of truth

The exam calls them Spark Declarative Pipelines. The Databricks docs call them Lakeflow pipelines. Both names refer to the same pipelines, and you can monitor them at three levels of detail. The Jobs & Pipelines list shows status indicators for the last five runs. The pipeline monitoring UI shows the graph colour-coded by status, per-table row counts, data quality metrics and backlog for the selected update. Under both of these sits the event log.

The observability page describes the event log as a structured Delta table that you query for programmatic or historical questions, such as update outcomes, data quality trends and resource usage. The same entries are available in three places: the monitoring UI, the Pipelines REST API, and direct SQL against the table. This lesson uses direct SQL. To be alerted on pipeline events, configure notifications. Set them directly on a pipeline that runs on its own schedule, or as job notifications when the pipeline runs inside a Lakeflow Job. Event hooks let you run custom actions, such as sending an alert, when an event is logged, and the event log records their status in hook_progress events. Configuring hooks is covered elsewhere.

Sources12

2.Finding and querying the event log

This section covers the default behaviour for Unity Catalog pipelines in the default publishing mode. Unity Catalog pipelines that use legacy publishing mode, and Hive metastore pipelines, have their own behaviour and syntax in separate docs, so do not assume everything below applies to them. The pipeline writes its event log to a hidden Delta table in the default catalog and schema configured for the pipeline. The table is named event_log_{pipeline_id}, where the pipeline ID has its dashes replaced by underscores. It is listed in system.information_schema.tables, but Catalog Explorer and the other workspace UI pages do not show it. By default, only the pipeline's run-as user can query it. To read it, pass the pipeline ID to the event_log() table-valued function:

SELECT * FROM event_log(<pipelineId>);

SELECT * FROM  ? (<pipelineId>);

To make the log a named table, publish it from the pipeline's Advanced settings. You give the table a name and can optionally choose a catalog and schema. The event log location is also where any Auto Loader queries in the pipeline keep their schema.

{
  "id": "ec2a0ff4-d2a5-4c8c-bf1d-d9f12f10e749",
  "name": "billing_pipeline",
  "event_log": {
    "catalog": "catalog_name",
    "schema": "schema_name",
    "name": "event_log_table_name"
  }
}

Before you give anyone else access, create a view over the table and grant on the view. The docs warn that some compute settings can expose schema metadata if the table itself is shared. The view also gives every monitoring query a stable name, and because Unity Catalog views support streaming queries, you can read it with spark.readStream. Never delete the event log, or the catalog or schema it is published in, because the pipeline may then fail to update.

CREATE VIEW event_log_raw
AS SELECT * FROM <catalog_name>.<schema_name>.<event_log_table_name>;

Sources2

3.Reading the event schema

Every row has the same top-level fields. id identifies the record. timestamp is in UTC. origin carries the pipeline_id, pipeline_name, update_id (the equivalent of a run ID), flow_name and dataset_name. message is human-readable text, and error holds error details when something failed. You will filter mostly on event_type and level. details is the main field for analysis: it is a JSON string whose layout depends on event_type, and you extract values from it with the : operator, as in details:flow_progress.status.

level has four values. INFO is informational, WARN is an unexpected but non-critical issue, ERROR is an event failure that may need attention, and METRICS covers the high-volume events described above. maturity_level tells you whether an event's schema is stable. STABLE and NULL both mean stable; NULL appears on records written before the field was added. EVOLVING means the schema may change. DEPRECATED means the runtime may stop emitting the event at any time. Databricks advises against building monitoring or alerts on EVOLVING or DEPRECATED fields. The event types you will use most often are listed below.

For failures, filter on level = 'ERROR'. The docs' example dashboard does this against the pipeline_events system table, which covers all pipelines in a region, and the query below counts errors per pipeline per hour so you can spot spikes. Read message and the error field on those rows to see what went wrong.

SELECT
  pipeline_id,
  date_trunc('HOUR', event_time) AS hour,
  count(*) AS num_errors
FROM
  system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
  level = 'ERROR'
  AND event_time >= current_timestamp() - INTERVAL 7 DAYS
GROUP BY
  pipeline_id,
  date_trunc('HOUR', event_time)
ORDER BY
  hour DESC

Sources34

4.Reporting update health from update events

To check pipeline health, start with the update. The docs' run-overview query combines two event types. The earliest create_update timestamp is the start time. The end time is the latest update_progress event whose state is terminal: COMPLETED, FAILED or CANCELED. If no terminal state exists yet, which is normal for a continuous pipeline, the query uses current_timestamp(). Both events are grouped by origin.update_id. The latest state per update is the update_progress row ranked first by timestamp descending, and that state is pulled out of the JSON like this:

FROM_JSON(details, 'struct<update_progress: struct<state: string>>').update_progress.state AS last_update_state,

Below the update, origin.flow_id tells you whether a flow is still refreshing incrementally. It stays the same across updates while the flow refreshes incrementally. It changes on a full refresh, a checkpoint reset or a full recomputation. When a materialized view unexpectedly refreshes in full, look at the planning_information details for that flow.

One access caveat applies to every query in this lesson. By default, only the pipeline's run-as user can query the hidden event log through event_log(), so a data engineer who is not that user needs access granted first, ideally through a view. When an update fails, filter the same log on level = 'ERROR' and read message and error alongside the update_progress state.

Sources32

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.If an event isn't in the pipeline UI's event list, the pipeline never recorded it.Why is that wrong?

   Events at the METRICS level are written only to the Delta table and are never shown in the UI. You have to query the event log to see them.

   Covered in Reading the event schema

2. 2.Any field in the event log is a safe basis for production alerts.Why is that wrong?

   Only STABLE (or NULL) maturity fields are safe to alert on. EVOLVING fields may change shape, and DEPRECATED events may stop being emitted.

   Covered in Reading the event schema

3. 3.If the event log isn't in Catalog Explorer, it wasn't created and you have to publish it before you can query it.Why is that wrong?

   By default the log is a hidden table. It still exists and can be queried through the event_log() function.

   Covered in Finding and querying the event log

4. 4.Dropping an old event log table to save storage is harmless housekeeping.Why is that wrong?

   The pipeline depends on its event log. Deleting the log, or the catalog or schema that holds it, can make future updates fail.

   Covered in Finding and querying the event log