Debugging Lakeflow Pipelines with the Event Log

1.Where the event log lives and who can read it

Most people first see the event log in the UI. When a pipeline update has errors, they appear in the bottom pane of the monitoring page, and a View logs button opens the event log for that run. In the Lakeflow Pipelines Editor the same errors show up in the Issues pane. Behind both views is an ordinary Delta table that you can query with SQL. For anything beyond a quick look, querying the table is the stronger skill.

By default, the pipeline writes its event log to a hidden Delta table in the pipeline's default catalog and schema. The table is named event_log_{pipeline_id}, where the pipeline's UUID has its dashes replaced by underscores. You can see the table in system.information_schema.tables, but not in Catalog Explorer or any other workspace UI page. The only way to read it is the event_log() table-valued function, called with the pipeline ID:

SELECT * FROM event_log(<pipelineId>);

Two operational rules follow. First, never delete the event log, or the catalog or schema it is published in. If you do, future updates of the pipeline can fail. Second, the event log keeps history the UI drops. Pipelines keep 60 days of past updates in the update history drop-down. Older updates disappear from that list but stay in the event log, so the table is where you go to investigate a failure from three months ago.

Sources12

2.Publishing the event log and sharing it through a view

If a team needs to query the log by name, you can publish it in the pipeline's Advanced settings. Give it a table name and, if you want, a catalog and schema. In the pipeline settings JSON this is the event_log object:

{
  "id": "ec2a0ff4-d2a5-4c8c-bf1d-d9f12f10e749",
  "name": "billing_pipeline",
  "event_log": {
    "catalog": "catalog_name",
    "schema": "schema_name",
    "name": "event_log_table_name"
  }
}

The event log location also serves as the schema location for any Auto Loader queries in the pipeline. That is one more reason to treat the table carefully. Before you widen access, Databricks recommends that you create a view over the table instead of sharing the table itself. With some compute settings, users who are given the table directly could gain access to schema metadata. The documentation's debugging queries all assume a view called event_log_raw. For an unpublished log, the view selects from event_log(<pipelineId>) instead of a table name.

CREATE VIEW event_log_raw
AS SELECT * FROM <catalog_name>.<schema_name>.<event_log_table_name>;

In Unity Catalog, views support streaming queries. That means you can follow the event log as it grows, for example to feed a monitor, instead of re-running batch queries.

df = spark. ? .table("event_log_raw")

Sources2

3.Reading an event: event_type, level, error and details

Every row has the same top-level fields. timestamp is in UTC. message is a readable summary. origin identifies the pipeline, update, flow and dataset. error holds the failure details when something went wrong. details is a JSON string whose shape depends on event_type, and it is the primary field for analysis. You parse it with the : operator or from_json. So when you debug, you usually filter on event_type and level, then dig into details and error.

The level field has four values. INFO marks routine events. WARN marks unexpected but non-critical issues. ERROR marks failures that might need your attention. The fourth value, METRICS, is easy to overlook: these high-volume events are stored only in the Delta table and are never shown in the pipelines UI. So a metric you can't find in the UI may still be in the table. Finally, check maturity_level before you build alerts on a field. Fields marked EVOLVING or DEPRECATED may change or stop appearing.

Sources3

4.From the system table to a fixed pipeline

The per-pipeline event log covers one pipeline. To look across all of them, Databricks recommends the pipeline_events system table, which is in Beta. It holds events for every pipeline in every workspace of a region. To read it, a user must either be both a metastore admin and an account admin, or hold USE and SELECT on the system schemas. The query below finds the most recent error for each pipeline in the last seven days, along with its outermost exception:

SELECT
  workspace_id,
  pipeline_id,
  event_time,
  message,
  error.exceptions[0].error_class AS exception_error_class,
  error.exceptions[0].sql_state   AS exception_sql_state
FROM
  system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
  level = 'ERROR'
  AND event_time >= current_timestamp() - INTERVAL 7 DAYS
QUALIFY
  ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id ORDER BY event_time DESC) = 1
ORDER BY
  event_time DESC

For a single failed update, the documentation lays out a workflow:

1. Start where the UI points you. That is the Issues pane in the editor, or the bottom pane of the monitoring page. 2. Find what failed in the Graph view. Failed tables and flows are highlighted there, so you can see whether the failure stayed in one table or spread to its downstream tables. 3. Query the event log for the full error. The UI can truncate messages, while the error and details columns hold the full stack trace. 4. Re-run only what failed. Refresh failed tables retries the failed tables plus everything downstream of them.

If the failure looks like a code or configuration problem, run Validate before a full update. Validate re-checks the graph and source code without materializing any data.

Sources41

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.The default event log table can be found and opened in Catalog Explorer like any other table.Why is that wrong?

   The default event log is hidden. It is listed in information_schema.tables but not shown in Catalog Explorer, and you read it with event_log().

   Covered in Where the event log lives and who can read it

2. 2.Once an update drops out of the 60-day update history, its events are gone.Why is that wrong?

   The UI keeps 60 days of updates, but older updates stay in the event log table.

   Covered in Where the event log lives and who can read it

3. 3.Every event in the log can also be seen in the pipelines UI.Why is that wrong?

   Events with level METRICS are stored only in the Delta table, so you have to query the log to see them.

   Covered in Reading an event: event_type, level, error and details

4. 4.A failure that reproduces in the editor will fail the same way when the pipeline runs on a schedule.Why is that wrong?

   Updates started manually from the editor have automatic retries off. Scheduled and API-triggered updates retry recoverable failures automatically.

   Covered in From the system table to a fixed pipeline