Alerting on System Tables, Audit Logs, AI Agents, and Lakeflow Job Branches

1.Usage and cost alerts on system.billing.usage

A Databricks SQL alert runs a query on a schedule and sends a notification when the result crosses a threshold. System tables expose platform telemetry as queryable tables, so one alert mechanism can also watch spend, warehouses, security events, and jobs. Start with cost. Users who have access to system tables can query billing records in system.billing.usage. In each record, usage_metadata identifies the resource involved, identity_metadata identifies the user or service principal, and billing_origin_product and product_features identify the product. custom_tags holds compute tags.

The answer is custom_tags. It includes tags added by serverless usage policies, which is how serverless usage gets attributed to a team. For alerts, choose a documented sample that has no parameters, since alerts can't take them. One example is the ranking of jobs by DBU usage:

SELECT
  usage_metadata.job_id as `Job ID`, sum(usage_quantity) as `Usage`
FROM
  system.billing.usage
WHERE
  usage_metadata.job_id IS NOT NULL
GROUP BY
  `Job ID`
ORDER BY
  `Usage` DESC

Because this query sorts the heaviest job to the top, a condition on the first value of Usage checks the most expensive job. Several other billing samples use bind variables such as custom_tags [:key] = :value or :usage_start_time. Before using one of those as an alert, write the values directly into the SQL.

Sources12

2.SQL warehouse and query health

Alerts on warehouse events or query history catch slow queries, failed sessions, and capacity problems. Four system tables cover this area, and Databricks publishes example queries for them that you can turn into alerts.

SELECT
  statement_id,
  executed_by,
  statement_type,
  execution_status,
  total_duration_ms,
  execution_duration_ms,
  compilation_duration_ms,
  waiting_at_capacity_duration_ms,
  read_rows,
  produced_rows,
  start_time,
  statement_text
FROM
  system.query.history
WHERE
  compute.warehouse_id = '<warehouse-id>'
  AND start_time >= NOW() - INTERVAL 1 DAY

Each column answers a different health question. total_duration_ms shows slow queries, execution_status shows failures, and waiting_at_capacity_duration_ms shows queries that waited because the warehouse was full. An alert set on one of these values tells you when it moves into a range you didn't expect.

Sources13

3.Audit and security events

For security monitoring, alert on audit log queries. The audit log system table, system.access.audit (Public Preview), records user_identity, service_name, action_name, request_params, response, and source_ip_address. Keep a few details in mind:

- Filter on event_date rather than event_time for better performance. - Account-level events record workspace_id as 0. - Most audit logs are only available in the region of the workspace. - Some request_params keys that hold SQL definitions are hidden unless you are an account admin or a member of databricks_pii_access.

The documented example monitors Genie Agents, formerly called Genie Spaces. You can then set an alert to automate the check.

SELECT
  user_identity.email as user_email,
  action_name,
  request_params.space_id,
  request_params.feedback_rating,
  *
FROM
  system.access.audit
WHERE
  service_name = ' ? '
  AND action_name = 'updateConversationMessageFeedback'
  AND event_date >= current_date() - interval 30 days

Sources145

4.AI agent quality

The alerts overview lists agents as a target: alert on agent quality metrics so that failures and new issues come to light during development and in operation. The agent observability docs describe where those metrics come from. Production monitoring runs MLflow scorers on agent traces and attaches the results to each evaluated trace as feedback. Traces stored in Unity Catalog are governed and queryable with SQL.

These sources name the pattern but don't include a sample alert query on scorer results. Treat it as the general rule applied to agents: write a SQL query that returns a quality metric, then put an alert condition on it. The Genie feedback-rating audit query from the previous section is one documented, alertable signal of agent quality from users.

Sources167

5.Alerts as tasks in Lakeflow Jobs, and branching

An alert can also run as a task in Lakeflow Jobs. The condition check then happens as part of a pipeline run instead of on a clock. The task needs an existing Databricks SQL alert, CAN RUN permission on that alert, and a serverless or pro SQL warehouse. The task evaluates the alert separately from the alert's own schedule, which the job run leaves untouched. Alert tasks don't support parameters, and they don't support legacy alerts.

resources:
  jobs:
    my_job:
      name: my_job
      tasks:
        - task_key: alert_task
          alert_task:
            workspace_path: ./my_alert.dbalert.json
            warehouse_id: 1a111111a1111aa1

Lakeflow Jobs provides two ways to branch. Run if dependencies run a task based on whether upstream tasks succeeded, failed, or completed. The If/else condition task tests a boolean expression whose operands can reference job parameters, task parameters, or task values such as {{tasks.<task_name>.values.<value_name>}}. Downstream tasks depend on <task-name> (true) or <task-name> (false). Operator choice matters. == and != compare their operands as strings, so 12.0 == 12 is false. >, >=, <, and <= compare them as numbers. The condition task doesn't need a cluster, and it supports neither retries nor notifications.

resources:
  jobs:
    my-job:
      name: my-job
      tasks:
        - task_key: condition_task
          condition_task:
            op:  ? 
            left: '{{job.repair_count}}'
            right: '5'
        - task_key: notebook_task
          depends_on:
            - task_key: condition_task
              outcome: 'true'
          notebook_task:
            notebook_path: ../src/notebook.ipynb

Sources18910

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.A SQL alert task fails when its alert condition is met, so a Run if dependency on failure catches triggered alerts.Why is that wrong?

   The task reports Succeeded whether or not the condition triggered. It reports Failed only when evaluation errors, for example a warehouse connection problem or a query error.

   Covered in Alerts as tasks in Lakeflow Jobs, and branching

2. 2.An If/else condition of {{tasks.check.values.score}} == 12 is true when the task value is 12.0.Why is that wrong?

   == and != compare their operands as strings, so 12.0 == 12 evaluates to false. Use a numeric operator such as >= when the comparison has to be numeric.

   Covered in Alerts as tasks in Lakeflow Jobs, and branching

3. 3.Adding an alert to a job as a SQL alert task replaces the alert's own schedule.Why is that wrong?

   The task evaluates the alert separately, and the schedule on the underlying alert keeps running unchanged.

   Covered in Alerts as tasks in Lakeflow Jobs, and branching

Practise it for real

Build a cost alert on system.billing.usage and run it as a SQL alert task in a Lakeflow job

1. 1.Click Alerts in the sidebar, then click Create Alert. Paste the jobs-by-DBU query (SELECT usage_metadata.job_id ... FROM system.billing.usage ... ORDER BY Usage DESC) and click Run all.

   Why: The alert owns its query, and this sample has no parameters, which alerts don't support.

   You should see: A result table of job IDs, sorted by Usage from highest to lowest.

2. 2.Pick a SQL warehouse in the compute selector. In Condition, choose the first value of the Usage column, the > operator, and a static threshold of your choice.

   Why: Because the query sorts descending, the first value is the most expensive job.

   You should see: The condition panel shows Usage > your threshold.

3. 3.Click Test condition, then lower the threshold until the alert would trigger.

   Why: This confirms the condition logic before any notification goes out.

   You should see: The preview reports TRIGGERED at a low threshold and OK at a high one.

4. 4.Add yourself under Notifications, set a schedule with the calendar icon, and click View alert.

   Why: A saved, scheduled alert is required before a job can use it.

   You should see: The details page shows the alert and its evaluation history, plus a Run now button.

5. 5.In Jobs & Pipelines, click Create, then Job. Add a SQL Alert task, select your alert, and choose a serverless or pro SQL warehouse. Run the job.

   Why: Running the alert as a task moves the check into a pipeline run, independent of the alert's own schedule.

   You should see: The task reports Succeeded whether your alert evaluated to OK or TRIGGERED.