Reading MLflow 3 Evaluation Results to Pick the Most Reliable Context Configuration

1.Pull per-scorer metrics for each evaluation run

Each call to mlflow.genai.evaluate() records an evaluation run in your experiment, and each scorer adds an aggregate metric to that run. In the Databricks "Evaluate and improve an agent" tutorial, two versions of a RAG email app were evaluated on the same dataset with the same judges. Their runs are compared using mlflow.search_runs(). Two details from that code matter. First, the runs are fetched one at a time, because the filter syntax doesn't support IN or OR. Second, only some columns are kept: quality metrics end in /mean, and anything under agent/ (latency and token counts) is excluded from the quality comparison.

# Fetch runs separately since mlflow.search_runs doesn't support IN or OR operators
run_v1_df = mlflow.search_runs(
    filter_string=f"run_id = '{eval_results_v1.run_id}'"
)
run_v2_df = mlflow.search_runs(
    filter_string=f"run_id = '{eval_results_v2.run_id}'"
)

# Extract metric columns (they end with /mean, not .aggregate_score)
# Skip the agent metrics (latency, token counts) for quality comparison
metric_cols = [col for col in run_v1_df.columns
               if col.startswith('metrics.') and col.endswith('/mean')
               and 'agent/' not in col]

Sources1

2.A higher average can hide a regression on the metric your task needs

The tutorial reports an overall average improvement of +0.143 (+20.0%), and on most dimensions V2 really is better. It is much more concise and more relevant. But retrieval_groundedness, which checks whether the response is supported by the retrieved context, regressed. A per-trace drill-down in the tutorial shows why. On one customer request, V1 passed and V2 failed. The judge's rationale for V2 notes a promised status update "which is not directly supported by the retrieved context." Which configuration is "most reliable" therefore depends on the task you were given. Work out which scorers measure that task, then compare the runs on those scorers first. The overall average is a secondary signal. The sources define reliability through scorer results on a shared dataset. They don't describe a method for measuring variance across repeated runs of the same configuration, so don't assume one exists.

Sources1

3.Compare traces side by side in the Evaluation runs tab

Aggregate numbers tell you where to look. The traces tell you why a score changed. The tutorial calls the UI comparison view "the primary way to verify that a new prompt version outperforms the previous one." You open the experiment, go to its evaluation runs, select the runs you want and compare them. The right pane then shows each trace from the selected runs side by side. Clicking a request identifier opens the full trace from each run. From there, See details shows each assessment and See detailed trace view shows the trace itself, which is where you read rationales like the one quoted above.

Sources1

4.Scanning many runs: chart view and parallel coordinates

When you are comparing more than two configurations, a table comparing two runs isn't enough. The experiment's chart view compares runs visually. By default, "charts on this page show the most recent 10 runs," so if your configuration sweep produced more than that, change the number of runs displayed. You can filter runs by parameter, metric or tag, sort them by a parameter, and group them by parameter value with the Group by menu. If each run logged its context settings as parameters, grouping by those settings lets you see how each setting relates to the scores. A parallel coordinates plot goes further: you choose the parameters and metrics, and it draws how each run's settings line up with its scores. Experiment and run metadata is also available in MLflow system tables, if you would rather query it with Databricks SQL.

Sources234

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.The configuration with the higher overall average score is the most reliable for any given task.Why is that wrong?

   An average can rise while the metric the task depends on regresses. In the tutorial, V2 gained 20% overall but lost on retrieval_groundedness. Look at the regressed metrics first.

   Covered in A higher average can hide a regression on the metric your task needs

2. 2.You can fetch all the runs to compare in one mlflow.search_runs call with an IN or OR filter on run_id.Why is that wrong?

   mlflow.search_runs doesn't support IN or OR operators, so the tutorial fetches each run separately and compares the resulting DataFrames.

   Covered in Pull per-scorer metrics for each evaluation run

3. 3.The chart view automatically shows every run in the experiment.Why is that wrong?

   By default the charts show only the 10 most recent runs. You have to change the number of runs displayed to see a larger configuration sweep.

   Covered in Scanning many runs: chart view and parallel coordinates

Practise it for real

Evaluate two context configurations of the same agent on one dataset and decide which one is more reliable for a grounding-sensitive task

1. 1.Wrap each configuration as its own function decorated with @mlflow.trace (for example, two prompt or retrieval variants)

   Why: Direct evaluation calls predict_fn and captures traces, so each configuration needs an entry point MLflow can call

   You should see: Two callables that take the same inputs keys as your evaluation dataset

2. 2.Run mlflow.genai.evaluate() for the first configuration inside mlflow.start_run(run_name="v1"), passing your evaluation dataset and a scorer list that includes RetrievalGroundedness

   Why: A named run is easy to find later, and the scorer list defines what reliable means for this task

   You should see: An evaluation run named v1 in your experiment, with per-scorer /mean metrics

3. 3.Repeat with run_name="v2" and the second configuration, keeping data and scorers exactly the same

   Why: Only predict_fn should change, so that score differences come from the configuration

   You should see: A second evaluation run whose metrics can be compared with v1 column by column

4. 4.Fetch each run with mlflow.search_runs(filter_string=f"run_id = '...'") and select metrics.* columns ending in /mean, excluding agent/

   Why: search_runs doesn't support IN or OR, and the agent/ columns are latency and token counts, not quality

   You should see: A side-by-side table of per-scorer means for v1 and v2

5. 5.In Evaluation runs, select both runs, choose Actions > Compare, and open the traces where the grounding metric regressed

   Why: Judge rationales explain why a trace failed, which tells you whether the regression matters for the task

   You should see: Per-trace assessments with rationales for each version