What you will be able to do
- Use trace data to tell whether a bad answer came from the retrieval step or the generation step
- Map low scores from MLflow's RetrievalRelevance, RetrievalSufficiency and RetrievalGroundedness judges to a likely failure mode
- Read the AI Search retrieval quality dashboard to spot ranking problems and failed queries
- Recognise when missing results point to a stale index rather than a relevance problem
Key concept
Locating the failing stage — Before you fix a RAG application, find out which stage produced the bad answer. A wrong answer can come from retrieval (the wrong context, or missing context) or from generation (good context the model used badly). Each case has a different fix.
1.First question: did retrieval fail, or did generation?
Databricks splits RAG quality issues into two kinds. Retrieval quality asks whether you are "retrieving the most relevant information for a given retrieval query". Generation quality asks whether the LLM made the best possible response from that information. The two overlap. Poor retrieval always drags generation down, so a wrong final answer doesn't tell you which stage is to blame. You need to see the intermediate step.
That is why evaluation and production logging capture more than the question and the answer. The guidance is that production logging "must track the inputs, outputs, and intermediate steps such as document retrieval". With the retrieved documents recorded next to the answer, you can look at a failing request and ask: was the needed information in the context? If it wasn't, you have a retrieval failure, and the rest of this lesson applies. If it was, the problem sits in the chain's generation step, and no change to the index or its permissions will fix it.
Checkpoint 1 of 5· Check yourself
Why does Databricks say production trace logging must record intermediate steps such as document retrieval, not just inputs and outputs?
Without the retrieved context in the log, a bad answer can't be attributed to a stage. The docs give root-cause diagnosis as the reason for logging intermediate steps.
“you need to know if the root cause of a low-quality answer is due to the retrieval step or a hallucination.”Source: docs.databricks.com
2.Reading MLflow retrieval judges as failure-mode signals
Reading traces by hand doesn't scale. MLflow's built-in LLM judges score the same question automatically across a whole evaluation set. Three of them look at retrieval, and each asks something different. RetrievalRelevance asks: "Is the retrieved context directly relevant to the user's request?" RetrievalSufficiency asks whether the context provides "all necessary information to generate a response that includes the ground truth facts". It needs expectations (ground truth), because it can only judge completeness against a known answer. RetrievalGroundedness checks the response, not the retrieval: is the answer supported by the context, or is the agent hallucinating?
The pattern across the three judges tells you which failure mode you're dealing with. Low relevance means the retriever is returning chunks that don't match the request, which points to irrelevant chunks or over-retrieval. Low sufficiency with acceptable relevance means the right kind of content arrives but something essential is missing. That points to a recall gap, and the cause could be poor ranking, a stale index, or content the retriever isn't allowed to see. Low groundedness while both retrieval judges look healthy sends you back to generation.
| Judge | Needs ground truth? | What it evaluates | Low score suggests |
|---|---|---|---|
| RetrievalRelevance | No | Is the retrieved context directly relevant to the user's request? | Irrelevant chunks or over-retrieval (a precision problem) |
| RetrievalSufficiency | Yes | Does the context provide all necessary information to generate a response that includes the ground truth facts? | Missing context (a recall problem) |
| RetrievalGroundedness | No | Is the response grounded in the information provided in the context? | A generation problem (hallucination), not a retrieval problem |
Checkpoint 2 of 5· Match them up
Match each MLflow judge to the question it answers
Tap a term, then the definition that fits it.
Relevance and sufficiency judge what the retriever returned. Groundedness judges whether the generator stayed faithful to that context.
“Is the response grounded in the information provided in the context? Is the agent hallucinating?”Source: docs.databricks.com
Checkpoint 3 of 5· Exam question
MLflow eval logs for a legal-research agent show high context relevance scores (retrieved chunks are on-topic) but low answer correctness, with the LLM judge repeatedly noting that a specific clause needed to answer the question is absent from any retrieved chunk even though it exists in the source contract. UC metadata confirms the contract document was indexed successfully. Which retrieval failure mode does this pattern indicate?
Correct answer: D — Under-retrieval caused by chunk boundaries splitting the needed clause away from the query terms that would otherwise surface it.
Sources3
3.Ranking and recall signals in the AI Search evaluation dashboard
MLflow judges score the whole agent. AI Search also has a built-in retrieval quality evaluation (Beta) that tests the index on its own. It needs a managed Delta Sync index. It generates queries from your documents, runs them through several retrieval strategies (ANN, hybrid and full-text), and runs each strategy "with and without the reranker". An LLM judge then scores every query and document pair on a 4-point relevance scale.
Some dashboard panels match specific failure modes. The failed-queries table lists "queries where the top-1 result was scored 0 (irrelevant)". These are the clearest examples of irrelevant results at the top. The line chart of average relevance by result position shows whether good documents are in the results but ranked too low. The table comparing base and reranker performance tells you whether reranking fixes the order. Read these results against your metric. If missing context leads to wrong answers, track Recall@k. If users need the exact answer at the top, track Precision@k.
Checkpoint 4 of 5· Put it in order
Put the stages of the AI Search retrieval quality evaluation pipeline in order
- 1.Score relevance of each query and document pair with an LLM judge
- 2.Compute metrics with confidence intervals and persist the results
- 3.Generate queries by sampling documents and using an LLM
- 4.Search across strategies, each with and without the reranker
The evaluation is a fixed four-stage pipeline: generate queries, search across strategies, score relevance, then compute metrics.
“An LLM judge evaluates every query and retrieved document pair on a 4-point relevance scale.”Source: docs.databricks.com
4.When the missing answer is recent: the stale-index signal
One kind of low sufficiency has a giveaway pattern. Questions about older material pass, and questions about recently added or changed documents fail. The source Delta table holds the new content, but the index doesn't. This is a freshness failure, not a relevance failure, and the fix is in how the index syncs.
AI Search indexes update in one of two ways. With Triggered Sync, "You call the API or Python SDK to trigger an index update". This is the cheapest option, but the index only changes when someone triggers it. With Continuous Sync, "The index is automatically updated with changes from the source Delta table with near real-time latency". This costs more because it provisions a streaming cluster. If a triggered index has fallen behind its source, trigger a sync, or switch to continuous if near-real-time freshness matters. A permission grant won't help, because the content isn't in the index at all.
Trigger an index update through the API or Python SDK, or move to Continuous Sync if the use case needs near-real-time freshness. The content was never synced, so permission grants and reranking can't bring it back.
Checkpoint 5 of 5· Check yourself
Which index configuration keeps an AI Search index current with its source Delta table without anyone calling an update API?
Continuous Sync updates the index automatically from changes in the source Delta table. Triggered Sync only updates when someone calls it.
“Continuous Sync: The index is automatically updated with changes from the source Delta table with near real-time latency.”Source: docs.databricks.com
Sources6
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.A low RetrievalGroundedness score means the retriever is returning the wrong documents.Why is that wrong?
Groundedness judges whether the response is supported by the retrieved context, which is a test of hallucination. If the relevance and sufficiency judges are healthy, the failure is in generation.
Covered in Reading MLflow retrieval judges as failure-mode signals
2.If new documents are missing from results, the retriever must lack permission to see them.Why is that wrong?
If the content is in the source table but missing from a triggered index, the index is stale. It only updates when someone triggers it.
Covered in When the missing answer is recent: the stale-index signal
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Retrieval quality: Are you retrieving the most relevant information for a given retrieval query?”
↩︎ First question: did retrieval fail, or did generation?“Issues here can manifest as hallucinations, inconsistent output, or failure to directly address the user query.”
↩︎ Prediction - 2.https://docs.databricks.com/aws/en/agents/tutorials/ai-cookbook/fundamentals-evaluation-monitoring-ragOfficial docs
“Your production logging must track the inputs, outputs, and intermediate steps such as document retrieval”
↩︎ First question: did retrieval fail, or did generation?“you need to know if the root cause of a low-quality answer is due to the retrieval step or a hallucination.”
↩︎ Key concept“you need to know if the root cause of a low-quality answer is due to the retrieval step or a hallucination.”
↩︎ Checkpoint - 3.
“Is the retrieved context directly relevant to the user's request?”
↩︎ Reading MLflow retrieval judges as failure-mode signals“Does the context provide all necessary information to generate a response that includes the ground truth facts?”
↩︎ Reading MLflow retrieval judges as failure-mode signals“Is the response grounded in the information provided in the context? Is the agent hallucinating?”
↩︎ Exam trap 1“Is the response grounded in the information provided in the context? Is the agent hallucinating?”
↩︎ Checkpoint - 4.
“a table of failed queries (queries where the top-1 result was scored 0 (irrelevant))”
↩︎ Ranking and recall signals in the AI Search evaluation dashboard“Each strategy is also evaluated with and without the reranker.”
↩︎ Ranking and recall signals in the AI Search evaluation dashboard“An LLM judge evaluates every query and retrieved document pair on a 4-point relevance scale.”
↩︎ Checkpoint - 5.
“RAG agents: Missing key context leads to incorrect answers or hallucinations.”
↩︎ Ranking and recall signals in the AI Search evaluation dashboard - 6.
“Continuous Sync: The index is automatically updated with changes from the source Delta table with near real-time latency.”
↩︎ When the missing answer is recent: the stale-index signal“Triggered Sync: You call the API or Python SDK to trigger an index update. This is the most cost-effective option.”
↩︎ Exam trap 2