Diagnosing memory retrieval failures with precision, recall and traces

1.Measure retrieval separately from the answer

Over-retrieval and under-retrieval both live in the retrieval step, but they show up in the answer only indirectly. An agent might answer correctly today while dragging ten irrelevant memories into every turn. That works until one of them misleads it. Databricks' guidance is that it is very important to collect both response and retrieval metrics, because only measuring both lets you tell the components apart.

There is a precondition: you need a reproducible evaluation system before you tune anything. A golden dataset, a synthetic evaluation set or ground-truth-free judges all qualify. Changing your memory search settings without measuring is guesswork, and it is easy to trade one failure for the other without noticing.

Sources12

2.Precision catches pollution, recall catches gaps

The two failure modes map onto the two classic retrieval metrics:

- Precision asks what percentage of the retrieved items are relevant to the request. Low precision means irrelevant material is getting in. That is over-retrieval. - Recall asks what percentage of the known-relevant items were retrieved. Low recall means needed material is missing. That is under-retrieval.

The cookbook example: two of three retrieved results were relevant, so precision was 0.66 (2/3). Those results covered two of four relevant documents, so recall was 0.5 (2/4). One retrieval can be mediocre on both.

There is a practical difference between the two metrics. Computing precision does not require knowing all the relevant items, so an LLM judge can score it on live traffic. Computing recall requires ground truth that lists every relevant item. In MLflow 3 the built-in judges follow the same split. RetrievalRelevance checks whether retrieved context is relevant to the request and needs no ground truth. RetrievalSufficiency checks whether the context provides all the information needed and does need expectations.

Sources13

3.Use traces to see what memory actually returned

Metrics tell you that a failure exists. Traces show you where it happened. An MLflow trace records one complete execution of the agent, including every intermediate step (LLM calls, retriever lookups, tool invocations), along with latency and errors. Each of those steps is a span with its inputs and outputs. For a memory tool, the span shows the query the agent sent and the entries that came back.

That is how you tell the failures apart in practice. A span that returned many entries unrelated to the request points to over-retrieval. A span that returned nothing, or returned entries that miss a fact you know is stored, points to under-retrieval. The search tool below returns the literal text No memory found. when nothing matches, and that is easy to spot in a span's outputs:

@function_tool
    def search_memory(query: str) -> str:
        """Search long-term memory for relevant facts about the user."""
        results = memory_store.search(actor_id=actor_id, query=query, limit=10)
        return "\n\n".join(f"{r.memory.path}: {r.memory.content}" for r in results) or "No memory found."

Missing history often appears only across turns. A fact from turn two that the agent forgets by turn eight is invisible when you score turns one at a time. Tagging traces with mlflow.trace.session groups the turns of one conversation, so you can evaluate the conversation end-to-end.

Sources4

4.Match the fix to the diagnosis

Once you know which failure you have, the levers follow from it. In managed memory, the first lever is the limit on each search. Lowering it trims pollution, and raising it closes gaps. Re-measure precision and recall after each change. The other levers are writing better description text on entries so BM25 search finds them, and scoping each search to the correct actor_id.

If your agent also retrieves from AI Search indexes, the retrieval-quality guide offers techniques that improve one side without simply sacrificing the other:

- Metadata filtering dramatically improves precision by eliminating irrelevant results, and it also improves recall within the filtered subset. - Reranking boosts precision by achieving high recall with fewer candidates. You retrieve broadly, then keep only the best results. - Hybrid search improves recall by catching both semantic and keyword matches.

Which side to favour depends on the use case. When missing key context leads to wrong answers, weight recall. When the user needs the exact item at the top, weight precision.

results = memory_store.search(actor_id="user-123", query="communication preferences",  ? =10)

Sources2

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.If the agent's final answers are correct, its memory retrieval does not need separate evaluation.Why is that wrong?

   Good responses can rest on faulty retrieval, so response metrics alone hide over- and under-retrieval. Collect retrieval metrics too.

   Covered in Measure retrieval separately from the answer

2. 2.You need a fully labelled ground-truth set before you can detect any retrieval problem.Why is that wrong?

   Precision, the over-retrieval signal, can be judged without knowing all relevant items. Only recall-style metrics need complete ground truth.

   Covered in Precision catches pollution, recall catches gaps