Retrieval vs Memory: Fixing Missing, Wrong or Forgotten Knowledge

1.The agent never had the knowledge: add retrieval

The simplest design, an LLM with a prompt, answers from what it learned in training. That works for generic questions, but such a model is often disconnected from your real business data. You can embed some data in the system prompt. When the knowledge is proprietary, changes often or is domain-specific, though, the more direct fix is retrieval-augmented generation. RAG also lets the model cite its sources, and its retrieval step can enforce access control based on the user's credentials.

So when an agent states things that were never true for your business, or were true once but have since changed, it is usually missing knowledge, not reasoning badly. A deterministic RAG chain runs the same three steps for every request, in a fixed order the developer defines. The model doesn't decide the order, which makes the chain predictable and easy to audit.

Sources12

2.The agent retrieved, but retrieved the wrong thing

Adding retrieval brings a new failure. When the harness pulls the wrong material from search or memory, the model can still answer confidently. Databricks splits RAG problems into retrieval quality (did the right information come back?) and generation quality (did the model use it well?). These overlap: generation can't recover from context that is missing key information or full of superfluous material.

The fix depends on the pattern of wrong results. The examples in the table come from the Databricks RAG quality guide.

Some of these fixes need changes in more than one place. Retrieval quality depends on both the offline data pipeline (parsing, chunking, metadata, embedding model) and the online chain (query transformation, number of chunks, re-ranking). Filter extraction is a clear example: it only works if the pipeline has extracted the metadata fields and the retriever accepts and applies the filters.

The same reasoning applies to structured data. If a Genie Agent picks the wrong table because tables overlap, remove the unnecessary ones. If it filters on "California" when the column stores "CA", turn on example values and value dictionaries so it can see the real values.

Sources3456

3.The agent forgot: sessions or memory?

Forgetting comes in two kinds, and Databricks has a separate managed store for each, both backed by Lakebase. A managed agent session holds the state of the interaction in progress, usually the ordered transcript of messages, tool calls and results. The agent reads it at the start of each turn and appends to it as the interaction continues. This lets an interaction survive a restart and rebuild its full context on a follow-up. Managed agent memory holds durable facts, preferences and decisions about a subject, which the agent recalls in later, unrelated conversations.

The two stores connect: as a session builds up, the agent or application extracts the durable facts and writes them to memory. The stores are independent, so clearing a session doesn't remove anything from memory. To let an agent decide when to save and recall, wrap the store's operations as tools and explain in the system prompt when to use them.

def make_memory_tools(memory_store, actor_id: str):
    @function_tool
    def search_memory(query: str) -> str:
        """Search long-term memory for relevant facts about the user."""
        results = memory_store.search(actor_id=actor_id, query=query, limit=10)
        return "\n\n".join(f"{r.memory.path}: {r.memory.content}" for r in results) or "No memory found."

    @function_tool
    def save_memory(path: str, content: str, description: str = "") -> str:
        """Save a durable, long-term memory about the user."""
        memory_store. ? (actor_id=actor_id, path=path, content=content, description=description)

Sources78

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Retrieval problems belong to the data pipeline, so re-chunking or re-embedding is always the fix.Why is that wrong?

   Retrieval quality also depends on the RAG chain: query rewriting, how many chunks are retrieved, and re-ranking. Some fixes, such as filter extraction, need changes to both.

   Covered in The agent retrieved, but retrieved the wrong thing

2. 2.Long-term memory is the right place for an agent's in-conversation history, and deleting a session also wipes what was learned.Why is that wrong?

   Short-term history within an interaction belongs in managed agent sessions. The two stores are independent, so deleting a session doesn't delete memory.

   Covered in The agent forgot: sessions or memory?

3. 3.When memory is exposed as a tool, the model can pass in whichever actor_id fits the request.Why is that wrong?

   The application sets actor_id from the verified end-user identity. The model must never choose whose memory it reads or writes.

   Covered in The agent forgot: sessions or memory?

Practise it for real

Save a user preference to a managed memory store and recall it with a natural-language search, as an agent would at the start of a new conversation.

1. 1.Make sure you have a memory store named support-agent-memory (creating a store provisions its Lakebase storage automatically), and set DATABRICKS_HOST and DATABRICKS_TOKEN.

   Why: Both calls below address the store by its display name.

   You should see: A workspace-scoped store you can write entries to.

2. 2.POST to /api/2.0/agents/memory-stores/support-agent-memory/entries with actor_id "user-123", path "/preferences/communication.md", content "Prefers email over phone." and description "Communication preferences".

   Why: actor_id says whose memory it is, path organises it, and description improves retrieval.

   You should see: A new entry, uniquely identified by actor_id, session_id and path.

3. 3.POST to /api/2.0/agents/memory-stores/support-agent-memory/entries:search with actor_id "user-123" and query "communication preferences".

   Why: This is the recall step that fixes 'the agent forgot between conversations'.

   You should see: The email-preference entry, returned among the most relevant entries for that actor.