What you will be able to do
- Name the one question that separates pre-inference retrieval from just-in-time agentic retrieval
- Put the stages of an embedding-based RAG chain in order and explain why the query and the chunks must share one embedding model
- Recognise the use cases where a deterministic, up-front retrieval chain is the right choice, and the ones where it falls short
Key concept
Who decides when retrieval happens — In pre-inference retrieval, the developer's code fetches context on every request before the LLM sees the prompt, so the model never chooses whether to retrieve. In just-in-time agentic retrieval, the LLM itself decides during reasoning whether to call a retrieval tool, which one to call, and with what query.
1.Two moments at which context can arrive
Every retrieval-augmented system has to get supporting data in front of the model, and Databricks describes the basic loop the same way each time: the user's request is used to query a knowledge base, the results are combined with the request into a prompt, and the LLM generates a response. That knowledge base can be 'a vector store, keyword search, or SQL database.' What separates the two strategies in this lesson is not where the data lives. It is *when* the retrieval runs and *who* triggers it.
Pre-inference retrieval puts the retrieval step in code, ahead of the model call. The developer has already decided that every request gets context, which index or source to use, and how many results to fetch. By the time the LLM is invoked, the context is already in the prompt.
Just-in-time agentic retrieval gives the retrieval capability to the model as a tool. The LLM reads the request, decides whether it needs outside data, forms its own query, and can call again if the first result is not good enough.
The plainest form of up-front loading has no retriever at all. Databricks' simplest design pattern is a standalone LLM where you 'customize behavior by providing a system prompt with your custom instructions or embedded data.' The data is pasted in before inference, but nothing is selected per request. Embedding-based retrieval improves on this by choosing the most relevant pieces for each request. The choice is still made by code before the model runs.
Checkpoint 1 of 5· Check yourself
A team pastes the same reference material into the system prompt of a standalone LLM for every request. How does this differ from embedding-based pre-inference retrieval?
A static system prompt supplies the same embedded data up front for every request. Embedding-based retrieval goes further by choosing the most relevant pieces for each request, still in code before the model runs.
“You can customize behavior by providing a system prompt with your custom instructions or embedded data.”Source: docs.databricks.com
It answers from training data plus whatever was hard-coded into the prompt. Databricks notes that this design is 'often disconnected from your real-world business data.' Per-request retrieval, whether pre-inference or agentic, exists to close that gap.
2.How an embedding-based pre-inference chain works
Pre-inference retrieval relies on work done before any user arrives. A data pipeline has to 'Pre-process and index documents, tables, or other data for fast and accurate retrieval.' On Databricks, that index is an AI Search index (formerly Vector Search) built from a source table such as a Delta table. You can set it up to sync automatically when the source table changes. That matters because a pre-inference chain can only return what is already in the index.
At inference time the RAG chain runs a fixed sequence. An optional preprocessing step can rewrite the user's query or pull keywords from it. The retrieval query is then embedded and compared with the stored chunk embeddings. The most similar chunks are returned and placed into a prompt template, and the LLM generates an answer grounded in them. An optional post-processing step can add citations or apply business rules.
Checkpoint 2 of 5· Put it in order
Put the stages of a RAG chain at inference time in order.
- 1.(Optional) Post-process the response, for example to add citations
- 2.(Optional) Preprocess the user query into a retrieval query
- 3.Send the augmented prompt to the LLM for generation
- 4.Embed the retrieval query and return the most similar chunks
- 5.Augment the prompt template with the retrieved context
The RAG chain is a fixed series of steps that runs at inference time. Retrieval and augmentation always happen before the LLM generates the answer, which is what makes this retrieval 'pre-inference'.
“The series, or chain of steps that are invoked at inference time is commonly referred to as the RAG chain.”Source: docs.databricks.com
The retrieval step works only if the query and the chunks are in the same vector space. The same embedding model must be used at both ends. Similarity is then a distance calculation, and the top results go forward. AI Search uses the HNSW algorithm for approximate nearest neighbor search and measures similarity with L2 distance. The developer fixes how many chunks to retrieve and how they are combined with the query. Databricks warns that these choices 'can significantly impact the model's ability to generate quality responses,' and in a pre-inference chain the model has no way to ask for more.
Checkpoint 3 of 5· Check yourself
In a pre-inference RAG chain, how is the user's retrieval query compared against the indexed document chunks?
Semantic comparison works only when the query and the chunks are embedded by the same model. The most similar chunks are then ranked and returned.
“the retrieval query is translated into an embedding using the same embedding model that was used to embed the document chunks during data preparation.”Source: docs.databricks.com
3.When up-front retrieval is the right choice
Databricks lists basic RAG as the standard example of a deterministic chain. It fits 'Well-defined tasks', 'Static pipelines such as basic RAG' and cases with 'No need for on-the-fly decisions.' The reasons are practical. Every request follows the same path, so the system is easy to test and audit. No LLM call is spent deciding whether or how to retrieve, so latency stays low.
| Dimension | Deterministic chain with up-front retrieval |
|---|---|
| When to use | Well-defined tasks with predictable workflows; consistency and auditing are top priorities |
| Latency | Typically lower, because there are fewer LLM calls for orchestration |
| Testing | Highest predictability and auditability; easier to test and validate |
| Weakness | Limited flexibility for diverse or unexpected requests; may need significant refactoring to add capabilities |
The weaknesses come from the same design. The chain retrieves even when the question needs no lookup, and it can't change its query when the first results miss. A request that needs a different source, such as a live order record instead of a policy document, falls outside the predefined path. Supporting it means more code branches, and Databricks notes these chains 'can become complex and difficult to maintain as logic branches grow.'
Checkpoint 4 of 5· Check yourself
A compliance team wants a document Q&A assistant where every answer follows an identical, auditable path and latency stays low. Which retrieval approach fits best?
The requirements are consistency, auditability and low latency. Those are exactly the reasons Databricks gives for choosing a deterministic chain, and basic RAG is its standard example.
“When you want to minimize latency by avoiding multiple LLM calls for orchestration decisions.”Source: docs.databricks.com
Checkpoint 5 of 5· Exam question
A team is building a support assistant over a 5,000-page product manual corpus that is revised only once per quarter. Users ask free-form natural-language questions, and the team wants sub-second answers with minimal per-query compute overhead. Which retrieval strategy best fits this use case?
Correct answer: A — Pre-inference retrieval using an AI Search embedding index built from the Delta table holding the manual content
- A. Correct. The corpus is large, unstructured, and changes infrequently, so pre-computing embeddings once and querying the resulting AI Search index at inference time gives fast semantic matches without repeating expensive indexing work per query.
- B. Incorrect. Issuing a fresh SQL query per request suits structured, frequently changing data, not semantic search over free-form text; it also cannot perform similarity matching the way an embedding index can.
- C. Incorrect. Recomputing embeddings for the entire corpus on every inference call wastes compute and adds latency that a static quarterly-updated index avoids entirely.
- D. Incorrect. Genie spaces are designed for structured tabular question answering via generated SQL, not semantic retrieval over long-form unstructured manual text.
Sources2
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Agentic retrieval is always the better default because it is more flexible than a fixed RAG chain.Why is that wrong?
Databricks advises starting simple. A deterministic chain is the right fit for well-defined tasks where predictability, auditability and low latency matter. Add agentic behavior only when the use case truly needs it.
Covered in When up-front retrieval is the right choice
2.The query can be embedded with any model at retrieval time, as long as the chunks were embedded once.Why is that wrong?
Semantic similarity is meaningful only if the query is embedded with the same model that embedded the document chunks during data preparation.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“a vector store, keyword search, or SQL database”
↩︎ Two moments at which context can arrive“Pre-process and index documents, tables, or other data for fast and accurate retrieval.”
↩︎ How an embedding-based pre-inference chain works - 2.
“You can customize behavior by providing a system prompt with your custom instructions or embedded data.”
↩︎ Two moments at which context can arrive“This design is good for simple or generic queries but is often disconnected from your real-world business data.”
↩︎ Two moments at which context can arrive“Static pipelines such as basic RAG No need for on-the-fly decisions”
↩︎ When up-front retrieval is the right choice“Can become complex and difficult to maintain as logic branches grow.”
↩︎ When up-front retrieval is the right choice“The LLM does not make decisions about which tools to call or in what order.”
↩︎ Key concept“Introduce more complex agentic behaviors when you truly need them for better flexibility or model-driven decisions.”
↩︎ Exam trap 1“The system follows a predefined workflow or "chain" for all requests, making it highly predictable.”
↩︎ Prediction“When you want to minimize latency by avoiding multiple LLM calls for orchestration decisions.”
↩︎ Checkpoint - 3.
“You can structure the index to automatically sync when the underlying source table is updated.”
↩︎ How an embedding-based pre-inference chain works - 4.https://docs.databricks.com/aws/en/agents/tutorials/ai-cookbook/fundamentals-inference-chain-ragOfficial docs
“determining how many chunks to retrieve in step 2 and how to combine them with the user's query in step 3 can significantly impact”
↩︎ How an embedding-based pre-inference chain works“the retrieval query is translated into an embedding using the same embedding model that was used to embed the document chunks during data preparation.”
↩︎ Exam trap 2“The series, or chain of steps that are invoked at inference time is commonly referred to as the RAG chain.”
↩︎ Checkpoint“the retrieval query is translated into an embedding using the same embedding model that was used to embed the document chunks during data preparation.”
↩︎ Checkpoint