What you will be able to do
- Choose between full-corpus context, agentic search with tools and a pre-built RAG pipeline from corpus size, query pattern, freshness and latency and cost limits
- Explain why hybrid embeddings plus BM25 retrieval handles both paraphrased questions and exact identifiers
- Specify a contextual-retrieval configuration, including how prompt caching keeps its one-time indexing cost down
- Pick reranking, fusion weights and chunk count to fit a given latency and per-query cost budget
Key concept
Fit-first retrieval selection — Before you build an index, ask whether the whole corpus fits comfortably in the context window. If it does, load it in a cached prompt. Choose indexed retrieval or agentic search only when the corpus is too big or changes too fast for that, and then let the query pattern and the latency budget decide between them.
1.Three ways to get knowledge in front of Claude
Every retrieval design answers one question: which tokens about your business reach the model on each request. There are three common answers. Full-corpus context puts the entire knowledge base in the prompt. A pre-built RAG pipeline chunks and indexes the corpus ahead of time, then adds the best-matching chunks to each prompt. Agentic search gives Claude tools such as grep, file reads or a custom search function, and lets it look things up while it works. Each one moves cost and latency to a different point in the request.
Whichever you choose, the retrieved material has to fit in the same budget. Everything in a request counts toward the context window. That includes the system prompt, the tool definitions, and every tool result an agent collects along the way. Current models such as Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5 have a 1M-token window, and other models, such as the deprecated Claude Sonnet 4.5, have 200k. The window is a ceiling, not a target, because recall weakens as the context fills. So the selection depends on four properties of the workload: how large the corpus is, how often it changes, what the questions look like, and how quickly and cheaply each answer has to come back.
| Architecture | Work done ahead of time | Work done per query | Wins when |
|---|---|---|---|
| Full-corpus context | None beyond caching the prompt | Reading the cached corpus | The corpus fits comfortably in the window and questions need several sections at once |
| Pre-built RAG (embeddings + BM25) | Chunking, embedding, building the BM25 index | Index lookup, rank fusion, optional reranking | The corpus is far too large for one prompt and answers must return quickly |
| Agentic search with tools | Little or none, no index to keep fresh | Several tool calls and model turns | Content changes constantly and latency is not tight |
Checkpoint 1 of 9· Match them up
Match each workload to the retrieval architecture that fits it best.
Tap a term, then the definition that fits it.
Full context wins when the corpus fits comfortably and answers need several sections. A pre-built index does its work ahead of time, so it scales to huge corpora and still answers quickly. Agentic search keeps no index that could go stale, but each answer needs several tool calls and model turns, so it fits fast-changing content when latency is not tight.
“primitives like glob and grep allow it to navigate its environment and retrieve files just-in-time, effectively bypassing the issues of stale indexing”Source: www.anthropic.com
2.When the whole corpus fits: load it and cache it
Check whether the corpus fits before you design an index. If the knowledge base fits in the prompt with room to spare, you can skip RAG and include the whole thing. Anthropic first gave this advice when windows were 200,000 tokens, roughly 500 pages. A 1M-token window raises that ceiling, but the reasoning stays the same. With nothing to chunk, no chunk can fail to be retrieved. That matters for questions that need several sections at once, which a top-k chunk lookup can only half answer.
Prompt caching is what makes this affordable at volume. You write the corpus to the cache once, paying a 25% premium over base input for the 5-minute TTL. Every later request reads it at about 10% of the base input price. Anthropic reported that caching cut latency by more than 2x and cost by up to 90%. When the manuals change each quarter, you refresh the cached prefix, and there is no embedding job to rerun.
Two limits stop you from stretching this too far. The first is context rot: as token count grows, accuracy and recall degrade, so a corpus that only just fits makes a weaker prompt than one that fits easily. The second is price on some models. On Claude Haiku 5.5, prompts over 100,000 tokens cost more, so a large corpus there costs more on every request. A corpus of tens of millions of tokens is out of reach entirely. Rebuilding a cached prefix every week does not get around the size of the window.
Checkpoint 2 of 9· Check yourself
An architect plans to load a 900,000-token knowledge base into a cached prompt on a model with a 1M-token window, because "it fits." What is the main concern with this plan?
The window is a ceiling, not a target. Context rot means accuracy and recall degrade as the token count grows, so full-corpus context is the right choice only when the corpus fits comfortably. The cache-write premium is paid once per write, and later reads cost about 10% of the base input price.
“As token count grows, accuracy and recall degrade, a phenomenon known as context rot.”Source: platform.claude.com
Checkpoint 3 of 9· Exam question
You are the architect for a product-support assistant built on the Claude Messages API. The knowledge base is 120 pages of product manuals, roughly 150,000 tokens, revised about once a quarter. Thousands of questions arrive daily, many of them need facts from several sections at once, and the model's context window comfortably exceeds the corpus. Which retrieval architecture is the most defensible starting point?
Correct answer: B — Load the full manual text into a cached prompt prefix and answer each question from it, refreshing it after each revision
- A. Embedding chunks and retrieving only the closest matches adds an index to build and keep current, and it can drop facts that sit in different sections, which these questions often need. Because the whole 150,000-token corpus fits in the context window, that retrieval layer adds risk without solving a size problem.
- B. Correct. The manuals fit comfortably in the context window, so placing them whole in a prompt prefix with a caching breakpoint gives the model every section at once. That suits multi-section questions and avoids retrieval misses and index upkeep. Cached reads keep cost and latency low across thousands of daily requests, and the prefix only needs rewriting about once a quarter.
- C. An agent with grep and file-read tools needs several tool round trips per question, which adds latency and token cost at thousands of requests a day. Keyword search can also miss related passages in other sections, and it is unnecessary when the corpus already fits in one prompt.
- D. Splitting a corpus that fits in one context across subagents multiplies token usage on every question. Each subagent sees only its own section, so facts that span sections can be lost when partial findings are merged. Nothing about a 150,000-token corpus requires that sharding.
3.Agentic search: retrieval at runtime with tools
Agentic search replaces the index with tools and judgment. The agent keeps lightweight references, such as file paths, stored queries and links, and loads data into context at runtime only when it needs it. Claude Code works this way. Its CLAUDE.md files go into context up front, and glob and grep fetch everything else just in time. Anthropic says this approach avoids the problems of stale indexing and complex syntax trees. In a codebase or folder that changes all the time, avoiding a stale index is a real advantage.
You pay for that flexibility in time and in tokens. Exploring at runtime is slower than retrieving pre-computed data, because each search is another model turn and another tool call. An agent without good tools and guidance can also waste context on dead ends. Every file it opens counts against the same window. That is why an agent doing grep and file reads over a 40-million-token contract archive is a poor fit when answers must return in a few seconds. It works best where the content changes faster than an index could be rebuilt and a slower answer is acceptable.
You do not have to pick one or the other. Some agents load the most important data up front for speed and explore the rest as needed. Anthropic suggests this hybrid may suit areas where content changes less often, such as legal or finance work. When your own search tool supplies the results, you can return them as search_result content blocks. Claude then cites them with the source and title you gave, which suits answers that must point back to a specific document.
Checkpoint 4 of 9· Check yourself
Your team's assistant answers questions about a monorepo that receives dozens of commits every hour, and users accept answers that take a minute. A colleague proposes a vector index rebuilt every night. What is the strongest argument for agentic search with grep and file reads instead?
Agentic search loads data just in time, so it avoids the stale-index problem when content changes faster than an index can be rebuilt. It is slower than retrieving pre-computed data, tool results count against the context window, and without good tools and guidance an agent wastes context on dead ends.
“primitives like glob and grep allow it to navigate its environment and retrieve files just-in-time, effectively bypassing the issues of stale indexing”Source: www.anthropic.com
4.Indexed RAG: embeddings and BM25 together
For a corpus too large for any prompt, a pre-built index lets you scale to enormous knowledge bases at a predictable per-query cost. The standard pipeline splits documents into chunks of at most a few hundred tokens. It embeds each chunk and stores the vectors so they can be searched by semantic similarity. Anthropic does not make its own embedding model. Voyage AI is one provider, and the factors to weigh when choosing one include lookup latency and whether you can customize the model for your domain.
Embeddings alone have a blind spot: exact strings. Suppose someone searches for "Error code TS-999". An embedding model may return passages about error codes in general and miss the one chunk that contains TS-999. BM25 is a lexical ranking function built on TF-IDF. It searches for that exact string. Fine-tuning the embedding model does not remove the blind spot, because the model still ranks by meaning rather than by matching the literal string. When users mix paraphrased questions ("who pays if delivery is late") with clause numbers or error codes, you need both kinds of signal.
A hybrid pipeline runs both searches and then merges the results:
- Chunk the corpus and create both TF-IDF encodings and semantic embeddings for each chunk.
- Take the top chunks from BM25 by exact match, and the top chunks from embeddings by semantic similarity.
- Combine and deduplicate the two lists with rank fusion.
- Add the top-K chunks to the prompt.
Each query is an index lookup plus a single model call, so answers come back in seconds even over tens of millions of tokens. A weekly re-ingest means re-indexing the changed chunks, not rebuilding a 40M-token prompt that would not fit anyway.
Checkpoint 5 of 9· Put it in order
A support corpus gets both paraphrased questions and exact error-code lookups such as "TS-999". Put the steps of the hybrid RAG pipeline that handles both in order.
- 1.Add the top-K fused chunks to the prompt
- 2.At query time, take the top chunks from BM25 by exact match and from embeddings by semantic similarity
- 3.Create TF-IDF encodings and semantic embeddings for each chunk
- 4.Combine and deduplicate the two result lists with rank fusion
- 5.Split the corpus into chunks of at most a few hundred tokens
Both indexes are built from the same chunks before any query arrives. At query time BM25 and the embeddings each return their top matches, and rank fusion merges them into one deduplicated list. Only after that are the top-K chunks added to the prompt, so the exact-match hits for identifiers are not lost.
“Combine and deduplicate results from (3) and (4) using rank fusion techniques;”Source: www.anthropic.com
Checkpoint 6 of 9· Exam question
A legal-tech company must let Claude answer questions over 40 million tokens of contract and policy documents. Questions mix paraphrased concepts ("who pays if delivery is late") with exact identifiers such as clause numbers and error codes. The corpus is re-ingested weekly, and answers must return in a few seconds. Which design best fits these constraints?
Correct answer: C — Index chunks with embeddings and BM25 keyword scoring, merge the two ranked lists, and send the top chunks to Claude
- A. Prompt caching cuts the cost and latency of a reused prefix, but the prefix must still fit in the context window. A 40-million-token corpus is far beyond any context window, so the documents cannot be loaded this way.
- B. Grep matches literal strings, so it finds clause numbers and error codes but misses paraphrased questions such as who pays for late delivery. Searching and opening files one call at a time over 40 million tokens also cannot be relied on to finish within a few seconds.
- C. Correct. Embedding similarity covers paraphrased concepts, and BM25 matches exact clause numbers and error codes. Merging the two ranked lists serves both kinds of question. A prebuilt index can be rebuilt after each weekly ingest and queried in well under a few seconds.
- D. Vector similarity handles paraphrase, but dense embeddings, fine-tuned or not, match exact tokens such as clause numbers and error codes unreliably. Without a keyword component, the identifier-style questions in the mix would often retrieve the wrong chunks.
5.Contextual retrieval: giving each chunk its context back
Chunking removes the context a passage needs. Consider a chunk from an SEC filing that reads "The company's revenue grew by 3% over the previous quarter." It does not say which company or which quarter. A question about ACME Corp in Q2 2023 may therefore never retrieve it. Contextual retrieval fixes this by asking Claude to write a short, chunk-specific explanation and prepending it to the chunk before it is embedded and before the BM25 index is built. These are two techniques: Contextual Embeddings and Contextual BM25.
<document>
{{WHOLE_DOCUMENT}}
</document>
Here is the chunk we want to situate within the whole document
<chunk>
{{CHUNK_CONTENT}}
</chunk>
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else.The added context is usually 50 to 100 tokens, and it has to be specific to the chunk. Adding a generic document summary to every chunk was tested and gave very limited gains. Copying one chunk's description onto its neighbours would also give them context that does not describe them. When done properly, contextual embeddings cut the top-20 retrieval failure rate by 35%. Contextual embeddings combined with contextual BM25 cut it by 49%, from 5.7% to 2.9%.
Use prompt caching. Put a cache breakpoint after the document, write it to the cache on the first chunk's call, and the calls for its remaining chunks read it at the cache-read price instead of paying full input price each time. Using 8k-token documents and 800-token chunks, Anthropic estimated the one-time cost at $1.02 per million document tokens. An 8k document is well above the minimum cacheable length, which is 512 tokens on current models such as Claude Opus 5.5 and Haiku 5.5. Cutting the document down to its first page would save tokens but lose the context the method needs. Sending chunks without their document leaves nothing to situate them against.
In a Lambda function that adds context to each document. You select it as a custom chunking option when you configure the Knowledge Base.
Checkpoint 7 of 9· Fill the gap
In the contextualizer prompt, which placeholder goes in the first tag so that Claude can write context specific to the chunk?
<document>
{{ ? }}
</document>
Here is the chunk we want to situate within the whole document
<chunk>
{{CHUNK_CONTENT}}
</chunk>
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else.Claude needs the whole document to explain where this one chunk fits, for example which company and which quarter it covers. A generic document summary was tested and gave very limited gains. A neighbouring chunk's context does not describe this chunk, and a first page leaves out the context the method depends on. Prompt caching is what keeps sending the whole document affordable.
Source: www.anthropic.comCheckpoint 8 of 9· Exam question
You are planning contextual retrieval for a 10-million-token corpus. Generating the situating context means sending Claude each source document together with each of its chunks, and the same document is sent once per chunk. Which technique keeps this one-time indexing cost down?
Correct answer: B — Mark the source document with a prompt-caching breakpoint so later chunk calls for that document read it from cache
- A. Copying one description onto neighbouring chunks gives them context about the wrong passage, which undermines the purpose of situating each chunk within its document.
- B. Correct. The full document is the repeated, unchanging prefix across all its chunk calls, so caching it lets each later call read it at the cached-read rate, which Anthropic cites as the way to make contextualisation affordable.
- C. A first-page excerpt removes the surrounding material the model needs to describe where the chunk sits, so the generated context becomes generic or wrong for later sections.
- D. Without the document the model has nothing to situate the chunk against, so the output is no better than the chunk alone, and dropping below the cache minimum disables caching rather than saving money.
6.Reranking, fusion weights and chunk count within a latency budget
Once the index exists, a few settings decide how much quality you get for the latency and cost you can spend. The numbers below come from a published Anthropic cookbook that builds a contextual-retrieval pipeline and tests it on a dataset of 9 codebases with 248 queries. Quality is reported as Pass@k: the share of queries where the correct chunk appears among the top k retrieved, so Pass@10 of 95% means the right chunk was in the top 10 for 95% of queries. The first setting is fusion weighting. The pipeline pulls the top 150 candidates from semantic search and from BM25 and merges the two ranked lists with weighted Reciprocal Rank Fusion. Fusion works from rank positions rather than raw scores, and each list's weight sets how much that list counts toward the merged ranking. The pipeline starts at 80% semantic and 20% BM25. Those weights are a starting point that balances semantic understanding against keyword precision. A corpus full of part numbers may need more weight on BM25, so that exact matches rise in the merged list. BM25 matches against the added context as well as the original chunk text.
Reranking adds a second stage. You over-retrieve about 10x the number of chunks you need, for example 100 chunks when you need 10. A reranking model scores each candidate and keeps the top-k. Voyage AI and Cohere both offer rerankers. On the cookbook's codebase set this raised Pass@10 from 92.34% to about 95%, and Anthropic's blog reports a 67% cut in failed retrievals when reranking is added to contextual retrieval. Reranking is not free. It adds roughly 100 to 200 ms per query, depending on candidate count, and about $0.002 per query in API cost.
The last setting is how many chunks to send. More chunks raise the chance that the right one is included, but extra material can distract the model. In Anthropic's tests, 20 chunks performed better than 5 or 10. Questions that need facts from more than one chunk depend on that headroom.
| Requirement | Configuration | Pass@10 | Ongoing cost |
|---|---|---|---|
| High-volume, cost-sensitive | Contextual embeddings alone | 92.34% | None per query; one-time ingestion (~$3 for this dataset with prompt caching) |
| Balanced production system | Contextual embeddings + hybrid search (BM25) | 93.21% | Elasticsearch infrastructure and maintenance, no per-query API cost |
| Maximum accuracy, latency-tolerant | Contextual embeddings + reranking | 95.26% | 100-200ms and ~$0.002 per query |
Pick the row from whichever constraint is tightest. Contextual embeddings gave the biggest single gain, 5 to 7 points, and they cost nothing per query. Hybrid search and reranking add the last 2 to 3 points of precision, but only if you can afford the extra infrastructure, latency or per-query cost. Hybrid search is still the safeguard whenever exact identifiers appear in queries, because it catches the matches embeddings miss.
Checkpoint 9 of 9· Match them up
Each team below already has a contextual-retrieval index. Pair each team's tightest constraint with the configuration that fits it.
Tap a term, then the definition that fits it.
Choose the configuration from the tightest constraint. Contextual embeddings cost nothing per query. Hybrid search adds infrastructure but no per-query API cost. Reranking gives the best accuracy but adds 100–200 ms and about $0.002 per query. The 80/20 fusion weights are only a starting point, so a corpus full of exact identifiers needs more weight on BM25.
“Choose your approach based on your requirements: High-volume, cost-sensitive: Contextual embeddings alone (92% Pass@10, no per-query costs) Maximum accuracy, latency-tolerant: Full reranking pipeline”Source: platform.claude.com
Sources
(8)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“every message in messages (including tool results, images, and documents), and your tool definitions”
↩︎ Three ways to get knowledge in front of Claude“Other Claude models, including Claude Sonnet 4.5 (deprecated), have a 200k-token context window.”
↩︎ Three ways to get knowledge in front of Claude“long-context requests are billed at standard pricing, except on Claude Haiku 5.5, where prompts over 100,000 tokens cost more.”
↩︎ When the whole corpus fits: load it and cache it“As token count grows, accuracy and recall degrade, a phenomenon known as context rot.”
↩︎ Exam trap 1“As token count grows, accuracy and recall degrade, a phenomenon known as context rot.”
↩︎ Checkpoint - 2.
“runtime exploration is slower than retrieving pre-computed data.”
↩︎ Three ways to get knowledge in front of Claude“effectively bypassing the issues of stale indexing and complex syntax trees.”
↩︎ Agentic search: retrieval at runtime with tools“Without proper guidance, an agent can waste context by misusing tools, chasing dead-ends, or failing to identify key information.”
↩︎ Agentic search: retrieval at runtime with tools“primitives like glob and grep allow it to navigate its environment and retrieve files just-in-time, effectively bypassing the issues of stale indexing”
↩︎ Checkpoint“primitives like glob and grep allow it to navigate its environment and retrieve files just-in-time, effectively bypassing the issues of stale indexing”
↩︎ Checkpoint - 3.https://www.anthropic.com/engineering/contextual-retrievalSecondary source
“If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base”
↩︎ When the whole corpus fits: load it and cache it“reducing latency by > 2x and costs by up to 90%”
↩︎ When the whole corpus fits: load it and cache it“Combine and deduplicate results from (3) and (4) using rank fusion techniques;”
↩︎ Indexed RAG: embeddings and BM25 together“Contextual Retrieval solves this problem by prepending chunk-specific explanatory context to each chunk before embedding”
↩︎ Contextual retrieval: giving each chunk its context back“You simply load the document into the cache once and then reference the previously cached content.”
↩︎ Contextual retrieval: giving each chunk its context back“the one-time cost to generate contextualized chunks is $1.02 per million document tokens.”
↩︎ Contextual retrieval: giving each chunk its context back“We tried delivering 5, 10, and 20 chunks, and found using 20 to be the most performant of these options”
↩︎ Reranking, fusion weights and chunk count within a latency budget“Sometimes the simplest solution is the best.”
↩︎ Key concept“BM25 looks for this specific text string to identify the relevant documentation.”
↩︎ Exam trap 2“adding generic document summaries to chunks (we experimented and saw very limited gains)”
↩︎ Exam trap 3 - 4.
“Cache writes: When new content is written to the cache (25 percent more than base input tokens for 5-minute TTL)”
↩︎ When the whole corpus fits: load it and cache it“Cache reads: When cached content is used (10 percent of base input token price, with per-model exceptions)”
↩︎ Contextual retrieval: giving each chunk its context back“512 tokens for Claude Fable 5.2”
↩︎ Contextual retrieval: giving each chunk its context back“Cache reads: When cached content is used (10 percent of base input token price, with per-model exceptions)”
↩︎ Prediction - 5.https://platform.claude.com/cookbook/capabilities-retrieval-augmented-generation-guideSecondary source
“Some of our questions require synthesis between more than one chunk in order to be answered correctly”
↩︎ When the whole corpus fits: load it and cache it - 6.
“From tool calls: Your custom tools return search results, which lets you build dynamic RAG applications.”
↩︎ Agentic search: retrieval at runtime with tools - 7.
“Anthropic does not offer its own embedding model.”
↩︎ Indexed RAG: embeddings and BM25 together“Inference performance: embedding lookup speed and end-to-end latency.”
↩︎ Indexed RAG: embeddings and BM25 together“Voyage AI makes embedding models and rerankers.”
↩︎ Reranking, fusion weights and chunk count within a latency budget - 8.
“Default: 80% weight to semantic search, 20% to BM25”
↩︎ Reranking, fusion weights and chunk count within a latency budget“Over-retrieve: Get 10x more results than needed (e.g., retrieve 100 chunks when we need 10)”
↩︎ Reranking, fusion weights and chunk count within a latency budget“High-volume, cost-sensitive: Contextual embeddings alone (92% Pass@10, no per-query costs)”
↩︎ Reranking, fusion weights and chunk count within a latency budget“Pass@k checks whether or not the 'golden document' was present in the first k documents retrieved for each query.”
↩︎ Reranking, fusion weights and chunk count within a latency budget“We use a pre-chunked dataset of 9 codebases”
↩︎ Reranking, fusion weights and chunk count within a latency budget“reranking adds ~100-200ms per query depending on candidate set size.”
↩︎ Exam trap 4“Choose your approach based on your requirements: High-volume, cost-sensitive: Contextual embeddings alone (92% Pass@10, no per-query costs) Maximum accuracy, latency-tolerant: Full reranking pipeline”
↩︎ Checkpoint