What you will be able to do
- Decide when a knowledge base should go straight into the prompt and when it needs a retrieval pipeline
- Pick semantic, lexical or hybrid retrieval based on whether queries carry exact identifiers or are phrased conceptually
- Tune a hybrid pipeline's fusion weights and embedding model to the data you are indexing
Key concept
Retrieval matched to data shape — No single retrieval method is best for every corpus or query. You pick and weight full-context prompting, semantic embeddings and exact-match lexical search based on how big the corpus is, what kind of content it holds, and how users phrase their queries.
1.First question: does the corpus fit in the prompt?
Before you choose between retrieval methods, decide whether you need retrieval at all. Anthropic's Contextual Retrieval post opens with the plainest option: if the knowledge base is smaller than 200,000 tokens (about 500 pages), put all of it in the prompt. Prompt caching makes this practical, because a large static prefix sent on every call can be cached between API calls. The post says caching cuts latency by more than 2x and cost by up to 90%.
If you take this route, how you lay out the prompt matters. Anthropic's prompting guidance for inputs of 20k+ tokens is to put the long material first and the question last. It also suggests wrapping each document in its own tags with source metadata.
Retrieval becomes necessary once the corpus grows past what fits in the context window. The standard RAG preprocessing then has three steps. Split the corpus into chunks, usually no more than a few hundred tokens each. Turn each chunk into a vector embedding. Store the vectors in a database that supports semantic-similarity search. At query time, the most similar chunks are added to the prompt. The rest of this page is about choosing what kind of search runs over that index.
Checkpoint 1 of 5· Check yourself
At roughly what size does Anthropic say a knowledge base outgrows the 'just put it in the prompt' approach?
The Contextual Retrieval post sets the threshold at 200,000 tokens, about 500 pages. Below that, the whole corpus can go in the prompt.
“smaller than 200,000 tokens (about 500 pages of material)”Source: www.anthropic.com
2.Semantic embeddings versus exact-match BM25
Embeddings represent meaning. A query and a document land close together in vector space when they are about the same thing, even if they share no words. In the embeddings guide's example, the query "When is Apple's conference call scheduled?" retrieves a document announcing the fourth-quarter results call by nearest-neighbour search. That is the right tool when users describe what they want in their own words.
Embeddings have a known blind spot, though: exact strings. The Contextual Retrieval post's example is a user searching a technical support database for "Error code TS-999". An embedding model may return content about error codes in general and miss the one document that contains TS-999. BM25 is a ranking function built on TF-IDF. It matches words and phrases lexically, so it finds that exact string. It also accounts for document length and saturates term frequency, so common words don't dominate the ranking.
| Query pattern | Example | Stronger signal |
|---|---|---|
| Conceptual, paraphrased question | When is Apple's conference call scheduled? | Semantic embeddings |
| Unique identifier or technical term | Error code TS-999 | BM25 lexical matching |
| Mixed traffic of both kinds | Support or code search over a large corpus | Hybrid: embeddings + BM25 fused |
Checkpoint 2 of 5· Check yourself
Users of an internal support tool mostly paste exact error codes such as 'TS-999'. A pure embedding search keeps returning generic error-handling pages. What is the best fix?
BM25 matches the literal text string. Embeddings capture meaning and can miss exact identifiers.
“BM25 looks for this specific text string to identify the relevant documentation.”Source: www.anthropic.com
If you build the semantic leg with Voyage AI, the provider Anthropic's embeddings guide uses, there is one detail that matters for retrieval. Queries and documents are embedded asymmetrically. Pass input_type="document" when you index chunks and input_type="query" when you embed the user's question. The guide says to set it on every retrieval task, because each value prepends a different internal prompt and gives better vectors for retrieval.
Checkpoint 3 of 5· Fill the gap
This line from the embeddings guide embeds a user's question for retrieval. Which value completes it?
query_embd = vo.embed([query], model="voyage-4", input_type=" ? ").embeddings[0]The question is embedded with input_type="query" and the indexed corpus with input_type="document". The guide says not to omit the parameter or set it to None.
Source: platform.claude.com3.Hybrid search and tuning to the data shape
Most real query traffic mixes both patterns, so the usual production answer is to run both searches and merge their results. The Contextual Retrieval post describes the pipeline step by step. Chunk the corpus. Build TF-IDF encodings and semantic embeddings for each chunk. Get top candidates from BM25 and from embeddings. Merge and deduplicate the two lists with rank fusion. Put the top-K chunks in the prompt.
Checkpoint 4 of 5· Put it in order
Put the steps of an embeddings + BM25 hybrid RAG pipeline in order.
- 1.Create TF-IDF encodings and semantic embeddings for each chunk
- 2.Retrieve top chunks by BM25 exact match and by embedding similarity
- 3.Combine and deduplicate the two result sets using rank fusion
- 4.Add the top-K chunks to the prompt to generate the response
- 5.Break the corpus into chunks of no more than a few hundred tokens
Both indexes are built from the same chunks. The two candidate lists are then fused and deduplicated before the top-K go to the model.
“Combine and deduplicate results from (3) and (4) using rank fusion techniques;”Source: www.anthropic.com
How you fuse the two lists is where you match the system to your data. The contextual embeddings cookbook takes 150 candidates from each method and merges them with weighted Reciprocal Rank Fusion. The default is 80% weight on semantic search and 20% on BM25. The cookbook says these weights are tunable: move weight toward BM25 when queries are full of identifiers and toward embeddings when they are conceptual. On its codebase dataset, BM25 contributed roughly 40–45% of the top results after fusion. Keyword matching does real work even when embeddings carry most of the weight.
The embedding model is a second setting that depends on the data. Anthropic's embeddings guide lists general-purpose Voyage models (voyage-4-large for quality, voyage-4-lite for latency and cost, voyage-4 balanced). It also lists domain-specific ones: voyage-code-3 for code, voyage-finance-2 for finance and voyage-law-2 for legal text. Choosing a domain model for a code or contracts corpus is a direct way to fit the semantic leg to the data. The guide adds that you should compare several embedding vendors rather than assume one fits every use case.
Checkpoint 5 of 5· Exam question
An internal ops agent has 8 tools total (deploy, rollback, restart, status, logs, scale, alert, notify), each under 100 tokens, and nearly every request needs several of them. Which tool-configuration approach fits this scenario?
Correct answer: A — Call all 8 tools directly through standard tool definitions without enabling tool search or deferred loading.
- A. Correct. With fewer than 10 tools, small definitions, and near-universal use per request, standard tool calling without tool search is the better fit; deferring these tools would add search overhead for no benefit.
- B. Tool search helps when catalogs are large or selection accuracy is degrading; forcing a search step for 8 tools used in nearly every request adds latency without solving a real problem.
- C. Deferring all 8 tools behind regex search removes the 3-5 frequently used tools from immediate context, which is discouraged when most of the catalog is needed on almost every call.
- D. Deferring the whole toolset behind one MCP connector still forces a search step before nearly every request can proceed, which is unnecessary at this small scale.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Any knowledge base Claude must answer from needs a chunk-embed-retrieve RAG pipeline.Why is that wrong?
Below about 200,000 tokens, Anthropic recommends putting the whole corpus in the prompt, with prompt caching to control cost and latency.
Covered in First question: does the corpus fit in the prompt?
2.A strong enough embedding model will reliably retrieve exact identifiers such as error codes or part numbers.Why is that wrong?
Embeddings capture meaning and can miss exact strings. Lexical BM25 is what matches identifiers and technical terms, which is why hybrid search exists.
Covered in Semantic embeddings versus exact-match BM25
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practicesOfficial docs
“Place your long documents and inputs near the top of your prompt, above your query, instructions, and examples.”
↩︎ First question: does the corpus fit in the prompt? - 2.https://www.anthropic.com/engineering/contextual-retrievalSecondary source
“reducing latency by > 2x and costs by up to 90%”
↩︎ First question: does the corpus fit in the prompt?“Break down the knowledge base (the “corpus” of documents) into smaller chunks of text, usually no more than a few hundred tokens;”
↩︎ First question: does the corpus fit in the prompt?“uses lexical matching to find precise word or phrase matches”
↩︎ Semantic embeddings versus exact-match BM25“you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods”
↩︎ Exam trap 1“While embedding models excel at capturing semantic relationships, they can miss crucial exact matches.”
↩︎ Exam trap 2“you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods”
↩︎ Prediction“smaller than 200,000 tokens (about 500 pages of material)”
↩︎ Checkpoint“BM25 looks for this specific text string to identify the relevant documentation.”
↩︎ Checkpoint“Combine and deduplicate results from (3) and (4) using rank fusion techniques;”
↩︎ Checkpoint - 3.
“The embeddings allow you to do semantic search / retrieval in the vector space.”
↩︎ Semantic embeddings versus exact-match BM25“Do not omit input_type or set input_type=None.”
↩︎ Semantic embeddings versus exact-match BM25“Code and programming documentation: voyage-code-3”
↩︎ Hybrid search and tuning to the data shape - 4.
“Default: 80% weight to semantic search, 20% to BM25”
↩︎ Hybrid search and tuning to the data shape“hybrid search catches keyword-specific queries that embeddings miss”
↩︎ Hybrid search and tuning to the data shape“The weighting system lets you balance between semantic understanding and keyword precision based on your data characteristics.”
↩︎ Key concept