CertSafari
    Anthropic Claude Certified Architect – Professional (CCAR-P) - December 7 onwards· Lessons

    Domain 3 · Lesson 17/39

    Retrieval architecture selection: full context, agentic search or RAG

    Retrieval Architecture Selection

    19 min read
    2.63% of exam
    8 sources
    Published 18 Nov 2026
    Docs as of 18 Nov 2026

    What you will be able to do

    • Choose between full-corpus context, agentic search with tools and a pre-built RAG pipeline from corpus size, query pattern, freshness and latency and cost limits
    • Explain why hybrid embeddings plus BM25 retrieval handles both paraphrased questions and exact identifiers
    • Specify a contextual-retrieval configuration, including how prompt caching keeps its one-time indexing cost down
    • Pick reranking, fusion weights and chunk count to fit a given latency and per-query cost budget

    Key concept

    Fit-first retrieval selection — Before you build an index, ask whether the whole corpus fits comfortably in the context window. If it does, load it in a cached prompt. Choose indexed retrieval or agentic search only when the corpus is too big or changes too fast for that, and then let the query pattern and the latency budget decide between them.

    1.Three ways to get knowledge in front of Claude

    Every retrieval design answers one question: which tokens about your business reach the model on each request. There are three common answers. Full-corpus context puts the entire knowledge base in the prompt. A pre-built RAG pipeline chunks and indexes the corpus ahead of time, then adds the best-matching chunks to each prompt. Agentic search gives Claude tools such as grep, file reads or a custom search function, and lets it look things up while it works. Each one moves cost and latency to a different point in the request.

    Whichever you choose, the retrieved material has to fit in the same budget. Everything in a request counts toward the context window. That includes the system prompt, the tool definitions, and every tool result an agent collects along the way. Current models such as Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5 have a 1M-token window, and other models, such as the deprecated Claude Sonnet 4.5, have 200k. The window is a ceiling, not a target, because recall weakens as the context fills. So the selection depends on four properties of the workload: how large the corpus is, how often it changes, what the questions look like, and how quickly and cheaply each answer has to come back.

    Where each architecture spends its cost and when it wins
    ArchitectureWork done ahead of timeWork done per queryWins when
    Full-corpus contextNone beyond caching the promptReading the cached corpusThe corpus fits comfortably in the window and questions need several sections at once
    Pre-built RAG (embeddings + BM25)Chunking, embedding, building the BM25 indexIndex lookup, rank fusion, optional rerankingThe corpus is far too large for one prompt and answers must return quickly
    Agentic search with toolsLittle or none, no index to keep freshSeveral tool calls and model turnsContent changes constantly and latency is not tight

    Checkpoint 1 of 9· Match them up

    Match each workload to the retrieval architecture that fits it best.

    Tap a term, then the definition that fits it.

    Sources12

    2.When the whole corpus fits: load it and cache it

    Check whether the corpus fits before you design an index. If the knowledge base fits in the prompt with room to spare, you can skip RAG and include the whole thing. Anthropic first gave this advice when windows were 200,000 tokens, roughly 500 pages. A 1M-token window raises that ceiling, but the reasoning stays the same. With nothing to chunk, no chunk can fail to be retrieved. That matters for questions that need several sections at once, which a top-k chunk lookup can only half answer.

    Prompt caching is what makes this affordable at volume. You write the corpus to the cache once, paying a 25% premium over base input for the 5-minute TTL. Every later request reads it at about 10% of the base input price. Anthropic reported that caching cut latency by more than 2x and cost by up to 90%. When the manuals change each quarter, you refresh the cached prefix, and there is no embedding job to rerun.

    Two limits stop you from stretching this too far. The first is context rot: as token count grows, accuracy and recall degrade, so a corpus that only just fits makes a weaker prompt than one that fits easily. The second is price on some models. On Claude Haiku 5.5, prompts over 100,000 tokens cost more, so a large corpus there costs more on every request. A corpus of tens of millions of tokens is out of reach entirely. Rebuilding a cached prefix every week does not get around the size of the window.

    Checkpoint 2 of 9· Check yourself

    An architect plans to load a 900,000-token knowledge base into a cached prompt on a model with a 1M-token window, because "it fits." What is the main concern with this plan?

    Checkpoint 3 of 9· Exam question

    You are the architect for a product-support assistant built on the Claude Messages API. The knowledge base is 120 pages of product manuals, roughly 150,000 tokens, revised about once a quarter. Thousands of questions arrive daily, many of them need facts from several sections at once, and the model's context window comfortably exceeds the corpus. Which retrieval architecture is the most defensible starting point?

    Sources3415

    Agentic search replaces the index with tools and judgment. The agent keeps lightweight references, such as file paths, stored queries and links, and loads data into context at runtime only when it needs it. Claude Code works this way. Its CLAUDE.md files go into context up front, and glob and grep fetch everything else just in time. Anthropic says this approach avoids the problems of stale indexing and complex syntax trees. In a codebase or folder that changes all the time, avoiding a stale index is a real advantage.

    You pay for that flexibility in time and in tokens. Exploring at runtime is slower than retrieving pre-computed data, because each search is another model turn and another tool call. An agent without good tools and guidance can also waste context on dead ends. Every file it opens counts against the same window. That is why an agent doing grep and file reads over a 40-million-token contract archive is a poor fit when answers must return in a few seconds. It works best where the content changes faster than an index could be rebuilt and a slower answer is acceptable.

    You do not have to pick one or the other. Some agents load the most important data up front for speed and explore the rest as needed. Anthropic suggests this hybrid may suit areas where content changes less often, such as legal or finance work. When your own search tool supplies the results, you can return them as search_result content blocks. Claude then cites them with the source and title you gave, which suits answers that must point back to a specific document.

    Checkpoint 4 of 9· Check yourself

    Your team's assistant answers questions about a monorepo that receives dozens of commits every hour, and users accept answers that take a minute. A colleague proposes a vector index rebuilt every night. What is the strongest argument for agentic search with grep and file reads instead?

    Sources26

    4.Indexed RAG: embeddings and BM25 together

    For a corpus too large for any prompt, a pre-built index lets you scale to enormous knowledge bases at a predictable per-query cost. The standard pipeline splits documents into chunks of at most a few hundred tokens. It embeds each chunk and stores the vectors so they can be searched by semantic similarity. Anthropic does not make its own embedding model. Voyage AI is one provider, and the factors to weigh when choosing one include lookup latency and whether you can customize the model for your domain.

    Embeddings alone have a blind spot: exact strings. Suppose someone searches for "Error code TS-999". An embedding model may return passages about error codes in general and miss the one chunk that contains TS-999. BM25 is a lexical ranking function built on TF-IDF. It searches for that exact string. Fine-tuning the embedding model does not remove the blind spot, because the model still ranks by meaning rather than by matching the literal string. When users mix paraphrased questions ("who pays if delivery is late") with clause numbers or error codes, you need both kinds of signal.

    A hybrid pipeline runs both searches and then merges the results:

    1. Chunk the corpus and create both TF-IDF encodings and semantic embeddings for each chunk.
    2. Take the top chunks from BM25 by exact match, and the top chunks from embeddings by semantic similarity.
    3. Combine and deduplicate the two lists with rank fusion.
    4. Add the top-K chunks to the prompt.

    Each query is an index lookup plus a single model call, so answers come back in seconds even over tens of millions of tokens. A weekly re-ingest means re-indexing the changed chunks, not rebuilding a 40M-token prompt that would not fit anyway.

    Checkpoint 5 of 9· Put it in order

    A support corpus gets both paraphrased questions and exact error-code lookups such as "TS-999". Put the steps of the hybrid RAG pipeline that handles both in order.

    1. 1.Add the top-K fused chunks to the prompt
    2. 2.At query time, take the top chunks from BM25 by exact match and from embeddings by semantic similarity
    3. 3.Create TF-IDF encodings and semantic embeddings for each chunk
    4. 4.Combine and deduplicate the two result lists with rank fusion
    5. 5.Split the corpus into chunks of at most a few hundred tokens

    Checkpoint 6 of 9· Exam question

    A legal-tech company must let Claude answer questions over 40 million tokens of contract and policy documents. Questions mix paraphrased concepts ("who pays if delivery is late") with exact identifiers such as clause numbers and error codes. The corpus is re-ingested weekly, and answers must return in a few seconds. Which design best fits these constraints?

    Sources73

    5.Contextual retrieval: giving each chunk its context back

    Chunking removes the context a passage needs. Consider a chunk from an SEC filing that reads "The company's revenue grew by 3% over the previous quarter." It does not say which company or which quarter. A question about ACME Corp in Q2 2023 may therefore never retrieve it. Contextual retrieval fixes this by asking Claude to write a short, chunk-specific explanation and prepending it to the chunk before it is embedded and before the BM25 index is built. These are two techniques: Contextual Embeddings and Contextual BM25.

    The contextualizer prompt Anthropic used with Claude 3 Haiku: the whole document plus one chunktext
    <document>
    {{WHOLE_DOCUMENT}}
    </document>
    Here is the chunk we want to situate within the whole document
    <chunk>
    {{CHUNK_CONTENT}}
    </chunk>
    Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else.

    The added context is usually 50 to 100 tokens, and it has to be specific to the chunk. Adding a generic document summary to every chunk was tested and gave very limited gains. Copying one chunk's description onto its neighbours would also give them context that does not describe them. When done properly, contextual embeddings cut the top-20 retrieval failure rate by 35%. Contextual embeddings combined with contextual BM25 cut it by 49%, from 5.7% to 2.9%.

    Use prompt caching. Put a cache breakpoint after the document, write it to the cache on the first chunk's call, and the calls for its remaining chunks read it at the cache-read price instead of paying full input price each time. Using 8k-token documents and 800-token chunks, Anthropic estimated the one-time cost at $1.02 per million document tokens. An 8k document is well above the minimum cacheable length, which is 512 tokens on current models such as Claude Opus 5.5 and Haiku 5.5. Cutting the document down to its first page would save tokens but lose the context the method needs. Sending chunks without their document leaves nothing to situate them against.

    Checkpoint 7 of 9· Fill the gap

    In the contextualizer prompt, which placeholder goes in the first tag so that Claude can write context specific to the chunk?

    <document>
    {{ ? }}
    </document>
    Here is the chunk we want to situate within the whole document
    <chunk>
    {{CHUNK_CONTENT}}
    </chunk>
    Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else.

    Checkpoint 8 of 9· Exam question

    You are planning contextual retrieval for a 10-million-token corpus. Generating the situating context means sending Claude each source document together with each of its chunks, and the same document is sent once per chunk. Which technique keeps this one-time indexing cost down?

    Sources34

    6.Reranking, fusion weights and chunk count within a latency budget

    Once the index exists, a few settings decide how much quality you get for the latency and cost you can spend. The numbers below come from a published Anthropic cookbook that builds a contextual-retrieval pipeline and tests it on a dataset of 9 codebases with 248 queries. Quality is reported as Pass@k: the share of queries where the correct chunk appears among the top k retrieved, so Pass@10 of 95% means the right chunk was in the top 10 for 95% of queries. The first setting is fusion weighting. The pipeline pulls the top 150 candidates from semantic search and from BM25 and merges the two ranked lists with weighted Reciprocal Rank Fusion. Fusion works from rank positions rather than raw scores, and each list's weight sets how much that list counts toward the merged ranking. The pipeline starts at 80% semantic and 20% BM25. Those weights are a starting point that balances semantic understanding against keyword precision. A corpus full of part numbers may need more weight on BM25, so that exact matches rise in the merged list. BM25 matches against the added context as well as the original chunk text.

    Reranking adds a second stage. You over-retrieve about 10x the number of chunks you need, for example 100 chunks when you need 10. A reranking model scores each candidate and keeps the top-k. Voyage AI and Cohere both offer rerankers. On the cookbook's codebase set this raised Pass@10 from 92.34% to about 95%, and Anthropic's blog reports a 67% cut in failed retrievals when reranking is added to contextual retrieval. Reranking is not free. It adds roughly 100 to 200 ms per query, depending on candidate count, and about $0.002 per query in API cost.

    The last setting is how many chunks to send. More chunks raise the chance that the right one is included, but extra material can distract the model. In Anthropic's tests, 20 chunks performed better than 5 or 10. Questions that need facts from more than one chunk depend on that headroom.

    Matching the indexed configuration to the binding constraint (cookbook codebase results)
    RequirementConfigurationPass@10Ongoing cost
    High-volume, cost-sensitiveContextual embeddings alone92.34%None per query; one-time ingestion (~$3 for this dataset with prompt caching)
    Balanced production systemContextual embeddings + hybrid search (BM25)93.21%Elasticsearch infrastructure and maintenance, no per-query API cost
    Maximum accuracy, latency-tolerantContextual embeddings + reranking95.26%100-200ms and ~$0.002 per query

    Pick the row from whichever constraint is tightest. Contextual embeddings gave the biggest single gain, 5 to 7 points, and they cost nothing per query. Hybrid search and reranking add the last 2 to 3 points of precision, but only if you can afford the extra infrastructure, latency or per-query cost. Hybrid search is still the safeguard whenever exact identifiers appear in queries, because it catches the matches embeddings miss.

    Checkpoint 9 of 9· Match them up

    Each team below already has a contextual-retrieval index. Pair each team's tightest constraint with the configuration that fits it.

    Tap a term, then the definition that fits it.

    Sources783

    Sources

    (8)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “every message in messages (including tool results, images, and documents), and your tool definitions”
      ↩︎ Three ways to get knowledge in front of Claude
      “Other Claude models, including Claude Sonnet 4.5 (deprecated), have a 200k-token context window.”
      ↩︎ Three ways to get knowledge in front of Claude
      “long-context requests are billed at standard pricing, except on Claude Haiku 5.5, where prompts over 100,000 tokens cost more.”
      ↩︎ When the whole corpus fits: load it and cache it
      “As token count grows, accuracy and recall degrade, a phenomenon known as context rot.”
      ↩︎ Exam trap 1
      “As token count grows, accuracy and recall degrade, a phenomenon known as context rot.”
      ↩︎ Checkpoint
    2. 2.
      “runtime exploration is slower than retrieving pre-computed data.”
      ↩︎ Three ways to get knowledge in front of Claude
      “effectively bypassing the issues of stale indexing and complex syntax trees.”
      ↩︎ Agentic search: retrieval at runtime with tools
      “Without proper guidance, an agent can waste context by misusing tools, chasing dead-ends, or failing to identify key information.”
      ↩︎ Agentic search: retrieval at runtime with tools
      “primitives like glob and grep allow it to navigate its environment and retrieve files just-in-time, effectively bypassing the issues of stale indexing”
      ↩︎ Checkpoint
      “primitives like glob and grep allow it to navigate its environment and retrieve files just-in-time, effectively bypassing the issues of stale indexing”
      ↩︎ Checkpoint
    3. 3.
      “If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base”
      ↩︎ When the whole corpus fits: load it and cache it
      “reducing latency by > 2x and costs by up to 90%”
      ↩︎ When the whole corpus fits: load it and cache it
      “Combine and deduplicate results from (3) and (4) using rank fusion techniques;”
      ↩︎ Indexed RAG: embeddings and BM25 together
      “Contextual Retrieval solves this problem by prepending chunk-specific explanatory context to each chunk before embedding”
      ↩︎ Contextual retrieval: giving each chunk its context back
      “You simply load the document into the cache once and then reference the previously cached content.”
      ↩︎ Contextual retrieval: giving each chunk its context back
      “the one-time cost to generate contextualized chunks is $1.02 per million document tokens.”
      ↩︎ Contextual retrieval: giving each chunk its context back
      “We tried delivering 5, 10, and 20 chunks, and found using 20 to be the most performant of these options”
      ↩︎ Reranking, fusion weights and chunk count within a latency budget
      “Sometimes the simplest solution is the best.”
      ↩︎ Key concept
      “BM25 looks for this specific text string to identify the relevant documentation.”
      ↩︎ Exam trap 2
      “adding generic document summaries to chunks (we experimented and saw very limited gains)”
      ↩︎ Exam trap 3
    4. 4.
      “Cache writes: When new content is written to the cache (25 percent more than base input tokens for 5-minute TTL)”
      ↩︎ When the whole corpus fits: load it and cache it
      “Cache reads: When cached content is used (10 percent of base input token price, with per-model exceptions)”
      ↩︎ Contextual retrieval: giving each chunk its context back
      “Cache reads: When cached content is used (10 percent of base input token price, with per-model exceptions)”
      ↩︎ Prediction
    5. 5.
      “Some of our questions require synthesis between more than one chunk in order to be answered correctly”
      ↩︎ When the whole corpus fits: load it and cache it
    6. 6.
      “From tool calls: Your custom tools return search results, which lets you build dynamic RAG applications.”
      ↩︎ Agentic search: retrieval at runtime with tools
    7. 7.
      “Anthropic does not offer its own embedding model.”
      ↩︎ Indexed RAG: embeddings and BM25 together
      “Inference performance: embedding lookup speed and end-to-end latency.”
      ↩︎ Indexed RAG: embeddings and BM25 together
      “Voyage AI makes embedding models and rerankers.”
      ↩︎ Reranking, fusion weights and chunk count within a latency budget
    8. 8.
      “Default: 80% weight to semantic search, 20% to BM25”
      ↩︎ Reranking, fusion weights and chunk count within a latency budget
      “Over-retrieve: Get 10x more results than needed (e.g., retrieve 100 chunks when we need 10)”
      ↩︎ Reranking, fusion weights and chunk count within a latency budget
      “High-volume, cost-sensitive: Contextual embeddings alone (92% Pass@10, no per-query costs)”
      ↩︎ Reranking, fusion weights and chunk count within a latency budget
      “Pass@k checks whether or not the 'golden document' was present in the first k documents retrieved for each query.”
      ↩︎ Reranking, fusion weights and chunk count within a latency budget
      “We use a pre-chunked dataset of 9 codebases”
      ↩︎ Reranking, fusion weights and chunk count within a latency budget
      “reranking adds ~100-200ms per query depending on candidate set size.”
      ↩︎ Exam trap 4
      “Choose your approach based on your requirements: High-volume, cost-sensitive: Contextual embeddings alone (92% Pass@10, no per-query costs) Maximum accuracy, latency-tolerant: Full reranking pipeline”
      ↩︎ Checkpoint

    Ready to test yourself?

    Practise Anthropic Claude Certified Architect – Professional (CCAR-P) - December 7 onwards in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.