What you will be able to do
- Explain why chunking is one of the first places to optimise a RAG application
- Distinguish fixed-size, paragraph-based, format-specific and semantic chunking by the boundaries each uses
- Choose a chunking strategy that fits the structure of a given corpus, including when to use ai_prep_search
Key concept
Semantically coherent chunk — A chunk whose boundaries follow the document's real units of meaning, such as paragraphs, headed sections or topic shifts, so it can stand alone as a meaningful piece of text. Choosing a chunking strategy mostly comes down to picking the boundary signal that gives you this kind of chunk for your documents.
1.Why chunking is a first-layer decision
In an unstructured data pipeline, chunking sits between preparing the documents and embedding them. Raw files are parsed, cleaned, deduplicated and filtered. Then the text is broken into smaller units, each unit is turned into a vector, and the vectors are indexed. Whatever the chunker produces is exactly what the retriever can later return. If a chunk cuts a procedure in half, or bundles three unrelated topics together, nothing further down the pipeline can fix that.
This is why the Databricks RAG cookbook calls chunking one of the first layers of optimisation. The goal has two parts. Chunks should be small enough that several retrieved chunks fit in the LLM's context window. They should also be focused enough that what gets retrieved isn't padded with distracting text. A chunk that is too broad wastes context and weakens the answer. A chunk that is too narrow drops the surrounding information that made it useful.
The cookbook lists five factors to weigh when you chunk: the strategy (how the text is divided), the chunk size, the overlap between neighbouring chunks, semantic coherence, and the metadata attached to each chunk. This page covers the first and the fourth: how the structure of your documents should drive the strategy. Size, overlap and metadata depend on the embedding model and the queries you expect, so they are treated separately.
Checkpoint 1 of 3· Check yourself
According to the Databricks RAG cookbook, what two goals does splitting large documents into semantically concentrated chunks serve?
The cookbook justifies chunking by context fit and by relevance. Cost, keyword matching, security and deduplication are dealt with elsewhere in the pipeline.
“ensures that retrieved data fits in the LLM's context while minimizing the inclusion of distracting or irrelevant information”Source: docs.databricks.com
Sources1
2.Four chunking strategies and the boundaries they use
The cookbook groups chunking strategies into four broad families. They differ in one thing: where they get their boundaries from. Fixed-size chunking ignores the content and cuts at a set count of characters or tokens. Paragraph-based chunking cuts at paragraph breaks, because a paragraph usually holds related information. Format-specific chunking uses markup the document already has, such as Markdown or HTML headers. Semantic chunking analyses the content itself, for example with topic modelling or embeddings, and cuts where the topic changes.
| Strategy | Boundary comes from | Example splitter named in the docs | Fit |
|---|---|---|---|
| Fixed-size | A set number of characters or tokens | CharacterTextSplitter | Quick to set up; rarely production-grade |
| Paragraph-based | Natural paragraph breaks | RecursiveCharacterTextSplitter | Plain prose where paragraphs hold related ideas |
| Format-specific | Existing markup such as Markdown or HTML headers | MarkdownHeaderTextSplitter | Documents with explicit section structure |
| Semantic | Topic shifts found by analysing the content | SemanticChunker | More work to build; better alignment with how the meaning divides |
The structure of your documents should decide which of these to use. If your source files are Markdown or HTML, the authors have already marked the section boundaries, so a header-based splitter uses them for free. If you have long, unbroken prose with no markup, paragraph breaks are the most reliable signal available. If topics drift within paragraphs, or the formatting is inconsistent, semantic chunking is worth the extra effort. The AI Search retrieval quality guide lists it as an advanced approach that groups sentences by similarity rather than by fixed size. Fixed-size splitting is still useful as a quick baseline, but the cookbook is clear that it rarely holds up in production.
Checkpoint 2 of 3· Exam question
A legal team's contract management RAG system indexes lengthy contracts whose clauses frequently reference definitions found in other sections. Users need precise, clause-level answers, but the agent also needs enough surrounding context to interpret cross-referenced terms correctly when generating a response. Which chunking strategy best addresses both needs?
Correct answer: B — Parent-child chunking, indexing small child chunks for retrieval and returning their parent chunks for context
- A. Incorrect. A single fixed chunk size with no overlap does not give the agent both a precise retrieval unit and a larger surrounding context for cross-referenced terms; it forces a single size to serve two competing needs.
- B. Correct. Parent-child (small-to-big) chunking indexes small, precise child chunks for retrieval so clause-level matches stay sharp, while returning the larger parent chunk that contains surrounding sections, giving the agent the context needed to resolve cross-references.
- C. Incorrect. Random-length chunking at arbitrary character positions is likely to split clauses and definitions mid-sentence, degrading both precision and context rather than improving either.
- D. Incorrect. Embedding each entire contract as a single vector loses the precision needed to retrieve a specific clause and dilutes the embedding with unrelated sections of the contract.
- E. Incorrect. Semantic chunking can group related clauses together, but on its own it does not provide a distinct larger context unit the way parent-child chunking does, so cross-referenced definitions outside the semantic group are still missed.
3.Preserving structure before and during the split
A structure-aware strategy only works if the structure survives parsing. If a PDF parser drops tables or flattens headings into one stream of text, a header-based splitter has nothing to split on. The AI Search retrieval quality guide treats parsing and chunking as one data-preparation problem. Its cleaning advice is to remove boilerplate such as headers, footers and page numbers, to keep document structure such as headings, lists and tables, and to keep semantic boundaries when chunking. For PDFs and complex documents, Databricks recommends ai_parse_document for high-quality parsing.
For teams that don't want to build their own splitter, Databricks has a managed option. ai_prep_search is a SQL AI function, currently in Beta, that takes either the structured output of ai_parse_document or plain text and Markdown. It splits the content into semantic chunks and adds document-level context to each one: title, section headers and page references. For chunks that contain a table, it also appends a table summary and natural-language questions the table can answer. That is aimed squarely at documents where tables carry the answers. Its options are a schema version and an optional metadata-extraction schema. It does not expose a chunk-size setting.
WITH parsed_documents AS (
SELECT ai_parse_document(content) AS parsed
FROM READ_FILES('/Volumes/mydata/documents/', format => 'binaryFile')
)
SELECT ai_prep_search(parsed) AS result
FROM parsed_documents;Checkpoint 3 of 3· Match them up
Match each document situation to the chunking approach the Databricks docs point toward
Tap a term, then the definition that fits it.
Each approach uses the strongest boundary signal the corpus offers: markup, topic shifts, or parsed table structure. Fixed-size splitting is easy to set up but is not recommended for production.
“Formats such as Markdown or HTML have an inherent structure that can define chunk boundaries (for example, markdown headers).”Source: docs.databricks.com
Because the headers have to survive parsing first. If the parser flattens headings or drops tables, the splitter has no boundaries to use. The guide notes that poor parsing, such as missing tables or broken formatting, directly hurts retrieval quality.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Fixed-size token chunking is a safe production default because every chunk comes out the same size.Why is that wrong?
Splitting at an arbitrary character or token count typically does not give semantically coherent chunks, and Databricks says it rarely works in production. Use the boundaries the document structure gives you instead.
Covered in Four chunking strategies and the boundaries they use
2.Chunking strategy can be chosen without looking at how the documents were parsed.Why is that wrong?
Structure-based chunking depends on headings, lists and tables surviving parsing. Poor parsing directly hurts retrieval quality, so parsing and chunking have to be designed together.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/agents/tutorials/ai-cookbook/quality-data-pipeline-ragOfficial docs
“making it one of the first layers of optimization in an RAG application”
↩︎ Why chunking is a first-layer decision“Use the natural paragraph boundaries in the text to define chunks.”
↩︎ Four chunking strategies and the boundaries they use“determine the most appropriate chunk boundaries based on topic shifts”
↩︎ Four chunking strategies and the boundaries they use“aim to create semantically coherent chunks that contain related information but can stand independently as a meaningful unit of text”
↩︎ Key concept“it will typically not result in consistent semantically coherent chunks”
↩︎ Exam trap 1“ensures that retrieved data fits in the LLM's context while minimizing the inclusion of distracting or irrelevant information”
↩︎ Checkpoint“This approach rarely works for production-grade applications.”
↩︎ Prediction“Formats such as Markdown or HTML have an inherent structure that can define chunk boundaries (for example, markdown headers).”
↩︎ Checkpoint - 2.
“Semantic chunking: Group sentences by similarity rather than fixed size.”
↩︎ Four chunking strategies and the boundaries they use“Preserve document structure (headings, lists, tables).”
↩︎ Preserving structure before and during the split“Poor parsing (missing tables, broken formatting) directly impacts retrieval quality.”
↩︎ Exam trap 2 - 3.
“the function splits content into semantic chunks, enriches each chunk with document-level context”
↩︎ Preserving structure before and during the split“For chunks that contain a table, the function appends this table summary and a set of related natural-language questions”
↩︎ Preserving structure before and during the split