Sizing Chunks for Embedding Limits and Query Types

1.The embedding model's context length is a hard ceiling

Every embedding model has a maximum token limit, and that limit is the upper bound on chunk size. Limits vary widely. The cookbook gives bge-large-en-v1.5 as an example with a limit of 512 tokens, while the AI Search example notebooks describe both the GTE model and the OpenAI embedding model as supporting up to 8192 tokens. Exceeding the limit doesn't raise an error. The chunk is truncated, so the end of a long chunk is effectively invisible to similarity search. The same model also embeds the user's query at query time, so the limit applies to both sides of the comparison. Check the limit before you set a chunk size, not after retrieval starts missing content.

A large limit is not a target. Even for the 8192-token models, Databricks recommends splitting into smaller chunks so the reasoning model receives a wider variety of examples. The OpenAI example notebook does this, chunking at 1024 tokens even though the model accepts 8192:

max_chunk_tokens = 1024
encoding = tiktoken.get_encoding("cl100k_base")

Sources12

2.Sizing chunks for the queries the agent will face

Below the embedding ceiling, the right size depends on what users will ask. The AI Search retrieval quality guide offers three starting points to test and the trade-off each one makes. Small chunks localise a specific fact well but lose the surrounding context. Large chunks keep context but make it harder to pinpoint the relevant passage. There is also a second ceiling: the retrieved chunks, taken together, must fit in the LLM's context window. The guide also warns that embeddings can fail to capture basic information in long contexts, citing DeepMind's LIMIT work, so larger is not automatically safer.

Some queries aren't about the chunk's own text at all. A question like "what does the brake manual say about emergency adjustments" refers to the document and the section, not just to the sentences inside the chunk. The guide's answer is to prepend a document title, summary and section name to the chunk text before embedding. This helps with queries that reference document-level concepts. ai_prep_search does the same thing automatically. The guide also says that rather than over-optimising chunk size, you'll usually get more from good parsing and from semantic metadata such as document summaries and section headers added to chunks.

# Prepend document summary to each chunk
chunk_with_context = f"""
Document: {doc_title}
Summary: {doc_summary}
Section: {section_name}
{chunk_content}
"""

Sources3

3.Parent-child chunking, overlap, and iterating

What if the agent needs both precise matching and rich context? Parent-child chunking, also called small-to-big retrieval, avoids having to pick one size. Each document is split into large parent chunks for context, and each parent is split again into small child chunks for precision. Both are stored on the same row. Search runs against the small child text, which localises well, and the result returns the parent text, which gives the LLM enough context to answer.

# Record child and parent chunks in your source table
for parent_chunk in create_chunks(doc, size=2048):  # Large for context
    for child_chunk in create_chunks(parent_chunk, size=512):  # Small for precision
        source_table.append({"text": child_chunk, "parent_text": parent_chunk})

# Search children, return parents
results = index.similarity_search(
    query_text="Is attention all you need?",
    num_results=10,
    columns=["text", " ? "]
)

Overlap deals with a smaller version of the same problem: a sentence that matters can sit right on a chunk boundary. Repeating some text between neighbouring chunks keeps continuity across the boundary and can improve retrieval. The cookbook's illustration uses RecursiveCharacterTextSplitter with chunk_size=100 and chunk_overlap=20, and points to ChunkViz as a way to see how size and overlap change the resulting chunks.

None of these settings is final. The cookbook says outright that there is no one-size-fits-all approach. Choose a strategy from the document structure, set a size within the embedding limit, adjust for your query mix, then evaluate and change it.

Sources13

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.A chunk longer than the embedding model's token limit makes indexing fail, so oversized chunks are easy to spot.Why is that wrong?

   Oversized chunks are truncated without an error. Information past the limit never makes it into the vector.

   Covered in The embedding model's context length is a hard ceiling

2. 2.If the embedding model supports 8192 tokens, chunks should be 8192 tokens to make full use of it.Why is that wrong?

   The limit is a ceiling. Databricks recommends smaller chunks so the reasoning model receives a wider variety of examples.

   Covered in The embedding model's context length is a hard ceiling

3. 3.Retrieval quality problems are best fixed by repeatedly fine-tuning the chunk size.Why is that wrong?

   The guide steers effort away from over-optimising chunk size and toward metadata extraction, high-quality parsing, and semantic context added to chunks.

   Covered in Sizing chunks for the queries the agent will face

Practise it for real

Produce structure-aware, context-enriched chunks from documents in a Unity Catalog volume using ai_parse_document and ai_prep_search

1. 1.On Databricks Runtime 18.2 or above (or serverless environment version 3 or above), with the ai_prep_search Beta enabled, run the WITH parsed_documents AS (SELECT ai_parse_document(content) ...) query against your volume path, ending in SELECT ai_prep_search(parsed) AS result.

   Why: Parsing first keeps headings and tables intact, so ai_prep_search can split on the document's real structure.

   You should see: One result row per document, holding the prepared chunks for that document.

2. 2.Add map('schema', '["company", "document_type", "fiscal_year"]') as the second argument, replacing the field names with ones that matter to your corpus.

   Why: The schema fixes which document-level fields go into every chunk, so queries about those fields can match.

   You should see: Each chunk carries the fields you named as metadata.

3. 3.Flatten the output with LATERAL variant_explode(prepped_documents.result:document.contents) AS chunk, selecting chunk.value:chunk_id and chunk.value:chunk_to_embed.

   Why: AI Search indexes one row per chunk, using chunk_to_embed as the embedding column and chunk_id as the primary key.

   You should see: One row per chunk. Each chunk_to_embed string starts with labelled fields such as Document Title, Section Header and Page Number, followed by the chunk content.