Cruxly › AI engineering › Retrieval and RAG

Chunking and Document Parsing: Structure-Aware Splits, Parent-Child, Late Chunking and Contextual Retrieval

parse first, count the overlap tax, and test five strategies on one span-labelled set

Prompt: "Our RAG answers are wrong on questions whose answers sit in tables, and right-looking but incomplete on policy PDFs. Someone suggests a bigger chunk size. How do you choose a chunk size, how do you handle tables and PDFs, what is contextual retrieval and what did it improve, and how would you compare two chunking strategies?"

40 minutes first. Work out the chunk count and the overlap tax for a 6,000-token document, and place one answer span on the chunk boundaries. The trap is tuning a number before looking at what the parser gave the chunker.

Parsing comes first: what the chunker is actually given

Most bad chunks are parsing failures that the chunker inherits. A PDF does not store paragraphs; it stores positioned runs of text. A two-column page extracted in raw order interleaves the left and right columns line by line, and no splitter can fix a sentence that begins in one column and continues in the other. Headers, footers and page numbers repeat on every page and end up in the middle of chunks. A scanned page has no text at all until OCR produces some, and OCR errors (rn read as m, dropped Devanagari matras) become permanent tokens in your index. Run these checks on twenty real documents before you tune any chunking parameter.

• Reading order. Use a layout-aware parser, then read ten extracted pages with your eyes. Look for interleaved columns and footnotes landing mid-sentence.
• Headings. Preserve heading levels and build a title path such as HR Policy > Leave > Maternity leave. It is the cheapest high-value metadata you have, and section 5 shows how to use it.
• Tables. A table cut at 512 tokens leaves the header row in one chunk and the numbers in the next. A chunk reading Gold | 1,200 | 18 has lost what 1,200 and 18 mean; the answer to "monthly fee for the Gold plan" is now unreachable. Either keep the table as one chunk (if it fits), or split by row groups and repeat the header on every piece, or serialise each row as a sentence (Gold plan: monthly fee 1,200 rupees, 18 months lock-in).
• Code blocks. Split at function or class boundaries; never inside a block. Include the file path and signature as the title path.
• Scanned pages and OCR. Run a quality gate: the ratio of tokens that are dictionary words, the average confidence the OCR engine reports, and the page's character count. A page with 12 characters or 40 percent unknown words is quarantined, not indexed. Ingestion Pipelines: Sync, Freshness, Permissions and Deletion makes this a pipeline stage.

Chunk on structure first and size second. Split at section boundaries, then paragraphs, then sentences, and only fall back to a hard token cut when a single paragraph exceeds the limit. That is what recursive splitters do, and the RAG From Scratch, No Framework lesson shows the blind 500-character cut that this rule prevents.

Five strategies, counted on one document

Fixed-size chunks with overlap are the baseline; the others buy something specific, and each has a price you can count. Use a 6,000-token document and 512-token chunks.

Fixed with overlap. Chunk 512, overlap 64, so the stride is 448. Number of chunks is ceil((6000 - 512) / 448) + 1 = ceil(12.25) + 1 = 14. Chunks 0 to 12 hold 512 tokens each and the last starts at token 5,824 and holds

1. Indexed tokens: 13 x 512 + 176 = 6,832, which is 13.9 percent more than the 6,000 in the document (a padded last chunk would make it 7,168 and 19 percent; count what your splitter really emits). With no overlap it is 12 chunks and exactly 6,000 tokens. That 13.9 percent is paid in embedding tokens, vector storage and BM25 postings, every time.

Recursive or structure-aware. Split on the largest separator that keeps pieces under the limit: section, paragraph, sentence. Chunk sizes vary (maybe 180 to 510 tokens), nothing is cut mid-sentence, and a heading stays with its text. The default to beat.

Semantic splitting. Embed each sentence and cut where adjacent-sentence similarity drops. It follows topic shifts, costs one embedding call per sentence at ingestion (a 6,000-token document is roughly 250 sentences), and is unstable: change the model or threshold and every boundary moves. It must earn its place in an ablation, not in a blog post.

Parent-child (small-to-big). Embed small child chunks for precise matching, but return the larger parent to the model. With 128-token children, the document yields ceil(6000 / 128) = 47 vectors; with 512-token parents, 12 parents to store as plain text. The search sees 47 sharp targets instead of 14 blurry ones, and the model sees the full surrounding section. The failure mode: two of the top five children share a parent, so you must deduplicate parents or you waste your context on the same text.

Sentence-window. Embed single sentences and return the sentence plus the two either side. It is the extreme of parent-child: the most precise matching and the largest vector count (about 250 for this document, more than 5x the 47 above and more than 17x the 14).

Chunk size is three constraints, not one number

Chunk size has to satisfy the embedding model, the reranker and the answer budget at once, and the best size usually comes from a sweep.

• Embedding limit. A model with a 512-token input limit silently drops everything after token 512; an 8,192-token model accepts a 3,000-token chunk, but that vector averages many topics and matches none of them sharply (Embeddings and Similarity: Cosine, Dot Product, and What a Vector Encodes).
• Reranker window. Cross-encoder rerankers read the query and the chunk together, and they have their own limit and a cost that grows with chunk length (the cost model is in Hybrid Search, Rerankers and Late Interaction). 50 candidates at 400 tokens is 20,000 tokens per query; at 1,000 tokens it is 50,000.
• Answer budget. If the generator receives the top 5 chunks of 512 tokens, that is 2,560 tokens of evidence. Raise the chunk size to 1,024 with the same k and it is 5,120. The "Lost in the Middle" study (Liu et al., 2023) found that models use information at the beginning and end of a long context better than in the middle, so more tokens of evidence is not free even when it fits.

Run the sweep, do not argue. Build the index at 128, 256, 512 and 1,024 tokens (structure-aware, constant overlap ratio), run the same golden set, and plot recall@k against tokens delivered to the model, not against k. A 128-token chunk at k = 20 and a 512-token chunk at k = 5 both deliver about 2,560 tokens. Often the best size depends on the question type: factual lookups prefer small chunks, "explain the process" questions prefer large ones. When one size cannot serve both, that is the case for parent-child retrieval above: search small, deliver big.

Generic advice has no number in it for a reason. A single "best" chunk size does not exist, and a vendor's recommendation was tuned on a vendor's documents. Anthropic's contextual retrieval post says chunks should generally be no more than a few hundred tokens, and uses 800 only for its cost example. Use that as a starting range, then measure.

Late chunking and contextual retrieval: giving a chunk its document back

A chunk embedded alone has lost the document around it, and two techniques put it back. The classic example: the chunk "The company's revenue grew by 3% over the previous quarter" has no company name and no quarter in it. A query for "ACME Corp Q2 2023 revenue growth" matches it weakly.

Late chunking (Günther et al., 2024, "Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models") reverses the order of operations. Run the whole document through a long-context embedding model so every token's vector has seen the document, then split those token vectors into chunks and mean-pool each chunk. The chunk embedding is "late" because the boundary is drawn after the transformer. It needs no extra training (the authors also propose fine-tuning) and no extra LLM calls, but it needs a long-context embedding model and a document short enough for it.

Contextual retrieval (Anthropic's September 2024 post) does it in text. For each chunk, an LLM reads the whole document and writes a short context, 50 to 100 tokens, which is prepended to the chunk before embedding and before BM25 indexing: "This chunk is from ACME Corp's Q2 2023 filing; previous quarter revenue was $314 million". The numbers below are Anthropic's own benchmark, so quote them as that and not as a law of nature.

:::key Verified October 2026 Anthropic's contextual retrieval post, retrieved October 2026, reports top-20 retrieval failure rates (lower is better) of 5.7% for baseline embeddings, 3.7% with contextual embeddings (a 35% reduction), 2.9% with contextual embeddings plus contextual BM25 (49%), and 1.9% when a reranker is added on top (67%; reranking started from the top 150 candidates and kept 20). It is a vendor evaluation on its own datasets. The one-time cost it quotes is about $1.02 per million document tokens, assuming 800-token chunks, 8k-token documents, 50 tokens of instructions and 100 tokens of generated context. It also says that a knowledge base under about 200,000 tokens may not need RAG at all. :::

Count the cost yourself. Our 6,000-token document makes 14 chunks and 14 LLM calls, each of which needs the whole document in context. Without caching the input is 14 x (6,000 + 512 + 50) = 91,868 tokens plus 14 x 100 = 1,400 generated tokens. With prompt caching the document is written once and read from cache on the other 13 calls. Under an assumed cache write at 1.25x and cache read at 0.1x the base input price (check current rates, and see Prompt Caching: Paying for the Same Tokens Once), the input cost is 6,000 x 1.25 + 562 + 13 x (6,000 x 0.1 + 562) = 23,168 token-equivalents, about 25 percent of the uncached figure. Generated context also grows each embedded chunk from 512 to about 612 tokens, a 19.5 percent increase in embedding tokens and storage, on top of any overlap.

What to say in the room. Contextual retrieval is a one-time ingestion cost that pays back on every query, so it is attractive for stable corpora and awkward for fast-changing ones where every edit regenerates context for every affected chunk. It helps most where chunks are ambiguous without their document (filings, contracts, manuals with repeated section names) and least on self-contained chunks like FAQ entries. Run it as one row in your ablation table.

Metadata that earns its keep, and how to compare chunkers honestly

Attach a small set of fields to every chunk, and embed the one that carries meaning. Fields worth their storage: title path (HR Policy > Leave > Maternity), source and page, document type, effective date, and access-control groups. The title path is worth prepending to the text that is embedded and BM25-indexed, which gives a cheap, deterministic version of what contextual retrieval does with an LLM. Date and type enable filters and recency boosts. Access-control metadata belongs in the chunk row so the database can enforce it, as Multi-Tenant, Multilingual RAG at Scale explains; the sync side is in Ingestion Pipelines. Every field you add must be maintained, so add one only when a query or a filter uses it.

Compare chunkers on answer spans, not chunk IDs. If your golden set says "chunk 17 is relevant", it is useless the moment you change the chunker. Label each query with the document and the answer's token span instead. A retrieved chunk counts as a hit if it contains that span (or most of it).

Worked example: an answer at tokens 1,000 to 1,100. With non-overlapping 512-token chunks, boundaries fall at 512 and 1,024, so the answer straddles chunks 1 (tokens 512 to 1,023) and 2 (1,024 to 1,535). Neither contains the full span, so a strict contains-the-answer metric calls this a miss, and the generator sees half a sentence. With 512/64 chunks (stride 448), chunk 2 spans tokens 896 to 1,407 and contains the whole answer: a hit. That is what overlap buys, and why the 13.9 percent cost can be worth it. The flip side of overlap: if chunks 2 and 3 both come back, they share 64 tokens. Merge adjacent hits into one passage before prompting so you are not paying twice for the shared text.

Run the ablation table. Hold embedding model, index and k constant. Rows: fixed 512/64; structure-aware; structure-aware + title path; parent-child; + contextual text. Columns: recall@k (span-based), tokens delivered, index size, ingestion cost. Add one more column: queries that got worse against the baseline. A chunker that lifts the average and breaks tables is a regression that the mean hides, so read the per-query diff for the failures, which is the Evaluation Foundations error-analysis habit.

How to say it in the interview

"Chunking starts with parsing. If the parser scrambles columns or separates a table from its header, no chunk size repairs it. So I split on document structure first, size second; I keep tables whole or repeat the header on every row group; and I add a title path to what gets embedded. Then I compare strategies on a golden set labelled by answer span, not by chunk ID, so any chunker can be scored. Overlap costs about 14 percent more index at 512/64, contextual retrieval cut Anthropic's top-20 retrieval failures by 35 to 67 percent on their own benchmark depending on what was layered on, and I would verify that on our data before paying for it."

Checklist

• Read ten extracted pages of real documents before tuning any chunk parameter
• Kept each table whole, or repeated its header on every row group, or serialised rows as sentences
• Added a quality gate for empty parses, garbled OCR and duplicate floods before indexing
• Split on section, paragraph and sentence before falling back to a token cut
• Counted the overlap tax (512/64 on 6,000 tokens is 13.9 percent more indexed tokens)
• Checked chunk size against the embedding input limit, the reranker window and the answer budget
• Swept 128 to 1,024 tokens and compared at equal tokens delivered to the model
• Deduplicated parents (parent-child) and merged adjacent overlapping hits before prompting
• Prepended the title path to embedded and BM25 text, and stored page, type, date and ACL as metadata
• Priced contextual retrieval with prompt caching and tested it as one ablation row
• Labelled golden queries by document and answer span, not by chunk ID
• Read the per-query regressions of every candidate chunker, not just the mean

Related lessons

• RAG From Scratch, No Framework
• Choosing, Evaluating and Fine-Tuning an Embedding Model
• ANN Indexes in Depth: HNSW, IVF-PQ, DiskANN and the Recall/Latency/Memory Triangle
• Hybrid Search, Rerankers and Late Interaction
• Query Understanding and Agentic Retrieval: Rewriting, Routing, Multi-Hop and GraphRAG
• Multi-Tenant, Multilingual RAG at Scale