A retrieval-augmented generation system can miss an answer that is plainly present in its source documents. One possible cause is the way those documents were divided before indexing. A rule appears in one chunk, its exception in another, and the table heading identifying the applicable year in a third. The generated answer can sound coherent while using an incomplete piece of evidence.
Chunking is the process of dividing content into retrievable units. The useful question is not simply how many tokens each unit contains. It is whether the unit preserves enough meaning to answer the intended questions, whether retrieval finds it, and whether the answer points back to the right source location.
Microsoft's Azure AI Search documentation describes fixed-size, structure-aware, semantic, and combined approaches. Those are implementation options, not proof that one method is universally best. This guide proposes an original test workflow rather than claiming to have benchmarked a particular service.
Define a complete evidence unit
Consider a fictional returns policy with a standard period, a shorter period for one product class, and a footnote excluding opened items. A retrievable unit containing only the standard period is inadequate for a question about that product class. It needs either the exception and footnote or a reliable connection to them.
List the question types first: direct lookup, exception lookup, table interpretation, comparison across versions, and cases where the documents provide no answer. Then identify which source passages a reviewer would need. This creates an evidence set against which chunking can be judged.
Compare three practical approaches
| Method | Main advantage | Failure to test |
|---|---|---|
| Fixed token windows | Predictable processing size | Cuts across rules or tables |
| Heading and paragraph boundaries | Preserves document organization | Overlong sections or weak headings |
| Semantic segmentation | Groups related ideas | Separates visually linked evidence |
Figure data and assumptions
Original hypothetical teaching data; see the caption and article for assumptions.
| Category | Windows |
|---|---|
| 0 tokens | 20.0 |
| 100 tokens | 25.0 |
| 200 tokens | 33.0 |
Start with a simple reproducible baseline. A more elaborate segmentation method can add processing cost and failure modes. It deserves adoption when it improves the actual retrieval task, not merely because the chunks read elegantly in isolation.
Measure coverage, not just average size
Suppose 30 test questions have human-identified evidence. In a fictional comparison, fixed windows retrieve complete evidence for 18, heading-based chunks for 24, and a combined method for 26. These invented results demonstrate the measurement, not expected performance.
Review the four failures in the combined method. If all involve table footnotes, the next experiment should preserve table context. Increasing every chunk's length could make retrieval less selective without fixing missing table structure. The failure category is more informative than the aggregate score alone.
Overlap has a cost
Overlap can preserve continuity when a sentence spans a boundary, but repeated text consumes indexing and context space. For a long document, a 500-token window with a 100-token overlap advances about 400 tokens per chunk. A 200-token overlap advances about 300. More overlap therefore creates more chunks even though the underlying document is unchanged.
The accompanying figure uses a 10,000-token teaching document. Under a simple sliding-window rule, 500-token chunks need 20 windows without overlap, 25 with 100-token overlap, and 33 with 200-token overlap. These counts assume the final short window is retained and no additional structural rules intervene.
Duplicate passages can also crowd the retrieved results. If five near-identical chunks occupy the top five positions, the answer may miss another essential source. Track source diversity and complete evidence coverage, not just the apparent relevance of each returned snippet.
Keep source identity attached
Store document title, source URL or stable identifier, version date, section, and page or paragraph location. Retain access permissions as part of the retrieval design. A relevant chunk must not become an accidental route to information the requester cannot access.
Distinguish a publication date from a policy's effective date. When an old and new policy conflict, the system needs a rule for selecting the applicable version. A generated citation to the correct title but wrong version is still a sourcing failure.
Run an end-to-end experiment
Freeze the source collection, embedding model, search configuration, question set, and answer instructions. Change only the chunking policy. For each question, record whether the required evidence was indexed, retrieved, supplied to the answer model, and used correctly.
Also record indexing time, storage, retrieved-token volume, response time, and source-location accuracy. A policy that finds two additional answers may be worth extra processing; it may also be unsuitable if it creates a large queue or repeatedly exposes obsolete versions. The trade-off belongs in the result.
Handle tables and images explicitly
A table cell without column labels is often meaningless. Preserve row and column context, units, dates, and footnotes. If extraction creates prose descriptions of diagrams, label those descriptions as derived representations and retain the underlying page for checking. Do not imply that an image caption is the original table.
For scanned files, evaluate extraction separately. No chunking strategy can recover text that never reached the index. A useful diagnostic question is whether a human can find the answer in the extracted representation, rather than only in the original PDF.
Design a question set that can expose the failure
A test made only of direct keyword lookups will favor a different document representation from one containing exceptions and cross-references. Build the set from the work readers actually need. Include a question whose answer fits one paragraph, one requiring a paragraph plus footnote, one requiring a table row and heading, and one that has no supported answer.
For each question, write down the minimum sufficient evidence before looking at retrieval results. Otherwise it is easy to declare a returned passage sufficient simply because the answer sounds plausible. A reviewer should be able to explain which words or cells support the conclusion and which source location resolves an exception.
Keep a protected portion for final evaluation. If every missed question immediately drives a chunking change, the collection becomes a development set. A fresh set is needed to estimate whether the improvement extends beyond the examples used to tune the system.
Diagnose misses in the right order
First ask whether the source file entered the pipeline. A missing upload, unsupported format, or indexing failure is not a segmentation problem. Next inspect whether extraction retained the relevant material. If the table's footnote vanished during conversion, a different chunk size cannot restore it.
Then inspect whether the necessary evidence exists together in an indexed representation. If it does, check retrieval rank and filtering. An access filter or stale-version rule can affect which passage is returned. Finally examine whether the answer model used the supplied evidence accurately. These stages create a causal troubleshooting sequence instead of a general instruction to make chunks bigger.
Record one primary failure stage and any contributing conditions for each case. This avoids counting one missing table as an extraction, retrieval, and generation defect without explaining that all three symptoms arose from the same initial loss.
The context budget is a constrained resource
Figure data and assumptions
Original hypothetical teaching data; see the caption and article for assumptions.
| Category | Complete evidence found |
|---|---|
| Fixed | 18.0 |
| Headings | 24.0 |
| Combined | 26.0 |
Suppose the answer stage can use 4,000 source tokens after other input requirements are accounted for. Returning eight 500-token chunks fills that allowance. If several chunks repeat the same passage, the useful evidence occupies less than the apparent 4,000 tokens.
One alternative is to retrieve more candidates, remove redundant material, and provide fewer but more complete evidence units. Another is to retrieve a short passage and then expand its surrounding context. These are architectural options to test, not guaranteed improvements. Expansion can introduce irrelevant or obsolete material if the surrounding section is not checked.
Keep the supplied source text distinct from instructions. A retrieved document can contain text that attempts to direct the assistant. Preserving context should not grant that text authority over the application's rules. WorldLensia's document prompt-injection checklist addresses the related trust boundary.
Parent and child relationships can preserve context
A short retrievable unit can retain a reference to a larger parent section. Search may identify the child, while the answer stage receives a bounded part of the parent. That can help when the direct matching text is narrow but an exception sits nearby.
The relationship needs stable identifiers and version management. If a child points to an old parent or an expanded passage comes from a later version, the evidence package can become internally inconsistent. Include document and section versions rather than relying only on a title string.
Also evaluate access at the expanded level. A permitted snippet must not become a route to an unauthorized surrounding document. The same permission discipline applies to any neighboring paragraphs or attached tables returned after the initial search.
Tables need tests with real structure
Create a fictional policy table containing product category, deadline, region, effective date, and an exception note. Ask questions that require all those dimensions. A retrieval result containing the correct number but wrong region should fail the complete-evidence check.
Repeat the test after exporting the table to the formats the pipeline accepts. A visually clear spreadsheet, a text-based PDF, and a scanned image can produce different extracted representations. Document the observed behavior of each format rather than assuming the same chunking policy sees the same information.
If a system generates a description of the table image, compare that description against the original cells. An extraction model's confident wording is another output to validate. Keep the original source available to reviewers so a transcription error can be traced.
Update and deletion behavior belongs in the experiment
A retrieval system is not finished when the first index is built. Replace a document with a revised version and check that new answers use the revision. Then withdraw a document and verify that it no longer supplies evidence where it should not.
This is particularly important when overlapping windows or derived descriptions create several records from one source. The update process should know which records belong to the source and retire the obsolete representations together. Otherwise a correct new policy can coexist with searchable old fragments.
Record the indexing delay and the policy for questions asked during that delay. A source date in a citation is useful only if the system's update behavior makes that date meaningful. Do not describe a knowledge base as current simply because one recent file appears in it.
Compare costs on the same workload
Measure total source volume, number of chunks, preprocessing time, embedding work, storage, retrieval time, and answer-input volume. Report the same source collection and question set for every candidate. Larger chunks can reduce record counts while increasing the text supplied per answer; the cost trade-off is not one-dimensional.
Separate one-time indexing from recurring query costs. A policy that costs more initially but improves many future questions can differ economically from one used on a document queried once. The relevant usage pattern should be explicit rather than assumed.
Do not copy an illustrative cost into a claim about current service pricing. Rates, caching rules, and model choices vary. The framework identifies the quantities to measure; actual billing must come from the applicable service and configuration.
A repeatable acceptance record
For every proposed policy, preserve the parsing settings, chunk boundaries, overlap rule, metadata schema, source-version policy, retrieval configuration, question set, evidence labels, and failure log. Save representative successful and unsuccessful results.
Define acceptance criteria before the final test. These might concern complete evidence coverage, valid source locations, permission correctness, latency, and manageable cost. The exact thresholds should reflect the actual workflow and consequences; the fictional counts in this article are not recommendations.
The final comparison should explain the remaining failures. If a method succeeds on ordinary prose but fails on scanned tables, its deployment scope can be limited accordingly. That is a useful result. Calling it universally superior would erase the very evidence the test produced.
Evaluate unanswerable questions as deliberately as answerable ones
A retrieval system can return something for almost any question, even when the collection contains no sufficient answer. Include questions about a policy that is absent, a future date the documents do not cover, and an exception never addressed in the source. The desired behavior may be to state the gap and request another source.
Score whether the answer correctly distinguishes absent evidence from a negative finding. “The documents do not say whether this is allowed” differs from “this is prohibited.” A chunking policy that makes related passages easy to find still needs an answer policy that respects those limits.
Keep these cases in the final evaluation instead of removing them as inconvenient. They test a central promise of source-grounded assistance: the system should not convert topical similarity into factual support.
Review citations at the passage level
A citation to the right document title can still point to an irrelevant page. Check the location, the relevant text, and the relationship between that text and the answer. For a table claim, preserve units and footnotes with the cited row.
If the user cannot access a private source, the system should follow the applicable sharing rules rather than revealing it through an expanded snippet. Evidence quality and permission correctness are separate checks, and both must pass.
What a good result looks like
Choose the simplest policy that consistently retrieves complete, current, permitted evidence for the intended questions. Keep difficult and unanswerable questions in the test. Save the configuration and failure log so the comparison can be repeated when sources change.
WorldLensia's RAG-versus-fine-tuning guide explains why a retrieval failure and a behavior failure need different remedies. Chunking belongs on the evidence side of that diagnosis. Improve it when the test shows that the evidence is being divided badly, not as a ritual before every prompt change.
Check the boundary cases
Test a very short document as well as a long one. A short policy may already form a complete evidence unit, so splitting it can add complexity without benefit. A very long section may need subdivision even if it has a clear heading. The decision should follow evidence needs and input constraints, not a rule that every file must produce the same number of chunks. Include empty extraction, duplicate files, and revised filenames in the operational test. Each can affect the result without changing the apparent topic of the source collection.
Sources and methodology
Microsoft Azure AI Search's chunking documentation was checked October 18, 2026. All question counts and window-size examples are original teaching scenarios. No live vendor performance test was conducted.
- Microsoft: Chunk documents for RAG — checked October 18, 2026.
Continue with Document prompt-injection checklist and Audit AI citations for related context.
To test whether retrieval supplies complete, supported evidence, see RAG Retrieval Evaluation: Check the Evidence Before Grading the Answer and Valid JSON Is Not a Verified Answer: Check AI Structured Outputs.