Clean documents before RAG
Last updated:
Cleaning a document for retrieval-augmented generation (RAG) means removing everything that is not content — the running headers, footers, page numbers and repeated boilerplate that a PDF carries on every page — before the remaining text is split into chunks, embedded and stored in a vector index. It is the least glamorous step in a RAG pipeline and the one that quietly decides whether retrieval works: an index built from uncleaned text stores the same furniture hundreds of times and retrieves it as if it were knowledge. This article explains why furniture poisons embeddings, how chunking actually works, why deduplication belongs before embedding, what to strip and what to keep, and how to evaluate whether cleaning helped. To do the cleaning itself, use the Document Cleaner and read your own results alongside this guide.
Why page furniture poisons embeddings
An embedding model turns a chunk of text into a vector that is supposed to capture what the chunk is about. Feed it a chunk that is half report title, confidentiality footer and page number, and the vector faithfully captures that too — which means every chunk in a long document drifts toward the same furniture, and chunks that should be far apart in vector space end up artificially close because they share a header. Retrieval then does exactly what it was asked to do: asked for the section about refund policy, it returns five chunks whose strongest shared signal is the company name in the running head. Nobody made a modelling error. The index was stocked with noise at full strength, repeated on every page, and noise that repeats is indistinguishable from signal to a system that counts similarity.
The damage is worse than dilution. Furniture chunks can win retrieval outright: a question phrased like the footer — about confidentiality, document control or versioning — matches the footer chunk that was embedded two hundred times better than it matches the one paragraph that actually answers it. Page numbers create a subtler failure, teaching the model that bare numbers are content, so a chunk ending in “42” can be quoted back as if the number meant something. And every furniture chunk occupies a slot in a finite top-k result set, pushing a genuinely relevant chunk out of the context the model is given. Cleaning is therefore not cosmetic tidying; it is removing a systematic competitor to your own content inside your own index.
Chunking basics: size, overlap and boundaries
Chunking is the decision about what unit gets embedded, and it has three knobs. Size trades focus against context: small chunks, a few hundred characters, embed one idea sharply and retrieve precisely, but may arrive without the sentence that gives the idea its conditions; large chunks, several thousand characters, carry their context with them but blend several ideas into one average vector, so none of them retrieves strongly. Most document corpora land somewhere in the middle — roughly 500 to 1,000 characters is a sane starting region — and the right answer is decided by testing your own questions against your own documents, not by copying a number from a framework default. The LangChain text splitter documentation is a good map of the splitter families, from character and recursive splitters to structure-aware ones.
Overlap is insurance against the second knob, boundaries. A splitter that cuts every 800 characters regardless of structure will eventually cut a definition in half, and neither half retrieves or reads well. Splitting on boundaries — paragraphs first, then sentences, then spaces, as recursive splitters do — keeps units whole most of the time, and a modest overlap of ten to twenty percent repeats the edge of one chunk at the start of the next so a cut that still happens does not orphan its context. The hierarchy in the LlamaIndex node parser documentation makes the same point from the loader side: parse into nodes that respect the document, then split. But none of this rescues furniture: overlap faithfully repeats the header into the next chunk too, which is why cleaning comes before chunking, never after.
Dedupe before you embed
Duplicates are the second systematic pollutant, and they come in two kinds. The first is whole-document duplication: the same PDF indexed twice under different file names, the old version left beside the new one, the email attachment saved next to the shared-drive copy. The second is inside the document: the disclaimer on the title page and again in the appendix, the summary repeated at the head of every chapter, the table pasted into both the body and the annex. Both kinds have the same effect on retrieval — duplicated text is embedded repeatedly, so it is retrieved repeatedly, crowding the top-k with copies of one passage while other passages never surface — and the same effect on cost, because you pay to embed and store every copy.
Deduplication is cheapest before embedding, when it is still a string comparison. Exact duplicates fall to normalisation: collapse whitespace, ignore page furniture you have already stripped, hash the result and keep one copy. Near-duplicates — the same paragraph with a changed date or a renumbered heading — need a similarity pass, but even a crude one catches the boilerplate that does the most damage. Two cautions keep dedupe honest. First, keep the first occurrence and record where the copies were; a warning repeated in two chapters may be a deliberate safety device, and removing it from the record entirely is a different decision from removing it from the index. Second, dedupe after cleaning, not before: two pages that differ only by their page number are duplicates of content wrapped in different furniture, and only the cleaned text reveals that.
What to strip, and what to keep
Strip what repeats without informing: running heads and feet, page numbers, watermarks, confidentiality banners, generated “printed on” lines, and the navigation crumbs a web-to-PDF export stamps on every page. Keep what structures: real headings, because they are the strongest chunk-boundary signal you have and often belong in the chunk metadata; captions, because a figure caption is frequently the only text that says what the figure shows; table content, handled carefully, because a stripped table loses the row and column headers that make its numbers mean anything; and lists, whose items are natural units. The borderline case is the table of contents. As chunks, contents lines are furniture — a list of titles and page numbers that matches every query vaguely and answers none — but as metadata the same contents page is a map of the document. Strip it from the embedded text, keep it in your notes about the corpus.
The same judgement applies inside the Document Cleaner itself: furniture and page numbers are ticked by default because they are almost always safe to remove, while duplicate paragraphs are never ticked by default because repetition is sometimes the point. Extraction honesty matters here too. A PDF is a structured format — the ISO PDF standard, ISO 32000, defines pages as content streams of drawing and text operations, not as flowing text — so any extractor approximates reading order, and a scanned PDF has no text layer at all until OCR gives it one. If your source documents matter for other reasons as well, the companion piece on your PDFs as part of your website accessibility covers the tagged structure that helps both screen readers and extractors.
Evaluating retrieval after cleaning
Cleaning is a hypothesis — retrieval will improve — and hypotheses about search quality are testable with embarrassingly little machinery. Assemble twenty to fifty real questions your users actually ask, with the passage that should answer each one noted by hand. Run them against the uncleaned index and the cleaned index with the same retriever, the same chunk settings and the same top-k, and compare two things: how often the right passage appears in the results at all, and how high it ranks when it does. That pair — recall and rank — is the scoreboard. If cleaning is working, furniture chunks stop appearing in results, the right passages move up, and the questions that used to fail because a footer outranked the answer start passing.
Then read the failures, because the numbers alone will flatter you. A cleaned index can gain recall while losing a question whose answer genuinely lived in a repeated summary you deduplicated away, or in a header that carried the product name every chunk needed. Those losses are information: they tell you to keep that element as metadata, or to prepend the document title to each chunk deliberately rather than embedding it accidentally two hundred times. Re-run the evaluation whenever the corpus, the splitter or the embedding model changes, and keep the question set versioned alongside the index. RAG quality is not a launch property; it is a maintenance property, and a small fixed test set is what turns “it feels better” into something you can defend.