Document Cleaner for RAG

What the cleaner removes, and why it matters

Every long PDF carries furniture: the running header with the report title, the footer with a confidentiality notice, the page number ticking up in the corner. For a human reader that furniture is orientation. For an embedding model it is noise repeated at full strength on every single page, so the same header line is chunked, embedded and stored dozens of times, diluting the vectors that should describe the actual content and surfacing in retrieval results as if it were information. This cleaner treats furniture as a detection problem with a simple, inspectable rule: a short line that appears on at least sixty percent of pages (and on at least two) is listed as a removal candidate, and you decide whether it goes. Nothing is removed silently — every candidate is shown with its text, its pages and its count before you tick or untick it.

Page numbers get their own candidate because they never repeat literally: page one says 1, page two says 2, so a repetition rule would never catch them, yet a chunk that ends with a stray number or a “Page 7 of 40” line teaches the embedding nothing and can confuse a model that quotes it back as a fact. The cleaner matches the common patterns — a lone number equal to the page it sits on, and the “N of M” and “N / M” forms — and groups every match into one candidate. Duplicate paragraphs are the third class: the boilerplate disclaimer, the repeated summary or the copy-pasted section that appears twice in one document. Duplicates are never ticked by default, because a repeated paragraph is sometimes deliberate — a warning that must appear in two chapters, for example — so the cleaner flags them and leaves the decision to you, keeping the first occurrence when you do remove them.

How extraction works, and where it is approximate

Extraction is approximate, and this page will not pretend otherwise. The cleaner decodes the PDF content streams locally, reads the text-showing operations and treats each run of shown text as one line. That model is fast and good enough for reports, manuals and papers produced by ordinary word processors and exporters, but it has honest limits: multi-column layouts may come out in storage order rather than reading order, ligatures and custom font encodings can garble individual characters, and a scanned PDF — whose pages are photographs, not text — yields no lines at all. When that happens the tool says so plainly instead of inventing output: a zero-line result means the document needs OCR before any cleaner can help. Furniture detection in this first version is also position-agnostic: a short line is judged by how many pages it repeats on, not by where it sits on the page, so a genuinely repeated short heading inside the body could be listed — which is exactly why every candidate is yours to untick.

The output is built for the next step of a RAG pipeline. The cleaned text download joins the surviving lines page by page with a blank line between pages, so you can read, diff or feed it to your own splitter. The JSONL download goes one step further: it packs the cleaned text into chunks of roughly eight hundred characters, splitting on paragraph boundaries rather than mid-sentence wherever it can, and writes one JSON object per line in the form {"text": "..."} — the shape most embedding scripts and vector-store loaders accept directly. Chunk size is a compromise, not a law: eight hundred characters is small enough that a chunk usually covers one idea, and large enough that the idea still carries its context. The companion article explains size, overlap and boundaries in more depth.

Privacy and scope are deliberately narrow. The file is parsed in this page, on your device; analysing it makes no network request, so internal reports, draft contracts and personal documents stay where they are. Version one accepts PDF only, because PDF is where the furniture problem is worst and where a browser-local parser can still recover the text reliably; word-processor and HTML sources are better cleaned in the tools that produced them, or converted to PDF first. The cleaner also does not judge quality: it will not summarise, rewrite or correct the text it keeps, and it does not decide whether a document is worth embedding at all. It does one mechanical job — strip what repeats, keep what is unique, hand you chunks — and shows its working at every step, so the judgement calls stay with the person who knows the document.

The three kinds of candidate, at a glance