PDF to Markdown for RAG: Clean Input, Better Retrieval
RAG is only as good as what you index. Raw PDF text (with its broken column boundaries, repeating headers and orphan page numbers) produces chunks that are simultaneously too noisy to embed cleanly and too disconnected to synthesise from. Convert to Markdown first and you get free chunk boundaries: every <code>##</code> is a natural section break.
Where PDF kills your RAG accuracy
The two failure modes are predictable. First, naive fixed-size chunking on PDF text routinely splits sentences mid-clause and joins unrelated columns: embeddings are then averaged over noise, and retrieval surfaces irrelevant chunks. Second, the chunks that are retrieved often contain page numbers and headers that confuse the LLM during synthesis ("the document mentions page 14 in answer 4…").
Markdown solves both. Headings give you semantic chunk boundaries that respect the document's own structure. Cleaner text gives you embeddings that cluster on meaning instead of layout artefacts.
Recommended chunking strategy
Split first by Markdown headings (header-aware splitter), then sub-split anything still over your token budget with a recursive character splitter. Typical settings: target 800 tokens, overlap 100. Keep the heading path as metadata on each chunk so the LLM gets context for free at synthesis time.
Two reasons. First, PDF text extraction produces noisy chunks (column noise, headers, page numbers) that pollute embeddings. Second, fixed-size chunking on that noisy text breaks documents in semantically meaningless places, so retrieval surfaces fragments instead of coherent answers.
How should I chunk Markdown for a RAG pipeline?
Start with header-aware chunking (split on #, ##, ###), then sub-split anything still above your token budget with a recursive character splitter. Keep the heading path as chunk metadata so the LLM has context during synthesis.
What chunk size works best with Markdown headers?
For most documents, target 600–1000 tokens per chunk with 50–150 tokens of overlap. Smaller chunks improve retrieval precision; larger chunks reduce hallucination during synthesis. Let your top-K and re-ranker compensate for the trade-off.
Does Markdown improve embedding quality for retrieval?
Yes: by removing layout noise, embeddings cluster on semantic content instead of formatting artefacts. We routinely see 10–25% improvement in top-K retrieval accuracy switching from raw PDF text to Markdown on the same documents.
Can I use this with LangChain or LlamaIndex directly?