PDF to Markdown for LangChain: Ready-to-Chunk Output
LangChain ships PyPDFLoader, UnstructuredPDFLoader, and a half-dozen other PDF loaders. They all wrestle with the same problem: PDF text extraction is unreliable, and you end up writing post-processing every time. Convert to Markdown upstream and you skip the wrestle entirely: feed it to TextLoader and chunk by headers.
The cleaner LangChain pipeline
The standard advice is to use UnstructuredPDFLoader, which is fine if you accept that you'll be cleaning up its output for the rest of your project. The alternative is to convert PDFs to Markdown once (via our converter or any pre-processing step), persist the .md, and use TextLoader from then on. Your loader becomes deterministic, your output is human-inspectable, and your chunker can rely on real headings.
Pair with MarkdownHeaderTextSplitter
The single biggest win is the splitter. MarkdownHeaderTextSplitter chunks on actual headings instead of guessing, so your chunks correspond to document sections, your heading path lives in metadata, and your retrieval-augmented prompts get free structural context.
Code example
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter
# 1. Load Markdown that you converted from PDF (e.g. via mdisbetter.com)
docs = TextLoader("paper.md").load()
md_text = docs[0].page_content
# 2. Split by Markdown headings — keeps section context as metadata
md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[
("#", "title"), ("##", "section"), ("###", "subsection"),
])
sections = md_splitter.split_text(md_text)
# 3. Sub-split long sections to fit your embedding model
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=120)
chunks = splitter.split_documents(sections)
Frequently asked questions
MarkdownHeaderTextSplitter vs RecursiveCharacterTextSplitter?
Use both, in that order. MarkdownHeaderTextSplitter respects document structure and adds heading metadata; RecursiveCharacterTextSplitter then handles any sections that overflow your token budget. Using only the latter loses all structural information.
Which LangChain document loader works best with Markdown?
TextLoader is the simplest and most predictable for pre-converted Markdown. UnstructuredMarkdownLoader gives you per-element documents (heading vs paragraph vs list), useful if you want fine-grained metadata before chunking.
How do I integrate MDisBetter into a LangChain pipeline?
Two patterns: (1) batch-convert PDFs to .md once, then load with TextLoader, best for stable corpora. (2) convert new documents through the web tool as they arrive and store the Markdown alongside the source, best for corpora that grow slowly. Both skip the messy PDF loaders entirely.
Is this better than UnstructuredPDFLoader?
For most cases, yes: the output is more deterministic and the chunks are cleaner. UnstructuredPDFLoader still wins on PDFs with embedded images you need described, since it can route them through a vision model.
What metadata is preserved in the Markdown output?
Headings (with the full path: H1 > H2 > H3), bold and italic emphasis, lists, links, fenced code blocks, and tables. Page numbers and running headers are stripped. You can pass the heading path through MarkdownHeaderTextSplitter as chunk metadata for free.