Building a PDF to Markdown Converter: Lessons Learned
Building a PDF-to-Markdown converter sounds straightforward, and is, for the simple cases. The interesting engineering is in the long tail: the PDFs where naive approaches fall apart. Here are the lessons that matter if you build one: which problems dominate, which approaches generalize, and where the effort pays off.
Split the work into stages
The naive first version is a single step: call a text-extraction library such as PyMuPDF and apply regex-based heuristics for structure recovery. It works on simple inputs and fails badly on the rest. A sturdier design is a pipeline in stages:
- Page analysis: classify each page (digital vs scanned, simple vs multi-column, with vs without tables)
- Layout extraction: spatial reading-order analysis, column detection, table-region detection
- Content extraction: text extraction with proper Unicode mapping, OCR fallback for scanned regions, equation recognition for math regions
- Markdown emission: structure recovery (headings, lists, tables), furniture stripping, output formatting
Each stage is replaceable; failures are localized and debuggable. The architectural cost (more moving parts) is paid back in maintainability: when a conversion goes wrong, you can usually trace it to one stage and fix it without affecting the others.
Lesson 1: Heuristics scale further than you think, then they don't
Rules take you surprisingly far: regex patterns for heading detection, geometric thresholds for column boundaries, hand-tuned weights for furniture stripping. Then the heuristics start fighting each other. Every fix for one document type breaks something else. The font-size threshold for "this is a heading" varies by publisher; a column-detection rule that works on academic two-column layouts breaks on magazine three-column ones.
The way out is to hand the structure-recovery layer to a layout-aware model: not necessarily a giant LLM, a focused classifier trained on labeled documents can do the job. Keep the rules for the simple cases, where they are faster, and let the model handle the long tail. A hybrid tends to beat either pure approach.
Lesson 2: OCR quality matters more than OCR engine choice
It is tempting to obsess over which OCR engine to use: Tesseract, PaddleOCR, Google Document AI, Surya. The engine matters, but what you feed it matters at least as much. Image preprocessing (deskewing, contrast normalization, noise reduction, resolution upscaling) decides how much any engine can recover from a skewed or noisy scan.
Before switching engines to fix bad OCR, fix the images. Different engines do have different strengths by language, so compare them on your own scans once the preprocessing is in place.
Lesson 3: Table extraction is genuinely hard
Tables are the failure mode users notice first and complain about loudest. There are four broad approaches, each covering a blind spot of the one before:
- Whitespace-only: detect tables by looking for grids of whitespace.
- Line-detection: find horizontal/vertical lines, infer cell structure. Breaks on borderless tables and on PDFs where "lines" are drawn as filled rectangles.
- Hybrid (lines + whitespace + clustering): combine all signals.
- Layout-aware model: a model trained on labeled table regions.
The model approach tends to generalize best: it doesn't care whether your table has lines, color backgrounds, indentation, or just whitespace. It can still be confused by tables with merged cells across multiple header rows, which remain the hard case.
Lesson 4: Real failures beat synthetic benchmarks
The most useful improvements come from inspecting real failed conversions. The kinds of PDFs that break converters are rarely the ones you anticipate:
- PDFs from old versions of LaTeX that emit pre-Unicode font encoding
- PDFs generated by 3D CAD software with non-standard text rendering
- PDFs with rotated or vertical text in headers
- Multi-script PDFs (English text with Arabic citations)
- Forms with overlapping fields
- Heavily-stamped or watermarked documents
None of these is easy to anticipate, and each one matters to someone. The lesson: build observability into your tool from day one, capture failed conversions (with consent), and work through them regularly. Synthetic benchmarks don't substitute for production failures.
Lesson 5: Speed vs quality is a real tradeoff
Users want both. For the hardest pages, the most accurate processing is slow, and running it on every page of a long document would make every conversion slow.
The answer is tiered processing: a quick analysis of every page, and the heavy path only for the pages that need it (scanned, dense layout, equation-heavy). Most pages take the fast path; the slow path only runs where it adds value.
Users judge end-to-end latency, not per-page latency. As long as the overall experience feels fast, the tradeoff holds.
Lesson 6: Markdown is the right output format
Markdown, HTML, JSON or a custom semantic format: each has advocates as the output of a converter.
Markdown wins because:
- Every downstream consumer (LLMs, docs sites, knowledge bases) accepts it
- It's human-readable for spot-checking
- It diffs cleanly in Git
- The format itself is small (no schema overhead)
- Conversion to HTML downstream is a single Pandoc command
JSON suits structured data pipelines, but for documents headed to an LLM, a docs site or a knowledge base, Markdown is the better primary output.
Lesson 7: The boring infrastructure matters most
Around the conversion pipeline, infrastructure decides whether people can use the tool at all:
- Clear error reporting, so users know exactly what failed and how to retry
- Predictable throughput under load
- A clear, documented policy on how long uploaded files are kept
- Long conversions that finish instead of hitting timeouts on big documents
None of this is glamorous. All of it gates whether a tool is usable in production. Conversion accuracy gets noticed only when it's wrong; the infrastructure is felt on every single conversion.
What to do from the start
Three things are cheaper early than late:
- Start with observability. It belongs there from day one, not bolted on later.
- Layer early. Pure heuristics work for a while, but time spent stretching them is time not spent building the model-based layer.
- Treat OCR preprocessing as first-class. The engine choice matters less than what you feed it.
If you're building a similar tool: invest in evaluation infrastructure first, build a hybrid (rules + model) architecture from the start, and treat preprocessing as a first-class concern.
Before you build, compare existing converters on your own documents: see how to benchmark PDF to Markdown converters. For the underlying format problem, see how PDF works internally.