Pricing Dashboard Sign up
Recent
· 8 min read · MDisBetter

Building a PDF to Markdown Converter: Lessons Learned

Building a PDF-to-Markdown converter sounds straightforward, and is, for the simple cases. The interesting engineering is in the long tail: the PDFs where naive approaches fall apart. Here are the lessons that matter if you build one: which problems dominate, which approaches generalize, and where the effort pays off.

Split the work into stages

The naive first version is a single step: call a text-extraction library such as PyMuPDF and apply regex-based heuristics for structure recovery. It works on simple inputs and fails badly on the rest. A sturdier design is a pipeline in stages:

  1. Page analysis: classify each page (digital vs scanned, simple vs multi-column, with vs without tables)
  2. Layout extraction: spatial reading-order analysis, column detection, table-region detection
  3. Content extraction: text extraction with proper Unicode mapping, OCR fallback for scanned regions, equation recognition for math regions
  4. Markdown emission: structure recovery (headings, lists, tables), furniture stripping, output formatting

Each stage is replaceable; failures are localized and debuggable. The architectural cost (more moving parts) is paid back in maintainability: when a conversion goes wrong, you can usually trace it to one stage and fix it without affecting the others.

Lesson 1: Heuristics scale further than you think, then they don't

Rules take you surprisingly far: regex patterns for heading detection, geometric thresholds for column boundaries, hand-tuned weights for furniture stripping. Then the heuristics start fighting each other. Every fix for one document type breaks something else. The font-size threshold for "this is a heading" varies by publisher; a column-detection rule that works on academic two-column layouts breaks on magazine three-column ones.

The way out is to hand the structure-recovery layer to a layout-aware model: not necessarily a giant LLM, a focused classifier trained on labeled documents can do the job. Keep the rules for the simple cases, where they are faster, and let the model handle the long tail. A hybrid tends to beat either pure approach.

Lesson 2: OCR quality matters more than OCR engine choice

It is tempting to obsess over which OCR engine to use: Tesseract, PaddleOCR, Google Document AI, Surya. The engine matters, but what you feed it matters at least as much. Image preprocessing (deskewing, contrast normalization, noise reduction, resolution upscaling) decides how much any engine can recover from a skewed or noisy scan.

Before switching engines to fix bad OCR, fix the images. Different engines do have different strengths by language, so compare them on your own scans once the preprocessing is in place.

Lesson 3: Table extraction is genuinely hard

Tables are the failure mode users notice first and complain about loudest. There are four broad approaches, each covering a blind spot of the one before:

  1. Whitespace-only: detect tables by looking for grids of whitespace.
  2. Line-detection: find horizontal/vertical lines, infer cell structure. Breaks on borderless tables and on PDFs where "lines" are drawn as filled rectangles.
  3. Hybrid (lines + whitespace + clustering): combine all signals.
  4. Layout-aware model: a model trained on labeled table regions.

The model approach tends to generalize best: it doesn't care whether your table has lines, color backgrounds, indentation, or just whitespace. It can still be confused by tables with merged cells across multiple header rows, which remain the hard case.

Lesson 4: Real failures beat synthetic benchmarks

The most useful improvements come from inspecting real failed conversions. The kinds of PDFs that break converters are rarely the ones you anticipate:

None of these is easy to anticipate, and each one matters to someone. The lesson: build observability into your tool from day one, capture failed conversions (with consent), and work through them regularly. Synthetic benchmarks don't substitute for production failures.

Lesson 5: Speed vs quality is a real tradeoff

Users want both. For the hardest pages, the most accurate processing is slow, and running it on every page of a long document would make every conversion slow.

The answer is tiered processing: a quick analysis of every page, and the heavy path only for the pages that need it (scanned, dense layout, equation-heavy). Most pages take the fast path; the slow path only runs where it adds value.

Users judge end-to-end latency, not per-page latency. As long as the overall experience feels fast, the tradeoff holds.

Lesson 6: Markdown is the right output format

Markdown, HTML, JSON or a custom semantic format: each has advocates as the output of a converter.

Markdown wins because:

JSON suits structured data pipelines, but for documents headed to an LLM, a docs site or a knowledge base, Markdown is the better primary output.

Lesson 7: The boring infrastructure matters most

Around the conversion pipeline, infrastructure decides whether people can use the tool at all:

None of this is glamorous. All of it gates whether a tool is usable in production. Conversion accuracy gets noticed only when it's wrong; the infrastructure is felt on every single conversion.

What to do from the start

Three things are cheaper early than late:

  1. Start with observability. It belongs there from day one, not bolted on later.
  2. Layer early. Pure heuristics work for a while, but time spent stretching them is time not spent building the model-based layer.
  3. Treat OCR preprocessing as first-class. The engine choice matters less than what you feed it.

If you're building a similar tool: invest in evaluation infrastructure first, build a hybrid (rules + model) architecture from the start, and treat preprocessing as a first-class concern.

Before you build, compare existing converters on your own documents: see how to benchmark PDF to Markdown converters. For the underlying format problem, see how PDF works internally.

Frequently asked questions

Should I build my own converter or use an existing one?
Test existing ones on your own documents first: open-source libraries such as Marker and Docling, or a hosted web tool such as ours. Build your own only if none of them handles your documents well enough, and then apply the lessons above.
What remains hardest in PDF to Markdown conversion?
Complex tables (merged cells, multi-row headers) and equations in scanned mathematical content. Both are hard for every approach described above, so check them closely in any converter's output.
Which parts of a converter need a GPU?
PDF parsing, rule-based layout analysis and Markdown emission run fine on a CPU. Model-based steps, such as OCR and layout models, are where a GPU buys speed. It is a cost vs latency tradeoff.