How to Benchmark PDF to Markdown Converters on Your Own Documents
Most "best PDF to Markdown" lists are vibes-based: a couple of tools, one document, a ranking. Even a careful ranking made on someone else's documents says little about yours. This guide gives you a method instead: how to build a small test set from your own PDFs, a scoring grid to apply to every output, the conditions that keep the comparison fair, and what each kind of converter does by construction.
Step 1: Build a test set from your own documents
Pick documents that look like your real workload, not the cleanest files you have. A useful set covers the categories you actually handle, for example academic papers, financial reports, legal contracts, product manuals and scanned documents. A handful of documents per category is enough to see patterns while still letting you inspect every output by hand.
If your files are confidential, run them only through tools you are allowed to send them to, or build the set from public documents with the same layout traits: the same kind of columns, tables and scan quality.
Step 2: Score every output on the same grid
Build an answer key from the original PDF before reading a converter's output. Record the exact facts and structures you need, then count these witnesses for every result:
| Dimension | Witness | Record |
|---|---|---|
| Content | Selected quotations and numeric values | Missing or altered witnesses, with source locations |
| Headings | Expected section names and hierarchy | Missing, false or incorrectly nested headings |
| Tables | Selected cells, row labels and totals | Wrong values or shifted cell associations |
| Reading order | Known passage sequence across columns | Missing or reordered passages |
| Scans and equations | A transcribed passage or known formula | Character or symbol errors, separately from layout |
Record the number checked as well as the errors. Inspect raw Markdown and rendered output. Keep conversion failures in the ledger instead of omitting them. If you use a 0-10 editorial score, define its anchors first and keep the same reviewer; counts with a source witness are easier to reproduce.
Compare the Markdown side by side with the source PDF rather than skimming the rendered output: a table that renders cleanly can still have shifted cells, and a paragraph that reads fluently can still come from the wrong column.
Step 3: Run every tool under the same conditions
Use default settings for a first pass and note the version of each tool and the date you ran it. Tuning improves most tools on specific document types, so if you tune one, tune them all. If speed matters to you, time the same document on each tool and note the hardware: a GPU library on your own machine and a hosted web service are not timed under the same conditions.
What each kind of tool does by construction
Before you score anything, it helps to know what each family of tools is designed to do. Differences inside a family are what your test set will reveal.
Hosted converters
MDisBetter is a hosted PDF to Markdown tool. Signup and available credits are required. Download the .md result and verify headings, table values, image descriptions and scans against the PDF; extraction is not guaranteed to preserve every feature.
LlamaParse is LlamaIndex's hosted parser, with per-page pricing and tight LlamaIndex integration. Adobe PDF Extract is Adobe's hosted service; it outputs JSON rather than Markdown, so plan for an extra conversion step.
Self-hosted layout-aware libraries
Marker and Docling provide document-conversion pipelines. Their selected backend, models and acceleration change setup, memory and runtime requirements. Marker documents CPU, GPU and MPS device choices; Docling offers multiple pipelines, including optional vision-language configurations. Check the version you install rather than relying on a fixed model-download size or a universal GPU requirement.
General-purpose converters
MarkItDown is Microsoft's general-purpose document converter: DOCX, XLSX, PPTX and PDF in one library. It is worth including if you want unified Office ingestion; since PDF is one input among many, score its tables and equations closely.
Text extraction primitives
Text extractors can provide a useful baseline, but their capabilities differ. PyMuPDF supports OCR through Tesseract when configured; do not label every text library as incapable of OCR. Plain pdftotext extraction and a later formatting step are different from a layout-aware Markdown converter. Pandoc does not directly read PDF. Record the actual pipeline and OCR settings you ran.
What to check by document type
Each category stresses a different dimension, so weight your scores accordingly:
- Academic papers: equations and multi-column reading order
- Financial reports: table preservation
- Legal contracts: clause numbering and structure
- Product manuals: mixed layouts and embedded images
- Scanned documents: OCR quality
If your workload concentrates on one document type, evaluate two or three candidates on that category rather than relying on anyone's overall ranking.
Keep a comparison ledger
Save one row per document and tool: input identifier, version, settings, outcome, witnesses checked, errors, elapsed time, charged usage and cleanup time. Repeat uncertain cases. Do not present an example scoring sheet as a measured result. This is an evaluation worksheet, with empty cells for your own observations.
Compare cost at your own volume
Hosted services charge per page or per credit. Self-hosted libraries are free to use but cost GPU time and the engineering hours to set them up and keep them running. Price both at your real monthly page count, including the maintenance time. Hosted services usually win at low volume and self-hosting can win at very high volume, but where the crossover sits depends on your documents, your hardware and your team.
Pitfalls that skew a benchmark
- Vendor bias: we make MDisBetter, so weigh anything we write about converters with appropriate skepticism, and apply the same caution to any vendor's ranking. Your own documents are the only neutral referee.
- Stale versions: converters ship new versions often, and a comparison from a few months ago may no longer hold. Record the version of every tool you run.
- Default settings: tuning improves every tool on specific document types. Compare defaults with defaults, or tuned with tuned.
- Easy documents: a clean single-column digital PDF converts well almost everywhere. The differences show up on columns, tables, equations and scans, so make sure your set contains them.
For a quicker decision aid focused on the top picks, see the Best PDF to Markdown tools 2026 page. For free options only, see 5 best free PDF to Markdown converters.