How to Measure PDF vs Markdown Token Counts on Your Own Documents

The claim that converting PDFs to Markdown saves tokens is widely repeated but rarely measured on the documents that actually matter to you. The size of the gap depends on the document: its layout, its columns, its tables, whether it was scanned. This guide gives you a method to measure it yourself, with the tokenizer of the model you use, and to turn the result into a cost estimate.

Step 1: pick a representative sample

Choose a few documents from each type you actually send to a model, for example:

• Academic papers (single + multi-column, with and without equations)
• Financial reports (10-K, earnings, equity research)
• Legal contracts (NDAs, service agreements)
• Product manuals (mixed layout with tables and code)
• Scanned documents (range of OCR quality)

Step 2: count tokens three ways

For each document, count tokens on three versions of the same content:

1. Current input tokens: the text your pipeline sends to the model today. If you upload PDFs to a chat interface, you can't see the text its internal extractor produces, so the plain-text extract below is your closest proxy.
2. Plain text tokens: tokens for naive text extraction (e.g., pdftotext)
3. Markdown tokens: tokens for the Markdown output of the converter you're evaluating

Use the tokenizer that matches your target model: tiktoken with the right encoding for OpenAI models (for example cl100k_base or o200k_base), Anthropic's tokenizer for Claude, Google's tokenizer for Gemini. Count all three versions of a document with the same tokenizer.

Step 3: record the results

Keep one row per document, with its type, its page count, the three token counts, and the reduction, computed as (current tokens - Markdown tokens) / current tokens.

Then summarize per document type rather than with a single average. Scanned files, multi-column papers, table-heavy reports and clean single-column contracts carry different amounts of layout noise, so they won't shrink by the same amount. Note the smallest and largest reductions as well as the typical one: the outliers tell you which documents deserve a closer look at the converted output.

Step 4: turn tokens into cost

Input prices as of mid-2026 (check each provider's current pricing page before relying on them):

• GPT-4o: $2.50 / 1M input tokens
• Claude Sonnet 4.6: $3.00 / 1M input tokens
• Gemini 2.5 Pro: $1.25 / 1M input tokens

Monthly input cost = conversations per month × input tokens per conversation × price per token. For example, 1,000 conversations a month at 60,000 input tokens each is 60 million tokens, or $150 a month at $2.50 per million. Run the formula twice, once with your current token count and once with your Markdown count: the difference is what the conversion saves you, and it scales linearly with volume.

Tokenizer differences across models

The same Markdown text tokenizes differently across models because they use different tokenizers, so absolute counts from one tokenizer don't transfer to another. If you compare several models, repeat the count with each model's tokenizer and check the relative reduction on each. For cost calculations, use the tokenizer matching your target model.

Why the savings appear at all

Three sources of token reduction:

Stripped page furniture

Repeating headers, footers, page numbers, copyright lines. Present on every page in raw PDF extraction; removed by Markdown conversion.

Collapsed whitespace and normalized encoding

Raw extraction emits multiple consecutive spaces, broken paragraphs from line wrapping, encoding markers leaked into text. Markdown is normalized.

Eliminated layout artifacts

Column-break artifacts, justified-text padding, font metadata leakage, hyphenation artifacts. None of which carry meaning; all of which tokenize.

The short version

Short on time? Three steps:

1. Get token counts on your PDFs as they currently feed to your LLM (use the relevant tokenizer)
2. Convert via our web converter for one-offs, or via OSS like Marker/Docling/PyMuPDF for batch (we don't ship a programmatic API today)
3. Get token counts on the resulting Markdown using the same tokenizer

Use our token counter tool for a quick comparison without installing tokenizer libraries. Expect the size of the gap to vary from one document to the next: that variation is exactly why it's worth measuring on your own files rather than relying on a published average.