How to Measure HTML vs Markdown Token Counts for LLMs on Your Own Pages

Everyone says "Markdown uses fewer tokens than HTML" but the actual ratio changes a lot from one page to the next, so a single headline number won't tell you what your own pages cost. This guide shows how to measure it on the URLs you care about with tiktoken (OpenAI's tokenizer), how to read the results by page type, how to turn them into cost math at GPT-4o pricing, and the edge cases where HTML preserves data Markdown can't.

Step 1: pick pages by category

Choose a few pages in each category you actually feed to a model, for example:

• Documentation: API reference pages, framework tutorials
• News: articles from the outlets you follow
• Wiki: encyclopedia entries, project wikis, manual pages
• Forum: discussion threads, Q&A pages, issue threads
• Blog: newsletter posts, long-form articles, personal dev blogs

Step 2: measure each page

For each page:

1. Fetch the raw HTML via curl (no JS rendering; what the server returns)
2. Convert the same URL to Markdown, for example with MDisBetter URL to Markdown
3. Count tokens for both versions with OpenAI's tiktoken, using the o200k_base encoding for the GPT-4o family

Compare like with like: measure the publicly served version of each page for both counts, and note which pages need JavaScript rendering to show their text, since a curl fetch won't see that content.

Step 3: read the results

For each page, record HTML tokens, Markdown tokens, and the ratio (HTML tokens divided by Markdown tokens). Then compute the mean and the median per category, and note the smallest and largest ratios: the outliers tell you which pages need a closer look.

Expect the categories to differ. The more a page is buried under ad scripts, related-story cards, social-share widgets and tracking pixels, the more there is to strip, which is common on modern news sites. Pages whose HTML is already fairly semantic, like many MediaWiki pages, have less chrome to strip, so the ratio is smaller.

Step 4: do the cost math at GPT-4o pricing

GPT-4o input pricing (2026): $2.50 per million tokens. Check the current rate on OpenAI's pricing page, then plug in your own counts:

• Cost per query = page tokens × $2.50 / 1,000,000
• Daily cost of a pipeline = pages per day × average page tokens × $2.50 / 1,000,000
• Monthly cost = daily cost × 30

Run each line twice, once with your HTML average and once with your Markdown average. The difference is what the markup costs you. Because it scales linearly with volume, a gap that's negligible on a handful of pages becomes a real budget line for a summarization pipeline or a RAG endpoint that handles thousands of pages.

Why is the gap so large?

HTML carries everything Markdown discards:

1. Tag overhead: <p class="text-base text-gray-700 leading-relaxed">Hello</p> spends far more tokens on markup than on its one word of content. Markdown is just Hello.
2. Inline styles: Tailwind-class-heavy pages are particularly bad; every element is decorated with several utility classes that mean nothing to an LLM.
3. Script tags: <script> blocks for analytics, ads, A/B tests can make up a large share of total page tokens.
4. JSON-LD blobs: structured data (schema.org markup) adds a block of tokens per page that is useful for SEO crawlers and useless for content understanding.
5. Inline SVG icons: every social-share button, navigation chevron, and decorative icon is dozens of tokens.
6. Hidden DOM: ARIA labels, screen-reader-only content, hidden modal markup; all in the page source, all counted as tokens, all discarded by Markdown extraction.

Markdown keeps the semantic essence: headings, lists, paragraphs, links, code, emphasis. Everything else is chrome.

Edge cases where HTML wins

Markdown is not strictly better for every input. Honest counterexamples:

Interactive widgets

If the page contains an interactive code editor (CodeSandbox embed, JSFiddle), an interactive chart (D3, Plotly), or a configurator (price calculator, form builder), the interactive behavior lives in HTML+JS and disappears in Markdown. The Markdown version captures the static text but not the interaction.

For LLM consumption this usually doesn't matter: the LLM can't interact with widgets anyway. But if your downstream task is "render this page exactly," Markdown is lossy.

Complex multi-column layouts

A magazine-style layout with sidebars, callouts, and floating images loses spatial structure when converted to Markdown. Linearization happens. For most LLM tasks ("summarize this article") this is fine because the LLM cares about content not layout. For visual archival, screenshot the HTML.

Tables with rowspan or colspan

Markdown tables don't support merged cells. Complex HTML tables with rowspan/colspan get flattened or broken. If your data is in a complex table, consider feeding the HTML table directly to the LLM (modern models read HTML tables fine) or extract to CSV/JSON first.

Pages where the data is in attributes

Schema.org product pages encode price, availability, and ratings in <meta> tags and JSON-LD blobs. A Markdown extraction strips those. If you're scraping structured data, parse the HTML directly with a structured-data extractor (e.g., extruct in Python) rather than converting to Markdown first.

The right pattern for LLM workflows

Convert to Markdown unless you have a specific reason not to. The token savings, cleanliness improvements, and downstream chunkability advantages dominate for the vast majority of LLM use cases.

For one-off URL conversion, paste into /convert/url-to-markdown. For batch conversion in a script, use Trafilatura locally: recipe in scrape a website to Markdown for RAG. For RAG pipelines specifically, see URL-to-Markdown for RAG and the knowledge base tutorial.

How does this compare to PDF vs Markdown?

The same dynamic shows up there, for different reasons: PDFs don't carry the script, style and tracking overhead, but raw extraction adds page furniture and layout artifacts that Markdown conversion drops. See how to measure PDF vs Markdown token counts for the equivalent method on documents. The combined story across web and document workflows is consistent: convert to Markdown before LLM ingestion, measure the saving on your own inputs, and get cleaner downstream behavior.

Beyond cost: quality wins

The cost case is the easy story. The quality case is bigger:

• Better attention: LLMs allocate attention across all tokens. Junk tokens (nav, footer, scripts) dilute the model's focus on the actual content. Cleaner input = better answers.
• Cleaner chunks for RAG: HTML chunks include boilerplate that pollutes embeddings. Markdown chunks are signal-only.
• Higher fidelity in long context: At 100K+ context windows, cost matters less but "needle in haystack" recall matters more. Less noise = better recall.

Tooling recommendation

tiktoken is the right token counter for OpenAI models. Anthropic's Claude family has its own tokenizer (the counts differ, but the markup Markdown removes is the same). Google Gemini uses a SentencePiece variant, same story.

Our token counter tool handles single files; to count a whole corpus in a script, pip install tiktoken is the canonical path. Whichever you use, measure with the tokenizer of the model you'll actually call before making cost decisions. Anyone running production LLM workloads against web content should have a Markdown-conversion step in the pipeline before any model call.

Measuring one page in a script

Pick one URL you care about. Run:

import requests, tiktoken

url = 'https://www.bbc.com/news/articles/some-article'
html = requests.get(url).text
enc = tiktoken.get_encoding('o200k_base')
print(f'HTML tokens: {len(enc.encode(html)):,}')

# Convert via the MDisBetter web tool, then:
md = open('downloaded.md').read()
print(f'Markdown tokens: {len(enc.encode(md)):,}')
print(f'Reduction: {len(enc.encode(html))/len(enc.encode(md)):.1f}x')

The exact ratio depends on the page. A low ratio usually means the page already ships fairly clean HTML; a very high ratio usually means the page is heavy on tracking, ad scripts and widgets.