URL to Markdown for Vector Databases — Web Content Embeddings
A vector database's retrieval quality is bounded by chunk quality, and chunk quality is bounded by input quality. Embedding raw HTML scraped from the web means embedding navigation menus, cookie banners, and ad slots alongside content — every chunk gets pulled toward boilerplate. Pre-convert URLs to Markdown and embeddings concentrate on what each page is actually about.
The web-content chunk-quality problem
Across Pinecone, Chroma, Weaviate, Qdrant, and pgvector, the public benchmarks all measure retrieval algorithm quality and largely ignore input quality. In practice, input quality is the single largest source of variance in retrieval accuracy. Embed a corpus of web articles as raw HTML text on the same vector DB as the same articles converted to Markdown, and top-K accuracy differs by 15-30 points.
Chunking strategy for web Markdown
Split first by H2 (article sections), then sub-split by H3 if your H2 sections are long. For prose-heavy articles without strong subheadings, fall back to a recursive character splitter at 600-1000 tokens. Always store the source URL and heading path as chunk metadata — it powers source attribution in synthesis and per-source filtering at query time.
Frequently asked questions
Why does chunking by H2/H3 work better than fixed-size chunking?
Because H2/H3 boundaries respect semantic units the author intended — a section is a coherent argument, not an arbitrary 800-token slice. Embeddings cluster around the section's theme rather than averaging across topics, and retrieval surfaces complete passages rather than fragments mid-thought.
How do I store source URLs as metadata in Pinecone?
On upsert, pass metadata={"source_url": url, "heading_path": "Section X > Subsection Y"} alongside each vector. Pinecone's metadata filtering then lets you scope retrieval to specific URLs (filter={"source_url": "https://example.com/article"}) without re-embedding.
Should I deduplicate similar web content before embedding?
For corpora that aggregate from multiple sources (news monitoring, competitive intelligence), yes — near-duplicates pollute top-K with redundant results. SimHash or MinHash on the Markdown content works well; embed first, then drop chunks above a similarity threshold within the same source domain.
How do I handle web pages that update over time?
Track a content hash per URL at ingestion. On re-crawl, if the hash changes, delete by metadata filter ({"source_url": url}) and re-insert. All major vector DBs support metadata-filtered deletion. For high-churn corpora, consider scheduled re-crawls with a freshness window.
Markdown chunks vs raw HTML chunks: how much does it actually matter?
On our internal web-content benchmarks, top-K retrieval accuracy improves 15-30% switching from BeautifulSoup-cleaned HTML to Markdown on the same documents in the same vector DB with the same embedding model. Most of the gain comes from removing boilerplate that polluted embedding clusters.
Your AI doesn't read PDFs directly. It first has to extract the text, decode the layout, ignore the metadata — before it can even start answering. A Markdown file removes all of those steps. Your AI reads it instantly. So you get faster responses, more accurate results, and zero information lost along the way.
Size-wise, it's 100 to 500 times lighter for the same content. A 15 MB PDF becomes a 30 KB .md file. So your AI knowledge base can hold hundreds of documents instead of a handful.
MDisBetter brings 19 free tools together for that — documents, videos, audio, web pages and prompts.
How does it work?
Drop a PDF, a video, an audio file or a URL. MDisBetter extracts the content and gives you a clean Markdown file. So you can send it to your AI, add it to your project files, or store it in your knowledge base — without losing anything.
Frequently Asked Questions
How do I convert a PDF to Markdown for free?
Upload your PDF to MDisBetter, click Convert, and get structured Markdown in seconds. No signup, no installation — it works directly in your browser. The free plan includes 50 credits per month.
Why is Markdown better than PDF for AI?
Markdown reduces token usage by up to 95% compared to PDF. AI models like ChatGPT and Claude process Markdown far more efficiently because it contains only content structure — no fonts, no layout data, no binary overhead.
What file types can MDisBetter convert?
MDisBetter converts PDF, Word (.docx), plain text, YouTube videos (transcript), audio files (MP3, WAV, M4A, OGG, FLAC, WEBM), and any web page URL to clean Markdown.
Is MDisBetter free?
Yes, free to start. The free plan includes 50 credits per month. All processing happens securely — your files are never stored.
Can I extract a YouTube transcript as Markdown?
Yes. Paste the YouTube video URL, click Convert, and get the full transcript structured as Markdown with headings and timestamps. Perfect for feeding video content to AI tools.
PDF → Markdown
Faithful and structured conversion of your documents
Text to MD, EPUB to MD, MD to PDF, MD Cleaner, Merger, Chunker, Token Counter, Context Builder
Free
—
Word to MD
0.5 credit
per page
Excel to MD
0.5 credit
per conversion
Single URL Scrape
0.5 credit
per call
Site Crawl
1 credit
per page
Translate
1 credit
per 10 000 chars (min 1, free re-translation on cache hit)
Prompt Optimizer
1 credit
per call
System Prompt Generator
1 credit
per call
Audio to MD
2 credits
per minute
Video to MD
2 credits
per minute
YouTube to MD
2 credits
per minute
Image OCR
4 credits
per image (0 on cache hit)
PDF to MD
4 credits
per page
PPTX to MD
4 credits
per slide
Questions
Yes! You get 50 credits every month to use any tool. Basic tools like MD Cleaner or Token Counter cost just 0.5 credit per use. When you run out, credits reset the next month or you can upgrade for more.
Wait for your monthly reset or upgrade to a higher plan. Credits renew on your billing date each month.
Yes, cancel anytime with one click. No questions asked. You keep access until the end of your billing period.
Pro gives you 30,000 credits for $29 — that's 30x more credits than Starter for just 3x the price. Every credit costs less, so you get far more value per dollar.