URL to Markdown for RAG — Web Scraping to Knowledge Base
Scraping the web for RAG is mostly scraping HTML. The result is a corpus full of nav menus, footer columns, cookie banners, and ad markers — embedded alongside the actual content. Convert URLs to Markdown first and the embeddings concentrate on what the page is about, not how it was decorated.
Convert each URL to Markdown first — through MDisBetter's web tool for one-offs, or through a self-rolled OSS pipeline (Trafilatura, html2text, Readability.py) for automation. Either way, you skip the per-site DOM-wrangling and end up with clean prose plus real headings.
Two paths: web tool or OSS
One-off ingestion (a handful of URLs at a time): paste each URL into /convert/url-to-markdown, click Convert, save the .md file, run it through your chunker. We don't currently expose a programmatic API — for batch automation you'll want to roll your own with the OSS tools below.
Recommended pipeline
URL list → extract main content (Trafilatura is the best-in-class OSS extractor) → convert to Markdown (html2text or markdownify) → chunk on H2/H3 headings (header-aware splitter) → embed → store in vector DB with the source URL and heading path as metadata. For PDF sources, see PDF to Markdown for RAG.
Code example
# Self-rolled URL-to-Markdown pipeline using OSS tools.
# Install: pip install trafilatura langchain-text-splitters
import trafilatura
from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter
# 1. Fetch + extract main content as Markdown (Trafilatura handles boilerplate stripping)
def url_to_markdown(url: str) -> str:
downloaded = trafilatura.fetch_url(url)
md = trafilatura.extract(
downloaded,
output_format="markdown",
include_links=True,
include_tables=True,
)
return md or ""
# 2. Chunk on Markdown headings, preserving heading path metadata
md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[
("#", "h1"), ("##", "h2"), ("###", "h3"),
])
# 3. Sub-split anything still over budget
char_splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
urls = ["https://example.com/article-1", "https://example.com/article-2"]
all_chunks = []
for url in urls:
md = url_to_markdown(url)
if not md:
continue
sections = md_splitter.split_text(md)
for s in sections:
s.metadata["source_url"] = url
all_chunks.extend(char_splitter.split_documents(sections))
# all_chunks is now ready to embed and upsert into your vector DB.
# For one-off URLs you can also paste into mdisbetter.com/convert/url-to-markdown
# and feed the downloaded .md file through the same splitter.
Frequently asked questions
Why is BeautifulSoup-scraped HTML bad for RAG?
Because the cleanup is per-site and brittle. You strip <script>, <style>, and a few common boilerplate selectors — and miss the next site's nav class, the comment thread on a third site, and the cookie banner on a fourth. Every chunk gets a different mix of boilerplate, polluting your embeddings.
How should I chunk web Markdown for retrieval?
Header-aware first, then recursive character splitter as a safety net. Target 600-1000 tokens per chunk, 50-150 overlap. Keep the source URL and heading path as chunk metadata — it makes retrieval debugging tractable and powers source citations in your RAG output.
Should I crawl recursively or convert one URL at a time?
Depends on the site. For documentation sites with clear navigation, crawl recursively from the index page. For news/blog content, convert specific URLs as you bookmark them. Recursive crawling without rate-limiting will get you blocked; respect robots.txt and add delays.
Does the converter handle JavaScript-rendered pages?
Yes — the converter runs a real browser engine before extracting, so React, Vue, Svelte, and other client-rendered pages produce Markdown the same way static pages do. This eliminates a major class of "missing content" failures common with raw requests.get().
How do I attribute sources in my RAG responses?
Store the source URL as chunk metadata at ingestion time, then include it in your synthesis prompt: "When citing, include the source URL from each chunk's metadata." Most RAG frameworks expose chunk metadata to the LLM directly during synthesis.
Your AI doesn't read PDFs directly. It first has to extract the text, decode the layout, ignore the metadata — before it can even start answering. A Markdown file removes all of those steps. Your AI reads it instantly. So you get faster responses, more accurate results, and zero information lost along the way.
Size-wise, it's 100 to 500 times lighter for the same content. A 15 MB PDF becomes a 30 KB .md file. So your AI knowledge base can hold hundreds of documents instead of a handful.
MDisBetter brings 19 free tools together for that — documents, videos, audio, web pages and prompts.
How does it work?
Drop a PDF, a video, an audio file or a URL. MDisBetter extracts the content and gives you a clean Markdown file. So you can send it to your AI, add it to your project files, or store it in your knowledge base — without losing anything.
Frequently Asked Questions
How do I convert a PDF to Markdown for free?
Upload your PDF to MDisBetter, click Convert, and get structured Markdown in seconds. No signup, no installation — it works directly in your browser. The free plan includes 50 credits per month.
Why is Markdown better than PDF for AI?
Markdown reduces token usage by up to 95% compared to PDF. AI models like ChatGPT and Claude process Markdown far more efficiently because it contains only content structure — no fonts, no layout data, no binary overhead.
What file types can MDisBetter convert?
MDisBetter converts PDF, Word (.docx), plain text, YouTube videos (transcript), audio files (MP3, WAV, M4A, OGG, FLAC, WEBM), and any web page URL to clean Markdown.
Is MDisBetter free?
Yes, free to start. The free plan includes 50 credits per month. All processing happens securely — your files are never stored.
Can I extract a YouTube transcript as Markdown?
Yes. Paste the YouTube video URL, click Convert, and get the full transcript structured as Markdown with headings and timestamps. Perfect for feeding video content to AI tools.
PDF → Markdown
Faithful and structured conversion of your documents
Text to MD, EPUB to MD, MD to PDF, MD Cleaner, Merger, Chunker, Token Counter, Context Builder
Free
—
Word to MD
0.5 credit
per page
Excel to MD
0.5 credit
per conversion
Single URL Scrape
0.5 credit
per call
Site Crawl
1 credit
per page
Translate
1 credit
per 10 000 chars (min 1, free re-translation on cache hit)
Prompt Optimizer
1 credit
per call
System Prompt Generator
1 credit
per call
Audio to MD
2 credits
per minute
Video to MD
2 credits
per minute
YouTube to MD
2 credits
per minute
Image OCR
4 credits
per image (0 on cache hit)
PDF to MD
4 credits
per page
PPTX to MD
4 credits
per slide
Questions
Yes! You get 50 credits every month to use any tool. Basic tools like MD Cleaner or Token Counter cost just 0.5 credit per use. When you run out, credits reset the next month or you can upgrade for more.
Wait for your monthly reset or upgrade to a higher plan. Credits renew on your billing date each month.
Yes, cancel anytime with one click. No questions asked. You keep access until the end of your billing period.
Pro gives you 30,000 credits for $29 — that's 30x more credits than Starter for just 3x the price. Every credit costs less, so you get far more value per dollar.