URL to Markdown for RAG: Web Scraping to Knowledge Base
Scraping the web for RAG is mostly scraping HTML. The result is a corpus full of nav menus, footer columns, cookie banners, and ad markers, embedded alongside the actual content. Convert URLs to Markdown first and the embeddings concentrate on what the page is about, not how it was decorated.
Convert each URL to Markdown first, through MDisBetter's web tool for one-offs, or through a self-rolled OSS pipeline (Trafilatura, html2text, Readability.py) for automation. Either way, you skip the per-site DOM-wrangling and end up with clean prose plus real headings.
Two paths: web tool or OSS
One-off ingestion (a handful of URLs at a time): paste each URL into /convert/url-to-markdown, click Convert, save the .md file, run it through your chunker. We don't currently expose a programmatic API: for batch automation you'll want to roll your own with the OSS tools below.
Recommended pipeline
URL list → extract main content (Trafilatura is the best-in-class OSS extractor) → convert to Markdown (html2text or markdownify) → chunk on H2/H3 headings (header-aware splitter) → embed → store in vector DB with the source URL and heading path as metadata. For PDF sources, see PDF to Markdown for RAG.
Code example
# Self-rolled URL-to-Markdown pipeline using OSS tools.
# Install: pip install trafilatura langchain-text-splitters
import trafilatura
from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter
# 1. Fetch + extract main content as Markdown (Trafilatura handles boilerplate stripping)
def url_to_markdown(url: str) -> str:
downloaded = trafilatura.fetch_url(url)
md = trafilatura.extract(
downloaded,
output_format="markdown",
include_links=True,
include_tables=True,
)
return md or ""
# 2. Chunk on Markdown headings, preserving heading path metadata
md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[
("#", "h1"), ("##", "h2"), ("###", "h3"),
])
# 3. Sub-split anything still over budget
char_splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
urls = ["https://example.com/article-1", "https://example.com/article-2"]
all_chunks = []
for url in urls:
md = url_to_markdown(url)
if not md:
continue
sections = md_splitter.split_text(md)
for s in sections:
s.metadata["source_url"] = url
all_chunks.extend(char_splitter.split_documents(sections))
# all_chunks is now ready to embed and upsert into your vector DB.
# For one-off URLs you can also paste into mdisbetter.com/convert/url-to-markdown
# and feed the downloaded .md file through the same splitter.
Frequently asked questions
Why is BeautifulSoup-scraped HTML bad for RAG?
Because the cleanup is per-site and brittle. You strip <script>, <style>, and a few common boilerplate selectors, and miss the next site's nav class, the comment thread on a third site, and the cookie banner on a fourth. Every chunk gets a different mix of boilerplate, polluting your embeddings.
How should I chunk web Markdown for retrieval?
Header-aware first, then recursive character splitter as a safety net. Target 600-1000 tokens per chunk, 50-150 overlap. Keep the source URL and heading path as chunk metadata: it makes retrieval debugging tractable and powers source citations in your RAG output.
Should I crawl recursively or convert one URL at a time?
Depends on the site. For documentation sites with clear navigation, crawl recursively from the index page. For news/blog content, convert specific URLs as you bookmark them. Recursive crawling without rate-limiting will get you blocked; respect robots.txt and add delays.
Does the converter handle JavaScript-rendered pages?
Yes: the converter runs a real browser engine before extracting, so React, Vue, Svelte, and other client-rendered pages produce Markdown the same way static pages do. This eliminates a major class of "missing content" failures common with raw requests.get().
How do I attribute sources in my RAG responses?
Store the source URL as chunk metadata at ingestion time, then include it in your synthesis prompt: "When citing, include the source URL from each chunk's metadata." Most RAG frameworks expose chunk metadata to the LLM directly during synthesis.