URL to Markdown for LangChain: Web Loader Alternative
LangChain's WebBaseLoader is BeautifulSoup with extra steps: same brittleness, same per-site cleanup, same pollution of every chunk with boilerplate. Pre-convert URLs to Markdown and you can use the much simpler TextLoader pipeline, with deterministic output and header-aware chunking ready out of the box.
What WebBaseLoader actually does (and why it disappoints)
Under the hood, WebBaseLoader does requests.get(), runs BeautifulSoup, and returns the page text. No JavaScript execution, no readability heuristics, no boilerplate stripping beyond what you configure manually with bs_kwargs. The output is unfiltered DOM text: usable, but you spend the rest of your pipeline cleaning it up.
The alternative is a pre-processing step: extract main content with a real readability library (Trafilatura, Readability.py, jusText), convert to Markdown (html2text, markdownify), persist the .md, and use TextLoader from then on. Your loader becomes deterministic, your output is human-inspectable, and your splitter can be MarkdownHeaderTextSplitter (which respects real document structure). For one-off URLs that don't justify a custom pipeline, paste them into mdisbetter.com/convert/url-to-markdown and feed the downloaded .md to TextLoader.
Pair with MarkdownHeaderTextSplitter
The chunker is where the win compounds. MarkdownHeaderTextSplitter chunks on real headings: your chunks correspond to article sections, the heading path lives in metadata, and your synthesis prompts get free structural context.
Code example
# Self-rolled URL-to-Markdown pre-processor for LangChain.
# Install: pip install trafilatura langchain-community langchain-text-splitters
import trafilatura
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import MarkdownHeaderTextSplitter
# 1. Fetch + extract Markdown (replaces WebBaseLoader + BeautifulSoup tinkering)
url = "https://example.com/article"
downloaded = trafilatura.fetch_url(url)
md_text = trafilatura.extract(
downloaded,
output_format="markdown",
include_links=True,
include_tables=True,
)
# Persist for re-runs
with open("article.md", "w", encoding="utf-8") as f:
f.write(md_text or "")
# 2. Load and split using LangChain's Markdown-native pipeline
docs = TextLoader("article.md", encoding="utf-8").load()
splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[
("#", "title"), ("##", "section"), ("###", "subsection"),
])
chunks = splitter.split_text(docs[0].page_content)
# Each chunk now carries heading-path metadata for retrieval.
# For ad-hoc URLs you don't want to script: convert at
# mdisbetter.com/convert/url-to-markdown, save the .md, point TextLoader at it.
Frequently asked questions
WebBaseLoader vs pre-converted Markdown: which is better?
For one-off ingestion of a clean static site, WebBaseLoader is simpler. For any production pipeline, pre-conversion wins: JS execution, boilerplate stripping, and deterministic output are not optional at scale, and rebuilding them on top of WebBaseLoader is a project of its own.
Does this replace BeautifulSoup in LangChain pipelines?
Effectively yes, for ingestion. The conversion step does the readability extraction, JS execution, and boilerplate stripping that you'd otherwise hand-write with BeautifulSoup selectors. You can keep BS4 for any post-processing on the resulting Markdown if needed.
Can I use this with LangChain's async ingestion?
Yes: wrap the OSS extraction step (Trafilatura, Readability.py + html2text) in an async helper using aiohttp or httpx.AsyncClient for the fetch, gather results, then feed to TextLoader / MarkdownHeaderTextSplitter as usual. The fetch is the only network-bound step; extraction and splitting are local CPU work.
How do I handle pagination across web URLs?
Two patterns: (1) collect all paginated URLs upfront and convert each, then concatenate; (2) for "infinite scroll" pages, the conversion handles the JS-rendered final state: you typically get all loaded content in one Markdown output. For multi-page articles, pattern (1) is cleaner.
What metadata does the Markdown carry through to chunks?
Heading path (H1 > H2 > H3) is added by MarkdownHeaderTextSplitter automatically. You can attach the source URL, fetch timestamp, or any custom metadata at ingestion time: pass them through to chunk.metadata before embedding so they're available at retrieval and synthesis.