URL to Markdown for LangChain — Web Loader Alternative
LangChain's WebBaseLoader is BeautifulSoup with extra steps — same brittleness, same per-site cleanup, same pollution of every chunk with boilerplate. Pre-convert URLs to Markdown and you can use the much simpler TextLoader pipeline, with deterministic output and header-aware chunking ready out of the box.
What WebBaseLoader actually does (and why it disappoints)
Under the hood, WebBaseLoader does requests.get(), runs BeautifulSoup, and returns the page text. No JavaScript execution, no readability heuristics, no boilerplate stripping beyond what you configure manually with bs_kwargs. The output is unfiltered DOM text — usable, but you spend the rest of your pipeline cleaning it up.
The alternative is a pre-processing step: extract main content with a real readability library (Trafilatura, Readability.py, jusText), convert to Markdown (html2text, markdownify), persist the .md, and use TextLoader from then on. Your loader becomes deterministic, your output is human-inspectable, and your splitter can be MarkdownHeaderTextSplitter (which respects real document structure). For one-off URLs that don't justify a custom pipeline, paste them into mdisbetter.com/convert/url-to-markdown and feed the downloaded .md to TextLoader.
Pair with MarkdownHeaderTextSplitter
The chunker is where the win compounds. MarkdownHeaderTextSplitter chunks on real headings — your chunks correspond to article sections, the heading path lives in metadata, and your synthesis prompts get free structural context.
Code example
# Self-rolled URL-to-Markdown pre-processor for LangChain.
# Install: pip install trafilatura langchain-community langchain-text-splitters
import trafilatura
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import MarkdownHeaderTextSplitter
# 1. Fetch + extract Markdown (replaces WebBaseLoader + BeautifulSoup tinkering)
url = "https://example.com/article"
downloaded = trafilatura.fetch_url(url)
md_text = trafilatura.extract(
downloaded,
output_format="markdown",
include_links=True,
include_tables=True,
)
# Persist for re-runs
with open("article.md", "w", encoding="utf-8") as f:
f.write(md_text or "")
# 2. Load and split using LangChain's Markdown-native pipeline
docs = TextLoader("article.md", encoding="utf-8").load()
splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[
("#", "title"), ("##", "section"), ("###", "subsection"),
])
chunks = splitter.split_text(docs[0].page_content)
# Each chunk now carries heading-path metadata for retrieval.
# For ad-hoc URLs you don't want to script: convert at
# mdisbetter.com/convert/url-to-markdown, save the .md, point TextLoader at it.
Frequently asked questions
WebBaseLoader vs pre-converted Markdown: which is better?
For one-off ingestion of a clean static site, WebBaseLoader is simpler. For any production pipeline, pre-conversion wins — JS execution, boilerplate stripping, and deterministic output are not optional at scale, and rebuilding them on top of WebBaseLoader is a project of its own.
Does this replace BeautifulSoup in LangChain pipelines?
Effectively yes — for ingestion. The conversion step does the readability extraction, JS execution, and boilerplate stripping that you'd otherwise hand-write with BeautifulSoup selectors. You can keep BS4 for any post-processing on the resulting Markdown if needed.
Can I use this with LangChain's async ingestion?
Yes — wrap the OSS extraction step (Trafilatura, Readability.py + html2text) in an async helper using aiohttp or httpx.AsyncClient for the fetch, gather results, then feed to TextLoader / MarkdownHeaderTextSplitter as usual. The fetch is the only network-bound step; extraction and splitting are local CPU work.
How do I handle pagination across web URLs?
Two patterns: (1) collect all paginated URLs upfront and convert each, then concatenate; (2) for "infinite scroll" pages, the conversion handles the JS-rendered final state — you typically get all loaded content in one Markdown output. For multi-page articles, pattern (1) is cleaner.
What metadata does the Markdown carry through to chunks?
Heading path (H1 > H2 > H3) is added by MarkdownHeaderTextSplitter automatically. You can attach the source URL, fetch timestamp, or any custom metadata at ingestion time — pass them through to chunk.metadata before embedding so they're available at retrieval and synthesis.
Your AI doesn't read PDFs directly. It first has to extract the text, decode the layout, ignore the metadata — before it can even start answering. A Markdown file removes all of those steps. Your AI reads it instantly. So you get faster responses, more accurate results, and zero information lost along the way.
Size-wise, it's 100 to 500 times lighter for the same content. A 15 MB PDF becomes a 30 KB .md file. So your AI knowledge base can hold hundreds of documents instead of a handful.
MDisBetter brings 19 free tools together for that — documents, videos, audio, web pages and prompts.
How does it work?
Drop a PDF, a video, an audio file or a URL. MDisBetter extracts the content and gives you a clean Markdown file. So you can send it to your AI, add it to your project files, or store it in your knowledge base — without losing anything.
Frequently Asked Questions
How do I convert a PDF to Markdown for free?
Upload your PDF to MDisBetter, click Convert, and get structured Markdown in seconds. No signup, no installation — it works directly in your browser. The free plan includes 50 credits per month.
Why is Markdown better than PDF for AI?
Markdown reduces token usage by up to 95% compared to PDF. AI models like ChatGPT and Claude process Markdown far more efficiently because it contains only content structure — no fonts, no layout data, no binary overhead.
What file types can MDisBetter convert?
MDisBetter converts PDF, Word (.docx), plain text, YouTube videos (transcript), audio files (MP3, WAV, M4A, OGG, FLAC, WEBM), and any web page URL to clean Markdown.
Is MDisBetter free?
Yes, free to start. The free plan includes 50 credits per month. All processing happens securely — your files are never stored.
Can I extract a YouTube transcript as Markdown?
Yes. Paste the YouTube video URL, click Convert, and get the full transcript structured as Markdown with headings and timestamps. Perfect for feeding video content to AI tools.
PDF → Markdown
Faithful and structured conversion of your documents
Text to MD, EPUB to MD, MD to PDF, MD Cleaner, Merger, Chunker, Token Counter, Context Builder
Free
—
Word to MD
0.5 credit
per page
Excel to MD
0.5 credit
per conversion
Single URL Scrape
0.5 credit
per call
Site Crawl
1 credit
per page
Translate
1 credit
per 10 000 chars (min 1, free re-translation on cache hit)
Prompt Optimizer
1 credit
per call
System Prompt Generator
1 credit
per call
Audio to MD
2 credits
per minute
Video to MD
2 credits
per minute
YouTube to MD
2 credits
per minute
Image OCR
4 credits
per image (0 on cache hit)
PDF to MD
4 credits
per page
PPTX to MD
4 credits
per slide
Questions
Yes! You get 50 credits every month to use any tool. Basic tools like MD Cleaner or Token Counter cost just 0.5 credit per use. When you run out, credits reset the next month or you can upgrade for more.
Wait for your monthly reset or upgrade to a higher plan. Credits renew on your billing date each month.
Yes, cancel anytime with one click. No questions asked. You keep access until the end of your billing period.
Pro gives you 30,000 credits for $29 — that's 30x more credits than Starter for just 3x the price. Every credit costs less, so you get far more value per dollar.