URL to Markdown for LlamaIndex: Clean Web Ingestion
LlamaIndex's SimpleWebPageReader is convenient and shallow: it fetches a URL, strips some tags, returns text. The text then flatlines through SentenceSplitter, losing every shred of section structure the page had. Pre-convert to Markdown and you can use MarkdownNodeParser, which builds nodes that mirror the page's real heading hierarchy.
What SimpleWebPageReader misses
SimpleWebPageReader extracts a flat blob of text and hands it to whatever node parser you configure. Default node parsing is sentence-level, which on web articles produces nodes with no hint that "Section 3" is part of "Chapter 2": every retrieved sentence is an orphan. Synthesis then has to re-derive context that was thrown away at ingestion.
Pre-converting to Markdown and using MarkdownNodeParser changes the math. Nodes form a tree: H1 → H2 → H3 → paragraph. Retrieval can climb the tree, your prompts can include parent context, your re-ranker has structural signal to work with.
Code example
# Self-rolled URL-to-Markdown ingestion for LlamaIndex.
# Install: pip install trafilatura llama-index
import trafilatura
from llama_index.core import Document, VectorStoreIndex
from llama_index.core.node_parser import MarkdownNodeParser
# 1. Fetch + extract Markdown (replaces SimpleWebPageReader)
def fetch_as_markdown(url: str) -> Document | None:
downloaded = trafilatura.fetch_url(url)
md = trafilatura.extract(
downloaded,
output_format="markdown",
include_links=True,
include_tables=True,
)
if not md:
return None
return Document(text=md, metadata={"source_url": url})
documents = [d for d in (fetch_as_markdown(u) for u in [
"https://example.com/docs/intro",
"https://example.com/docs/advanced",
]) if d is not None]
# 2. Parse into hierarchical nodes via Markdown structure
parser = MarkdownNodeParser()
nodes = parser.get_nodes_from_documents(documents)
# 3. Index — each node carries its heading path AND source URL
index = VectorStoreIndex(nodes)
query_engine = index.as_query_engine(similarity_top_k=5)
# For one-off URLs you don't want to script, paste into
# mdisbetter.com/convert/url-to-markdown and load the .md with SimpleDirectoryReader.
Frequently asked questions
SimpleWebPageReader vs pre-converted Markdown for LlamaIndex?
SimpleWebPageReader is fine for prototypes and clean static sites. For production, JS-heavy pages, or any corpus with consistent structural needs, pre-conversion gives you deterministic ingestion plus hierarchical MarkdownNodeParser nodes: strictly better for retrieval quality.
How does MarkdownNodeParser handle web content?
Identically to any other Markdown source: it builds a node tree mirroring the heading hierarchy. Web content, once converted, looks no different to MarkdownNodeParser than a hand-written README. The hierarchy enables auto-merging-retriever and parent-context patterns out of the box.
Can I attach source URLs as node metadata?
Yes: pass metadata={"source_url": url} when constructing the Document, and it propagates to every node parsed from that document. Your synthesis prompt can then cite the originating URL for each retrieved chunk.
What about web pages with code blocks (Stack Overflow, docs)?
Code blocks in the source HTML are converted to fenced Markdown code blocks (with language hints when the source uses them). MarkdownNodeParser preserves them as discrete nodes, which means retrieval can surface code snippets as units rather than splitting them mid-function.
How do I re-ingest when web pages change?
Re-convert the URLs, delete the affected nodes from your index by source_url metadata, and re-insert. LlamaIndex's vector stores all support metadata-filtered deletion: Pinecone, Chroma, Weaviate, and Qdrant included.