Most enterprise knowledge lives in .docx: policies, specs, SOPs, contracts, training materials, board memos. None of it is semantically searchable until it becomes structured text in a vector database. Convert each Word document to Markdown, chunk by heading hierarchy, embed, and the corpus becomes queryable.
Where Word-to-RAG pipelines fall apart without Markdown
Two failure modes show up immediately. First, the standard Docx2txt loaders flatten heading structure: H1/H2/H3 become indistinguishable runs of text, so chunking by character count slices through topic boundaries. Second, the XML overhead in raw .docx confuses embedding models that expect natural-language input.
Pre-converting to clean Markdown solves both. Heading hierarchy is preserved as #/##/### markers, which every modern Markdown-aware splitter respects as semantic boundaries. The XML envelope is stripped: embeddings encode prose, not metadata.
Honest workflow note
The web tool at Word to Markdown converts one document at a time. For a corpus of 20-200 documents, this is a manageable progressive workflow: convert as you onboard each policy or spec into the knowledge base. For true mass migration of thousands of documents, run Pandoc locally (pandoc input.docx -o output.md in a shell loop) or use python-docx for programmatic conversion. The web tool is the right surface for ad-hoc and progressive enterprise use; local OSS is the right surface for batch automation.
Recommended pipeline
Convert each .docx to .md (web tool for progressive, Pandoc locally for batch). Split first by H1/H2 (top-level document and section), then sub-split anything over 800 tokens with a recursive character splitter. Keep the document title and section path as chunk metadata: your retrieval can filter by document, scope to specific sections, or boost by metadata. Building a multi-source pipeline? Combine with PDFs, web pages, audio, and video the same way.
Code example
# Local pipeline: load the .md you downloaded from mdisbetter.com,
# chunk by section headings, embed, upsert.
# Install: pip install langchain-text-splitters
from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter
# 1. Load the converted Word document
with open("data-retention-policy.md", "r", encoding="utf-8") as f:
md_text = f.read()
# 2. Split by section headings — each chunk is one coherent unit
md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[
("#", "document_title"),
("##", "section"),
("###", "subsection"),
])
section_chunks = md_splitter.split_text(md_text)
# 3. Sub-split any over-budget sections, preserving heading metadata
char_splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=80)
final_chunks = char_splitter.split_documents(section_chunks)
# Each chunk now has document_title + section metadata.
# Upsert to Pinecone, Chroma, Weaviate, or Qdrant.
Frequently asked questions
Is the web tool suitable for converting thousands of Word documents?
No: the web tool converts one file at a time and is the wrong surface for true mass migration. For 1000+ documents, run Pandoc locally (pandoc input.docx -o output.md in a script) or use python-docx programmatically. The web tool is the right surface for progressive enterprise onboarding (10-50 documents at a time), ad-hoc spot conversions, and pipeline prototyping.
Why is Markdown better than Docx2txt for RAG ingestion?
Docx2txt flattens heading hierarchy: H1/H2/H3 become indistinguishable runs of plain text. Markdown preserves headings as #/##/###, which Markdown-aware splitters use as semantic chunk boundaries. The result: chunks that respect document structure rather than slice arbitrarily through it.
How should I chunk Word-derived Markdown for retrieval?
Split first on ## (section boundaries), then sub-split anything over 600-1000 tokens with a recursive character splitter. Keep document title and section path as chunk metadata. Retrieval can then filter by document, scope to specific sections, or boost by metadata fields like document type or owner team.
What about contracts and policies with deeply nested numbered sections?
Word's nested numbering (1.1.2.3) typically becomes either nested H3/H4/H5 headings or numbered lists in the Markdown output, depending on how the source document was styled. For deeply structured legal documents, spend a moment in Word adding heading styles to each numbered section before converting: payoff at retrieval time is significant.
Pinecone, Chroma, Weaviate, Qdrant: which for an enterprise document corpus?
All four work. Pinecone for managed simplicity. Chroma for local development and prototyping. Weaviate when you want hybrid retrieval (policy documents have many exact phrases worth lexical matching). Qdrant when filter-heavy queries dominate (scope to specific document types, owners, or date ranges).