URL to Markdown for Researchers — Archive Web Sources
Half the citations in any given working paper point to URLs that no longer resolve. The other half point to pages that have been silently edited since you cited them. Converting each web source to Markdown at the moment of citation gives you a frozen, plain-text, annotatable record — the digital equivalent of photocopying the journal article.
Why this is hard without the right tool
- Web sources disappear (link rot) — Pew estimates ~25% of cited URLs die within 5 years
- Pages get edited after you cite them; your quote no longer matches the live page
- Need plain-text versions you can quote, annotate, and load into qualitative coding tools
- HTML pages don't paste cleanly into Zotero or NVivo — you get nav menus and footer cruft
- Paywalled content is hard to capture once your library access lapses
Recommended workflow
- Convert each cited URL to Markdown at the moment of citation
- Save with a YAML front matter block: source URL, fetch date, archive.org snapshot link
- Drop into your reference manager (Zotero, Obsidian, Roam) or qualitative coding tool
- Annotate inline with Markdown highlights and footnotes
- Cite the local Markdown in your manuscript, with the source URL preserved in the metadata
Frequently asked questions
Is a Markdown archive citable in academic work?
The cited source is the original URL, not the local copy. The Markdown archive is your evidence that the page said what you say it said on the date you accessed it — analogous to a photocopy of a journal article. Pair with an archive.org snapshot URL in your citation for full reproducibility.
How do I handle paywalled academic content?
For one-off captures, the SingleFile or Save Page WE browser extension respects your existing logged-in session and emits standalone HTML you can then convert in the MDisBetter web tool. For scripted captures, use <code>requests</code> with your library proxy or institutional cookie, then run the returned HTML through Trafilatura or Readability.py. We don't bypass paywalls — we (and the OSS tools) just clean what your authenticated session returns.
Can I batch-convert all URLs from a Zotero library?
Yes — export your Zotero library as CSV, extract the URL column, run a Python loop using Trafilatura (or Readability.py + html2text) to fetch and convert each. A 500-source review typically processes in 15–30 minutes. The output is a folder of Markdown files you can re-import into Zotero as attached notes. We don't expose a programmatic API today, so the loop lives in your script, not in our service.
What about dynamic content (interactive charts, comments)?
Static text and structure come through cleanly. Interactive D3/Plotly charts are captured as the underlying data when exposed in the DOM, otherwise as a placeholder. Comment threads can be included or excluded via a flag — most academic citations want only the article body.
How do I annotate the converted Markdown for qualitative coding?
Use Obsidian with the Highlightr plugin for colour-coded annotation, or load the Markdown into NVivo / Atlas.ti / MaxQDA which all accept plain text. Markdown's simplicity is a feature here — coding works on the words, not on HTML structure.