URL to Markdown for Academic Web Research

Roughly one in four web citations in published academic articles dies within a few years of publication — link rot is so well-documented it has its own literature. For a working researcher, this is not an abstract problem. It means the source you cite today might return a 404 the day a reviewer tries to verify it, the blog post that anchored a key argument might quietly change, and the institutional report you depended on might disappear when the institution restructures its website. URL-to-Markdown converts each web source into a portable, archivable, searchable plaintext file the moment you encounter it — making your literature corpus durable, greppable, and ready to feed into the same AI workflows you already use for PDFs.

Why web sources need their own workflow

Most researcher tooling — Zotero, Mendeley, EndNote, Paperpile — was built around the assumption that sources are PDFs. That assumption was reasonable when 90% of cited material was peer-reviewed journal articles. It is increasingly false. Modern literature reviews routinely cite government white papers (HTML), preprints with embedded interactive figures (HTML), institutional blog posts (HTML), congressional testimony (HTML), corporate transparency reports (HTML), policy briefings (HTML), Substack essays from domain experts (HTML), and the entire universe of grey literature that never gets a DOI.

Each of these is a citation hazard for three reasons:

• Link rot: the canonical URL stops resolving. Multiple studies in the information science literature have measured this; the consensus is that web citations decay rapidly enough to be a serious threat to reproducibility.
• Content drift: the URL still resolves, but the content has changed since you cited it. The reader following your reference sees something different from what you read.
• Format hostility: even when the page is still there, navbar/footer/cookie banner clutter makes the source hard to re-read, hard to feed to an AI, hard to quote.

Saving a Markdown copy at the moment of citation solves all three. The cleaned plaintext sits next to your PDF library, gets backed up the same way, and outlives the original URL.

Step 1: Build a web-source corpus alongside your PDF library

If you already use the workflow in PDF to Markdown for researchers, the structure for web sources mirrors it exactly. One folder for converted PDF papers, one folder for converted web sources, one folder for your own permanent notes:

Research Vault/
  Sources/
    PDF/
      Smith2026 - Transformers.md
      Wasserman2024 - All of Statistics.md
    Web/
      EU-AI-Act-2024-impact-assessment.md
      Karpathy2025 - Software is changing again.md
      OECD-2025-AI-policy-observatory.md
  Permanent/
    Generative AI policy convergence in OECD.md

Drop each web URL into our URL-to-Markdown converter. The output strips navigation chrome, preserves quotes and headings, and captures the page's metadata in a YAML frontmatter block — the same shape you already use for PDFs:

---
title: Software is changing again
author: Andrej Karpathy
source_url: https://karpathy.bearblog.dev/software-is-changing-again/
accessed: 2026-05-10
fetched_status: 200 OK
tags: [llms, software-engineering, opinion]
---

The accessed date is the citation timestamp; the file itself is the snapshot. If the URL later 404s or the author edits the post, your archived Markdown is unchanged.

Step 2: Beat link rot at the moment of citation

The discipline that separates a reproducible literature review from a fragile one is converting at read time, not at write-up time. By the time you sit down to draft, the URL you skimmed three months ago may already be dead.

A workflow that scales: every time you open a web source you might cite, paste its URL into the converter and dump the output into Sources/Web/Inbox/. Promote to Sources/Web/Cited/ when you actually quote it. The cost per save is single-digit seconds; the cost of losing a key source mid-review is much higher.

For a belt-and-suspenders approach, also submit the URL to the Internet Archive's Wayback Machine (web.archive.org/save/<url>) — but the local Markdown copy is what you'll actually re-read, quote from, and feed to AI. The Wayback snapshot is the public-facing receipt; the Markdown is the working copy.

Step 3: Feed the corpus to AI for literature synthesis

Modern frontier models comfortably ingest 30-50 medium-length web sources in a single context window. The same prompts that work on a folder of converted papers work on a folder of converted web sources:

• "Across these 20 policy briefings, what are the recurring concerns about model evaluation?"
• "Identify every empirical claim made in these sources and group by whether they cite primary data, secondary data, or no data."
• "Compare how these three institutions frame the same regulatory question."

For mixed corpora — your sources are a combination of PDFs and web pages — the workflow is the same: convert everything to Markdown, drop it all in one folder, point Claude/Gemini/GPT at it. The model treats both equally well because both are now plaintext. See the PDF workflow for researchers for the parallel pipeline.

Step 4: Citation export from web sources

The metadata block at the top of each converted file contains everything you need to generate a citation: title, author (when extractable), publication, URL, access date. To produce a BibTeX entry programmatically:

import yaml, re
from pathlib import Path

def bibtex_from_md(md_path):
    text = Path(md_path).read_text(encoding="utf-8")
    fm_match = re.match(r'---\n(.*?)\n---', text, re.DOTALL)
    fm = yaml.safe_load(fm_match.group(1))
    key = re.sub(r'\W+', '', fm.get('author','anon').split()[0]) + str(fm.get('accessed','')[:4])
    return f"""@misc{{{key},
  title = {{{fm['title']}}},
  author = {{{fm.get('author','Unknown')}}},
  url = {{{fm['source_url']}}},
  urldate = {{{fm['accessed']}}}
}}"""

for f in Path('Sources/Web/Cited').glob('*.md'):
    print(bibtex_from_md(f))

Pipe the output into your references.bib file. APA/MLA/Chicago variants are template substitutions away. The point is that your bibliography becomes a build artifact derived from your sources folder, not a hand-maintained list that drifts out of sync.

Step 5: Build reading lists that survive your career

One underappreciated benefit: a Markdown corpus is the only format that's plausibly readable in 30 years. Mendeley accounts get deactivated. Notion changes its export format. Evernote gets sold. Plain Markdown files in a folder you control will open in any text editor that exists in 2056.

For senior researchers building a personal canon — the 200 papers and 400 web sources that define your subfield — converting everything to Markdown is the only durable answer. The folder gets backed up to two clouds and one external drive; the corpus is yours regardless of what any vendor does next.

Concrete workflow: a literature review on AI governance

You're writing a review article on emerging AI governance frameworks. Your sources will be roughly 40% peer-reviewed papers (PDFs from journals and arXiv), 60% web-based: government white papers, NIST publications, EU Commission reports, think-tank briefings, and a handful of widely-cited blog posts from policy researchers.

1. Week 1-2 (gathering): every time you open a relevant URL, paste it into URL-to-Markdown. Output drops into Sources/Web/Inbox/. PDFs go through the parallel PDF pipeline.
2. Week 3 (triage): skim the Inbox; promote relevant sources to Cited/, archive the rest in Background/.
3. Week 4 (synthesis): drop the entire Cited/ folder into a Claude Project. Generate a thematic map: which sources address which sub-questions, which contradict each other, which cite which.
4. Week 5-6 (drafting): write the review with the corpus open in Obsidian. Cross-link permanent notes to source files. Quote directly from the Markdown — no PDF page numbers to track, just heading anchors.
5. Week 7 (bibliography + verification): auto-generate the .bib file from frontmatter. Check that every cited URL still resolves; if any 404, your archived Markdown is the source of record and you note the access date.

Six weeks for a review that would have taken twelve, with a corpus that's reproducible and a bibliography that's bit-for-bit verifiable.

Tools and integrations specific to academic web research

• Obsidian + Citations plugin: same vault for PDF and web sources, BibTeX integration, autocomplete cite keys
• Zotero + Better BibTeX + Zotfile: Zotero remains source of truth for canonical metadata; sync converted Markdown to a vault folder per collection
• Quarto: write the review in Markdown with embedded R/Python for any quantitative summaries; export to PDF for submission
• archivebox: self-hosted Wayback Machine alternative for full-fidelity HTML/PDF/screenshot snapshots alongside your Markdown copies

The pattern across all of this is the same: Markdown is the substrate that lets your research workflow be programmable, durable, and AI-ready. URL-to-Markdown is what makes the web half of your corpus first-class citizens alongside the PDF half.

Citation styles, page numbers, and the web-source quirk

One question that comes up in peer review for any literature review heavy in web sources: how do you cite a section of a long web page when there are no page numbers? APA, MLA, and Chicago all converge on the same answer — quote the section heading or paragraph number when the URL doesn't anchor directly. Markdown helps here because the converted file preserves the heading hierarchy explicitly. A citation like "(OECD, 2025, sec. 'Compute Governance')" maps cleanly to a specific ## Compute Governance heading in your archived Markdown, which a reader following the URL can locate in the original. The Markdown copy also lets you do the inverse — for any quoted passage, grep across the corpus to confirm the source attribution before submission.

Reproducibility for systematic reviews

For systematic reviews and meta-analyses with PRISMA-style protocols, the search-and-screening trail must be reproducible by independent reviewers. Storing the full text of every screened source as Markdown — alongside your screening decisions — gives a co-reviewer everything they need to repeat the process from your supplementary materials. The fetched_status and accessed_at fields in the frontmatter are exactly the metadata fields PRISMA reporting checklists ask for. A folder of timestamped Markdown sources is the cleanest possible appendix for a systematic review's data availability statement.