How Researchers Use Web Archives for Academic Studies

The web changes and disappears fast. A 2024 Pew Research Center study found that 38% of web pages that existed in 2013 were no longer available a decade later. For researchers, web archives are both a way to cite pages reliably and a source of data in their own right.

1. Citing pages that might disappear

Links in papers break over time, a problem called link rot. Researchers now often cite an archived copy alongside the live link.

• Save the page with Save Page Now on the Wayback Machine, or use Perma.cc, which many universities and law journals use for citations.
• Cite the archived address and the capture date, as well as the original address.
• For pages you rely on heavily, save your own copy too.

2. Studying how the web changes

• How a company's messaging, prices or policies changed over the years.
• How government pages described an issue before and after a policy shift.
• How news stories were edited after publication.
• How web design and technology evolved.

Wayback Quick Access shows whether any page has captures and opens the full list in one click, a quick way to see how far back a source goes.

3. Collecting data at scale

• The Wayback Machine's CDX API lists every capture of an address or website, which researchers use to sample pages across time.
• Common Crawl publishes huge free crawls of the web for analysis.
• Archive-It lets universities build their own collections around a topic, such as an election or a disaster.
• National libraries run web archives of their countries' sites, often with research access.

The limits of archives

• Incomplete: popular pages are captured often, obscure ones rarely or never.
• Interactive content: pages built with JavaScript, logins or search forms often aren't captured properly.
• Uneven timing: captures may be months apart, so a change could have happened any time in between.
• Removals: some captures are taken down at the owner's request.

Report these limits in your methods, and check important findings against other sources. See verifying claims with archives.

Ethics

Archived pages can include personal information that people later removed. Researchers working with archived social media or personal sites should follow their institution's ethics rules, even though the material is public.

Keeping track of sources

Note why each source matters as you go. Sticky Notes Anywhere keeps notes on the pages themselves, stored in your browser. See also cached vs archived pages.