A few months ago, The New York Times and several other major media outlets quietly updated their robots.txt files to block the Wayback Machine from archiving their sites. When questioned, news publishers and media corporations point to a familiar scapegoat: AI training.
They claim that blocking web crawlers protects their copyrighted journalism from being scraped to train Large Language Models (LLMs). But if you look beneath the surface, that explanation falls apart quickly. AI training is a convenient excuse—and the real consequence of this policy is a far bigger threat to digital history and public accountability.
Why the “AI Training” Excuse Fails
The claim that blocking the Wayback Machine stops AI scrapers doesn’t hold up under scrutiny for two key reasons:
- The Internet Archive Is Rate-Limited: AI labs require massive scale and fast access to ingest terabytes of raw text. The Internet Archive aggressively rate-limits automated requests to protect its servers. Any company attempting to scrape the Wayback Machine to train an LLM would hit rate limits almost immediately, making it one of the most inefficient ways to harvest AI data.
- It Doesn’t Bypass Paywalls or Hurt Revenue: The Internet Archive stores snapshot HTML exactly as it appeared at the moment of capture. It does not bypass paywalls to display premium content to non-subscribers. If an article was behind a paywall when archived, the paywall remains intact in the snapshot.
By blocking the Wayback Machine, news outlets aren’t stopping AI companies; they are simply preventing researchers, journalists, and everyday citizens from tracking edits, verifying quotes, and preserving history.
The Danger of a Single Archival Monopoly
With the Wayback Machine blocked by major publishers, researchers are forced to turn to alternative web archiving tools. But a quick audit reveals a troubling reality: almost all of them are blocked or broken, leaving a single, controversial platform standing.
| Archival Service | Status for NYT / Major Outlets | Technical & Operational Notes |
| Wayback Machine | Blocked | Blocked via robots.txt compliance. |
| Megalodon | Blocked | Returns 403 Forbidden / Access Denied errors. |
| Ghost Archive | Blocked | Blocked by publisher CAPTCHAs, rendering automated captures impossible. |
| Arqhivo.qt | Blocked | Blocked by anti-bot measures and CAPTCHA screens. |
| Archive.today | Allowed | Bypasses robots.txt, but operates as a single point of failure with a history of page tampering. |
This dynamic creates an accidental monopoly for Archive.today.
Unlike the Internet Archive—which is a transparent, non-profit academic institution—Archive.today operates in a gray area. Over the years, users and researchers have documented instance-level tampering, modified HTML, and missing header data on Archive.today captures.
When a single, closed-source service becomes the only place left to archive public record news, the integrity of digital history is compromised. If an archived article can be silently altered or modified after capture, it loses its power as a reliable historical record.
Why This Matters
When news organizations scrub their history from open archives, they erode public trust. News articles are frequently edited, silent corrections are made, and controversial headlines are swapped overnight. Without transparent, non-profit institutions like the Wayback Machine capturing these changes in real time, the public loses the ability to hold major media institutions accountable.
Blocking the Wayback Machine doesn’t stop AI. It only stops the public from remembering.
Leave a Reply