Common Crawl: The 2007 Nonprofit Archive Behind Most AI Models, Now Facing Publisher Revolt Over Copyright

Almost nobody outside of AI research has heard of Common Crawl, yet a huge share of the AI models you've used were trained in part on its archive. This small nonprofit has quietly supplied the raw substrate behind some of the biggest names in AI for nearly two decades.

Since 2007, this small nonprofit has been quietly crawling the public web and giving away the archive for free. That obscurity ended in 2026, when some of the largest news publishers in the world decided Common Crawl wasn't a research project anymore. They called it a copyright problem, and they want it fixed.

What Common Crawl Actually Is

Common Crawl is a 501(c)(3) nonprofit, founded in 2007, that has crawled the public web for close to two decades and gives the resulting archive away for free.

Why This Obscure Archive Actually Matters

Common Crawl isn't itself an AI company, and it doesn't train models, but it supplies the raw material that a huge share of the AI industry builds on. That role makes it one of the most consequential pieces of infrastructure in AI that almost nobody outside the field has heard of:

• OpenAI's own GPT-3 research paper reported that Common Crawl-derived data made up the majority of that model's training tokens

• Widely used cleaned datasets like FineWeb, C4, and OSCAR, staples of open-source model training, are all built directly on top of Common Crawl's raw archive

• Its funders and downstream users include OpenAI, Google, Anthropic, Nvidia, Meta, and Amazon

That's what makes Common Crawl different from any single AI lab's crawler. Blocking GPTBot only affects OpenAI. Blocking, or not blocking, Common Crawl affects a meaningful share of the entire open-model ecosystem at once, since so many training pipelines start from the same archive.

How The Archive Is Actually Built

Common Crawl's crawler, CCBot, works from a frontier of more than 1 trillion known URLs but only samples a small fraction of it in any given cycle. Each monthly release typically captures somewhere between 2 and 5 billion pages, a mix of previously seen pages revisited to track how they've changed and newly discovered ones, rather than any attempt to exhaustively crawl the entire web at once.

• The full historical archive now exceeds 300 billion captured pages, spanning more than a dozen petabytes of raw data collected since crawling began in 2008

• To decide which pages are worth prioritizing out of that trillion-URL frontier, Common Crawl uses Harmonic Centrality, a graph-based measure of how close a page sits to the structural core of the web, alongside PageRank

• The crawler focuses on text-based formats, primarily HTML and PDFs, rather than image or video files, which keeps the dataset's size manageable relative to the web it's sampling

• CCBot publishes the IP ranges it crawls from, identifies itself by user-agent, and says it honors robots.txt directives and rate-limits requests to avoid overloading any single server

That last point is the organization's core pitch for why it should exist at all: a single, disciplined crawler respecting server limits is a lighter load on the web than dozens of AI companies each running their own uncoordinated scrapers against the same sites.

How Common Crawl Handles Sensitive And Illegal Content

Common Crawl says it does not run aggressive automatic filters across its raw archive, in order to preserve the dataset's research value, but it does act on specific reports of illegal or dangerous material. That reactive process, rather than proactive scanning of billions of pages, is also part of why complete removal of any single publisher's flagged content has proven slower than a one-time request might suggest.

• The organization has described working with outside researchers to identify and remove child sexual abuse material and non-consensual intimate imagery when flagged

• It has also said it removes exposed personal data covered by regulations like the GDPR, along with accidentally leaked cryptographic keys and credentials, once such material is brought to its attention

• Publishers who prefer not to be archived at all can add their domains to Common Crawl's public Opt-Out Ledger, formerly called the Opt-Out Registry, so that downstream model developers can identify and exclude those sites from their own training runs

That ledger is real, and it's grown to include entries from the BBC, The Guardian, the Financial Times, The Washington Post, Reuters, and hundreds of other outlets. What it doesn't do, according to reporting cited later in this piece, is come with any enforcement mechanism forcing AI companies who already downloaded a dataset to actually delete a publisher's content from it.

The Publisher Revolt, Step By Step

What had been a slow-burning standards dispute turned into a formal legal confrontation over the course of about seven months in 2025 and 2026.

That last point is the argument that changes the stakes. An opt-out system puts the burden on publishers to notice they've been scraped and ask to be removed. Digital Content Next's position flips that: Common Crawl should have to ask first. If that framing gains legal traction, it doesn't just affect Common Crawl, it questions the default assumption most web archiving and AI training has operated under for two decades.

Common Crawl's Defense

Rich Skrenta has pushed back directly and repeatedly against the accusation that Common Crawl has misled publishers about removing their content. In his November 2025 response to The Atlantic, he said the organization communicates honestly with publishers who contact it and has always operated in good faith and in public view, in line with its mission to serve the public good. He's rejected the idea that Common Crawl has become an extension of AI industry interests, pointing out that no donor, corporate or otherwise, controls what the archive collects, publishes, or removes.

His deeper argument is about the archive itself, not any single publisher's dispute: pulling historical material out of a shared, decades-spanning web archive doesn't just resolve one company's complaint, it starts dismantling one of the only comprehensive public records of what the web actually looked like at a given point in time. Once that precedent is set for one publisher, he's warned, it's set for everyone who wants their history rewritten too.

The Part Neither Side Fully Controls

Even where everyone agrees on what should happen, the technical and structural reality is messier than a letter can fix. The New York Times and Danish publishers represented by the Danish Rights Alliance both secured agreements to have their content removed from Common Crawl's archives, and The Atlantic later reported that content from both was still turning up in the dataset months afterward. That's not necessarily bad faith, decades of crawl data spread across enormous archive files is genuinely difficult to scrub completely, but it means a removal agreement and an actual removal are currently two different things.

Digital Content Next's lawyers have gone further, saying they're reviewing whether Common Crawl's past statements about complying with removal requests, later followed by claims that technical costs and delays prevented full removal, may have been inaccurate or misleading. And the Opt-Out Ledger itself has drawn criticism for being easy to sign and hard to enforce: it's listed as one of dozens of subsections on Common Crawl's site, carries no directive to the developers actually downloading the data, and does nothing to un-scrape content that was captured before a publisher ever opted out.

Blocking future crawling is more straightforward, disallowing CCBot in robots.txt does stop new scraping, but it comes with its own cost. Research published in early 2026 found publishers who blocked AI crawlers saw roughly a 23% monthly traffic decline, without a matching drop in how often they were still cited by AI tools. Blocking protects future content. It doesn't recover past content, and it isn't free.

The Counter-Argument: Why Some Say Brands Should Stay Crawlable

Not everyone in this debate is arguing to be removed. A separate line of argument, mostly from marketers and AI-visibility consultants rather than publishers, says opting out of Common Crawl carries its own cost: invisibility to the conversational AI tools increasingly replacing traditional search.

The logic runs like this: as more people ask AI assistants for recommendations instead of typing queries into a search bar, a brand's visibility depends on whether it exists inside the datasets those assistants were trained on in the first place. A company that blocks CCBot doesn't just skip out on AI training, in this view, it risks having AI systems answer questions about its industry with no knowledge that the company exists at all.

It's worth naming as a real trade-off, and worth being clear about who's making the argument: companies selling "AI visibility" and "AI SEO" services have an obvious commercial interest in persuading brands to stay maximally crawlable. The traffic-decline data earlier in this piece cuts the other way for content creators specifically, so neither position is a free lunch, and the honest answer is that publishers and brands are currently choosing between two different kinds of exposure, not between exposure and safety.

What This Means For The Open Web Going Forward

Common Crawl was built on a premise that made sense in 2007, and that premise is what's now being tested. Crawling the web once and sharing the result saves everyone else from hammering the same servers with redundant scrapers, and keeps web-scale data out of the hands of only the largest search companies. That premise didn't anticipate a world where the same shared archive would become the default training substrate for an entire industry worth trillions of dollars, built on content its original authors never priced for that use.

Whatever gets resolved between Common Crawl and this wave of publisher demands will likely set the template other archives, libraries, and open datasets get judged against next. The open web's future partly depends on whether "open" can still mean "freely available to read" without also meaning "freely available to train billion-dollar AI models on," and right now, nobody involved has fully answered that question.

Common Crawl And The Open Web: FAQ

Jans Bock-Schroeder

Publisher & Founder of AI Angst

Coming from the world of art, photography, and the luxury market, Jans launched AI Angst in 2025 to explore the cultural, ethical, and psychological impacts of artificial intelligence. His work bridges creative vision with critical technology analysis, offering clarity in an era of rapid technological change.