Common Crawl
Non-profit that publishes a free archive of crawled web pages.
Founded in 2007, Common Crawl gives researchers access to an archive of web pages collected since 2008. Its data supplies raw material for language-model training collections.
Last refreshed: 22 October 2026 · Appears in 1 active topic
Timeline for Common Crawl
Mentioned in: USA Today Co. sues OpenAI, not Microsoft
Media's AI PivotBackground
Common Crawl is a non-profit organisation that collects web pages and makes the resulting archive available without an access charge. Researchers can use page content, metadata and extracted text without first operating a large web crawler themselves. Founded in 2007, it has preserved crawl material since 2008. Its collection supports work on language, search and the structure of the web. [Common Crawl](https://www.commoncrawl.org/about).
The archive also supplies material for language-model datasets. A downstream collection can select and filter Common Crawl records, so that collection’s contents depend on further processing decisions. Common Crawl collects and distributes the underlying crawl data. Its CCBot crawler follows Robots.txt instructions, according to the organisation. Those instructions concern automated access to websites; they do not describe every later use of material already collected.
What does Common Crawl provide for AI training?
How does Common Crawl collect web pages?
How much does access to Common Crawl cost?
Written with AI from Lowdown's published reporting. Not individually reviewed by an editor. Updated 22 October 2026. Editorial standards.