Public datasets · Large-scale public web crawl corpus
Common Crawl provides raw public web crawl data for discovery research, web measurement and content analysis.
Put it to work
Common Crawl provides raw public web crawl data for discovery research, web measurement and content analysis.
Choose one question the available fields and coverage can answer. Check a small relevant slice before committing to a larger analysis or publishing a comparison.
What it covers
Geography: Coverage depends on the source's documented geographic and time scope
Granularity: Provider-described dataset or API records available at the indicated geography and time scope.
Source type: Large-scale public web crawl corpus
Access and cost
Review the source page for registration, API-key, quota and usage terms before accessing or redistributing data.
Cost: Public access is described, but any registration, account and downstream hosting costs should be checked.
Update cadence: Release cadence varies by provider and dataset; confirm the specific collection or reference period.
Before you use it
Crawl files are very large; coverage and freshness depend on each collection and do not guarantee a complete or current web index.