GlobalGlobalglobalglobalAPPLICATION/DOWNLOADwebOther (Open)Oct 10, 20132013-10-10
Dataset
A corpus of web crawl data composed of 5 billion web pages.
A corpus of web crawl data composed of 5 billion web pages. This data set is freely available on Amazon S3 at s3://aws-publicdatasets/common-crawl/crawl-002/ and formatted in the ARC (.arc) file format.
About
A corpus of web crawl data composed of 5 billion web pages. This data set is freely available on Amazon S3 at s3://aws-publicdatasets/common-crawl/crawl-002/ and formatted in the ARC (.arc) file format.
Common Crawl is a non-profit organization that builds and maintains an open repository of web crawl data for the purpose of driving innovation in research, education and technology. This data set contains web crawl data from 5 billion web pages and is released under the Common Crawl Terms of Use.
Download & API
About the Common Crawl Corpusapplication/download31 KB
A 1-pager describing the corpus, its format, link to terms of use, what you can do with the data, and resources for learning more.
Sources & license
License
- Other (Open)