DataHub v1 is the original DataHub catalog of 11,000+ open datasets, now migrated to PortalJS Arc.
Dataset

The ClueWeb09 Dataset

The ClueWeb09 dataset was created to support research on information retrieval and related human language technologies. It consists of about 1 billion web pages in ten languages that were collected in January and February 2009. The dataset is used by several tracks of the TREC conference.

About

The ClueWeb09 dataset was created to support research on information retrieval and related human language technologies. It consists of about 1 billion web pages in ten languages that were collected in January and February 2009. The dataset is used by several tracks of the TREC conference. 

1,040,809,705 web pages, in 10 languages
5 TB, compressed. (25 TB, uncompressed.)
Download & API
tiki-download_file.phpexample/warc
ClueWeb09 English Sample
Download
tiki-download_file.phpexample/warc
ClueWeb09 Spanish Sample
Download
Sources & license
Sources