RDFa, Microdata, and Microformat Data Set

Overview page of the data sets. Currently three different extractions are made from 2010, August 2012 and November 2013. The data is available as download from our Web Server.

This document provides statistics about the Web Data Commons RDFa, Microdata and Microformats data sets which have been extracted from the November 2013 release of the Common Crawl. In summary, we found structured data within 585 million HTML pages out of the 2.24 billion pages contained in the crawl (26%). These pages originate from 1.7 million different pay-level-domains out of the 12.8 million pay-level-domains covered by the crawl (13%). Altogether, the extracted data sets consist of 17.2 billion RDF quads. Instructions on how to download the RDFa, Microdata, and Microformats data sets are given on the page how to get the data.

This document provides statistics about the Web Data Commons data set which has been extracted from the August 2012 version of the Common Crawl. Apart from the analysis of the different formats and the raw statistics presented in this document, we have also created additional statistics and analysis which can be found here. To allow an easy access to the data we created an additiona section, explaining how to get the data in detail.