Croissant: A Metadata Format for ML-Ready Datasets
Authors:
Mubashara Akhtar,
Omar Benjelloun,
Costanza Conforti,
Pieter Gijsbers,
Joan Giner-Miguelez,
Nitisha Jain,
Michael Kuchnik,
Quentin Lhoest,
Pierre Marcenac,
Manil Maskey,
Peter Mattson,
Luis Oala,
Pierre Ruyssen,
Rajat Shinde,
Elena Simperl,
Goeffry Thomas,
Slava Tykhonov,
Joaquin Vanschoren,
Jos van der Velde,
Steffen Vogler,
Carole-Jean Wu
Abstract:
Data is a critical resource for Machine Learning (ML), yet working with data remains a key friction point. This paper introduces Croissant, a metadata format for datasets that simplifies how data is used by ML tools and frameworks. Croissant makes datasets more discoverable, portable and interoperable, thereby addressing significant challenges in ML data management and responsible AI. Croissant is…
▽ More
Data is a critical resource for Machine Learning (ML), yet working with data remains a key friction point. This paper introduces Croissant, a metadata format for datasets that simplifies how data is used by ML tools and frameworks. Croissant makes datasets more discoverable, portable and interoperable, thereby addressing significant challenges in ML data management and responsible AI. Croissant is already supported by several popular dataset repositories, spanning hundreds of thousands of datasets, ready to be loaded into the most popular ML frameworks.
△ Less
Submitted 30 May, 2024; v1 submitted 28 March, 2024;
originally announced March 2024.
Google Dataset Search by the Numbers
Authors:
Omar Benjelloun,
Shiyu Chen,
Natasha Noy
Abstract:
Scientists, governments, and companies increasingly publish datasets on the Web. Google's Dataset Search extracts dataset metadata -- expressed using schema.org and similar vocabularies -- from Web pages in order to make datasets discoverable. Since we started the work on Dataset Search in 2016, the number of datasets described in schema.org has grown from about 500K to almost 30M. Thus, this corp…
▽ More
Scientists, governments, and companies increasingly publish datasets on the Web. Google's Dataset Search extracts dataset metadata -- expressed using schema.org and similar vocabularies -- from Web pages in order to make datasets discoverable. Since we started the work on Dataset Search in 2016, the number of datasets described in schema.org has grown from about 500K to almost 30M. Thus, this corpus has become a valuable snapshot of data on the Web. To the best of our knowledge, this corpus is the largest and most diverse of its kind. We analyze this corpus and discuss where the datasets originate from, what topics they cover, which form they take, and what people searching for datasets are interested in. Based on this analysis, we identify gaps and possible future work to help make data more discoverable.
△ Less
Submitted 11 June, 2020;
originally announced June 2020.
