Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach
Authors:
Huy V. Vo,
Vasil Khalidov,
Timothée Darcet,
Théo Moutakanni,
Nikita Smetanin,
Marc Szafraniec,
Hugo Touvron,
Camille Couprie,
Maxime Oquab,
Armand Joulin,
Hervé Jégou,
Patrick Labatut,
Piotr Bojanowski
Abstract:
Self-supervised features are the cornerstone of modern machine learning systems. They are typically pre-trained on data collections whose construction and curation typically require extensive human effort. This manual process has some limitations similar to those encountered in supervised learning, e.g., the crowd-sourced selection of data is costly and time-consuming, preventing scaling the datas…
▽ More
Self-supervised features are the cornerstone of modern machine learning systems. They are typically pre-trained on data collections whose construction and curation typically require extensive human effort. This manual process has some limitations similar to those encountered in supervised learning, e.g., the crowd-sourced selection of data is costly and time-consuming, preventing scaling the dataset size. In this work, we consider the problem of automatic curation of high-quality datasets for self-supervised pre-training. We posit that such datasets should be large, diverse and balanced, and propose a clustering-based approach for building ones satisfying all these criteria. Our method involves successive and hierarchical applications of $k$-means on a large and diverse data repository to obtain clusters that distribute uniformly among data concepts, followed by a hierarchical, balanced sampling step from these clusters. Extensive experiments on three different data domains including web-based images, satellite images and text show that features trained on our automatically curated datasets outperform those trained on uncurated data while being on par or better than ones trained on manually curated data. Code is available at https://github.com/facebookresearch/ssl-data-curation.
△ Less
Submitted 28 June, 2024; v1 submitted 24 May, 2024;
originally announced May 2024.
Avoiding Echo-Responses in a Retrieval-Based Conversation System
Authors:
Denis Fedorenko,
Nikita Smetanin,
Artem Rodichev
Abstract:
Retrieval-based conversation systems generally tend to highly rank responses that are semantically similar or even identical to the given conversation context. While the system's goal is to find the most appropriate response, rather than the most semantically similar one, this tendency results in low-quality responses. We refer to this challenge as the echoing problem. To mitigate this problem, we…
▽ More
Retrieval-based conversation systems generally tend to highly rank responses that are semantically similar or even identical to the given conversation context. While the system's goal is to find the most appropriate response, rather than the most semantically similar one, this tendency results in low-quality responses. We refer to this challenge as the echoing problem. To mitigate this problem, we utilize a hard negative mining approach at the training stage. The evaluation shows that the resulting model reduces echoing and achieves better results in terms of Average Precision and Recall@N metrics, compared to the models trained without the proposed approach.
△ Less
Submitted 8 October, 2018; v1 submitted 15 December, 2017;
originally announced December 2017.
