Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasets
Authors:
Dominique Beaini,
Shenyang Huang,
Joao Alex Cunha,
Zhiyi Li,
Gabriela Moisescu-Pareja,
Oleksandr Dymov,
Samuel Maddrell-Mander,
Callum McLean,
Frederik Wenkel,
Luis Müller,
Jama Hussein Mohamud,
Ali Parviz,
Michael Craig,
Michał Koziarski,
Jiarui Lu,
Zhaocheng Zhu,
Cristian Gabellini,
Kerstin Klaser,
Josef Dean,
Cas Wognum,
Maciej Sypetkowski,
Guillaume Rabusseau,
Reihaneh Rabbany,
Jian Tang,
Christopher Morris
, et al. (10 additional authors not shown)
Abstract:
Recently, pre-trained foundation models have enabled significant advancements in multiple fields. In molecular machine learning, however, where datasets are often hand-curated, and hence typically small, the lack of datasets with labeled features, and codebases to manage those datasets, has hindered the development of foundation models. In this work, we present seven novel datasets categorized by…
▽ More
Recently, pre-trained foundation models have enabled significant advancements in multiple fields. In molecular machine learning, however, where datasets are often hand-curated, and hence typically small, the lack of datasets with labeled features, and codebases to manage those datasets, has hindered the development of foundation models. In this work, we present seven novel datasets categorized by size into three distinct categories: ToyMix, LargeMix and UltraLarge. These datasets push the boundaries in both the scale and the diversity of supervised labels for molecular learning. They cover nearly 100 million molecules and over 3000 sparsely defined tasks, totaling more than 13 billion individual labels of both quantum and biological nature. In comparison, our datasets contain 300 times more data points than the widely used OGB-LSC PCQM4Mv2 dataset, and 13 times more than the quantum-only QM1B dataset. In addition, to support the development of foundational models based on our proposed datasets, we present the Graphium graph machine learning library which simplifies the process of building and training molecular machine learning models for multi-task and multi-level molecular datasets. Finally, we present a range of baseline results as a starting point of multi-task and multi-level training on these datasets. Empirically, we observe that performance on low-resource biological datasets show improvement by also training on large amounts of quantum data. This indicates that there may be potential in multi-task and multi-level training of a foundation model and fine-tuning it to resource-constrained downstream tasks.
△ Less
Submitted 18 October, 2023; v1 submitted 6 October, 2023;
originally announced October 2023.
Fast Development of ASR in African Languages using Self Supervised Speech Representation Learning
Authors:
Jama Hussein Mohamud,
Lloyd Acquaye Thompson,
Aissatou Ndoye,
Laurent Besacier
Abstract:
This paper describes the results of an informal collaboration launched during the African Master of Machine Intelligence (AMMI) in June 2020. After a series of lectures and labs on speech data collection using mobile applications and on self-supervised representation learning from speech, a small group of students and the lecturer continued working on automatic speech recognition (ASR) project for…
▽ More
This paper describes the results of an informal collaboration launched during the African Master of Machine Intelligence (AMMI) in June 2020. After a series of lectures and labs on speech data collection using mobile applications and on self-supervised representation learning from speech, a small group of students and the lecturer continued working on automatic speech recognition (ASR) project for three languages: Wolof, Ga, and Somali. This paper describes how data was collected and ASR systems developed with a small amount (1h) of transcribed speech as training data. In these low resource conditions, pre-training a model on large amounts of raw speech was fundamental for the efficiency of ASR systems developed.
△ Less
Submitted 16 March, 2021;
originally announced March 2021.
