-
Contrastive Learning for Cross-modal Artist Retrieval
Authors:
Andres Ferraro,
Jaehun Kim,
Sergio Oramas,
Andreas Ehmann,
Fabien Gouyon
Abstract:
Music retrieval and recommendation applications often rely on content features encoded as embeddings, which provide vector representations of items in a music dataset. Numerous complementary embeddings can be derived from processing items originally represented in several modalities, e.g., audio signals, user interaction data, or editorial data. However, data of any given modality might not be ava…
▽ More
Music retrieval and recommendation applications often rely on content features encoded as embeddings, which provide vector representations of items in a music dataset. Numerous complementary embeddings can be derived from processing items originally represented in several modalities, e.g., audio signals, user interaction data, or editorial data. However, data of any given modality might not be available for all items in any music dataset. In this work, we propose a method based on contrastive learning to combine embeddings from multiple modalities and explore the impact of the presence or absence of embeddings from diverse modalities in an artist similarity task. Experiments on two datasets suggest that our contrastive method outperforms single-modality embeddings and baseline algorithms for combining modalities, both in terms of artist retrieval accuracy and coverage. Improvements with respect to other methods are particularly significant for less popular query artists. We demonstrate our method successfully combines complementary information from diverse modalities, and is more robust to missing modality data (i.e., it better handles the retrieval of artists with different modality embeddings than the query artist's).
△ Less
Submitted 12 August, 2023;
originally announced August 2023.
-
Supervised and Unsupervised Learning of Audio Representations for Music Understanding
Authors:
Matthew C. McCallum,
Filip Korzeniowski,
Sergio Oramas,
Fabien Gouyon,
Andreas F. Ehmann
Abstract:
In this work, we provide a broad comparative analysis of strategies for pre-training audio understanding models for several tasks in the music domain, including labelling of genre, era, origin, mood, instrumentation, key, pitch, vocal characteristics, tempo and sonority. Specifically, we explore how the domain of pre-training datasets (music or generic audio) and the pre-training methodology (supe…
▽ More
In this work, we provide a broad comparative analysis of strategies for pre-training audio understanding models for several tasks in the music domain, including labelling of genre, era, origin, mood, instrumentation, key, pitch, vocal characteristics, tempo and sonority. Specifically, we explore how the domain of pre-training datasets (music or generic audio) and the pre-training methodology (supervised or unsupervised) affects the adequacy of the resulting audio embeddings for downstream tasks.
We show that models trained via supervised learning on large-scale expert-annotated music datasets achieve state-of-the-art performance in a wide range of music labelling tasks, each with novel content and vocabularies. This can be done in an efficient manner with models containing less than 100 million parameters that require no fine-tuning or reparameterization for downstream tasks, making this approach practical for industry-scale audio catalogs.
Within the class of unsupervised learning strategies, we show that the domain of the training dataset can significantly impact the performance of representations learned by the model. We find that restricting the domain of the pre-training dataset to music allows for training with smaller batch sizes while achieving state-of-the-art in unsupervised learning -- and in some cases, supervised learning -- for music understanding.
We also corroborate that, while achieving state-of-the-art performance on many tasks, supervised learning can cause models to specialize to the supervised information provided, somewhat compromising a model's generality.
△ Less
Submitted 7 October, 2022;
originally announced October 2022.
-
Artist Similarity with Graph Neural Networks
Authors:
Filip Korzeniowski,
Sergio Oramas,
Fabien Gouyon
Abstract:
Artist similarity plays an important role in organizing, understanding, and subsequently, facilitating discovery in large collections of music. In this paper, we present a hybrid approach to computing similarity between artists using graph neural networks trained with triplet loss. The novelty of using a graph neural network architecture is to combine the topology of a graph of artist connections…
▽ More
Artist similarity plays an important role in organizing, understanding, and subsequently, facilitating discovery in large collections of music. In this paper, we present a hybrid approach to computing similarity between artists using graph neural networks trained with triplet loss. The novelty of using a graph neural network architecture is to combine the topology of a graph of artist connections with content features to embed artists into a vector space that encodes similarity. To evaluate the proposed method, we compile the new OLGA dataset, which contains artist similarities from AllMusic, together with content features from AcousticBrainz. With 17,673 artists, this is the largest academic artist similarity dataset that includes content-based features to date. Moreover, we also showcase the scalability of our approach by experimenting with a much larger proprietary dataset. Results show the superiority of the proposed approach over current state-of-the-art methods for music similarity. Finally, we hope that the OLGA dataset will facilitate research on data-driven models for artist similarity.
△ Less
Submitted 30 July, 2021;
originally announced July 2021.
-
Multimodal Metric Learning for Tag-based Music Retrieval
Authors:
Minz Won,
Sergio Oramas,
Oriol Nieto,
Fabien Gouyon,
Xavier Serra
Abstract:
Tag-based music retrieval is crucial to browse large-scale music libraries efficiently. Hence, automatic music tagging has been actively explored, mostly as a classification task, which has an inherent limitation: a fixed vocabulary. On the other hand, metric learning enables flexible vocabularies by using pretrained word embeddings as side information. Also, metric learning has already proven its…
▽ More
Tag-based music retrieval is crucial to browse large-scale music libraries efficiently. Hence, automatic music tagging has been actively explored, mostly as a classification task, which has an inherent limitation: a fixed vocabulary. On the other hand, metric learning enables flexible vocabularies by using pretrained word embeddings as side information. Also, metric learning has already proven its suitability for cross-modal retrieval tasks in other domains (e.g., text-to-image) by jointly learning a multimodal embedding space. In this paper, we investigate three ideas to successfully introduce multimodal metric learning for tag-based music retrieval: elaborate triplet sampling, acoustic and cultural music information, and domain-specific word embeddings. Our experimental results show that the proposed ideas enhance the retrieval system quantitatively, and qualitatively. Furthermore, we release the MSD500, a subset of the Million Song Dataset (MSD) containing 500 cleaned tags, 7 manually annotated tag categories, and user taste profiles.
△ Less
Submitted 29 October, 2020;
originally announced October 2020.
-
Assessing the impact of machine intelligence on human behaviour: an interdisciplinary endeavour
Authors:
Emilia Gómez,
Carlos Castillo,
Vicky Charisi,
Verónica Dahl,
Gustavo Deco,
Blagoj Delipetrev,
Nicole Dewandre,
Miguel Ángel González-Ballester,
Fabien Gouyon,
José Hernández-Orallo,
Perfecto Herrera,
Anders Jonsson,
Ansgar Koene,
Martha Larson,
Ramón López de Mántaras,
Bertin Martens,
Marius Miron,
Rubén Moreno-Bote,
Nuria Oliver,
Antonio Puertas Gallardo,
Heike Schweitzer,
Nuria Sebastian,
Xavier Serra,
Joan Serrà,
Songül Tolan
, et al. (1 additional authors not shown)
Abstract:
This document contains the outcome of the first Human behaviour and machine intelligence (HUMAINT) workshop that took place 5-6 March 2018 in Barcelona, Spain. The workshop was organized in the context of a new research programme at the Centre for Advanced Studies, Joint Research Centre of the European Commission, which focuses on studying the potential impact of artificial intelligence on human b…
▽ More
This document contains the outcome of the first Human behaviour and machine intelligence (HUMAINT) workshop that took place 5-6 March 2018 in Barcelona, Spain. The workshop was organized in the context of a new research programme at the Centre for Advanced Studies, Joint Research Centre of the European Commission, which focuses on studying the potential impact of artificial intelligence on human behaviour. The workshop gathered an interdisciplinary group of experts to establish the state of the art research in the field and a list of future research challenges to be addressed on the topic of human and machine intelligence, algorithm's potential impact on human cognitive capabilities and decision making, and evaluation and regulation needs. The document is made of short position statements and identification of challenges provided by each expert, and incorporates the result of the discussions carried out during the workshop. In the conclusion section, we provide a list of emerging research topics and strategies to be addressed in the near future.
△ Less
Submitted 7 June, 2018;
originally announced June 2018.
-
On Evaluation Validity in Music Autotagging
Authors:
Fabien Gouyon,
Bob L. Sturm,
Joao Lobato Oliveira,
Nuno Hespanhol,
Thibault Langlois
Abstract:
Music autotagging, an established problem in Music Information Retrieval, aims to alleviate the human cost required to manually annotate collections of recorded music with textual labels by automating the process. Many autotagging systems have been proposed and evaluated by procedures and datasets that are now standard (used in MIREX, for instance). Very little work, however, has been dedicated to…
▽ More
Music autotagging, an established problem in Music Information Retrieval, aims to alleviate the human cost required to manually annotate collections of recorded music with textual labels by automating the process. Many autotagging systems have been proposed and evaluated by procedures and datasets that are now standard (used in MIREX, for instance). Very little work, however, has been dedicated to determine what these evaluations really mean about an autotagging system, or the comparison of two systems, for the problem of annotating music in the real world. In this article, we are concerned with explaining the figure of merit of an autotagging system evaluated with a standard approach. Specifically, does the figure of merit, or a comparison of figures of merit, warrant a conclusion about how well autotagging systems have learned to describe music with a specific vocabulary? The main contributions of this paper are a formalization of the notion of validity in autotagging evaluation, and a method to test it in general. We demonstrate the practical use of our method in experiments with three specific state-of-the-art autotagging systems --all of which are reproducible using the linked code and data. Our experiments show for these specific systems in a simple and objective two-class task that the standard evaluation approach does not provide valid indicators of their performance.
△ Less
Submitted 30 September, 2014;
originally announced October 2014.
