Showing 1–6 of 6 results for author: Jonason, N

SYMPLEX: Controllable Symbolic Music Generation using Simplex Diffusion with Vocabulary Priors

Authors: Nicolas Jonason, Luca Casini, Bob L. T. Sturm

Abstract: We present a new approach for fast and controllable generation of symbolic music based on the simplex diffusion, which is essentially a diffusion process operating on probabilities rather than the signal space. This objective has been applied in domains such as natural language processing but here we apply it to generating 4-bar multi-instrument music loops using an orderless representation. We sh… ▽ More We present a new approach for fast and controllable generation of symbolic music based on the simplex diffusion, which is essentially a diffusion process operating on probabilities rather than the signal space. This objective has been applied in domains such as natural language processing but here we apply it to generating 4-bar multi-instrument music loops using an orderless representation. We show that our model can be steered with vocabulary priors, which affords a considerable level control over the music generation process, for instance, infilling in time and pitch and choice of instrumentation -- all without task-specific model adaptation or applying extrinsic control. △ Less

Submitted 21 May, 2024; originally announced May 2024.

Retrieval Augmented Generation of Symbolic Music with LLMs

Authors: Nicolas Jonason, Luca Casini, Carl Thomé, Bob L. T. Sturm

Abstract: We explore the use of large language models (LLMs) for music generation using a retrieval system to select relevant examples. We find promising initial results for music generation in a dialogue with the user, especially considering the ease with which such a system can be implemented. The code is available online. We explore the use of large language models (LLMs) for music generation using a retrieval system to select relevant examples. We find promising initial results for music generation in a dialogue with the user, especially considering the ease with which such a system can be implemented. The code is available online. △ Less

Submitted 28 December, 2023; v1 submitted 17 November, 2023; originally announced November 2023.

Comments: LBD @ ISMIR 2023

DDSP-based Neural Waveform Synthesis of Polyphonic Guitar Performance from String-wise MIDI Input

Authors: Nicolas Jonason, Xin Wang, Erica Cooper, Lauri Juvela, Bob L. T. Sturm, Junichi Yamagishi

Abstract: We explore the use of neural synthesis for acoustic guitar from string-wise MIDI input. We propose four different systems and compare them with both objective metrics and subjective evaluation against natural audio and a sample-based baseline. We iteratively develop these four systems by making various considerations on the architecture and intermediate tasks, such as predicting pitch and loudness… ▽ More We explore the use of neural synthesis for acoustic guitar from string-wise MIDI input. We propose four different systems and compare them with both objective metrics and subjective evaluation against natural audio and a sample-based baseline. We iteratively develop these four systems by making various considerations on the architecture and intermediate tasks, such as predicting pitch and loudness control features. We find that formulating the control feature prediction task as a classification task rather than a regression task yields better results. Furthermore, we find that our simplest proposed system, which directly predicts synthesis parameters from MIDI input performs the best out of the four proposed systems. Audio examples are available at https://erl-j.github.io/neural-guitar-web-supplement. △ Less

Submitted 14 September, 2023; originally announced September 2023.

Exploring Softly Masked Language Modelling for Controllable Symbolic Music Generation

Authors: Nicolas Jonason, Bob L. T. Sturm

Abstract: This document presents some early explorations of applying Softly Masked Language Modelling (SMLM) to symbolic music generation. SMLM can be seen as a generalisation of masked language modelling (MLM), where instead of each element of the input set being either known or unknown, each element can be known, unknown or partly known. We demonstrate some results of applying SMLM to constrained symbolic… ▽ More This document presents some early explorations of applying Softly Masked Language Modelling (SMLM) to symbolic music generation. SMLM can be seen as a generalisation of masked language modelling (MLM), where instead of each element of the input set being either known or unknown, each element can be known, unknown or partly known. We demonstrate some results of applying SMLM to constrained symbolic music generation using a transformer encoder architecture. Several audio examples are available at https://erl-j.github.io/smlm-web-supplement/ △ Less

Submitted 11 May, 2023; v1 submitted 5 May, 2023; originally announced May 2023.

Comments: Version 1.1

Audio Latent Space Cartography

Authors: Nicolas Jonason, Bob L. T. Sturm

Abstract: We explore the generation of visualisations of audio latent spaces using an audio-to-image generation pipeline. We believe this can help with the interpretability of audio latent spaces. We demonstrate a variety of results on the NSynth dataset. A web demo is available. We explore the generation of visualisations of audio latent spaces using an audio-to-image generation pipeline. We believe this can help with the interpretability of audio latent spaces. We demonstrate a variety of results on the NSynth dataset. A web demo is available. △ Less

Submitted 7 December, 2022; v1 submitted 5 December, 2022; originally announced December 2022.

Comments: Late Breaking / Demo, ISMIR 2022 (https://ismir2022program.ismir.net/lbd_413.html)

ACM Class: J.5

TimbreCLIP: Connecting Timbre to Text and Images

Authors: Nicolas Jonason, Bob L. T. Sturm

Abstract: We present work in progress on TimbreCLIP, an audio-text cross modal embedding trained on single instrument notes. We evaluate the models with a cross-modal retrieval task on synth patches. Finally, we demonstrate the application of TimbreCLIP on two tasks: text-driven audio equalization and timbre to image generation. We present work in progress on TimbreCLIP, an audio-text cross modal embedding trained on single instrument notes. We evaluate the models with a cross-modal retrieval task on synth patches. Finally, we demonstrate the application of TimbreCLIP on two tasks: text-driven audio equalization and timbre to image generation. △ Less

Submitted 21 November, 2022; originally announced November 2022.

Comments: Submitted to AAAI workshop on creative AI across modalities

ACM Class: J.5