Abstract

In the pursuit of develo** expressive music performance models using artificial intelligence, this paper introduces DExter, a new approach leveraging diffusion probabilistic models to render Western classical piano performances. In this approach, performance parameters are represented in a continuous expression space and a diffusion model is trained to predict these continuous parameters while being conditioned on the musical score. Furthermore, DExter also enables the generation of interpretations (expressive variations of a performance) guided by perceptually meaningful features by conditioning jointly on score and perceptual feature representations. Consequently, we find that our model is useful for learning expressive performance, generating perceptually steered performances, and transferring performance styles. We assess the model through quantitative and qualitative analyses, focusing on specific performance metrics regarding dimensions like asynchrony and articulation, as well as through listening tests comparing generated performances with different human interpretations. Results show that DExter is able to capture the time-varying correlation of the expressive parameters, and compares well to existing rendering models in subjectively evaluated ratings. The perceptual-feature-conditioned generation and transferring capabilities of DExter are verified by a proxy model predicting perceptual characteristics of differently steered performances.

1 1 0 https://doi.org/ DExter: Learning and Controlling Performance Expression with Diffusion Models Title Huan Zhang ¹, Shreyan Chowdhury ², Carlos Eduardo Cancino-Chacón ², **hua Liang¹, Simon Dixon ¹, Gerhard Widmer² Huan Zhang, Shreyan Chowdhury, Carlos Eduardo Cancino-Chacón, **hua Liang, Simon Dixon and Gerhard Widmer Zhang, H.; Chowdhury, S.; Cancino-Chacón, C. E.; Liang, J.; Dixon, S; Widmer, G Correspondence: [email protected]

1 Introduction

A trained musician can take a piece of music and interpret it in their own way, moulding and varying the emotional expression of the piece by subtly changing performance parameters. Parametric dimensions include timing, dynamics, articulation, and use of devices like sustain pedals in piano. Studying such expression patterns has long been of keen interest to musicians, educators and researchers, and it presents a compelling inquiry into exploring whether such intricate expressions can be accurately encapsulated and replicated by computational systems Widmer et al. (2003). The accurate replication of human musical expression by machines not only bridges the gap between traditional and technological approaches to music but also opens new avenues for interactive performances Cancino-Chacón et al. (2023) and music education systems Morsi et al. (2024). Leveraging such technology can enhance musical training, allowing students and professionals alike to experiment with and learn from dynamically generated expression, thus broadening both creative perspectives and educational methods. In this paper, we aim at rendering such an expressive performance of a piece of music from its score using a machine learning model. We propose DExter, a Diffusion-based Expressive performance generat(o)r, which predicts expression parameters conditioned on the score. In addition, we investigate whether the rendering process can also be conditioned on desired high-level performance characteristics given in the form of mid-level perceptual features Aljanaki (2018); Chowdhury et al. (2019), in this way permitting us to control general performance character, as well as to explore the potential of style transfer within the varied space of human expressive performances. In this context, we leverage the conditional design of diffusion models as well as the diffusion chain to serve as a mechanism to regulate the extent of transferred information from a source performance to a target performance.

This paper offers three contributions: 1) we propose a diffusion-based method for learning and conditioning the expression parameters in Western classical solo piano performance; 2) we conduct a comprehensive quantitative evaluation on the rendered outputs along with other renderers in the literature (re-trained to make for a fair comparison), taking into account multiple expressive dimensions such as asynchrony and articulation; and 3) we explore mid-level conditioned generation and style transfer with our model and conduct an experimental study on the conditioning effects.

2 Related Work

2.1 Expressive Performance Rendering

Expressive performance rendering has long been a challenge for Music Information Retrieval (MIR) research. While the role of machine learning in such a task was recognised early on Widmer et al. (2003), several rule-based methods have been proposed and investigated over the years Widmer and Goebl (2004); Cancino-Chacón et al. (2018); Kirke and Miranda (2013). Early experiments in deep-learning based performance rendering Cancino-Chacón (2018); Maezawa et al. (2019); Jeong et al. (2019) use traditional sequence modeling architectures like RNNs and LSTMs with modifications focusing on the music hierarchy and score features being applied as inputs. Recently, transformer-based systems Rhyu et al. (2022); Borovik and Viro (2023) have been proposed for controllable rendering, predicting different aspects of performance such as the shape of expressive attributes Rhyu et al. (2022) and performance direction markings in the score Borovik and Viro (2023). All of the above systems predict descriptors designed to capture expressive aspects of musical performance, typically tokens representing local tempo and timing deviations. However, such tokenized and quantized encodings of performance parameters are not lossless and can result in a large vocabulary to train Zhang and Dixon (2023).

Regarding the evaluation of performance rendering systems, there has been a growing criticism of the practice of evaluating against a single ground truth and ignoring the variations in interpretations Peter et al. (2023), as reconstruction-based error analysis has inherent limitations on fidelity and diversity Plasser et al. (2023); Peter et al. (2023). To mitigate this problem, we will evaluate the rendered performances with respect to a multitude of performance parameter dimensions, and against multiple different human interpretations.

2.2 Diffusion Models in Music

Diffusion Probabilistic Models (DPMs) generate data by inverting a Markovian data corruption process. DPMs have demonstrated impressive results first in the vision domain by generating text-controlled images Ramesh et al. (2021), and then also in the audio domain, with the most promising applications involving generation of high-fidelity audio samples Kong et al. (2021); Chen et al. (2021) and synthesis of speech Kim et al. (2022) and music Hawthorne et al. (2022).

Symbolic music, however, seems to be a more challenging target for DPMs – the challenge is to fit their probabilistic formulation into discrete data distributions. Mittal et al. (2021) train continuous DDPMs (Denoising Diffusion Probabilistic Models) on sequences of latent MusicVAE Roberts et al. (2018) embeddings, in order to achieve generation of monophonic melodies. Plasser et al. (2023) build upon the MusicVAE-like token representation and directly apply discrete denoising diffusion probabilistic models (D3PM). Another representation suitable for learning symbolic music Zhang et al. (2023) under the DPM framework is the piano roll: Cheuk et al. (2023) managed to transcribe music by generating a piano roll using an audio spectrogram as condition. Min et al. (2023) also achieved piano roll generation with more diverse control such as infilling music context and high-level guidance of chords and texture.

Our work places music DPMs into a niche spot: while the rendering is applied on symbolic data (discrete notes), DExter predicts continuously varying expressive parameters which are then applied at the note level. Generation of the continuous expressive parameters facilitates fine-grained control of performance parameters of each note without the reverse diffusion process having to learn a quantized representation space.

Figure 1: Training (left) and inference (right) phases of the diffusion framework. Training starts with p_codec and corrupts the $x_{0}$ by injecting noise; the UNet model takes in the corrupted $x_{t}$, conditions $c_{s}$ and $c_{c}$ to predict the injected noise, which is then used to reconstruct $\hat{x}_{0}$. Loss is calculated for both noise prediction and p_codec reconstruction. The inference process (right) starts with a random sample from $\mathcal{N}(\mathbf{0},\mathbf{I})$; the model iteratively predicts the noise and reconstructs $\hat{x}_{0}$, conditioned on the same $c_{s}$ and $c_{c}$. Alternatively for transferal, we initialize the process from another performance $y_{0}$, corrupting it for $t_{0}$ steps and denoising for the remaining $T-t_{0}$ steps.

3 Methodology

In this section, we first introduce the representations for expressive parameters, musical score, and perceptual features used to train DExter. We call these codecs, which are described in detail in Section 4.1. We then explain our diffusion framework that learns these representations, followed by the training and inference architectures and conditioning methodology.

3.1 The Codecs

We represent score information (note onset, duration, pitch, and voice), performance parameters (beat period, velocity, timing, articulation ratio, and pedal), and mid-level perceptual features (melodiousness, articulation, rhythmic complexity, rhythmic stability, dissonance, tonal stability, and minorness) as two-dimensional spectrogram-like matrices of (mostly real-valued, except for the score codec) numeric values. We call these the score codec (s_codec), the performance codec (p_codec), and the perceptual features codec (c_codec) respectively. The task of our diffusion model is to predict a p_codec conditioned on the s_codec and c_codec. Detailed descriptions of the composition of the codecs are given in Section 4.1.

3.2 Diffusion Framework

We frame the expression rendering problem as the task of learning a continuous space of performance expression parameters. Diffusion models Ho et al. (2020) consist of two processes: i) a forward process that transforms each data sample into a standard Gaussian noisy sample step-by-step with a predefined noise schedule; and ii) a reverse process where the model learns to denoise pure-noise inputs gradually, generating samples from the learned training data distribution. In effect, our model aims to convert Gaussian noise $x_{t}$ into a posterior performance codec $\hat{x}_{0}$, conditioned on a score codec $c_{s}:=$ s_codec, and a perceptual features codec $c_{c}:=$ c_codec.

The diffusion forward pass $q(x_{t}|x_{0})$ produces a noisy version of the performance codec. With the noise $\epsilon\in\mathcal{N}(\mathbf{0},\mathbf{I})$ sampled from a standard Gaussian distribution, we blend it with the input sample $x_{0}$ using $\beta$ as a scaling factor intended to ultimately achieve zero mean and unit variance of the fully-noised result. Specifically, the sampling process applies a linear noise schedule with $\beta\in[0.0001,0.2]$. As we would like to perform multiple steps simultaneously, reparameterization is applied to derive a closed-form equation, given that $\alpha_{t}=1-\beta_{t}$ and $\bar{\alpha}=\prod^{t}_{s=1}\alpha_{s}$.

$$x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon$$ (1)

During training, model $f_{\theta}(x_{t},t,c_{s},c_{c})$ learns to predict the injected noise $\hat{\epsilon}_{\theta}$ given a random timestep $t$ and its noised codec version $x_{t}$ calculated in the forward pass. $t$ is sampled from $[1,T]$; we use $T=1000$ in our experiments. Then, we use the predicted noise $\hat{\epsilon}_{\theta}$ to reconstruct the predicted initial codec $\hat{x}_{0}$ by inverting Eq. 1.

The training objective combines the noise estimation and codec reconstruction as shown in Eq. 2. Although the noise prediction is theoretically enough for training the model, empirically we found that constraining on the reconstructed codec yields better performance, with weighting $h=0.2$.

$$L(\theta)=\parallel\epsilon-\hat{\epsilon}_{\theta}\parallel^{2}+h\parallel x_{0}-\hat{x}_{0}\parallel^{2}$$ (2)

During inference, we start from a Gaussian noise distribution $p(x_{t})\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ and iteratively generate the codec posterior through $p_{\theta}(\hat{x}_{t-1}|x_{t},c_{s},c_{c})=\mathcal{N}(\mu_{\theta,t}(x_{t},t,c_{s},c_{c}),\sigma^{2}_{t}\mathbf{I})$ until $\hat{x}_{0}$ is reached. As the model $f_{\theta}$ estimates noise $\hat{\epsilon}_{\theta}$, we use it to construct the model mean $\mu_{\theta,t}$ and the posterior variance is predetermined by the noise schedule. Full construction of model mean and posterior variance is given in Eq. 3 and Eq. 4:

$$\mu_{\theta,t}(x_{t},t,c_{s},c_{c})=\sqrt{\frac{1}{\alpha_{t}}}(x_{t}-\frac{\hat{\epsilon}_{\theta}(1-\alpha_{t})}{\sqrt{1-\bar{\alpha}_{t}}})$$ (3)

$$\sigma^{2}_{t}=(1-\alpha_{t})\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}$$ (4)

Figure 2: Diagram of the UNet conditioning module in the network.

3.3 Architecture and Conditioning

We employ a 2-D UNet as the backbone of our architecture; the detailed layer and insertion structure can be found in Fig. 2. The conditioning on score information and perceptual feature information is enforced by a joint conditioning layer that projects the score dimensions and perceptual dimensions (5 and 7 respectively – see definition of codecs below) onto 512 dimensions. The diffusion timestep $t$ is encoded via sinusoidal position embeddings. The input codec and the conditioning codecs are downsampled and upsampled through ResNet blocks and 2D convolutions. Attention layers are interleaved at bottlenecks.

Before narrowing down on the above described architecture, we experimented with a DiffWave-based architecture Kong et al. (2021) which uses a series of 12 residual layers of 1D convolution. For conditioning, we also experimented with FiLM Perez et al. (2018); Kim and Serra (2023) which yields comparable results to the UNet model. We found that our present architecture gives the best trade-off between model simplicity and performance.

Classifier-Free Guidance (CFG) Ho and Salimans (2021) is widely used for conditioning diffusion models to achieve controllable generation, which we also adopt. During training, a dropout layer is applied to the conditions $c_{s}$ and $c_{c}$ to randomly mask out the conditions with probability $p$, in order to simultaneously train the conditional model $f_{\theta}(x_{t},t,c_{s},c_{c})$ and unconditional model $f_{\theta}(x_{t},t)$. We fixed $p=0.1$ in our training. In inference, a weight parameter $w$ is applied as the guidance scale to a combined prediction.

$$\hat{\epsilon}=w\hat{\epsilon}(x_{t},t)+(1-w)\hat{\epsilon}(x_{t},t,c_{s},c_{c})$$ (5)

Dataset	Pieces	Performances	Duration	MIDI	Repertoire
Vienna4x22 (Goebl (1999))	4	88	2h 18m	recorded	Excerpts from 4 pieces by F. Chopin (Op. 10 No. 3, Op. 38), W. A. Mozart (KV331, first mov.), and F. Schubert (D. 783 No. 15)
(n)ASAP (Peter et al. (2023))	235	1067	94h 30m	recorded	Common Practice Period solo piano works by 15 composers
ATEPP-subset* (Zhang et al. (2022))	1580	11677	1000h	transcribed	Solo piano works by 25 composers, ranging from Baroque to Modern era

Table 1: Overview of datasets used in experiments.

4 Data, Representation, and Processing

4.1 Input and Target Encodings

The performance codec (p_codec) – our prediction target – was originally proposed in the expressive rendering framework Basis Mixer Cancino-Chacón (2018), where four expressive parameters are computed for each note $n$ appearing in the score. These parameters of the p_codec encoding the expression controls modify properties of notes in a MIDI piano performance, thus changing speed and loudness of the performance with time. Combined with score information, the full expressive performance can be reconstructed in a lossless fashion. We expanded the original Performance Codec v.1.0 Cancino-Chacón (2018); Cancino-Chacón et al. (2023) by defining an additional parameter for sustain pedal control. The resulting five (note-wise) performance parameters are as follows:

• Beat period: the ratio of the inter-onset intervals (IOI) between two consecutive notes of the performance and the score. This parameter is computed for each onset $o_{k}$ instead of each note $n_{i}$. It is defined as:
  $x_{\mathit{bp}}(o_{k})=\frac{\text{IOI}_{\mathit{perf}}(o_{k})}{\text{IOI}_{\mathit{score}}(o_{k})}=\frac{\hat{o}^{\mathit{perf}}_{k+1}-\hat{o}^{\mathit{perf}}_{k}}{o_{k+1}-o_{k}}$, where $\hat{o}^{\mathit{perf}}_{k}$ is the actual performed onset time, in seconds, corresponding to score onset $o_{k}$ in beats, calculated as the average onset time of all notes played at score onset position $o_{k}$.

• Velocity: $x_{\mathit{vel}}(n_{i})=\frac{\text{vel}(n_{i})}{127}$, where vel is the MIDI velocity of a played note.

• Timing: $x_{\mathit{tim}}(n_{i})=\Delta_{t}(n_{i})=\hat{o}^{\mathit{perf}}(n_{i})-\text{onset}(n_{i})$, the average onset time of all notes played at score onset position (used in beat period) minus the performance onset time of $n_{i}$. Taking beat period as the ‘tempo grid’ notion Gillick et al. (2021), timing would then refer to the micro-deviation of each note relative to the grid.

• Articulation ratio: $x_{\mathit{art}}(n_{i})=\frac{\text{dur}^{\mathit{perf}}(n_{i})}{\text{dur}(n_{i})\cdot x_{\mathit{bp}}(n_{i})}$, measures the fraction of the expected note duration that is actually played.

• Pedal: $x_{\mathit{ped}}(n_{i})=\frac{\text{ped}(n_{i})}{127}$, where $\text{ped}(n_{i})$ is the discrete MIDI pedal value at the onset of $n_{i}$. Note that pedal encoding is not lossless since changes of value between note onsets will not be captured.

The p_codec is fully invertible in that the full event information from the MIDI file [Pitch, Onset, Duration, Velocity] can be reconstructed given the p_codec and score s_codec.

The Score codec (s_codec) represents the musical score and is derived from the note array from the partitura package Cancino-Chacón et al. (2022). Aligned with the p_codec at the note level, it contains four score parameters for each note: (notated) Onset, Duration, Pitch, and Voice, resulting in a 2D matrix of dimension $4\times n$ where n is the number of notes. The score is indispensable for performance conditioning, as it defines the musical content of the piece.

The Perceptual features codec (c_codec), which we use as steering inputs for the performance generation, are representations of the so-called mid-level perceptual features Aljanaki (2018), namely – melodiousness, articulation, rhythm complexity, rhythm stability, dissonance, tonal stability, and ‘minorness’ (or mode). They describe musical qualities that most listeners can easily perceive. Taking cue from previous research Chowdhury et al. (2019); Chowdhury (2022) showing that these features effectively represent musical factors underlying a wide range of emotions and capture variations in expressive character between different performances of a piece Chowdhury and Widmer (2021), we incorporate these as the performance steering inputs. In our scenario, these features are calculated by a previously trained specialised model Chowdhury et al. (2019), over the recorded audio performance data of Vienna$4\times 22$, (n)ASAP, and ATEPP datasets (see Sec. 4.4). The values are calculated from successive overlap** 15s windows with hop size of 5s. Each computed window is then aligned with the score note array to broadcast into c_codec, a 2D matrix of dimension $7\times n$.

4.2 Processing

Given that there could be slight variations in each performance (missing and extra notes relative to the score), we perform padding based on the score note array so that each pair of performances is perfectly aligned. To accommodate pieces of different lengths, we train our network on segments of $N$ notes where shorter segments are padded. In our experiment, we take $N=200$ (which corresponds to about 10 to 20 seconds of music depending on the tempo and note density).

4.3 Mixup Augmentation

Mixup Zhang et al. (2018) is a data augmentation scheme that regularizes a network to favor simple linear behavior between training examples. To strengthen our model’s ability to model different interpretations, we fuse p_codec pairs $x_{1}$ and $x_{2}$ (codecs representing two different performances of the same piece) and their corresponding c_codec using Eq. 6, where $\lambda$ is a scaling factor varying between $[0,1]$.

$$x_{1,2}=\lambda x_{1}+(1-\lambda)x_{2}$$ (6)

After the mixup augmentation, our dataset consists of 170k segments; the interpolated data are only used in training.

4.4 Datasets and Training Setup

We used three datasets of expressive performances (from the Western classical music solo piano repertoire): Vienna$4\times 22$ Goebl (1999), (n)ASAP Peter et al. (2023), and ATEPP Zhang et al. (2022). Each dataset includes audio, performance MIDI, score in MusicXML format, and their alignment. Information and a comparison of these sets can be found in Table 1. The training is based on ATEPP and 80% of (n)ASAP and Vienna$4\times 22$ data, while the testing set (used in all subsequent experiments in Sec. 5) contains the remaining 20% of (n)ASAP and Vienna$4\times 22$ data. The latter two datasets were recorded on computer-controlled grand pianos and are thus more accurate and precise than the ATEPP data, which were obtained through curated audio transcription.

For the training of the network as mentioned in Sec. 3, we use the Adam optimizer with a learning rate $5\times 10^{-5}$, and employ early stop** with a patience of 50 epochs.

	Basis Mixer Cancino-Chacón (2018)	VirtuosoNet Jeong et al. (2019)	DExter (Ours)
Deviation multiple ($\rightarrow$0)
articulation.key_overlap_ratio	0.62$\pm$3.15	1.92$\pm$2.72	2.24$\pm$2.76
asynchrony.pitch_correlation	-1.19$\pm$1.41	-1.67$\pm$1.25	-1.17$\pm$1.66
asynchrony.delta	4.10$\pm$1.78	4.38$\pm$1.63	4.43$\pm$2.21
dynamics.agreement	-0.07$\pm$1.33	-0.40$\pm$1.30	-0.002$\pm$1.05
dynamics.consistency	-0.67$\pm$1.64	-1.07$\pm$1.56	0.73$\pm$2.21
dynamics.ramp_correlation	-0.36$\pm$2.54	0.65$\pm$1.96	-0.28$\pm$2.44
pedal.onset_value	-	-1.16$\pm$1.68	-1.39$\pm$2.13
tempo_curve	-0.10$\pm$2.44	0.52$\pm$2.48	0.72$\pm$2.65
velocity_curve	-0.67$\pm$1.48	0.15$\pm$1.02	1.48$\pm$2.12
KL Divergence ($\downarrow$)
articulation.key_overlap_ratio	0.92$\pm$2.15	1.66$\pm$6.89	1.64$\pm$3.63
asynchrony.pitch_correlation	0.14$\pm$0.278	0.13$\pm$1.25	0.20$\pm$0.33
asynchrony.delta	4.04$\pm$5.15	4.83$\pm$9.58	1.29$\pm$3.16
dynamics.agreement	0.10$\pm$0.04	0.09$\pm$0.04	0.06$\pm$0.04
dynamics.consistency	0.12$\pm$0.23	0.06$\pm$0.07	0.28$\pm$0.49
dynamics.ramp_correlation	1.54$\pm$5.43	0.35$\pm$1.01	0.42$\pm$1.13
pedal.onset_value	-	0.34$\pm$1.45	0.33$\pm$0.36
tempo_curve	0.98$\pm$2.55	0.65$\pm$1.86	1.26$\pm$5.66
velocity_curve	0.16$\pm$0.21	0.10$\pm$0.06	0.71$\pm$1.37
Pearson’s Correlation ($\uparrow$)
articulation.key_overlap_ratio	-0.01$\pm$0.13	0.05$\pm$0.16	0.11$\pm$0.17
asynchrony.pitch_correlation	0.33$\pm$0.25	0.55$\pm$0.17	0.57$\pm$0.25
asynchrony.delta	0.17$\pm$0.22	0.29$\pm$0.19	0.28$\pm$0.21
dynamics.agreement	0.02$\pm$0.87	0.04$\pm$0.84	0.11$\pm$0.79
dynamics.consistency	0.92$\pm$0.17	0.91$\pm$0.13	0.92$\pm$0.15
dynamics.ramp_correlation	0.04$\pm$0.76	0.12$\pm$0.80	0.14$\pm$0.73
pedal.onset_value	-	0.01$\pm$0.13	0.02$\pm$0.14
tempo_curve	0.02$\pm$0.13	0.09$\pm$0.19	0.19$\pm$0.17
velocity_curve	0.08$\pm$0.23	0.21$\pm$0.27	0.27$\pm$0.23

Table 2: Quantitative expression metrics in the categories of articulation, asynchrony, dynamics, pedaling, plus global tempo and velocity curves. Columns represent different models, and rows are divided into blocks according to the three different evaluation metrics, with each block detailing the outcomes for all features. Note that each generated performance is compared with multiple human ground truth interpretations.

5 Evaluation

In this section, we present quantitative evaluation of generated performances without and with steering, followed by evaluation of performance transfer, and an investigation into the effect of varying the conditioning weight. Finally, we also describe our qualitative study employing a listening test and human participants and present the results.

5.1 Quantitative Evaluation

In this subsection, we evaluate our generated samples’ expressiveness by comparing core expression attributes with ground truth performances. This experiment is conducted on the aforementioned testing set, with condition of s_codec and audio performance inferred c_codec as described in Sec. 4.1. With respect to the critique of reconstruction-based evaluation Peter et al. (2023), we compare with various interpretations of ground truth (the testing set consists of about 5.3 human performances for each piece, on average).

5.1.1 Assessed Attributes

The expression attributes we assess are derived from the tempo and velocity curves (joint-onset level), joint-onset asynchrony, articulation, dynamics and pedalling. While a detailed documentation of the selected attributes can be found on the project page, we provide a summary below:

• Tempo curve: Onset-level tempo (inverse of local inter-beat-intervals), with values averaged across notes on the joint onset. (tem_curve)

• Velocity curve: Onset-level velocity, with values averaged across notes on the joint onset. (vel_curve)

• Asynchrony: The absolute difference in seconds between the earliest and latest note in a joint onset (asy.delta). We also measure the pitch correlation (asy.p_cor) between the pitch and micro-timing within the joint onset, inspired by the melody lead phenomenon Goebl (2001).

• Articulation: Key overlap ratio (art.kor) Bresin and Umberto Battel (2000), measured at each note transition; overlap time (or gap time if staccato) divided by the IOI between the two notes.

• Dynamics: Comparing performed velocity and score marking (, , etc.), and measuring their agreement (dyn.agr) and consistency (dyn.con) as proposed by Kosta et al. (2018). We also propose the ramp correlation (dyn.r_cor) for changing markings (hairpins) since Kosta et al. (2018) only worked with constant markings. The ramp correlation computes the amount of agreement between the performed velocities with respect to their cresc. or decresc. ramp, if the markings exist.

• Pedals: We measure the sustain pedal value at the note onset (ped.onval). Actually, sustain pedal change is a continuous stream of values and changes in pedal position often happen between note attacks; however, sampling at the note onsets simplifies the computation and allows for a consistent assessment across models.

5.1.2 Metrics

For each expression dimension, we measure three metrics between the generated performance and the ground truth space:

Standard deviation multiple: This metric computes the deviation of an attribute of the rendered output from the mean of multiple human performances on a beat-level basis. Different from absolute deviation, this measure incorporates the flexibility of interpretations: when human interpretations already contain large differences, a larger discrepancy can be tolerated. But if human players tend to agree on the interpretation, we would expect the rendered output to fit more closely to ground truth values. Additionally, we retain the sign (direction) of deviation, so that negative values indicate deviations in the direction of slower tempo or softer dynamics, for example.

KL divergence takes all the note-level or beat-level attributes, and compares their divergence with the ground truth attributes as an overall distribution. (Note that the ground truth attribute distributions are aggregated from multiple interpretations.) The KL divergence is calculated by estimating the two distributions using Monte Carlo sampling ($N=300$) and computing the relative entropy between them.

Pearson’s correlation is measured between the feature sequences of generated and ground truth performances. In contrast to the previous two metrics, this metric captures the time-varying similarity between the performance attributes.

5.1.3 Results

In Table 2, we compare our model with the samples from two existing performance rendering systems, BasisMixer Cancino-Chacón (2018) (BM) , and VirtuosoNet Jeong et al. (2019) (VN) . Results are rendered on the same testing set as ours and shown as mean and standard deviation.

Overall, DExter shows commendable results particularly in correlation across almost all performance dimensions, especially in capturing the global curve of tempo and velocity. This latter effect (learning the overall musical shape) could be attributed to the diffusion model predicting the time-varying codec in one pass, in contrast to autoregressive approaches. However, it is evident that DExter has room for improvement in terms of deviation and divergence: DExter’s outputs demonstrate more volatile changes of parameters that are less smooth than other renderings.

Meanwhile, each model exhibits distinct strengths across various performance dimensions. BM’s outputs have articulation that is closer to human ground truths, and this can be attributed to BM being more conservative in its use of expressive devices, using smaller deviations from mechanical reproduction of the score. VN also excels in modelling the dynamics in agreement with the score markings. It is also notable that both models with sustain pedal prediction did a poor job in mimicking human pedaling techniques, with almost no time-wise correlation and on average one standard deviation away from the gound truth. Another area where all models struggle is the asynchrony time (asy.delta: $\mathtt{\sim}$4 deviations away from human benchmark), highlighting the need of refining the micro-timing aspect in performance rendering models.

Figure 3: Steering the expressive characteristics of generated performances by using mid-level features (c_codec, see Sec. 4.1) as conditions. For each piece, a performance is generated with the c_codec derived from an actual performance of the piece, and two further performances are generated with one of the mid-level features doubled (2$\times$) or halved (0.5$\times$). The average difference between the halved and unmodified, and between the doubled and unmodified conditions are plotted here.

5.2 Expressive Steering with Perceptual Features

Our framework of conditioned performance generation allows us to explore conditioning the performance generation on additional features. As described in Sec. 4.1, we use mid-level perceptual features (encoded as c_codec) as steering inputs to guide the expressive character of the generated performance.

To gauge DExter’s sensitivity to changes in these features we use the perceptual feature recognition model of Chowdhury et al. (2019) as a proxy for human perception. However, as that model had originally been trained on audio input, we wanted to eliminate the effect of acoustic artifacts introduced by rendering MIDI to audio, we decided to fit a MIDI-to-perceptual-features model to serve as the proxy instead. Details on this proxy model are given in the Appendix.

Steering performance generation is done by manipulating individual dimensions of the perceptual features, aiming to induce measurable corresponding effects in the resulting outputs. For each test sample and across all seven perceptual attributes, we first generate performances using the unmodified target $c_{c}$. These target perceptual feature values could be randomly initialized in practice, or derived from an actual performance. We take the feature values derived from actual performances and modify the values to steer the generation in particular expressive directions, thus generating “alternate" performances of the original performance. We either halve one feature, $\frac{1}{2}c_{c}$, or double one feature, $2c_{c}$, at a time.

Fig. 3 displays the proxy model’s predictions on the generated outputs. We observe that for the first four features melodiousness, articulation, rhythm complexity, rhythm stability, the adjustments applied to the input conditions manifest as anticipated directional changes (the $c_{c,\mathit{double}}$ group leads the $c_{c,\mathit{half}}$ group 12.2% in terms of their absolute value), providing evidence of the model’s responsive behavior to the controlled feature alterations.

The other three dimensions – notably, dissonance – exhibit less consistent patterns in alignment with the input modifications. That seems reasonable, as harmonic and tonality-related properties are more a function of a piece itself, rather than any specific interpretation of it.

5.3 Transferring from a Source Performance

As suggested by Liu et al. (2023) and Zhang et al. (2023), style transfer can be achieved in a diffusion framework by using, as a starting point for generation, a shallowly noised version of the source information. Given the large amount of music overlap in our datasets, we can test this by forming data pairs that consist of two interpretations of the same piece, to be used as the source and target p_codec in this experiment.

Given a source performance codec $x_{\mathit{src}}$, we calculate its noisy version $x_{t_{0}}$ with a predefined time step $t_{0}\leq T$ according to the forward process shown in Equation 1. By using $x_{t_{0}}$ as the starting point for the reverse process of a pretrained model, we enable the manipulation of performance $x_{\mathit{src}}$ with target mid-level condition and shared score condition $c_{\mathit{(s,tgt)}}$ in a shallow reverse process $p_{\theta}(\hat{x}_{0:t_{0}}|x_{t_{0}},c_{\mathit{(s,tgt)}})$, as illustrated in Fig. 1 (top, right). With the transfer experiments, we attempt to understand the following two questions:

1. Does transferring help with the final generation quality compared with rendering from scratch?

hline1-2,6 = -, $t$ & dev: tem ($\downarrow$) dev: vel ($\downarrow$) cor: tem ($\uparrow$) cor: vel ($\uparrow$)
$T$ 0.72 $\pm$2.65 1.48$\pm$ 2.13 0.19$\pm$0.17 0.27$\pm$0.23
$\frac{3T}{4}$ 0.68$\pm$2.55 1.40$\pm$2.16 0.19$\pm$0.16 0.28$\pm$0.21
$\frac{T}{2}$ 0.74$\pm$2.49 1.33$\pm$2.10 0.15$\pm$0.17 0.21$\pm$0.22
$\frac{T}{4}$ 0.87$\pm$2.69 1.50$\pm$2.11 0.11$\pm$0.16 0.18$\pm$0.21

Table 3: Deviation and correlation of test set relative to ground truth space (same analysis as in Section 5.1) of tempo and velocity curves.

In the transfer experiment, we combine pairs of ground truth performances of the same piece segment $x_{\mathit{src}},x_{\mathit{tgt}}$ where $s_{\mathit{src}}=s_{\mathit{tgt}}$. The same testing set as the previous sections is used, and the source performance is randomly taken from the ground truth. We experimented with different transfer steps of $t_{0}\in\{T,\frac{3T}{4},\frac{T}{2},\frac{T}{4}\}$, and report the global metrics of tempo curve and velocity curve with their deviation and correlation.

What we observed in Table 3 is that transferring from a source performance slightly helps with initialization. Specifically, employing a denoiser for three-quarters of the diffusion steps – ideally preserving around one-quarter of the source’s characteristics – yielded the highest quality outcomes. However, transfer quality does not steadily improve with the retained information from source: the $\frac{T}{4}$-step transfer results in ambiguous outputs that do not align well with the given score.

2. Does a transferred rendering sound ‘closer’ to the source or the target?

Similar to Sec. 5.2, we wish to measure the transfer proximity using the predicted perceptual features. The radar plots in Fig. 4 show the seven perceptual feature dimensions predicted by the proxy, illustrating the perceptual distance between source, target, and generated performance for three different transfer gradations, $\frac{T}{4},\frac{T}{2},\frac{3T}{4}$. At $\frac{T}{4}$ steps, the predicted performance deviates from both source and target, which fits our previous observation that insufficient denoising steps result in ambiguous outputs. As the transfer step increases, there is a discernible shift in the predicted output towards the target profile across most perceptual dimensions.

Figure 4: Seven dimensions of perceptual features (AR: articuation; RC: rhythm complexity; RS: rhythm stability; TS: tonal stability; DI: dissonance; MI: Minorness; ME: Melodiousness) predicted using the proxy for output, source, and target, averaged across the testing set. The three plots correspond to transfer steps of $0.25T,0.5T,0.75T$.

5.4 Effect of Varying Conditioning Weights

In this experiment, we look at the effect of the conditioning weight $w$ on the generated results. As described in Sec. 3.3 and Eq. 5, the scale of classifier-free guidance $w$ is the ratio that combines the prediction with and without (masked by 0) the $c_{s}$ and $c_{c}$ conditioning. While the conditional and unconditional models are jointly trained in the training phase, the weighting parameter $w$ is only introduced in the sampling phase and the optimal $w$ is not trivial to find. In Tab. 4, conditioning weights $w=0.5,1.2,2,3$ are compared, while other settings are the same as in Section 5.1. Experimental results are best for $w=1.2$. Interestingly, with greater scale of classifier guidance, the generative results exhibit larger fluctuations in expressive parameters and less stability.

hline1-2,6 = -, $w$ & dev: tem ($\downarrow$) dev: vel($\rightarrow$0) cor: tem($\uparrow$) cor: vel ($\uparrow$)
$0.5$ 1.11 $\pm$2.46 -2.47$\pm$ 1.10 0.11$\pm$0.16 0.02$\pm$0.22
$1.2$ 0.72$\pm$2.65 1.48$\pm$2.12 0.19$\pm$0.16 0.28$\pm$0.21
$2$ 1.33$\pm$2.37 3.02$\pm$1.53 0.04$\pm$0.15 0.13$\pm$0.24
$3$ 1.86$\pm$1.81 4.63$\pm$1.15 0.04$\pm$0.14 0.10$\pm$0.23

Table 4: Deviation and correlation of test set relative to ground truth space (same analysis as in Section 5.1) of tempo and velocity curves.

5.5 Qualitative study

We evaluated the naturalness and expressiveness of the rendered performances through a listening test. For samples from eight selected pieces, we compared the following: 1) two human performances, with relatively distant interpretations; 2) renderings made with Basis Mixer and VirtuosoNet, as described in Section 5.1; and 3) a rendering from the proposed model DExter. The performances (including the ground truths) were rendered to audio using a Yamaha Disklavier, which produced similar pedal/articulation-related artifacts both in the human and the machine performances. 82 participants listened to the performances and evaluated them on a 100-point Likert scale, rating the overall naturalness and expression of the output as one score. The performances used for the test can be found on the demo page.

Figure 5: Mean ratings for the eight pieces evaluated in the listening test, for each model and human (Ground Truth, GT) performances.

The results of the listening test, shown in Fig. 5, give a nuanced view of the performance rendering capabilities of the models in comparison to the ground truth. In terms of the mean rating, DExter demonstrates better performances of the pieces of Chopin, even sometimes comparable to the ground truth (Barcarolle). However, it is outperformed by Basis Mixer or VirtuosoNet in the case of older compositions (Bach Fugue and Beethoven Sonatas). Overall, in terms of the mean rating scores, there is still a gap between the generative outputs and GT (51.81), while DExter (48.54) slightly outperforms VirtuosoNet (48.31) and Basis Mixer (46.33). It is also surprising to observe that GT does not always secure the highest ratings. In the case of the Chopin Etude, at least, it might be explained by the fact that the human pianists suffer from physical limitations in technically demanding passages, while the generative models do not.

Besides the numerical ratings, we also asked three musically trained participants for explicit feedback on their ratings. In addition to some positive comments, we also received quite specific and useful negative feedback, such as (specifically referring to polyphonic music such as Bach) no clear voicing among the lines and poor balance between hands. Given currently dominating ‘flattened’ representations such as our p_codec or tokens in Transformer models, learning the vertical structure of music remains a challenge to rendering models Zhang et al. (2023).

6 Conclusion and future work

In this paper, we introduce a novel diffusion-based model, DExter, for learning and controlling performance expression in solo piano music. Besides rendering performances at a comparable level of quality to existing models in quantitative measurement of expressive characteristics, DExter is also capable of style transfer between interpretations and conditioning the rendered expression with perceptual variables.

As a future direction, we would like to improve on the inference speed of DExter (currently 40 seconds inference time for a 95 second piece on single RTX 6000 GPU). To accelerate sampling of 1k step iterative process we would explore techniques like DDIM or learning in a latent space. On the other hand, we would also like to explore other conditioning inputs like text that could allow for more explicit controls over the rendered outputs.

7 Acknowledgment

This work is supported by the UKRI Centre for Doctoral Training in Artificial Intelligence and Music, funded by UK Research and Innovation [grant number EP/S022694/1], also by the European Research Council (ERC) under the EU’s Horizon 2020 research and innovation programme, grant agreement No. 101019375 (Whither Music?). We also like to thank Daesam Jeong for contributing VirtuosoNet and for comparative discussion, as well as Maximillian Hofmann for the Basis Mixer trained with LSTM.

8 Appendix

8.1 MIDI to perceptual features model

The MIDI to perceptual features model is used as proxy for human mid-level perception in the experiments described in Sec. 5.2 and Sec. 5.3. It takes in a rendered MIDI and outputs a 7-dimensional perceptual features, for each window of 15 seconds. The specifications are as follows:

• Data: The data used to train this oracle is ASAP performance MIDI, along with the audio perceptual features computed predicted from ASAP performance audio by the mid-level feature recognition model of Chowdhury and Widmer (2021), for 15 seconds windows.

• Representation: Each 15 seconds MIDI window is transformed into a piano-roll matrix of dimension 800 * 131 (128 pitches + 3 pedal channels), with MIDI velocity as matrix value.

• Architecture: The network consists of two residual blocks, each containing two convolution layers. A final projection layers is attached at the end to output the 7-dimension perceptual features.

• Training: Adam optimizer with learning rate $1e-3$. After 20 epochs the training converges with validation loss 0.038.

-0cm

References