DEX-TTS: Diffusion-based EXpressive Text-to-Speech with Style Modeling on Time Variability

Before introducing our methods, we review the diffusion used in the study. The diffusion model consists of two processes: the diffusion process, which adds Gaussian noise to the original data, and the reverse process, which removes Gaussian noise to generate samples. Diffusion process yields noisy data $\{x\}_{t=0}^{T}$, where $p_{0}(x)$ is a data distribution $p_{data}(x)$ and $p_{T}(x)$ is indistinguishable from pure Gaussian noise. Song et al. [33] addressed the diffusion process as a stochastic process over time $t$ and defined diffusion process with Stochastic Differential Equations (SDE) as $dx=f(x,t)dt+g(t)dw$ where $w$ is Brownian motion, and $f(\cdot,t)$ and $g(\cdot)$ are drift and diffusion coefficients. Song et al. [33] also presented that a probability flow Ordinary Differential Equations (ODE) corresponds to the deterministic process of SDE, and it is defined as below:

The deterministic process is determined from the SDE when the score predicted by the score function $\triangledown_{x}logp_{t}(x)$ is known. For the reverse process, a numerical ODE solver such as Euler can be used.

EDM [34] defines the score function as $\triangledown_{x}logp_{t}(x)=(D_{\theta}(x,t)-x_{t})/\sigma^{2}_{t}$ given $\sigma^{2}_{t}$ is $\int g(t)^{2}dt$, and $D_{\theta}$ is a denoiser network trained by denoising error $||D_{\theta}(x_{t},t)-x||_{2}^{2}$. To train a denoiser while avoiding gradient variation, EDM introduces pre-conditioning and $t$-dependent skip connection which are also investigated in CoMoSpeech [28] where the schedule $\sigma_{t}$ is $t$.

$F_{\theta}$ is the network before conditioning. We follow Equation 2 with the parameter settings in [34] to build diffusion. In practice, we can forward text and style representations into $D_{\theta}$ and $F_{\theta}$ to condition the denoising process, as described in [28].

Given the input phonemes, the text encoder extracts the text representation $h_{text}$. The text encoder consists of 8 layers, each composed of Transformer encoder structure [32] that includes Multi-Head Self-Attention (MHSA) and Feed-Forward Network (FFN). To enhance the encoder, we incorporate relative position embedding, RoPE [35], into the attention mechanism and apply the swish gate used in RetNet [36] after the attention operation. Since text varies over time, $h^{e}_{v}$, defined as T-V styles, can be effectively utilized to condition styles for $h_{text}$. We utilize Adaptive Layer Normalization (AdaLN) [37] after each MHSA and FFN to inject $h^{e}_{v}$, extracted by the T-V encoder given a reference input, into $h_{text}$. See Section A.1 for more details.

We adopt a convolution-based Duration Predictor (DP) [9] in which $h_{text}$ extracted by the text encoder is used. DP predicts the duration $\hat{d}$ which maps $h_{text}$ to frames of the mel-spectrogram for the initial mel-spectrogram representation $h_{mel}$ used as the condition input in the diffusion decoder. Aligner is trained by the Monotonic Alignment Search (MAS) algorithm.

Given time $t$ and corresponding noise $x_{t}$ generated by the diffusion process, the diffusion decoder synthesizes a denoised mel-spectrogram $\hat{x}$. Here, the initial mel-spectrogram representation $h_{mel}$ and styles $h_{inv,v}$ are utilized as conditioning information. For diffusion representation $h_{diff}$, we concatenate $x_{t}$, $h_{mel}$, and $t$, and pass it to the diffusion decoder, where $t$ is projected by sinusoidal encoding and linear layers.

The diffusion decoder comprises convolution blocks, adapters, and DiT blocks [30]. To leverage powerful denoising in the latent space, we utilize up and down convolution blocks, as in [15], to decrease and increase the resolution of $h_{diff}$. In the bottleneck, each adapter incorporates $h_{inv,v}$ into $h_{diff}$ to reflect each style information. Furthermore, we forward $t$ as an additional condition into adapters to effectively reflect styles during the iterative denoising process.

After adapters, we utilize DiT blocks to enhance latent representations. To effectively exploit DiT blocks, we introduce overlap** patchify and convolution-frequency (conv-freq) patch embedding. Unlike previous methods, we allow overlap** between patches, mitigating boundary artifacts between patches and enabling natural speech synthesis. For patchify, a convolution layer with a kernel size of $2\times P-1$ and stride of $P$ is used given patch size $P$. Let $h\in\mathbb{R}^{C\times F\times T}$ be the representations after adapters, where $C$, $F$, and $T$ are the hidden, frequency, and time sizes respectively. Given $F_{2}=F/P$ and $T_{2}=T/P$, the convolution layer patchifies $h$ into $h_{p}\in\mathbb{R}^{C\times F_{2}\times T_{2}}$.

Before converting the spatial dimensions of $h_{p}$ into a sequence for DiT inputs, embeddings are added to each frequency and time dimension. Since the speech length is variable, patch embedding should be able to handle unseen lengths during training. For the time axis, we apply a convolution layer to $h_{p}$ and take a time-wise average to obtain the relative positional embeddings $PE_{T}\in\mathbb{R}^{C\times 1\times T_{2}}$. On the other hand, we use fixed-size learnable parameters as frequency embedding $PE_{F}\in\mathbb{R}^{C\times F_{2}\times 1}$ since the frequency size is not variable in speech synthesis. $PE_{T}$ and $PE_{F}$ are added to $h_{p}$, and the spatial dimension is converted into a sequence to be used as input for DiT. By the above embedding approach, we can get robust embedding for variable lengths compared to conventional embeddings obtained by fixed-size parameters or sinusoidal encoding. After the DiT block, the up-convolution block predicts a denoised mel-spectrogram from latent features.

As depicted in Figure 1.c, the T-IV encoder consists of a few residual convolution blocks to extract the representation from the reference speech. To maintain individual characteristics regardless of the temporal information within a batch, we utilize Instance Normalization (IN) after each block. Inspired by [38], we use multi-level feature maps as T-IV styles $h_{inv}\in\mathbb{R}^{L\times C\times T}$, where $L$ is the number of layers in the T-IV encoder. Since $h_{inv}$ comprises all stacked feature maps from the convolution block, it contains T-IV information across the convolution blocks.

For T-IV Adatpor, we apply AdaIN, which can reflect styles regardless of temporal information, to inject $h_{inv}$ into $h_{diff}$ during the denoising process in the diffusion decoder. Given mean $\mu$ and standard deviation $\sigma$ as bias and scale for AdaIN, the process is defined as follows:

where $\mu$ and $\sigma$ are obtained using $h_{inv}$. Since $h_{inv}$ contains the feature maps of all layers, we compute the channel-wise mean and standard deviation for each layer and utilize attention pooling (AP) [38] to obtain representative statistics ($\mu$ and $\sigma$) for AdaIN. Furthermore, we include time $t$ in the pooling process to ensure adaptive operation at each time step during the denoising process. The process for extracting $\mu$ and $\sigma$ is represented as follows:

where $W_{ap}$ is the linear weight for AP. Through AP, we can extract common features across multi-level feature maps to utilize as a general T-IV style. Furthermore, time conditioning enables adaptive style incorporation at each timestep.

Similar to the T-IV encoder, the T-V encoder contains a few residual convolution blocks, but we employ Layer Normalization (LN) instead of IN to preserve temporal relationships in each instance. The T-V encoder extracts two styles $h^{e,d}_{v}$ for each text encoder and diffusion decoder. We obtain $h^{e}_{v}$ by applying convolution blocks to the reference and adding the pitch information $h_{f0}$. We use $h_{f0}$ to reflect changes in speech over time, and it is extracted by applying GRU layers to the log fundamental frequency of the reference speech. After channel-wise pooling on $h^{e}_{v}$ to obtain the overall T-V style information, $h^{e}_{v}$ is forwarded to the text encoder.

For $h^{d}_{v}$, we additionally apply Vector Quantization (VQ) to the output of convolution blocks. For VQ, we utilize a latent discrete codebook $e\in\mathbb{R}^{K\times D}$, where $K$ is the codebook size and $D$ is the dimension size. The VQ layer maps the outputs to a discrete space based on the distance to the codebook. This process removes noise from a continuous space, obtaining well-refined style information that can be used as generalized features. After adding the pitch information, $h^{d}_{v}$ is passed to the decoder without pooling to preserve temporal information.

In the text encoder, we apply AdaLN to reflect the overall style $h^{e}_{v}$ while preserving the temporal aspects of the text representation. AdaLN with $h_{text}$ in the encoder is defined as follows:

where $g(\cdot)$ and $b(\cdot)$ are linear layers for scaling and bias, respectively.

To reflect T-V styles $h^{d}_{v}$ to $h_{diff}$ while preserving temporal information, we design the T-V adapter with cross-attention. We use $h_{diff}$ as the query and $h^{d}_{v}$ as the key and values for cross-attention (CA), and it is defined as below.

where $W_{q,k,v}$ denotes the linear weight. As presented in Equation 6, IN is applied to $h_{diff}$ for the query. It maintains instance-level features for computing attention scores and enables reflecting suitable T-V style for each instance.

To evaluate the proposed method, we use the VCTK dataset [40], an English multi-speaker dataset, consisting of approximately 400 utterances per 109 speakers. We split the dataset into about 70%, 15%, and 15% for the train, validation, and test sets, respectively, based on speakers to consider both the seen and unseen (zero-shot) scenarios. For the zero-shot scenario, 10 unseen speakers are used. In addition, we conduct experiments on the Emotional Speech Dataset (ESD) [41] to verify whether the models can reflect styles using expressive reference speech. The ESD contains 10 English and 10 Chinese speakers with 400 sentences per speaker for five emotions (happy, sad, neutral, surprise, and angry). We only use English speakers and keep the same split ratio as that in the VCTK dataset. Two unseen speakers are used for the zero-shot scenario. Considering real-world applications, we design both parallel and non-parallel test scenarios based on whether the input text is the same as the text of the reference speech. For the experimental results, we record the average performances of the parallel and non-parallel scenarios. Finally, all datasets are resampled to 22 kHz.

For comparison, we set the following systems as baselines: 1) Ref, the reference audio. 2) MetaStyleSpeech [24], multi-speaker adaptive TTS with meta-learning. 3) YourTTS [25], VITS-based zero-shot multi-speaker TTS with the pre-trained speaker encoder. 4) GenerSpeech [27], style transfer method for out-of-domain TTS. 5) StyleTTS [26], style-based TTS with transferable aligner and AdaIN. Except for YourTTS (end-to-end TTS), the generated mel-spectrograms are transformed into waveforms by the pre-trained HiFi-GAN [42]. We record the performance of the baselines after training with their codes.

For training, we take 1000 and 1500 epochs for the VCTK and ESD datasets, respectively. An Adam optimizer with a learning rate of $10^{-4}$ and batch size of 32 are used. Regarding the model hyperparameters of the diffusion decoder, we take a patch size $P$ of 2, number of DiT blocks $N$ of 4, and hidden size $C$ of 64. The T-IV and T-V encoders use the 6 layers $L$, and their dimension sizes are matched with the diffusion decoder. We set a codebook size $K$ of 512 and dimension size $D$ of 192 for the VQ layer in the T-V encoder. We extract mel-spectrograms with 80 mel bins based on the FFT size of 1024, hop size of 256, and window size of 1024, which is compatible with the HiFi-GAN vocoder used in our TTS system. For the diffusion denoising steps, we use 50 Number of Function Evaluations (NFE) with the Euler solver (See Section B.3 to find results depending on NFE). All experiments are conducted on a single NVIDIA 3090 GPU. Codes are available here.²²2Codes are available at https://github.com/winddori2002/DEX-TTS/

We consider objective and subjective evaluation metrics. As objective metrics, we utilize Word Error Rate (WER %) and Cosine Similarity (COS). WER represents how accurately the model synthesizes the given text, and it is calculated as the error between the predicted text, obtained by applying the pre-trained Wav2Vec 2.0 [43] to the synthesized speech, and the given text. On the other hand, COS indicates the similarity in the feature space between the synthesized and reference speech, and it is calculated using a pre-trained speaker verification model.³³3https://github.com/resemble-ai/Resemblyzer For convenience, we show COS multiplied by 100 in the experimental results. For the subjective metrics, we adopt Mean Opinion Score for naturalness and similarity (MOS-N and S). We use Amazon Mechanical Turk (AMT) and ask participants to score on a scale from 1 to 5. They assess the synthesized speech for its naturalness by listening to it, or they compare the synthesized speech with reference speech to evaluate similarity. For every MOS evaluation, we randomly select 30 utterances for each model and guarantee at least 27 participants.