Temporal-Channel Modeling in Multi-head Self-Attention
for Synthetic Speech Detection

Abstract

Recent synthetic speech detectors leveraging the Transformer model have superior performance compared to the convolutional neural network counterparts. This improvement could be due to the powerful modeling ability of the multi-head self-attention (MHSA) in the Transformer model, which learns the temporal relationship of each input token. However, artifacts of synthetic speech can be located in specific regions of both frequency channels and temporal segments, while MHSA neglects this temporal-channel dependency of the input sequence. In this work, we proposed a Temporal-Channel Modeling (TCM) module to enhance MHSA’s capability for capturing temporal-channel dependencies. Experimental results on the ASVspoof 2021 show that with only 0.03M additional parameters, the TCM module can outperform the state-of-the-art system by 9.25% in EER. Further ablation study reveals that utilizing both temporal and channel information yields the most improvement for detecting synthetic speech.

1 Introduction

Powered by advanced deep generative neural networks, recent text-to-speech (TTS) and voice conversion (VC) systems have the ability to generate highly realistic synthetic human voices. Although this application can benefit many areas including data augmentation [1], criminals can utilize these fake speeches for malicious purposes leading to financial fraud, political conflict, and impersonation. Due to that, synthetic speech detection has been an active research field [2, 3]. To capture local synthetic artifacts, convolutional neural networks (CNNs) have conventionally served as the foundational architecture for SSD models. This approach covers a wide range of CNNs including LCNNs [4, 5], residual-connected ResNet [6, 7], and other variants [8, 9]. However, CNN-based models exhibit limitations in capturing the long-range dependencies of the input sequence. To overcome this, numerous studies employ Transformer models [10, 11, 12], yielding improved performance over CNN-based SSD models.

Notably, the recent SSD model [13], which combines the rich sequence representation of a self-supervised learning (SSL) model XLSR and the transformer-based Conformer architecture, achieves the state-of-the-art result in the ASVspoof 2021 corpus. This improvement can be attributed to the powerful modeling capability of the multi-head self-attention (MHSA) mechanism. It is conjecture that artifact details of synthetic speech can be located in specific regions of both temporal and spectral domain [14, 15, 16]. Therefore, incorporating the relationship between temporal and spectral information can provide a more complete and accurate representation for detecting artifacts in synthetic speech. By leveraging the temporal and spectral dependencies, several SSD systems [17, 18] exhibit improved capabilities in detecting deepfake speech. However, the MHSA in transformer-based SSD systems focuses on computing dot product between input tokens along the temporal dimension, hence it may overlook the dependencies between the temporal and channel dimensions of input sequences, which can be crucial for SSD tasks.

To better leverage the temporal and channel interaction of the input sequence for the XLSR-Conformer system, we propose the Temporal-Channel Modeling (TCM) module in the multi-head self-attention of the Conformer model. Our TCM module is based on the head tokens design, in which each head token represents the information on the channel dimension. The idea of head tokens is first proposed in [19] to enhance the interaction between the representation of attention heads in the MHSA and have improved the performance of Vision Transformer trained in the small-scale image classification dataset. However, in this work, head tokens aim to facilitate the correlation between temporal and channel dependencies by interacting them with the temporal tokens during MHSA. We also modify the original head token design by enriching the classification token with both temporal and channel information. The proposed TCM module, wherein with a marginal increase in parameters, improves the performance of the state-of-the-art XLSR-Conformer system on the ASV2021 eval set. Through empirical evaluation of the contribution of each component in the TCM module, temporal information from input tokens and channel information from head tokens both play an important role in the improvement of the TCM module.

Figure 1: The overall architecture of the baseline XLSR-Conformer and our proposed temporal-channel modeling (TCM) module. The TCM module is used to replace the multi-head self-attention (MHSA) of each Conformer block in the baseline XLSR-Conformer. The TCM module architecture includes three main parts: Head Token Generation, Multi-Head Self-Attention, and Classification Token Enrichment. The objective of TCM is to generate the head token for channel information and then integrate the temporal-channel dependency into the original temporal tokens for better synthetic speech detection.

2 Method

2.1 Baseline XLSR-Conformer

We adopt the state-of-the-art XLSR-Conformer [13] as our baseline architecture. As illustrated in Figure 1.a, it leverages the pre-trained XLSR [20], a variant of the wav2vec 2.0 model. Benefiting from the large-scale architecture and training on extensive data in an SSL manner, SSL models including XLSR can extract rich speech representations that have been useful for numerous speech tasks [21, 22, 23, 24, 25] including synthetic speech detection [26]. XLSR comprises two main components: a CNN front-end to transfer the 1D raw waveform into 2D temporal-channel representation, and $24$ transformer encoder layers for capturing the global relationship of the speech. The shape of the output speech representation is $(T\times D)$, where $T$ denotes the temporal length and $D$ is the channel dimension of XLSR representation.

After that, the XLSR representation is projected to $D$-dimensional and concatenated with the learnable classification token $CLS$ to form an input sequence $X\in\mathbb{R}^{(T+1)\times D}$ for the Conformer model. The Conformer model consists of $L$ Conformer blocks, each Conformer block includes the MHSA, feed-forward module, and the additional Convolutional layer to capture local dependencies within the speech representation. Finally, the $CLS$ token is detached from the Conformer model’s output representation to determine whether the input speech is bona fide or spoof.

2.2 Temporal-Channel Modeling module

The study of [19] introduces the concept of head token design, which initially focuses on fostering interaction between attention heads in multi-head self-attention (MHSA) and has improved the performance of image classification models trained on limited datasets. While the Temporal-Channel Modeling (TCM) approach is inspired by the head token design, its goal is to assist multi-head self-attention in capturing temporal-channel dependencies which can be essential for detecting synthetic speech. The proposed TCM module replaces the original MHSA of each Conformer block in the baseline model. As shown in Figure 1.b, the TCM architecture comprises three parts: Head Token Generation, Multi-Head Self-Attention, and Classification Token Enrichment. Similar to MHSA, TCM will not change the shape of the input and output token sequences for each Conformer block.

2.2.1 Head Token Generation

The Temporal-Channel Modeling module begins with the Head Token Generation component, designed to generate head tokens that represent the channel information of the input. These tokens interact with temporal information in subsequent steps. As shown in Figure 1.c, the input sequence of the Head Token Generation component consists of classification token $CLS$ and temporal tokens $X\in\mathbb{R}^{(T+1)\times D}$. It first undergoes the head token generation process where $X$ is reshaped into $H$ segments of $d=D/H$ dimensions along the channel axis, where $H$ is the number of attention heads in MHSA. Subsequently, each segment undergoes temporal average pooling and concatenates together, followed by the linear projection consisting of a fully connected layer and the GeLU function to project back to $D$-dimension channel representation. Since these steps are similar to the MHSA transformation process, by projecting the input sequence into distinct attention heads, these embeddings are designated as head tokens, representing different parts of the channel dimension. To distinguish head tokens from input tokens, we add a learnable head token embedding with the shape of $(H\times D)$ to head tokens. After obtaining head tokens, they are concatenated with the input sequence along the temporal dimension, forming a new temporal-channel token sequence with the length of $(T+H+1)$ to the MHSA.

2.2.2 Multi-Head Self-Attention

The multi-head self-attention mechanism within our TCM operates similarly to conventional multi-head self-attention but with the input sequence containing both temporal and channel tokens, rather than just temporal tokens. To learn the temporal-channel interaction for spoofing detection, the multi-head self-attention mechanism transforms the temporal-channel tokens into query $Q$, key $K$, and value $V$. This is achieved by projecting temporal-channel tokens $H$ times using corresponding linear projection matrices $W_{i}^{Q}$, $W_{i}^{K}$, $W_{i}^{V}$, resulting in $d$-dimensional channel representations, where $i$ represents the index of the head within the MHSA. With the scaled dot product, the self-attention operator then calculates appropriate weights for each token along the temporal axis based on its relevance to each other, and this process is repeated in parallel across $H$ attention heads. Subsequently, the output of each head is concatenated and subjected to a final linear projection denoted as $W^{O}$, yielding the output embedding. The multi-head self-attention can be represented by the following equation:

$$\text{MultiHead}(X) =\text{Concat}(\text{head}_{1},\ldots,\text{head}_{H})W^{O}$$

$$\text{where }\text{head}_{i} =\text{softmax}(\frac{XW_{i}^{Q}\cdot(XW_{i}^{K})^{T}}{\sqrt{d}})\cdot XW_{i}^{V}$$ (1)

Given that the self-attention of each head is independently computed along the temporal axis of the input sequence, if the input sequence only contains temporal tokens, the model may lack the interaction between the temporal and channel dimensions in the MHSA. However, in the proposed method, head tokens representing channel information are put together with temporal tokens, hence MHSA can learn temporal-channel dependencies by attending to different parts including temporal and head tokens of the input sequence.

2.2.3 Classification Token Enrichment

Although the classification token $CLS$ can attend the information from both temporal and channel tokens during MHSA, we further enrich the $CLS$ token with both temporal and channel tokens with the Classification Token Enrichment component module because the $CLS$ token is directly used for the final prediction, and the information from both tokens can be both crucial for detecting artifacts. Figure 1.d illustrates the Classification Token Enrichment component of the proposed TCM module. Firstly, the temporal and head tokens are segregated from the MHSA output and subjected to average pooling to get the mean temporal token and mean head token. After that, instead of considering only the mean head token in the original head token design [19], our TCM module also enriches the classification token $CLS$ with the mean temporal token. Finally, the enriched classification token is concatenated with the temporal tokens to form the output sequence, kee** the same shape of $(T+1)\times D$ as the input sequence.

3 Experiments

System	Params (M)	LA (Fix)		LA (Var)		DF (Fix)	DF (Var)
		EER (%)	min t-DCF	EER (%)	min t-DCF	EER (%)	EER (%)
RawNet2 [27]	25.43	9.50	0.4257	-	-	22.38	-
AASIST [17]	0.30	5.59	0.3398	-	-	-	-
RawFormer [11]	0.37	4.98	0.3186	4.53	0.3088	-	-
XLSR-AASIST [26]	317.84	1.00	0.2120	-	-	3.69	-
XLSR-Conformer [13]	319.74	1.38	0.2216	0.97	0.2116	2.27	2.58
XLSR-Conformer (reproduce)	319.74	1.40	0.2226	1.26	0.2200	2.79	2.98
XLSR-Conformer + TCM	319.77	1.03	0.2130	1.18	0.2172	2.06	2.25

Table 1: Performance comparison with the state-of-the-art systems on the ASVspoof 2021 eval set with fixed-length (Fix) and variable-length (Var) utterance evaluation (Bold denotes the best result, underline denotes the second-best result, and dash denotes the results are unavailable).

3.1 Dataset and metrics

While the training and development data are from the ASVspoof 2019 [28] logical access (LA) track containing clean speech with text-to-speech and voice conversion attacks, we evaluated our method on the ASVspoof 2021 [29] logical access (LA) and deep fake (DF) tasks. ASVspoof 2021 LA eval set includes 2 known and 11 unknown and the speech data is distorted by various codec and compression variations, mimicking real-world scenarios. Additionally, ASVspoof 2021 introduced a new DF eval consisting of two new additional sets of source data compared to the LA set. Our primary evaluation metrics are the common-used equal error rate (EER) [30] and minimum normalized tandem detection cost function (t-DCF).

3.2 Implementation details

In the training step, the audio data are cropped or concatenated giving segments of approximately 4 seconds duration (64,600 samples). We used the Adam optimizer with a learning rate of $10^{-6}$ with a weight decay of $10^{-4}$ to optimize a weighted cross-entropy loss. The batch size for the training step is set to 20. The final result is reported using the model checkpoint created by averaging the top-5 best validation performance models. Early stop** is applied when the cross entropy loss in the validation set did not improve for 7 epochs. All of the experiments are trained with one Nvidia A40 GPU with the same random seed. In terms of model architecture, following our baseline [13], the pre-trained SSL model XLSR is utilized as an upstream model to extract intermediate representation from the raw input signal.

To be comparable with [13, 26], the signal noise injection data augmentation technique RawBoost [31] is utilized in our experiments. The configuration and parameters of RawBoost used in our experiment are similar to the original paper. Following our baseline system [13], we trained two separate SSD systems with two different Rawboost settings to evaluate on the LA and DF track, respectively. In the LA track, the SSD system is trained with the RawBoost technique combining linear and non-linear convolutive noise and impulsive signal-dependent additive noise strategies. On the other hand, the stationary signal-independent additive, randomly colored noise, is added during the training in the DF track.

3.3 Results

3.3.1 Comparison with the state-of-the-art systems

Table 1 compares the performance of the proposed TCM with our reproduced state-of-the-art XLSR-Conformer and other existing competitive systems on the ASVspoof21 LA and DF evaluation set. In the fixed-length input evaluation on the LA track, adding the proposed TCM module can achieve 25% EER improvement than the baseline XLSR-Conformer for the pooled EER (1.03 % vs 1.40%). While XLSR-Conformer with TCM achieved comparable performance to the top-performing LA system XLSR-AASIST [26] in the LA track, it attained a new state-of-the-art result of 2.06% EER in the DF track, surpassing the previous best-reported result of XLSR-Conformer by 9.25% in the fixed-length input evaluation. Similar gains can be observed in variable-length utterance evaluation. Notably, while achieving noticeable improvement, our TCM module is lightweight since it adds only $0.03M$ parameters to the XLSR-Conformer system. In the following sections, we conduct experiments on the reproduced XLSR-Conformer for further analysis of the robustness and efficiency of TCM.

3.3.2 Transformer and Conformer comparison

To verify the robustness and effectiveness of the proposed TCM in the SSD task, we replaced the Conformer block with the Transformer one and conducted the study in Table 2. We notice that TCM can bring relatively stable improvement for both Conformer and Transformer structures. This can indicate that the learned temporal-channel dependency in TCM can be beneficial for detecting spoofed artifacts regardless of the transformer-based architectures. Furthermore, the Conformer-based system yielded superior performance compared to the corresponding Transformer in the baseline setting as well as with the TCM module. These demonstrate that the local information captured by the Convolution module in Conformer is important for the SSD task.

	21LA		21DF
System	Fix	Var	Fix	Var
XLSR-Transformer	1.60	1.44	2.24	2.49
XLSR-Transformer + TCM	1.51	1.91	2.02	2.34
XLSR-Conformer	1.40	1.26	2.33	2.48
XLSR-Conformer + TCM	1.03	1.18	2.06	2.25

Table 2: EER (%) results to evaluate the robustness of TCM for the Transformer and Conformer Block.

3.3.3 Multi-head attention

Table 3 further studies the effect of the different numbers of heads with and without TCM on the ASV2021 LA & DF eval set. The system with 4 heads can lead to the best performance. (1.03 % EER on LA track and 2.06 % EER on DF track.). The improvement by TCM is robust for most cases, except the DF eval track with 8 heads. It is important that an increase in the number of attention heads does not necessarily ensure improved results.

3.3.4 Ablation study

Table 4 presents an analysis of the contributions of each component within our TCM module on the ASV2021 DF evaluation set. We observed that the inclusion of head token embeddings leads to a slight improvement in system performance. Conversely, the performance of the TCM module experiences a notable decline, from 2.06% to 2.40% EER, when head tokens are excluded from the multi-head attention mechanism or when the mean head token (mean HT) is omitted from addition to the $CLS$ token. A similar trend is observed when the mean temporal token (mean TT) is excluded from enriching the $CLS$ token. Notably, when both mean HT and mean TT are absent, the performance drops significantly to 3.25% EER, indicating a deterioration compared to the baseline system. These findings underscore the importance of leveraging both temporal and channel information, as represented by temporal tokens and head tokens, in the task of detecting synthetic speech.

Track	System	EER (%)
		H=4	H=6	H=8
LA	XLSR-Conformer	1.40	1.14	1.72
	XLSR-Conformer + TCM	1.03	1.13	1.06
DF	XLSR-Conformer	2.79	2.87	3.11
	XLSR-Conformer + TCM	2.06	2.84	3.81

Table 3: Our methods with different numbers of heads on ASV2021 LA & DF eval set.

	EER (%)
XLSR-Conformer w/o TCM (baseline)	2.79
XLSR-Conformer + TCM	2.06
w/o HT embedding	2.08
w/o HT in MHSA	2.40
w/o adding mean HT to $CLS$	2.41
w/o adding mean TT to $CLS$	2.33
w/o adding mean HT & Mean TT to $CLS$	3.25

Table 4: Ablation study of each component in our proposed TCM on ASV2021 DF evaluation set. HT and TT represent head tokens and temporal tokens, respectively.

4 Conclusion

In this paper, we propose a Temporal-Channel Modeling module for MHSA-based synthetic speech detection systems. Our method integrates the channel representation head token into the temporal input token within the multi-head self-attention, which forces the model to learn the temporal-channel dependencies from the input sequence. The XLSR-Conformer using our TCM module outperforms the state-of-the-art performance and outperforms competing methods on the ASVspoof 2021 eval set. Additionally, the ablation study validates the effectiveness of our proposed method and demonstrates the importance of temporal-channel modeling in synthetic speech detection.

5 Acknowledgement

This research is supported by the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No.: AISG-TC-2023-011-SGIL). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore. The computational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore (https://www.nscc.sg).