Emphasized Non-Target Speaker Knowledge in Knowledge Distillation
for Automatic Speaker Verification

Abstract

Knowledge distillation (KD) is used to enhance automatic speaker verification performance by ensuring consistency between large teacher networks and lightweight student networks at the embedding level or label level. However, the conventional label-level KD overlooks the significant knowledge from non-target speakers, particularly their classification probabilities, which can be crucial for automatic speaker verification. In this paper, we first demonstrate that leveraging a larger number of training non-target speakers improves the performance of automatic speaker verification models. Inspired by this finding about the importance of non-target speakers’ knowledge, we modified the conventional label-level KD by disentangling and emphasizing the classification probabilities of non-target speakers during knowledge distillation. The proposed method is applied to three different student model architectures and achieves an average of 13.67% improvement in EER on the VoxCeleb dataset compared to embedding-level and conventional label-level KD methods.

Index Terms—  knowledge distillation, label-level knowledge distillation, automatic speaker verification

1 Introduction

Automatic speaker verification (ASV) is the process of authenticating an individual’s claimed identity based on voice characteristics. By leveraging large-scale neural networks trained on abundant unlabelled speech data, self-supervised learning (SSL) models have revolutionized various speech processing tasks [1, 2, 3], including ASV [4, 5]. However, these models are computationally expensive. To better utilize SSL models, knowledge distillation can be employed to transfer the robust speech representation to smaller student models. In ASV, KD encompasses two common approaches: one is embedding-level method [6, 7, 8, 9], which attempts to make student models mimic the teacher’s intermediate feature embedding by reducing the distance between representation spaces; the other is label-level method [6, 10], which focuses on minimizing the Kullback–Leibler divergence between the output probabilities of the teacher and student networks.

In the training step of an ASV model, the objective is to classify input speech into target speaker (the ground-truth speaker) and avoid assigning it to non-target speakers (incorrect speakers). While the importance of the target speaker is evident, non-target speakers can also enhance the model’s discriminability since there would be numerous non-target speakers sharing similar voice characteristics with the target speaker. In past studies, [11] compared ASV models performance trained on two training sets with different numbers of speakers but the same number of utterances, and found that a larger number of speakers improved the performance. Similarly, in face recognition, [12] also observed that an increasing number of training non-target classes improved model performance within a fixed-size training set. Building on these observations, we hypothesize that integrating knowledge from non-target speakers can enhance ASV model performance. However, the conventional label-level KD considers correlations among the teacher’s output probabilities of all speakers, the importance of non-target speakers’ probabilities can be overshadowed by the target speaker with high classification confidence in the teacher model. Based on this hypothesis, the conventional label-level KD approach for ASV can be improved by emphasizing the knowledge of non-target speakers.

To validate the assumptions above, this paper initially shows an experiment illustrating the importance of non-target speakers in ASV. When the number of training utterances remains the same, we observe that an increasing number of non-target training speakers leads to better results. Based on this observation, we investigate the significance of non-target speakers in the conventional label-level KD for ASV models. Following Decoupled Knowledge Distillation (DKD) [13], we segregate the output classification probabilities of the teacher and student models into two distinct probabilities of target and non-target speakers. Subsequently, the probabilities of non-target speakers are emphasized during KD using a specific weight. We utilize the large-scale SSL model WavLM-TDNN [14] as our teacher model and employ three different network architectures: x-vector [15], ResNet34 [16], and CAM++ [17] as student models. Our experiments show that DKD with an emphasis on the non-target speakers’ output probabilities, outperforms both embedding-level and conventional label-level KD methods across student models.

2 Methodology

2.1 The impact of non-target speakers for ASV

Fig. 1: Vox1-O results (EER %) of x-vector model trained on a fixed number of utterances but varying numbers of speakers.

To validate the hypothesis that a larger set of non-target speakers benefits ASV models, we conducted a toy experiment. We trained the x-vector model using a fixed 100,000 training utterances of the VoxCeleb 2 dev set [18]. These training utterances are evenly distributed among each training speaker, hence increasing the number of speakers will lead to fewer training utterances per speaker. As depicted in Figure 1, the performance of the x-vector model consistently improves with an increasing number of speakers in the training set. This indicates that involving more non-target speakers enhances the model’s ability to distinguish the target speaker from others. Inspired by this finding, we further extract and emphasize non-target speaker knowledge during the knowledge distillation process.

Fig. 2: Our Decoupled Knowledge Distillation (DKD) with an emphasis on non-target speaker knowledge in comparison with the embedding-level knowledge distillation (using cosine distance loss $\mathcal{L}_{\text{COS}}$) and the conventional label-level knowledge distillation (using Kullback–Leibler divergence loss $\mathcal{L}_{\text{KD}}$). $\mathcal{T}$, $\mathcal{S}$, $K$, and $\tau$ denote the teacher model, the student model, the number of training speakers, and the target speaker, respectively. $p_{i}$, $p_{\bar{\tau}}$, and $\hat{p}_{i}$ are respectively defined as Eq.(1) and Eq.(4). $\mathcal{L}_{\text{TSKD}}$, $\mathcal{L}_{\text{NSKD}}$ and $\gamma$ are defined as Eq.(6) and Eq.(8), respectively.

2.2 Rethinking conventional label-level KD

Following the reformulation of the conventional label-level KD for the computer vision task in [13], we interpret the conventional label-level KD loss for automatic speaker verification. In the training phase of the ASV model, the model’s output is classification probabilities $\mathbf{p}$ over the set $\mathcal{K}$ of K training speakers, in which the probability $p_{i}$ of the i-th speaker is computed using the softmax function to transform the logits vector $z_{i}$ into a probability distribution as follows:

$$p_{i}=\frac{e^{z_{i}}}{\sum_{j=1}^{K}e^{z_{j}}}$$ (1)

In the conventional label-level KD, the student model tries to mimic the teacher model by minimizing the Kullback-Leibler Divergence $D_{\text{KL}}$ between the student ($\mathcal{S}$) and teacher ($\mathcal{T}$) output probability distributions. The $D_{\text{KL}}$ loss is defined as:

$$\mathcal{L}_{\text{KD}}=D_{\text{KL}}(\mathbf{p}^{\mathcal{T}}\|\mathbf{p}^{\mathcal{S}})=\sum_{i\in K}p^{\mathcal{T}}_{i}\log(\frac{p^{\mathcal{T}}_{i}}{p^{\mathcal{S}}_{i}})$$ (2)

where $\mathbf{p}^{\mathcal{T}},\mathbf{p}^{\mathcal{S}}$ denote the output probabilities of the teacher and student networks, respectively. We further split the set $\mathcal{K}$ of indexes $\{i=1\dots K\}$ in $\mathcal{L}_{\text{KD}}$ into the target speaker $\tau$ and a set of non-target speakers $\mathcal{K}\backslash\{\tau\}$ as:

$$\mathcal{L}_{\text{KD}}=p_{\tau}^{\mathcal{T}}\log(\frac{p_{\tau}^{\mathcal{T}}}{p_{\tau}^{\mathcal{S}}})+\sum_{i\in\mathcal{K}\backslash\{\tau\}}p_{i}^{\mathcal{T}}\log(\frac{p_{i}^{\mathcal{T}}}{p_{i}^{\mathcal{S}}})$$ (3)

We define the probability of classifying a speaker belonging to $\mathcal{K}\backslash\{\tau\}$ as $p_{\bar{\tau}}$, and the probability of predicting a specific non-target speaker $i\neq\tau$ over all non-target speakers as $\hat{p}_{i}$:

$$p_{\bar{\tau}}=\frac{\sum_{i\in\mathcal{K}\backslash\{\tau\}}e^{z_{i}}}{\sum_{j\in K}e^{z_{j}}},\quad\hat{p}_{i}=\frac{e^{z_{i}}}{\sum_{j\in\mathcal{K}\backslash\{\tau\}}e^{z_{j}}}$$ (4)

From (1) and (4), we replace $p_{i}=p_{\bar{\tau}}\hat{p}_{i}$ in (3):

$$\mathcal{L}_{\text{KD}} =p_{\tau}^{\mathcal{T}}\log(\frac{p_{\tau}^{\mathcal{T}}}{p_{\tau}^{\mathcal{S}}})+p_{\bar{\tau}}^{\mathcal{T}}\sum_{i\in\mathcal{K}\backslash\{\tau\}}\hat{p}_{i}^{\mathcal{T}}\log(\frac{p_{\bar{\tau}}^{\mathcal{T}}\hat{p}_{i}^{\mathcal{T}}}{p_{\bar{\tau}}^{\mathcal{S}}\hat{p}_{i}^{\mathcal{S}}})$$

$$=p_{\tau}^{\mathcal{T}}\log(\frac{p_{\tau}^{\mathcal{T}}}{p_{\tau}^{\mathcal{S}}})+p_{\bar{\tau}}^{\mathcal{T}}\sum_{i\in\mathcal{K}\backslash\{\tau\}}\hat{p}_{i}^{\mathcal{T}}\log(\frac{p_{\bar{\tau}}^{\mathcal{T}}}{p_{\bar{\tau}}^{\mathcal{S}}})$$

$$+p_{\bar{\tau}}^{\mathcal{T}}\sum_{i\in\mathcal{K}\backslash\{\tau\}}\hat{p}_{i}^{\mathcal{T}}\log(\frac{\hat{p}_{i}^{\mathcal{T}}}{\hat{p}_{i}^{\mathcal{S}}})$$ (5)

Since $p_{\bar{\tau}}^{\mathcal{T}}$, $p_{\bar{\tau}}^{\mathcal{S}}$ are independent to the class index i and $\sum_{i\in\mathcal{K}\backslash\{\tau\}}\hat{p}_{i}^{\mathcal{T}}=1$, we can simplify (2.2) to:

$$\mathcal{L}_{\text{KD}} =\underbrace{p_{\tau}^{\mathcal{T}}\log(\frac{p_{\tau}^{\mathcal{T}}}{p_{\tau}^{\mathcal{S}}})+p_{\bar{\tau}}^{\mathcal{T}}\log(\frac{p_{\bar{\tau}}^{\mathcal{T}}}{p_{\bar{\tau}}^{\mathcal{S}}})}_{D_{\text{KL}}(\mathbf{b}^{\mathcal{T}}\|\mathbf{b}^{\mathcal{S}})}+p_{\bar{\tau}}^{\mathcal{T}}\underbrace{\sum_{i\in\mathcal{K}\backslash\{\tau\}}\hat{p}_{i}^{\mathcal{T}}\log(\frac{\hat{p}_{i}^{\mathcal{T}}}{\hat{p}_{i}^{\mathcal{S}}})}_{D_{\text{KL}}(\mathbf{\hat{p}}^{\mathcal{T}}\|\mathbf{\hat{p}}^{\mathcal{S}})}$$ (6)

From (6), the conventional label-level KD can be re-formulated into the sum of two terms: 1) Target Speaker Knowledge Distillation (TSKD) loss $\mathcal{L}_{\text{TSKD}}$: the $D_{\text{KL}}$ over the binary classification probability $\mathbf{b}\in\mathbb{R}^{2}$ of the target speaker and all non-target speakers, and 2) Non-Target Speaker Knowledge Distillation (NSKD) loss $\mathcal{L}_{\text{NSKD}}$: the $D_{\text{KL}}$ of the multi-class classification probability $\mathbf{\hat{p}}\in\mathbb{R}^{K-1}$ between $K-1$ non-target speakers as shown in (7) and Fig.2.

$$D_{\text{KL}}(\mathbf{p}^{\mathcal{T}}\|\mathbf{p}^{\mathcal{S}})=\underbrace{D_{\text{KL}}(\mathbf{b}^{\mathcal{T}}\|\mathbf{b}^{\mathcal{S}})}_{\mathcal{L}_{\text{TSKD}}}+(1-p_{\tau}^{\mathcal{T}})\underbrace{D_{\text{KL}}(\mathbf{\hat{p}}^{\mathcal{T}}\|\mathbf{\hat{p}}^{\mathcal{S}})}_{\mathcal{L}_{\text{NSKD}}}$$ (7)

From the above equation, when the teacher model predicts the target speaker accurately, a large value of $p_{\tau}^{\mathcal{T}}$ results in a smaller $(1-p_{\tau}^{\mathcal{T}})$, which leads to the suppression of $\mathcal{L}_{\text{NSKD}}$. This could potentially hinder the distillation of knowledge from non-target speakers in the label-level KD method.

2.3 Decoupled Knowledge Distillation with an emphasis on non-target speaker knowledge

In Section 2.1, it was demonstrated that leveraging more non-target speaker knowledge can enhance the performance of ASV models. In other words, $\mathcal{L}_{\text{NSKD}}$ may play a crucial role in the knowledge transfer from the teacher to student models. Decoupled Knowledge Distillation (DKD) [13] proposed a modification to remove the dependency factor $(1-p_{\tau}^{\mathcal{T}})$ in (7) by introducing hyperparameters to balance the $\mathcal{L}_{\text{TSKD}}$ and $\mathcal{L}_{\text{NSKD}}$. However, to place a greater emphasis on $\mathcal{L}_{\text{NSKD}}$, we adjusted the original DKD method by simply replacing $(1-p_{\tau}^{\mathcal{T}})$ with the hyperparameter $\gamma$ in the following manner:

$$\mathcal{L}_{\text{DKD}}=\mathcal{L}_{\text{TSKD}}+\gamma\mathcal{L}_{\text{NSKD}}$$ (8)

Finally, the DKD loss $\mathcal{L}_{\text{DKD}}$ is combined with the classification loss to optimize the student model. Fig. 2 illustrates the comparison between DKD with an emphasis on non-target speaker knowledge, embedding-level, and conventional label-level knowledge distillation.

Table 1: Results on the VoxCeleb1 test sets. $COS$ and $KLD$ denote embedding-level and conventional label-level KD

System	Params (M)	FLOPs (G)	Distillation Method	EER (%) / minDCF
				Vox1-O	Vox1-E	Vox1-H
Teacher model WavLM-TDNN [14]	316.62	$\sim$26	-	0.383 /  -	0.480 /  -	0.986 /  -
TDNN-based Student model x-vector [15]	4.61	0.53	-	1.835 /  -	1.822 /  -	3.110 /  -
			$COS$	1.760 / 0.189	1.742 / 0.185	2.879 / 0.255
			$KLD$	1.585 / 0.171	1.589 / 0.171	2.704 / 0.244
			Ours	1.319 / 0.160	1.388 / 0.155	2.440 / 0.226
CNN-based Student model ResNet34 [16]	6.64	4.55	-	0.862 / 0.089	1.035 / 0.112	1.827 / 0.176
			$COS$	0.829 / 0.088	0.943 / 0.107	1.694 / 0.164
			$KLD$	0.771 / 0.086	0.939 / 0.103	1.728 / 0.166
			Ours	0.766 / 0.101	0.850 / 0.096	1.615 / 0.161
D-TDNN-based Student model CAM++[17]	7.18	1.72	-	0.718 /  -	0.879 /  -	1.735 /  -
			$COS$	0.713 / 0.118	0.901 / 0.108	1.768 / 0.182
			$KLD$	0.633 / 0.101	0.790 / 0.093	1.572 / 0.159
			Ours	0.590 / 0.118	0.735 / 0.085	1.494 / 0.148

3 Experiments Setup

3.1 Dataset

We utilized the VoxCeleb2 dev dataset [18] for training and evaluated the performance on three test trials, Vox1-O, Vox1-E, and Vox1-H. During training, we applied data augmentation using the MUSAN noise corpus [19] and RIRs reverberation [20], with a probability of 0.6.

3.2 Model

The teacher model is the SSL-based ASV system [14] combining WavLM Large and ECAPA-TDNN [21]. On the other hand, we utilized various network architectures for our student models including TDNN-based x-vector [15], CNN-based ResNet-34 [16], and D-TDNN-based CAM++ [17].

3.3 Training and Evaluation

During the training, each audio sample was randomly cropped to a 2-second segment, then 80-dimensional Fbank features were extracted using a frame length of 25 ms and a frameshift of 10 ms. For the classification loss function, we employed the AAM-softmax [22] with a scale of 32 and a margin scheduler. In the proposed KD method, from the ablation study in Section 4.2, the value $\gamma$ in (8) is set to 2.0 in all the remaining experiments. For evaluation, speaker embeddings were scored using cosine similarity and score normalization. Performance is reported on two metrics: Equal Error Rate (EER) and the minimum of the normalized detection cost function (MinDCF) with $P_{target}=0.01$ and $C_{fa}=C_{miss}=1$. All experiments are conducted using Wespeaker toolkit [23].

4 Results and Analysis

4.1 Results of the proposed method

Table 1 presents a comparison of the performance of teacher and student models trained solely with classification loss, along with the results of different knowledge distillation methods. Although both embedding-level and conventional label-level KD methods outperform student networks trained solely with classification loss, the improvement remains limited. Under the limited number of parameters and floating-point operations (FLOPs), the smallest student model x-vector using DKD emphasizing non-target speakers exhibits the largest improvement of 28.12% in Vox1-O EER, compared to its baseline trained with classification loss only. Moreover, our proposed method enables the state-of-the-art CAM++ model to further boost its performance with an EER of 0.590%, while the model’s size and FLOPs are respectively 97.73% and 93.39% smaller than the teacher model. Lastly, all three student networks trained using our proposed method have a better result than the embedding-level and conventional label-level KD methods, especially in challenging sets like Vox1-E and Vox1-H. This indicates that DKD emphasizing non-target speaker probabilities effectively improves the performance of student models.

4.2 Ablation Study: The impact of $\mathcal{L}_{\text{NSKD}}$

We conducted an ablation study on the hyperparameters $\gamma$ in the DKD formula to show how the robustness of the proposed method varies. To save computational cost, we solely present the results of the student model x-vector, which are summarized in Table 2. When emphasizing the $\mathcal{L}_{\text{NSKD}}$ with non-zero values of $\gamma$, all the results exceed the performance of $\gamma=1-p_{\tau}^{\mathcal{T}}$, which is equivalent to the result of the conventional label-level knowledge distillation method. It is observed that removing $NSKD$ by assigning $\gamma=0$ obtains a worse result than the conventional label-level knowledge distillation. In alignment with the findings from Section 2.1, an increasing value of $\gamma$ leads to better performance, implying the increased significance of $\mathcal{L}_{\text{NSKD}}$. Notably, the best hyperparameter configuration of $\gamma=2$ achieved an average of 13% improvements in EER compared to the conventional label-level knowledge distillation.

Table 2: Results of x-vector using different $\gamma$ values in Eq.( 8)

$\gamma$	EER (%) / minDCF
	Vox1-O	Vox1-E	Vox1-H
$1-p_{\tau}^{\mathcal{T}}$	1.585 / 0.171	1.589 / 0.171	2.704 / 0.244
$0.0$	1.622 / 0.152	1.646 / 0.175	2.786 / 0.252
$1.0$	1.463 / 0.166	1.452 / 0.155	2.520 / 0.225
$2.0$	1.319 / 0.160	1.388 / 0.155	2.440 / 0.226
$4.0$	1.361 / 0.143	1.415 / 0.156	2.511 / 0.229

5 Conclusion

This paper has shown the benefit of leveraging non-target speakers for training automatic speaker verification models. Based on this finding, we modified the conventional label-level KD to emphasize the classification probabilities of non-target speakers, which involves splitting and amplifying the non-target speaker’s probabilities during the knowledge distillation process. Experimental results on the VoxCeleb test sets show an average of 13.67% improvement in EER of the proposed method compared to other knowledge distillation methods across three different architecture student models.

6 Acknowledgement

The computational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore (https://www.nscc.sg).