Sound Event Detection and Localization with Distance Estimation

Abstract

Sound Event Detection and Localization (SELD) is a combined task of identifying sound events and their corresponding direction-of-arrival (DOA). While this task has numerous applications and has been extensively researched in recent years, it fails to provide full information about the sound source position. In this paper, we overcome this problem by extending the task to Sound Event Detection, Localization with Distance Estimation (3D SELD). We study two ways of integrating distance estimation within the SELD core - a multi-task approach, in which the problem is tackled by a separate model output, and a single-task approach obtained by extending the multi-ACCDOA method to include distance information. We investigate both methods for the Ambisonic and binaural versions of STARSS23: Sony-TAU Realistic Spatial Soundscapes 2023. Moreover, our study involves experiments on the loss function related to the distance estimation part. Our results show that it is possible to perform 3D SELD without any degradation of performance in sound event detection and DOA estimation.

I Introduction

Computational Auditory Scene Analysis (CASA) has emerged as a prominent area of study in recent years [1]. The automated examination of audio content holds substantial potential for diverse practical applications, including speech recognition [2], autonomous robots [3], surveillance systems [4], and support systems for the hearing-impaired [5]. CASA encompasses a spectrum of audio-related tasks, ranging from acoustic scene classification [6] and audio tagging [7] to sound source localization [8] and sound event detection [9]. Most tasks are currently approached by Deep Neural Network (DNN) models.

Although existing research has predominantly focused on individual tasks, an evolutionary progression towards the development of complex scene analysis systems involves the creation of models capable of simultaneously addressing multiple objectives. This progressive approach has been observed in recent research, as exemplified by the exploration of joint sound event detection and localization (SELD) [10]. SELD combines the task of identifying the temporal activities of sound events, altogether with their direction of arrival (DOA) and textual label. However, this approach does not take advantage of full spatial information by limiting it to the DOA only. In many cases, performing Sound Distance Estimation (SDE) would be also important to obtain the explicit position of the sound source in space.

Research on DNN-based SELD has shown multiple ways of solving the problem of matching two tasks for different scenarios. In [10], the authors proposed a Convolutional Recurrent Neural Network (CRNN) with a two branch solution, in which SED and DOA estimation are solved with independent classwise outputs. To allow for detection of multiple events at the same time, a track-wise output has been proposed in [11, 12]. The activity-coupled Cartesian DOA (ACCDOA) scales the Cartesian DOA vector by its corresponding event activity, overcoming the need for a multi-task approach [13]. Finally, the multi-ACCDOA method takes advantage of the trackwise approach and the ACCDOA format, allowing for independent detection of same-class events [14].

Regarding SDE, studies on DNN-based approaches have been limited mostly to the binaural format. Most of them use a classification approach, in which the distance is expressed as a finite set of pre-defined distances in the close area up to 4 meters [15, 16]. In [17], the authors studied multiple loss functions to perform distance estimation with an activity detection branch for a tetrahedral microphone array. Few studies investigated the performance of distance estimation in conjunction with DOA estimation [18, 19, 20], however no approach has been made to merge SDE with SELD.

In this paper, we investigate the joint task of Sound Event Detection, Localization and Distance Estimation. We study two ways of performing all three tasks jointly. First, we examine a multi-task approach, in which two separate branches are responsible for SELD and SDE. Second, we propose the multi activity-coupled Cartesian Distance and DOA (multi-ACCDDOA) method, which is an an extension of the known multi-ACCDOA format, by including the distance in the estimated vector. For both approaches, we study the influence of several loss functions to determine which is the most suitable for the joint task. Experiments are conducted for both First Order Ambisonics (FOA) and binaural recordings to investigate the potential performance of the task in a more limited audio format. To the authors’ knowledge, this is the first study investigating joint modelling of all three tasks.

II Method

II-A Features

TABLE I: Input parameters for the models.

Audio data format	CH	T	F	P
Ambisonics	7	250	64	[4, 4, 2]
Binaural	4	250	512	[8, 8, 4]

A feature input matrix of shape $CH\times T\times F$ is fed to the model, where $CH$, $T$ and $F$ stand for the number of channels, time sequence length in frames and number of features respectively. The set of features utilized to train the models depends on the audio format under investigation as summarized in Table I. In this paper, we study the Ambisonic and binaural formats. Each file is split into clips of length $T=250$. A complex spectrogram of the signal is obtained using a Short-Time Fourier Transform (STFT) with a Hamming window of length 40 ms and 50% overlap. This results in $F=512$ frequency bins.

For Ambisonics, we accumulate the magnitude spectrograms from 4 channels into $F=64$ mel energies. In order to explore spatial cues, we extract 3 intensity vector matrices as in [21], accumulated along the mel frequencies to fit the required number of features. The overall input matrix sums up to $CH=7$ feature channels.

For the binaural format, we extract the mean magnitude spectrogram from both binaural channels. To represent spatial information about the signal, we extract sines and cosines of Interaural Phase Differences (IPD), which provide a smooth representation of phase values and avoid phase wrap**. This feature has been shown to perform successfully in binaural DOA estimation and SDE [20, 22]. On top of that, we use ILDs, which constitute another major binaural cue that becomes important above 1.5 kHz. This set of features results in $CH=4$ feature channels that are fed to the model.

II-B Model

Figure 1: Architecture of the deep neural network.

We employ a convolutional recurrent neural network (CRNN) model type which is common for SELD. To perform 3D SELD, we modify the model outputs to contain the included distance estimation part. The architecture of the utilized model is depicted in Fig. 1. First, the feature input matrix is processed by three 2D convolutional blocks, each consisting of 128 filter kernels, batch normalization and max-pooling across the feature dimension. Additionally, the first layer involves pooling across the time dimension with the rate of 5. The pooling rate across the featue dimension depends on the utilized feature set and is meant to reduce the feature dimension to 4 in the last convolutional block. Hence, for Ambisonics the rates are equal to $P=[4,4,2]$, whereas for the binaural format it is $P=[8,8,4]$. Next, the feature maps are passed to two bi-directional gated recurrent units (GRUs) and two Multi-Head Attention layers, with 8 heads each.

The output of the model consists of $Q$ branches, where $Q$ depends on the number of performed tasks. Each branch is composed of two fully connected (FC) layers, the former containing 128 neurons, and the latter outputting $C_{q}$ values. The parameters of the last layers are determined by the utilized method of including SDE within the 3D SELD framework. Here, we propose two techniques:

I. Multi-ACCDDOA: we modify the single task multi-ACCDOA approach proposed in [14]. Compared with the former, we extend the 3-element DOA vector to include the distance estimate as well. For $N$ tracks, $C$ classes and $T$ frames, we define the output as $y_{nct}=[a_{nct}R_{nct},D_{nct}]$, where $n,c,t$ indicate the output track number, target class and time frame, $a_{nct}\in\{0,1\}$ stands for the detection activity, $R_{nct}\in\langle-1,1\rangle$ are to the DOA vectors and $D_{nct}\in\langle 0,\infty)$ corresponds to distance values. The dimensions hold the following characteristics: $\textbf{a, D}\in\mathbb{R}^{NxCxT},\textbf{R}\in\mathbb{R}^{3xNxCxT}$, and $||\textbf{R}_{nct}||=1$. We model up to $N=3$, hence $O_{1}=156$. The whole output is linear to contain the range of both DOA and distance values. The multi-ACCDDOA model is trained using Auxiliary Duplicating Permutation Invariant Training (ADPIT) as in [14]. The final loss function is defined as:

$$\mathcal{L}^{ADPIT}=\frac{1}{CT}\sum_{c}^{C}\sum_{t}^{T}\min_{\alpha\in Perm[ct]}l_{\alpha,ct}^{ACCDDOA},$$ (1)

$$l_{\alpha,ct}^{ACCDDOA}=\frac{1}{N}\sum_{n}^{N}\mathcal{L}(y_{\alpha,nct},\hat{y}_{\alpha,nct}),$$ (2)

where $\mathcal{L}(\cdot)$ is a loss function of choice, $\alpha$ is one possible track permutation and $Perm[ct]$ is the set of all possible permutations.

II. Multi-task (MT): here, the output is split into $Q=2$ branches. The first branch performs SELD using the classwise ACCDOA approach as described in [13]. The output is defined as $y_{1,ct}=a_{ct}R_{ct}$. Since the models estimate the DOA vector as $x,y,z$ Cartesian coordinates for each class and $C=13$, this results in $O_{1}=39$. The second branch is responsible for classwise SDE, where $y_{2,ct}=D_{ct}$ and $O_{2}=13$. The ACCDOA output is normalized with a tanh activation, whereas the distance branch uses a Rectified Linear Unit. The matrix dimensions are defined as follows: $\textbf{a, D}\in\mathbb{R}^{CxT},\textbf{R}\in\mathbb{R}^{3xCxT}$. The final loss is a sum of both branches:

$$\mathcal{L}^{MT}=\frac{1}{CT}\sum_{c}^{C}\sum_{t}^{T}(\mathcal{L}_{1}(y_{1,ct},\hat{y}_{1,ct})+\mathcal{L}_{2}(y_{2,ct},\hat{y}_{2,ct})),$$ (3)

where $\mathcal{L}_{1}(\cdot)$ and $\mathcal{L}_{2}(\cdot)$ are the losses of the both branches.

We note that the output of the MT approach does not allow for differentiating between overlap** sources of the same class, whereas the multi-ACCDDOA approach overcomes this problem. The output parameters are summarized in Table II. Models are implemented in PyTorch [23] and trained using the Adam optimizer for 250 epochs with 75 epochs of patience.

TABLE II: Output parameters for different models.

Method	Q	Oq	Output activation
Multi-task	2	[39, 13]	[tanh, ReLU]
Multi-ACCDDOA	1	156	linear

II-C Loss functions

In recent works on SELD using ACCDOA, the mean squared error (MSE) has been established as the most common loss function. However, the inclusion of the distance estimation part introduces a different value range, for which other loss functions might be more appropriate. The standard MSE function prioritizes sound sources which are further away from the origin, since large distances create a more significant error. In [17] the authors introduced a relative regressor function, which evens out the error across ground truth distances, therefore penalizing the SDE branch more fairly. Here, we investigate the proposed loss functions for a joint 3D SELD model. The investigated functions include:

• Mean Squared Error:

  $MSE=\frac{1}{M}\sum_{m=0}^{M-1}(y[m]-\hat{y}[m])^{2},$

• Mean Absolute Error:

  $MAE=\frac{1}{M}\sum_{m=0}^{M-1}|y[m]-\hat{y}[m]|,$

• Mean Square Percent Error:

  $MSPE=\frac{1}{M}\sum_{m=0}^{M-1}(\frac{y[m]-\hat{y}[m]}{\hat{y}[m]})^{2},$

• Mean Absolute Percent Error:

  $MAPE=\frac{1}{M}\sum_{m=0}^{M-1}|\frac{y[m]-\hat{y}[m]}{\hat{y}[m]}|,$

where $M$ stands for the number of estimated values, $y$ and $\hat{y}$ correspond to the predicted and ground truth values. For the MT model, we investigate all aforementioned loss functions for the SDE branch ($\mathcal{L}_{2}$). Since scaling the DOA vectors with distance values would put a larger weight to close sources, the ACCDOA branch ($\mathcal{L}_{1}$) keeps the MSE loss function for all scenarios. For similar reasons, the multi-ACCDDOA approach ($\mathcal{L}$) is tested only for the MSE and MAE losses.

III Experiments

III-A Data

For experiments, we use the Ambisonic audio-only version of the STARSS23 dataset. The dataset includes 7 hours and 22 minutes of real recordings, which is split intro training data (90 clips) and testing data (78 clips). There are 13 sound event classes present, including female speech, male speech, clap**, telephone, laughter, domestic sounds, footsteps, door, music, musical instrument, water tap, bell and knock. The scenarios include up to 3 overlap** sound sources. For more details, refer to the original paper [24]. In order to increase the amount of training data, we synthesized more mixtures using the data generator described in [25]. The amount of additional data sums to 1200 clips of one-minute mixtures with the same maximum polyphony level as the original dataset. The data was synthesized using sounds from FSD50k [26]. In order to provide experiments for the binaural format, we convert the whole dataset from Ambisonic to binaural using the spaudiopy library [27]. Decoding is performed via magnitude least squared matching to a measured set of Head Related Transfer Functions as described in [28].

III-B Evaluation metrics

To evaluate our models, we use the SELD metrics from the DCASE Challenge 2023 Task 3. These include the Error Rate (ER), $F_{1}$ score for SED, and the DOA error and the localization recall for localization. The detection metrics are location aware, i.e., the events are counted only if the assigned source falls within ±20° of the ground truth DOA, whereas localization metrics are calculated only for true positives. On top of the well established SELD metrics, we add the distance error, which is defined as the mean absolute error between ground truth and predicted distances. The metrics are calculated in one second segments using micro-averaging and the matching between ground truth and predictions is done via the Hungarian algorithm referring to the angular distance between sources. For more details, see [29].

III-C Results

TABLE III: Results obtained for Ambisonics.

Method	SELD loss	Dist. loss	ER	$F_{1}[\%]$	DOA error [°]	Recall $[\%]$	Dist. error [m]
Multi-task	MSE	MSE	0.63 [0.59, 0.67]	41.4 [36.58, 47.04]	22.5 [19.18, 25.58]	61.0 [57.46, 65.57]	0.95 [0.82, 1.05]
	MSE	MAE	0.64 [0.60, 0.68]	43.6 [39.18, 48.60]	21.6 [18.63, 24.36]	41.10 [37.36, 45.64]	0.93 [0.80 , 1.02]
	MSE	MSPE	0.63 [0.59, 0.68]	44.1 [38.92, 48.72]	23.2 [18.31, 28.13]	64.7 [61.64, 68.68]	0.89 [0.77 , 0.99]
	MSE	MAPE	0.65 [0.61, 0.68]	43.5 [38.97, 47.80]	22.0 [18.94 , 24.80]	64.5 [61.18, 68.72]	0.88 [0.75, 0.97]
Multi-ACCDDOA	MSE		0.65 [0.61, 0.70]	44.2 [39.45, 48.65]	22.9 [19.33 , 26.46]	68.4 [65.15, 72.33]	0.92 [0.80 , 1.01]
	MAE		0.86 [0.82, 0.91]	21.5 [13.98, 28.47]	17.7 [14.09 , 21.05]	19.1 [12.44, 24.90]	0.74 [0.54 , 0.93]

TABLE IV: Results obtained for binaural audio.

Method	SELD loss	Dist. loss	ER	$F_{1}[\%]$	DOA error [°]	Recall $[\%]$	Dist. error [m]
Multi-task	MSE	MSE	0.82 [0.79, 0.86]	20.0 [15.40, 24.40]	41.1 [34.63 , 47.77]	45.6 [41.89, 49.02]	1.02 [0.90 , 1.12]
	MSE	MAE	0.85 [0.81, 0.87]	16.5 [13.91, 20.29]	38.6 [32.40 , 43.28]	36.7 [33.61, 40.89]	1.04 [0.90 , 1.15]
	MSE	MSPE	0.85 [0.81, 0.89]	19.3 [15.70, 23.91]	38.9 [31.91 , 44.28]	38.9 [35.52, 43.56]	1.01 [0.87 , 1.12]
	MSE	MAPE	0.87 [0.84, 0.91]	18.5 [15.16, 22.23]	38.1 [32.77 , 42.45]	42.2 [38.59, 45.83]	0.98 [0.86 , 1.09]
Multi-ACCDDOA	MSE		0.87 [0.82, 0.91]	21.1 [17.38, 25.26]	39.7 [32.58 , 46.34]	48.0 [44.75, 51.25]	0.99 [0.88 , 1.09]
	MAE		0.97 [0.94, 0.99]	5.4 [2.46, 8.24]	44.5 [36.84 , 51.93]	16.3 [10.12, 21.72]	0.75 [0.59 , 0.90]f

Tables III and IV show the results obtained for the Ambisonic and binaural datasets, respectively. The results are reported for the testing set using jackknife estimation to obtain confidence intervals with significance level of 0.05.

As can be seen from most metrics, the performance of the models are significantly worse for the binaural dataset than for the Ambisonic dataset. For the multi-task approach, the error rate went up from the range of [0.63, 0.65] to [0.82, 0.87], whereas the $F_{1}$ score decrased from over 40% to the range of [16.5, 20.0]%. A similarly large effect can be seen for the localization metrics, where the DOA error ranges from 38.1° to 41.1° and the recall from 36.7% to 45.6%. This is a partly expected effect. Binaural recordings consist of only two channels as compared with four channels for FOA. Moreover, the cone of confusion effect and high directivity of the ears might impact the performance significantly. These problems can be largely overcome by utilizing a moving receiver as shown in [20, 22]. Interestingly, distance estimation has been affected to a much lesser extent than other tasks. For the MT approach, the distance error stays in the range between 0.98m to 1.04m, which is roughly a 10% increase from the Ambisonic results. These results indicate that efficient distance estimation can be achieved even with binaural recordings, which shows a potential for future research.

As can be seen for both audio formats, for the multi-task approach most differences between loss functions appear for the distance error. This is expected given the ACCDOA branch uses the same loss for all scenarios. For MSE and MAE, the Ambisonic model achieves a similar distance error of 0.95 m and 0.93 m, respectively. This is a reasonable performance compared with other metrics. Introducing the distance scaling brings the error further down, to 0.89m for MSPE and 0.88m for MAPE. Similarly, the binaural model achieves the lowest distance error of 0.98 m for MAPE. The SELD metrics show very limited influence of the distance loss on the training of the rest of the multi-task model. For FOA data, all scenarios achieve a fairly similar error rate between 0.63 and 0.65 and an $F_{1}$ score between 41.4% for MSE and 44.1% for MSPE. Interestingly, a higher impact can be seen for the localization metrics. Despite the DOA error staying between 21.6° and 23.2°, the recall varies between different scenarios. Compared with 61.0% for MSE, the MAE loss results in a recall of 41.1%, showing a decreased efficiency in estimated the correct number of sources. Also for binaural data, the MAE loss decreases the recall from 45.6% to 36.7%. However, for FOA the distance-scaled losses visibly improve the localization recall to 64.7% and 64.5% for MSPE and MAPE, respectively.

Larger differences between loss functions can be observed for the multi-ACCDDOA approach. The single task model trained with MSE achieved an ER and DOA error which are comparable with the multi-task approach. However, amongst all results obtained for the Ambisonic format, this model achieves the best $F_{1}$ score of 44.2% and recall of 68.4%. The distance error of 0.92m is similar to the one achieved for the multi-task with MSE. Similarly, the binaural model obtains the best $F_{1}$ score of 21.1% and recall of 48.0% using this method.

Unsurprisingly, changing the loss function to MAE affects all tasks to a much larger extent when using this method. First, the models achieves a distance error of 0.74m which is the best result across all experiments, showing that using the absolute error seems to be a better fit for estimating the distance. The loss affects positively the DOA error as well, for which the value goes down to 17.7°. However, detection metrics show a significant decrease of performance of the SED part - ER increases to 0.86, whereas the $F_{1}$ score goes down to 21.5%. Similarly, recall goes down to 19.1%, which is the worst result overall. For binaural data, the $F_{1}$ score drops to 5.4%, which is an unacceptable performance. Such large differences between MSE and MAE show a discrepancy between the SDE and SELD parts, for which different loss functions are optimal.

Across both datasets, best SELD performance is achieved for the multi-ACCDDOA approach using the MSE loss function. Regarding the fact that this method also allows for the detection of multiple overlap** of the same class, we consider this format superior to the multi-task approach. However, best distance error is achieved when using MAE loss. For future studies, we propose to investigate a mixed loss function, which would connect the benefits of using MSE for SELD and MAE for distance estimation. Alternatively, a different task definition might be used, combining the track-wise approach of multi-ACCDDOA with a multi-task output representation.

IV Conclusions

In this study, we investigate the joint task of Sound Event Detection, Localization and Distance Estimation. We propose two methods of solving this problem - using a multi-task classwise approach and using the multi-ACCDDOA method. The methods are investigated using several loss functions for an Ambisonic and a binaural dataset. Our experiments show best results when using a multi-ACCDDOA approach with the MSE loss function. However, we note a disparity between SELD and distance estimation, where the latter performs better when using a MAE loss. Future studies could propose a mixed loss function to further improve the results.