Emotion and Intent Joint Understanding in Multimodal Conversation: A Benchmarking Dataset

Table 1 provides an overview and summary of some of the most important datasets related to our work. To highlight the significance of our MC-EIU dataset, we conduct a detailed comparison of these datasets based on annotation, modality, language diversity, and accessibility.

Annotation Diversity: The EmpatheticDialogues Rashkin et al. (2019), EmotionLines Chen et al. (2018), DailyDialog Li et al. (2017), EmoV-DB Adigwe et al. (2018), IEMOCAP, MELD, MEISD, and M3ED are widely used for emotion recognition, but these datasets primarily focus on emotion labels and lack annotation diversity as they do not include intent labels. Similarly, the Behance Intent Discovery dataset Maharana et al. (2022) and MIntRec Zhang et al. (2022a) are limited in annotation diversity because they are only used for intent recognition and do not provide emotion labels. In comparison, the Twitter Customer SupportHerzig et al. (2016), ED Welivita and Pu (2020), OSED, and EmoInt-MD contain the advantage of including both emotion and intent labels.

Modality Diversity: The EmpatheticDialogues, Emotionlines, DailyDialog, Twitter Customer Support Herzig et al. (2016), ED Welivita and Pu (2020), and OSED offer a vast collection of conversational textual data. The EmoV-DB is built with speech data in five emotion categories. However, these datasets only provide information on a single modality and do not include multiple modalities simultaneously. Consequently, the absence of support for multiple modalities limits the modality diversity of these datasets. Different from them, the IEMOCAP, MELD, MEISD, M3ED, Behance Intent Discovery dataset Maharana et al. (2022), MIntRec Zhang et al. (2022a), and EmoInt-MD offer a broader range of modalities, including textual, acoustic, and visual data.

Language Diversity: Most datasets in Table 1 mainly comprise English data, while the M3ED dataset solely consists of Mandarin data. Consequently, these datasets exhibit limitations in terms of language diversity. Only EmoV-DB stands out as it includes both English and French data.

Accessibility: All the datasets listed in Table 1 are accessible except for MEISD and Twitter. It is worth mentioning that the dataset from the provided open-source address in EmoInt-MD is currently unavailable, which poses a challenge in accessing the data.

In a nutshell, there is a lack of available data on the aforementioned four aspects of the MC-EIU task. Our dataset addresses these gaps and provides data that fulfill all four aspects.

Multi-task Prediction: Multi-task learning aims to leverage the valuable information in multiple related tasks to enhance generalization performance across all tasks Yang and Hospedales (2017), which has a wide range of applications in fields such as Emotion Recognition Hazarika et al. (2020), Object Detection Li et al. (2023), Conversational Speech Synthesis Liu et al. (2023), etc. Multi-task learning methods can be broadly classified into Hard Parameter Sharing (HPS) and Soft Parameter Sharing (SPS) Zhang et al. (2022b); Pahari and Shimada (2022). HPS serves as the foundational deep neural network that is shared across different tasks, while each task utilizes its own distinct top layer Zhang et al. (2022b); Hazarika et al. (2020). SPS allows the model to share some parameters between different tasks while preserving task-specific parameters Pahari and Shimada (2022).

It should be noted that compared to HPS, the advantage of SPS lies in the ability of the network to utilize shared features among different tasks by sharing bottom-layer parameters. It allows for better utilization of interactive information between tasks Pahari and Shimada (2022).

Emotion-Intent Interaction: Previous works confirm that there is a strong interaction between emotion and intent Welivita and Pu (2020); Welivita et al. (2020); Singh et al. (2022); Deng et al. (2023). For instance, Singh et al. (2022) emphasizes that the emotions of the speaker can be influenced by particular intents in dialogues. To explore the interaction between emotion and intention, they built the EmoInt-Trans model that follows the HPS framework. It uses a MISA network Hazarika et al. (2020) as the backbone, where the emotion and intent prediction tasks share all the parameters of this backbone. However, the complex relationship between emotion and intent poses a challenge for HPS, as it may struggle to accommodate the distinct characteristics of each task. Note that the biggest difference between our EI² model and EmoInt-Trans is that we employ an SPS mode to learn the deep-level interactive information for emotion and intent.

To simulate the emotional conversation scenarios in real-world situations Zhao et al. (2022); Singh et al. (2022), we select emotional video clips from 3 English (716 episodes) TV series and 4 Chinese (119 episodes) TV series across genres such as family, romance, crime, etc.¹¹1More details are shown in the Appendix. All videos are accompanied by corresponding subtitle files. Afterward, we preprocess the data to obtain the desired format and filter out low-quality data that does not meet the criteria. Specifically, 1) We first design Regular Expression scripts to extract text transcription and the timestamps from the subtitle; 2) Then, we leverage VideoFileClip²²2https://moviepy-tburrows13.readthedocs.io/en/improve-docs/ref/VideoClip/VideoFileClip.html to partition the videos into multiple clips based on the timestamps; 3) At last, we engage crowd workers to pick out high-quality conversational segments from the same conversational scene. Note that we provide the following specific guidelines to the crowd workers based on relevant work Zhao et al. (2022): a) Conversational scenes should exclusively involve interaction between two speakers; b) The selected conversational scenes should be free from noise or special effects sounds to ensure video quality is not affected; c) Each conversational segment should encompass at least two rounds of interaction between the speakers to provide abundant context information; d) The text content should be accurate and align with the video clips.

Annotation Scheme: The annotation scheme consists of emotion, intent, and speaker annotation. For emotion annotation, we choose Ekman’s six basic emotions (happy, surprise, sad, disgust, anger, and fear), along with the neutral label, which is a 7-emotion annotation scheme widely used in previous works Zhao et al. (2022); Busso et al. (2008); Poria et al. (2019). For intent annotation, we follow Welivita and Pu (2020); Welivita et al. (2020) and annotate each utterance based on nine intents: questioning, agreeing, acknowledging, sympathizing, encouraging, consoling, suggesting, wishing, and neutral. For speaker annotation, we follow the scheme described in Zhao et al. (2022) and assign the labels “0” and “1” to annotate the speaker for each dialog. The above annotation scheme ensures consistency with real conversation scenarios Zhao et al. (2022); Poria et al. (2019).

We recruit postgraduate students from the Foreign Languages College as annotators for our project³³3Although these annotators’ native language is Chinese, they have passed the Test for English Majors Grade Eight (TEM-8), indicating their proficiency in English language skills, including listening, speaking, reading, and writing, comparable to native English speakers.. Prior to commencing the annotation process, all annotators undergo a training and assessment phase. Only those who obtain a passing score are selected to proceed to the data annotation stage. Annotators who do not meet the passing criteria are required to undergo additional training and reassessment until they successfully pass the assessment.

Annotation Process: We recruited 21 annotators and divided them into seven groups. Each group consists of three volunteers, and non-duplicate data is assigned to each group for annotation. This ensures that each conversation data is annotated by three volunteers. Each annotator can access the current utterance and history in the multimodal conversation, including text, audio, and video clips during the annotation process. The annotators are asked to select the appropriate emotion and intent labels from the annotation scheme after watching the video clip. Simultaneously, annotators are required to assign speaker labels to each utterance in the dialog based on the speaking order. Suppose annotators encounter difficulty in assigning a specific category to a video clip. In that case, they categorize the utterance as other, and the corresponding dialog containing this utterance will not be included in our dataset.

We employ a majority voting strategy on all utterance annotations to derive the final emotion and intent labels. If at least two annotators provide the same annotation for one utterance, it is considered the final label. If all three annotators have different annotations, an additional emotion expert is consulted to confirm the final label.

The statistics of the MC-EIU dataset are presented in Table 2. It comprises a total of 56,012 utterances, including 4,013 conversations in English and 957 conversations in Mandarin. The total duration of the MC-EIU set is 53.06 hours. We partition the dataset into the train, valid, and test sets with a ratio of 7:1:2 as stated in Zhao et al. (2022); Poria et al. (2019). To assess the reliability of our data annotation, we compute Fleiss’s Kappa ($\kappa$) Fleiss et al. (2013) for emotion and intent annotation independently, as presented in Table 3. Note that $\kappa$ of our dataset is higher than or comparable to that of previous studies, which serves as strong evidence for the reliability and accuracy of our annotation.

We further explore the correlation between emotion and intent. We created two 7x9 two-dimensional matrices, with each element representing the number of samples in the dataset for each “emotion-intent” pair. Using the sample count as the radius, we visualize circles at the corresponding matrix positions. The resulting visualization is depicted in Figure 1. Larger circles indicate more samples and higher correlation. It can be observed that emotions and intents are not strictly one-to-one correspondences, and different intents vary in their influence on specific emotions, and vice versa. Take Figure.1 (a) as an example, compared to “Hap-Sym", “Hap-Agr" occurs at a higher frequency, indicating that “Agreeing" is more likely to drive the expression of “Happy". Similarly, compared to “Dis-Que", “Sad-Que" has a higher occurrence rate, suggesting that “Questioning" is more closely associated with “Sad" compared to “Disgust". Additionally, we observe that the correlation between emotion and intent in the English dataset is more intricate compared to the Mandarin dataset. For example, the emotion "Sur" is associated with all intent categories in the English dataset, whereas in the Mandarin dataset, it is only linked to 6 intent categories ("Que", "Agr", "Con", "Sug", "Wis", and "Neu"). Such a complex relationship introduces a significant challenge to the MC-EIU task. We make more details of data annotation finalization in the Appendix.

To model the multimodal dialog history and the deep-level interaction between emotion and intent, we propose an EI² network as shown in Figure 2. The network consists of the following components: 1) Emotion & Intent Encoders aim to generate the multimodal emotion and intent representations for current utterance; 2) the Multimodal History Encoder is responsible for capturing the multimodal contextual semantic information from the multimodal history; 3) the Emotion-Intent Interaction Encoder is proposed to learn the deep interaction between emotions and intents in a conversation; 4) Emotion & Intent Classifiers aim to make predictions based on the emotion-intent interactive information.

Emotion & Intent Encoders: In contrast to previous work Singh et al. (2022) that extracts general semantic representations from utterances, we add separate feature encoders for emotion and intent understanding to extract more explicit emotion and intent features.

The emotion and intent encoders share a similar structure, which includes a Visual Encoder, a Textual Encoder, an Acoustic Encoder, and a Transformer fusion network. Assume that $u_{n}^{a}$, $u_{n}^{t}$, $u_{n}^{v}$ represent the acoustic, textual, and visual features of the $n$-th utterance $u_{n}$, the multimodal emotion representation $f^{*}_{e}$ and intent representation $f^{*}_{i}$ can be expressed as:

where $s\in\{e,i\}$, $F^{a}_{s}$ is the Acoustic Encoder based on LSTM Sak et al. (2014) and max-pooling, $F^{v}_{s}$ is the Visual Encoder that has the same structure as $F^{a}_{s}$, $F^{t}_{s}$ is the TextCNN-based Text Encoder Kim (2014) adopted, and the $F^{*}_{s}$ means the Transformer fusion network.

It is worth noting that the modules with fire symbols in Figure 2 indicate the need for pre-training our intent and emotion encoders. Further details will be provided in Section 4.2.

Multimodal History Encoder: Different from EmoInt-Trans Singh et al. (2022), which solely models the context information of adjacent utterances, our Multimodal History Encoder takes into account a broader range of historical information. For the current utterance $u_{n}$, the multimodal conversational history information $f_{h}$ can be represented as:

where $m\in\{v,a,t\}$, $F_{h}^{m}$ denotes the GRU-based History Encoder Lee et al. (2023). Subsequently, $f_{h}$ is independently fused with $f_{e}$ and $f_{i}$ to obtain the robust emotion feature $f_{e}$ and intent feature $f_{i}$: $f_{s}=f^{*}_{s}+f_{h}$.

Emotion-Intent Interaction Encoder: The approach of simply sharing the hidden state is not sufficient to achieve explicit information transfer between two tasks. Therefore, we propose the Emotion-Intent Interaction Encoder to learn the deep interactive information between them by considering their complex correlations. As shown in Figure 2, the Emotion-Intent Interaction Encoder contains two branches for emotion or intent prediction, where each branch consists of Binary Correlation Attention, Triple Interaction Attention, and Gate Regulator.

Binary Correlation Attention first employs cross-attention to learn the mutual influence between emotions and intentions, which can also be referred to as the binary correlation between the two tasks. Specifically, we first adopt linear projection to map the $f_{e}$ and $f_{i}$ to generate the corresponding $Q$, $K$, and $V$. Then, the attention mechanism is used to extract the correlation between $f_{e}$ and $f_{i}$:

where $\gamma,\beta\in\{e,i\}$, if $\gamma$ denotes $e$, then, $\beta$ denotes $i$, and vice versa.

Triple Interaction Attention further explores the deep interactive information between the two tasks by integrating the binary correlation and task-specific information from each branch to obtain a more comprehensive and in-depth task interaction feature. Inspired by Jiang et al. (2023), we propose a triple interaction attention mechanism to compute cascaded interaction feature representations. It takes the outputs of binary correlation attention, along with the $f_{e}$ and $f_{i}$, as inputs; the deep interactive information $f_{\gamma-\beta-\gamma}$ of two tasks can be obtained by:

Gate Regulator: It utilizes a gating mechanism to automatically learn the weights of the binary correlation features and ternary interaction features, to model the potential impact of the correlation between different emotion-intent pairs in Figure 1 on the interaction features. This allows us to adjust the contribution of the binary correlation to the final recognition of emotions or intents. Specifically, the gate regulator first adds $f_{\gamma-\beta-\gamma}$ and $f_{\gamma-\beta}$ together. Afterward, the Sigmoid function is applied to obtain a control gate value between them. Lastly, $f_{\gamma-\beta}$ is multiplied by the control gate value to effectively adjust the weight of the correlation information:

where $g^{*}_{\gamma}$ represents the final output of Emotion-Intent Interaction Encoder. With the collaboration of these three components, our EI² model captures deep-level interaction between emotion and intent.

Emotion & Intent Classifiers: To preserve the specific information of the emotion and intent representations while incorporating the deep interactive information, we perform the residual connection to combine the $g^{*}$ with the emotion feature $f_{e}$ and the intent feature $f_{i}$ separately before making the final prediction. At last, the Emotion and Intent Classifiers take the results of the residual connection to predict the final emotion and intent categories, respectively:

We first pre-train the emotion and intent encoders to ensure the effective extraction of emotion and intent information. given the presence of category imbalance in the dataset (shown in the Appendix), we employ the Focal Loss (FL) Lin et al. (2017) as the loss function $\mathcal{L}_{pre}$ in the pre-training phase to constrain the prediction of the emotion and intent to be close to the Ground Truth $\hat{P}_{e}$ and $\hat{P}_{i}$ of emotion and intent and to improve the model’s ability to focus on the categories of a small sample:

where $CLS_{e}$ and $CLS_{i}$ represent the emotion and intent classifiers during the pre-training stage.

During the training phase of EI², we initialize the emotion and intent encoders with pre-trained weights. These encoders are then further updated during the EI² training. At last, we adopt a joint training approach and also utilize FL loss as the final loss function $\mathcal{L}_{total}$ for the MC-EIU task.

To validate our MC-EIU datasets and the proposed EI² network, we develop four MC-EIU systems based on state-of-the-art models: bc-LSTM Poria et al. (2017b) is widely employed in conversational sentiment recognition tasks. It adopts the bi-directional LSTM and multi-head attention mechanism, enabling the model to capture both contextual information and abundant semantic information within utterances. We add additional emotion and intent classifiers to make bc-LSTM support multi-task prediction. MMIN Zhao et al. (2021) incorporates a cascade residual autocoder network and cyclic consistency constraints to learning the robust multimodal joint representation for multimodal emotion recognition. We expand the MMIN model by incorporating an intent classifier, enabling it to recognize emotion and intent simultaneously. MISA Hazarika et al. (2020) introduces a modality-invariant and -specific feature representation learning mechanism to achieve robust multimodal emotion recognition. We also incorporate an additional intent classifier for MISA to support the MC-EIU task. COGMEN Joshi et al. (2022) is a multimodal sentiment analysis system based on the contextual Graph Neural Network (GNN) for predicting the sentiment of each speaker per utterance in a conversation. We integrated an extra intent classifier following the feature fusion layer to enable concurrent recognition of sentiment and intent information. EmoInt-Trans Singh et al. (2022) is the state-of-the-art baseline for the MC-EIU task. It adopts MISA Hazarika et al. (2020) as the backbone, which utilizes adjacent sentences as contextual information.

We choose the Weighted Average F-score (WAF) Poria et al. (2019) as the evaluation metrics. More implementation details are shown in the Appendix.

We first compare the recognition performance of our EI² and the baselines in both English and Mandarin. As shown in Table 4, we observe that EI² achieves the highest WAF in both emotion and intent recognition in both languages, clearly outperforming the baseline systems. For example, the WAF of emotion and intent for EI² in English are 42.09% and 45.53%, respectively. In Mandarin, the scores are 55.08% for emotion and 61.63% for intent. These results demonstrate the effectiveness of our method. By learning from multimodal dialog history and employing soft parameter sharing to capture the interaction between emotion and intent, our model achieves impressive joint recognition performance for emotion and intent.

We design three different ablation experiments, that are module ablation, task ablation, and modality ablation to further validate the components of EI² system.

Module Ablation: We conducted ablation experiments on the multimodal history encoder, emotion-intent interaction encoder, gate regulator, and the pretraining strategy for emotion&intent encoders: 1) w/o History: we remove the multimodal history encoder of EI²; 2) w/o Interaction: the features outputted by the emotion and intent encoders are directly used for final prediction; 3) w/o Gate: we remove the gate regulator; 4) w/o FL: we replace Focal Loss referenced in Section 4.2 with the Cross-Entropy Loss; 5)w/o Pre-training: we randomly initialize the parameters of the Emotion and Intent Encoders during the EI² training.

As shown in Table 4, EI² outperforms all the ablation models on both datasets. This provides compelling evidence for the following assertions: 1) The inclusion of conversation history helps analyze the current speaker’s emotions and intent states; 2) By modeling and integrating deep interactive information, the joint understanding task achieves improved performance, and further demonstrates the significance of modeling the interactive information between intent and emotion for joint understanding; 3) After reducing the gating mechanism, the model’s performance experienced a significant decline, highlighting the crucial role of the gate regulator in adjusting the weights of binary correlation to the final recognition of emotion or intent; 4) The pre-training of emotion and intent encoders resulted in a notable performance improvement, suggesting their capacity to learn substantial semantic information and distinct features for emotions and intents through pre-training.

Task Ablation: To investigate the effectiveness of the joint recognition task further, we conduct independent emotion and intent recognition tasks on the MC-EIU dataset. The results of these separate tasks are presented in Table 5. We can observe that the joint task outperforms the recognition results of the single task. This further substantiates the strong correlation between intent and emotion and that the joint task can improve prediction accuracy through the deep interaction between emotion and intent.

Modality Ablation: We conduct modality ablation experiments to verify the impact of different modal data, as depicted in Table 6. In the unimodal experiments, we find that the textual modality exhibits the highest overall performance, while the visual modality demonstrates the poorest performance. This aligns with previous research Fu et al. (2022), which concludes that the textual modality contains the most comprehensive semantic information, while the visual modality is comparatively limited in this regard. Additionally, the model’s performance improves as the number of modalities fused increases, with the highest performance achieved when all three modalities are combined. This highlights the effectiveness of integrating complementary information from different modalities.