-
Audio-Visual Compound Expression Recognition Method based on Late Modality Fusion and Rule-based Decision
Authors:
Elena Ryumina,
Maxim Markitantov,
Dmitry Ryumin,
Heysem Kaya,
Alexey Karpov
Abstract:
This paper presents the results of the SUN team for the Compound Expressions Recognition Challenge of the 6th ABAW Competition. We propose a novel audio-visual method for compound expression recognition. Our method relies on emotion recognition models that fuse modalities at the emotion probability level, while decisions regarding the prediction of compound expressions are based on predefined rule…
▽ More
This paper presents the results of the SUN team for the Compound Expressions Recognition Challenge of the 6th ABAW Competition. We propose a novel audio-visual method for compound expression recognition. Our method relies on emotion recognition models that fuse modalities at the emotion probability level, while decisions regarding the prediction of compound expressions are based on predefined rules. Notably, our method does not use any training data specific to the target task. Thus, the problem is a zero-shot classification task. The method is evaluated in multi-corpus training and cross-corpus validation setups. Using our proposed method is achieved an F1-score value equals to 22.01% on the C-EXPR-DB test subset. Our findings from the challenge demonstrate that the proposed method can potentially form a basis for develo** intelligent tools for annotating audio-visual data in the context of human's basic and compound emotions.
△ Less
Submitted 29 March, 2024; v1 submitted 19 March, 2024;
originally announced March 2024.
-
SUN Team's Contribution to ABAW 2024 Competition: Audio-visual Valence-Arousal Estimation and Expression Recognition
Authors:
Denis Dresvyanskiy,
Maxim Markitantov,
Jiawei Yu,
Peitong Li,
Heysem Kaya,
Alexey Karpov
Abstract:
As emotions play a central role in human communication, automatic emotion recognition has attracted increasing attention in the last two decades. While multimodal systems enjoy high performances on lab-controlled data, they are still far from providing ecological validity on non-lab-controlled, namely 'in-the-wild' data. This work investigates audiovisual deep learning approaches for emotion recog…
▽ More
As emotions play a central role in human communication, automatic emotion recognition has attracted increasing attention in the last two decades. While multimodal systems enjoy high performances on lab-controlled data, they are still far from providing ecological validity on non-lab-controlled, namely 'in-the-wild' data. This work investigates audiovisual deep learning approaches for emotion recognition in-the-wild problem. We particularly explore the effectiveness of architectures based on fine-tuned Convolutional Neural Networks (CNN) and Public Dimensional Emotion Model (PDEM), for video and audio modality, respectively. We compare alternative temporal modeling and fusion strategies using the embeddings from these multi-stage trained modality-specific Deep Neural Networks (DNN). We report results on the AffWild2 dataset under Affective Behavior Analysis in-the-Wild 2024 (ABAW'24) challenge protocol.
△ Less
Submitted 19 March, 2024;
originally announced March 2024.
-
An Audio-Video Deep and Transfer Learning Framework for Multimodal Emotion Recognition in the wild
Authors:
Denis Dresvyanskiy,
Elena Ryumina,
Heysem Kaya,
Maxim Markitantov,
Alexey Karpov,
Wolfgang Minker
Abstract:
In this paper, we present our contribution to ABAW facial expression challenge. We report the proposed system and the official challenge results adhering to the challenge protocol. Using end-to-end deep learning and benefiting from transfer learning approaches, we reached a test set challenge performance measure of 42.10%.
In this paper, we present our contribution to ABAW facial expression challenge. We report the proposed system and the official challenge results adhering to the challenge protocol. Using end-to-end deep learning and benefiting from transfer learning approaches, we reached a test set challenge performance measure of 42.10%.
△ Less
Submitted 2 November, 2020; v1 submitted 7 October, 2020;
originally announced October 2020.
