Dyadformer: A Multi-modal Transformer for Long-Range Modeling of Dyadic Interactions
Authors:
David Curto,
Albert Clapés,
Javier Selva,
Sorina Smeureanu,
Julio C. S. Jacques Junior,
David Gallardo-Pujol,
Georgina Guilera,
David Leiva,
Thomas B. Moeslund,
Sergio Escalera,
Cristina Palmero
Abstract:
Personality computing has become an emerging topic in computer vision, due to the wide range of applications it can be used for. However, most works on the topic have focused on analyzing the individual, even when applied to interaction scenarios, and for short periods of time. To address these limitations, we present the Dyadformer, a novel multi-modal multi-subject Transformer architecture to mo…
▽ More
Personality computing has become an emerging topic in computer vision, due to the wide range of applications it can be used for. However, most works on the topic have focused on analyzing the individual, even when applied to interaction scenarios, and for short periods of time. To address these limitations, we present the Dyadformer, a novel multi-modal multi-subject Transformer architecture to model individual and interpersonal features in dyadic interactions using variable time windows, thus allowing the capture of long-term interdependencies. Our proposed cross-subject layer allows the network to explicitly model interactions among subjects through attentional operations. This proof-of-concept approach shows how multi-modality and joint modeling of both interactants for longer periods of time helps to predict individual attributes. With Dyadformer, we improve state-of-the-art self-reported personality inference results on individual subjects on the UDIVA v0.5 dataset.
△ Less
Submitted 20 September, 2021;
originally announced September 2021.
Context-Aware Personality Inference in Dyadic Scenarios: Introducing the UDIVA Dataset
Authors:
Cristina Palmero,
Javier Selva,
Sorina Smeureanu,
Julio C. S. Jacques Junior,
Albert Clapés,
Alexa Moseguí,
Zejian Zhang,
David Gallardo,
Georgina Guilera,
David Leiva,
Sergio Escalera
Abstract:
This paper introduces UDIVA, a new non-acted dataset of face-to-face dyadic interactions, where interlocutors perform competitive and collaborative tasks with different behavior elicitation and cognitive workload. The dataset consists of 90.5 hours of dyadic interactions among 147 participants distributed in 188 sessions, recorded using multiple audiovisual and physiological sensors. Currently, it…
▽ More
This paper introduces UDIVA, a new non-acted dataset of face-to-face dyadic interactions, where interlocutors perform competitive and collaborative tasks with different behavior elicitation and cognitive workload. The dataset consists of 90.5 hours of dyadic interactions among 147 participants distributed in 188 sessions, recorded using multiple audiovisual and physiological sensors. Currently, it includes sociodemographic, self- and peer-reported personality, internal state, and relationship profiling from participants. As an initial analysis on UDIVA, we propose a transformer-based method for self-reported personality inference in dyadic scenarios, which uses audiovisual data and different sources of context from both interlocutors to regress a target person's personality traits. Preliminary results from an incremental study show consistent improvements when using all available context information.
△ Less
Submitted 28 December, 2020;
originally announced December 2020.
