-
HuMoT: Human Motion Representation using Topology-Agnostic Transformers for Character Animation Retargeting
Authors:
Lucas Mourot,
Ludovic Hoyet,
François Le Clerc,
Pierre Hellier
Abstract:
Motion retargeting is the long-standing problem in character animation that consists in transferring and adapting the motion of a source character to another target character. A typical application is the creation of motion sequences from off-the-shelf motions by transferring them onto new characters. Motion retargeting is also promising to increase interoperability of existing animation systems a…
▽ More
Motion retargeting is the long-standing problem in character animation that consists in transferring and adapting the motion of a source character to another target character. A typical application is the creation of motion sequences from off-the-shelf motions by transferring them onto new characters. Motion retargeting is also promising to increase interoperability of existing animation systems and motion databases, as they often differ in the structure of the skeleton(s) considered. Moreover, since the goal of motion retargeting is to abstract and transfer motion dynamics, effective solutions might provide expressive and powerful human motion models in which operations such as cleaning or editing are easier. In this article, we present a novel neural network architecture for retargeting that extracts an abstract representation of human motion agnostic to skeleton topology and morphology. Based on transformers, our model is able to encode and decode motion sequences with variable morphology and topology -- extending the current scope of retargeting -- while supporting skeleton topologies not seen during the training phase. More specifically, our model is structured as an autoencoder, and encoding and decoding are separately conditioned on skeleton templates to extract and control morphology and topology. Beyond motion retargeting, our model has many applications since our abstract representation is a convenient space to embed motion data from different sources. It may potentially be benefical to a number of data-driven methods, allowing them to combine scarce specialised motion datasets (e.g. with style or contact annotations) and larger general motion datasets, for improved performance and generalisation ability. Moreover, we show that our model can be useful for other applications beyond retargeting, including motion denoising and joint upsampling.
△ Less
Submitted 15 June, 2023; v1 submitted 30 May, 2023;
originally announced May 2023.
-
UnderPressure: Deep Learning for Foot Contact Detection, Ground Reaction Force Estimation and Footskate Cleanup
Authors:
Lucas Mourot,
Ludovic Hoyet,
François Le Clerc,
Pierre Hellier
Abstract:
Human motion synthesis and editing are essential to many applications like film post-production. However, they often introduce artefacts in motions, which can be detrimental to the perceived realism. In particular, footskating is a frequent and disturbing artefact requiring foot contacts knowledge to be cleaned up. Current approaches to obtain foot contact labels rely either on unreliable threshol…
▽ More
Human motion synthesis and editing are essential to many applications like film post-production. However, they often introduce artefacts in motions, which can be detrimental to the perceived realism. In particular, footskating is a frequent and disturbing artefact requiring foot contacts knowledge to be cleaned up. Current approaches to obtain foot contact labels rely either on unreliable threshold-based heuristics or on tedious manual annotation. In this article, we address foot contact label detection from motion with a deep learning. To this end, we first publicly release UnderPressure, a novel motion capture database labelled with pressure insoles data serving as reliable knowledge of foot contact with the ground. Then, we design and train a deep neural network to estimate ground reaction forces exerted on the feet from motion data and then derive accurate foot contact labels. The evaluation of our model shows that we significantly outperform heuristic approaches based on height and velocity thresholds and that our approach is much more robust on motion sequences suffering from perturbations like noise or footskate. We further propose a fully automatic workflow for footskate cleanup: foot contact labels are first derived from estimated ground reaction forces. Then, footskate is removed by solving foot constraints through an optimisation-based inverse kinematics (IK) approach that ensures consistency with the estimated ground reaction forces. Beyond footskate cleanup, both the database and the method we propose could help to improve many approaches based on foot contact labels or ground reaction forces, including inverse dynamics problems like motion reconstruction and learning of deep motion models in motion synthesis or character animation. Our implementation, pre-trained model as well as links to database can be found at https://github.com/InterDigitalInc/UnderPressure.
△ Less
Submitted 9 August, 2022;
originally announced August 2022.
-
A Survey on Deep Learning for Skeleton-Based Human Animation
Authors:
L. Mourot,
L. Hoyet,
F. Le Clerc,
François Schnitzler,
Pierre Hellier
Abstract:
Human character animation is often critical in entertainment content production, including video games, virtual reality or fiction films. To this end, deep neural networks drive most recent advances through deep learning and deep reinforcement learning. In this article, we propose a comprehensive survey on the state-of-the-art approaches based on either deep learning or deep reinforcement learning…
▽ More
Human character animation is often critical in entertainment content production, including video games, virtual reality or fiction films. To this end, deep neural networks drive most recent advances through deep learning and deep reinforcement learning. In this article, we propose a comprehensive survey on the state-of-the-art approaches based on either deep learning or deep reinforcement learning in skeleton-based human character animation. First, we introduce motion data representations, most common human motion datasets and how basic deep models can be enhanced to foster learning of spatial and temporal patterns in motion data. Second, we cover state-of-the-art approaches divided into three large families of applications in human animation pipelines: motion synthesis, character control and motion editing. Finally, we discuss the limitations of the current state-of-the-art methods based on deep learning and/or deep reinforcement learning in skeletal human character animation and possible directions of future research to alleviate current limitations and meet animators' needs.
△ Less
Submitted 23 November, 2021; v1 submitted 13 October, 2021;
originally announced October 2021.
-
JUMPS: Joints Upsampling Method for Pose Sequences
Authors:
Lucas Mourot,
François Le Clerc,
Cédric Thébault,
Pierre Hellier
Abstract:
Human Pose Estimation is a low-level task useful forsurveillance, human action recognition, and scene understandingat large. It also offers promising perspectives for the animationof synthetic characters. For all these applications, and especiallythe latter, estimating the positions of many joints is desirablefor improved performance and realism. To this purpose, wepropose a novel method called JU…
▽ More
Human Pose Estimation is a low-level task useful forsurveillance, human action recognition, and scene understandingat large. It also offers promising perspectives for the animationof synthetic characters. For all these applications, and especiallythe latter, estimating the positions of many joints is desirablefor improved performance and realism. To this purpose, wepropose a novel method called JUMPS for increasing the numberof joints in 2D pose estimates and recovering occluded ormissing joints. We believe this is the first attempt to addressthe issue. We build on a deep generative model that combines aGenerative Adversarial Network (GAN) and an encoder. TheGAN learns the distribution of high-resolution human posesequences, the encoder maps the input low-resolution sequencesto its latent space. Inpainting is obtained by computing the latentrepresentation whose decoding by the GAN generator optimallymatches the joints locations at the input. Post-processing a 2Dpose sequence using our method provides a richer representationof the character motion. We show experimentally that thelocalization accuracy of the additional joints is on average onpar with the original pose estimates.
△ Less
Submitted 14 October, 2020; v1 submitted 2 July, 2020;
originally announced July 2020.
