-
EPOCH: Jointly Estimating the 3D Pose of Cameras and Humans
Authors:
Nicola Garau,
Giulia Martinelli,
Niccolò Bisagno,
Denis Tomè,
Carsten Stoll
Abstract:
Monocular Human Pose Estimation (HPE) aims at determining the 3D positions of human joints from a single 2D image captured by a camera. However, a single 2D point in the image may correspond to multiple points in 3D space. Typically, the uniqueness of the 2D-3D relationship is approximated using an orthographic or weak-perspective camera model. In this study, instead of relying on approximations,…
▽ More
Monocular Human Pose Estimation (HPE) aims at determining the 3D positions of human joints from a single 2D image captured by a camera. However, a single 2D point in the image may correspond to multiple points in 3D space. Typically, the uniqueness of the 2D-3D relationship is approximated using an orthographic or weak-perspective camera model. In this study, instead of relying on approximations, we advocate for utilizing the full perspective camera model. This involves estimating camera parameters and establishing a precise, unambiguous 2D-3D relationship. To do so, we introduce the EPOCH framework, comprising two main components: the pose lifter network (LiftNet) and the pose regressor network (RegNet). LiftNet utilizes the full perspective camera model to precisely estimate the 3D pose in an unsupervised manner. It takes a 2D pose and camera parameters as inputs and produces the corresponding 3D pose estimation. These inputs are obtained from RegNet, which starts from a single image and provides estimates for the 2D pose and camera parameters. RegNet utilizes only 2D pose data as weak supervision. Internally, RegNet predicts a 3D pose, which is then projected to 2D using the estimated camera parameters. This process enables RegNet to establish the unambiguous 2D-3D relationship. Our experiments show that modeling the lifting as an unsupervised task with a camera in-the-loop results in better generalization to unseen data. We obtain state-of-the-art results for the 3D HPE on the Human3.6M and MPI-INF-3DHP datasets. Our code is available at: [Github link upon acceptance, see supplementary materials].
△ Less
Submitted 28 June, 2024;
originally announced June 2024.
-
HumMUSS: Human Motion Understanding using State Space Models
Authors:
Arnab Kumar Mondal,
Stefano Alletto,
Denis Tome
Abstract:
Understanding human motion from video is essential for a range of applications, including pose estimation, mesh recovery and action recognition. While state-of-the-art methods predominantly rely on transformer-based architectures, these approaches have limitations in practical scenarios. Transformers are slower when sequentially predicting on a continuous stream of frames in real-time, and do not…
▽ More
Understanding human motion from video is essential for a range of applications, including pose estimation, mesh recovery and action recognition. While state-of-the-art methods predominantly rely on transformer-based architectures, these approaches have limitations in practical scenarios. Transformers are slower when sequentially predicting on a continuous stream of frames in real-time, and do not generalize to new frame rates. In light of these constraints, we propose a novel attention-free spatiotemporal model for human motion understanding building upon recent advancements in state space models. Our model not only matches the performance of transformer-based models in various motion understanding tasks but also brings added benefits like adaptability to different video frame rates and enhanced training speed when working with longer sequence of keypoints. Moreover, the proposed model supports both offline and real-time applications. For real-time sequential prediction, our model is both memory efficient and several times faster than transformer-based approaches while maintaining their high accuracy.
△ Less
Submitted 16 April, 2024;
originally announced April 2024.
-
SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Camera
Authors:
Denis Tome,
Thiemo Alldieck,
Patrick Peluse,
Gerard Pons-Moll,
Lourdes Agapito,
Hernan Badino,
Fernando De la Torre
Abstract:
We present a solution to egocentric 3D body pose estimation from monocular images captured from downward looking fish-eye cameras installed on the rim of a head mounted VR device. This unusual viewpoint leads to images with unique visual appearance, with severe self-occlusions and perspective distortions that result in drastic differences in resolution between lower and upper body. We propose an e…
▽ More
We present a solution to egocentric 3D body pose estimation from monocular images captured from downward looking fish-eye cameras installed on the rim of a head mounted VR device. This unusual viewpoint leads to images with unique visual appearance, with severe self-occlusions and perspective distortions that result in drastic differences in resolution between lower and upper body. We propose an encoder-decoder architecture with a novel multi-branch decoder designed to account for the varying uncertainty in 2D predictions. The quantitative evaluation, on synthetic and real-world datasets, shows that our strategy leads to substantial improvements in accuracy over state of the art egocentric approaches. To tackle the lack of labelled data we also introduced a large photo-realistic synthetic dataset. xR-EgoPose offers high quality renderings of people with diverse skintones, body shapes and clothing, performing a range of actions. Our experiments show that the high variability in our new synthetic training corpus leads to good generalization to real world footage and to state of theart results on real world datasets with ground truth. Moreover, an evaluation on the Human3.6M benchmark shows that the performance of our method is on par with top performing approaches on the more classic problem of 3D human pose from a third person viewpoint.
△ Less
Submitted 2 November, 2020;
originally announced November 2020.
-
xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera
Authors:
Denis Tome,
Patrick Peluse,
Lourdes Agapito,
Hernan Badino
Abstract:
We present a new solution to egocentric 3D body pose estimation from monocular images captured from a downward looking fish-eye camera installed on the rim of a head mounted virtual reality device. This unusual viewpoint, just 2 cm. away from the user's face, leads to images with unique visual appearance, characterized by severe self-occlusions and strong perspective distortions that result in a d…
▽ More
We present a new solution to egocentric 3D body pose estimation from monocular images captured from a downward looking fish-eye camera installed on the rim of a head mounted virtual reality device. This unusual viewpoint, just 2 cm. away from the user's face, leads to images with unique visual appearance, characterized by severe self-occlusions and strong perspective distortions that result in a drastic difference in resolution between lower and upper body. Our contribution is two-fold. Firstly, we propose a new encoder-decoder architecture with a novel dual branch decoder designed specifically to account for the varying uncertainty in the 2D joint locations. Our quantitative evaluation, both on synthetic and real-world datasets, shows that our strategy leads to substantial improvements in accuracy over state of the art egocentric pose estimation approaches. Our second contribution is a new large-scale photorealistic synthetic dataset - xR-EgoPose - offering 383K frames of high quality renderings of people with a diversity of skin tones, body shapes, clothing, in a variety of backgrounds and lighting conditions, performing a range of actions. Our experiments show that the high variability in our new synthetic training corpus leads to good generalization to real world footage and to state of the art results on real world datasets with ground truth. Moreover, an evaluation on the Human3.6M benchmark shows that the performance of our method is on par with top performing approaches on the more classic problem of 3D human pose from a third person viewpoint.
△ Less
Submitted 23 July, 2019;
originally announced July 2019.
-
Rethinking Pose in 3D: Multi-stage Refinement and Recovery for Markerless Motion Capture
Authors:
Denis Tome,
Matteo Toso,
Lourdes Agapito,
Chris Russell
Abstract:
We propose a CNN-based approach for multi-camera markerless motion capture of the human body. Unlike existing methods that first perform pose estimation on individual cameras and generate 3D models as post-processing, our approach makes use of 3D reasoning throughout a multi-stage approach. This novelty allows us to use provisional 3D models of human pose to rethink where the joints should be loca…
▽ More
We propose a CNN-based approach for multi-camera markerless motion capture of the human body. Unlike existing methods that first perform pose estimation on individual cameras and generate 3D models as post-processing, our approach makes use of 3D reasoning throughout a multi-stage approach. This novelty allows us to use provisional 3D models of human pose to rethink where the joints should be located in the image and to recover from past mistakes. Our principled refinement of 3D human poses lets us make use of image cues, even from images where we previously misdetected joints, to refine our estimates as part of an end-to-end approach. Finally, we demonstrate how the high-quality output of our multi-camera setup can be used as an additional training source to improve the accuracy of existing single camera models.
△ Less
Submitted 4 August, 2018;
originally announced August 2018.
-
Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image
Authors:
Denis Tome,
Chris Russell,
Lourdes Agapito
Abstract:
We propose a unified formulation for the problem of 3D human pose estimation from a single raw RGB image that reasons jointly about 2D joint estimation and 3D pose reconstruction to improve both tasks. We take an integrated approach that fuses probabilistic knowledge of 3D human pose with a multi-stage CNN architecture and uses the knowledge of plausible 3D landmark locations to refine the search…
▽ More
We propose a unified formulation for the problem of 3D human pose estimation from a single raw RGB image that reasons jointly about 2D joint estimation and 3D pose reconstruction to improve both tasks. We take an integrated approach that fuses probabilistic knowledge of 3D human pose with a multi-stage CNN architecture and uses the knowledge of plausible 3D landmark locations to refine the search for better 2D locations. The entire process is trained end-to-end, is extremely efficient and obtains state- of-the-art results on Human3.6M outperforming previous approaches both on 2D and 3D errors.
△ Less
Submitted 11 October, 2017; v1 submitted 1 January, 2017;
originally announced January 2017.
-
Deep convolutional neural networks for pedestrian detection
Authors:
Denis Tomè,
Federico Monti,
Luca Baroffio,
Luca Bondi,
Marco Tagliasacchi,
Stefano Tubaro
Abstract:
Pedestrian detection is a popular research topic due to its paramount importance for a number of applications, especially in the fields of automotive, surveillance and robotics. Despite the significant improvements, pedestrian detection is still an open challenge that calls for more and more accurate algorithms. In the last few years, deep learning and in particular convolutional neural networks e…
▽ More
Pedestrian detection is a popular research topic due to its paramount importance for a number of applications, especially in the fields of automotive, surveillance and robotics. Despite the significant improvements, pedestrian detection is still an open challenge that calls for more and more accurate algorithms. In the last few years, deep learning and in particular convolutional neural networks emerged as the state of the art in terms of accuracy for a number of computer vision tasks such as image classification, object detection and segmentation, often outperforming the previous gold standards by a large margin. In this paper, we propose a pedestrian detection system based on deep learning, adapting a general-purpose convolutional network to the task at hand. By thoroughly analyzing and optimizing each step of the detection pipeline we propose an architecture that outperforms traditional methods, achieving a task accuracy close to that of state-of-the-art approaches, while requiring a low computational time. Finally, we tested the system on an NVIDIA Jetson TK1, a 192-core platform that is envisioned to be a forerunner computational brain of future self-driving cars.
△ Less
Submitted 7 March, 2016; v1 submitted 13 October, 2015;
originally announced October 2015.
-
Maximizing the Link Throughput between Smart-meters and Aggregators as Secondary Users under Power and Outage Constraints
Authors:
Pedro H. J. Nardelli,
Mauricio de Castro Tomé,
Hirley Alves,
Carlos H. M. de Lima,
Matti Latva-aho
Abstract:
This paper assesses the communication link from smart meters to aggregators as (unlicensed) secondary users that transmit their data over the (licensed) primary uplink channel. The proposed scenario assumes: (i) meters' and aggregators' positions are fixed so highly directional antennas are employed, (ii) secondary users transmit with limited power in relation to the primary, (iii) meters' transmi…
▽ More
This paper assesses the communication link from smart meters to aggregators as (unlicensed) secondary users that transmit their data over the (licensed) primary uplink channel. The proposed scenario assumes: (i) meters' and aggregators' positions are fixed so highly directional antennas are employed, (ii) secondary users transmit with limited power in relation to the primary, (iii) meters' transmissions are coordinated to avoid packet collisions, and (iv) the secondary links' robustness is guaranteed by an outage constraint. Under these assumptions, the interference caused by secondary users in both primary (base-stations) and other secondary users can be neglected. As unlicensed users, however, meter-aggregator links do experience interference from the mobile users of the primary network, whose positions and traffic activity are unknown. To cope with this uncertainty, we model the mobile users spatial distribution as a Poisson point process. We then derive a closed-form solution for the maximum achievable throughput with respect to a reference secondary link subject to transmit power and outage constraints. Our numerical results illustrate the effects of such constraints on the optimal throughput, evincing that more frequent outage events improve the system performance in the scenario under study. We also show that relatively high outage probabilities have little effect on the reconstruction of the average power demand curve that is transmitted from the smart-meter to the aggregator.
△ Less
Submitted 16 February, 2016; v1 submitted 16 June, 2015;
originally announced June 2015.
