EPOCH: Jointly Estimating the 3D Pose of Cameras and Humans

Abstract

Monocular Human Pose Estimation (HPE) aims at determining the 3D positions of human joints from a single 2D image captured by a camera. However, a single 2D point in the image may correspond to multiple points in 3D space. Typically, the uniqueness of the 2D-3D relationship is approximated using an orthographic or weak-perspective camera model. In this study, instead of relying on approximations, we advocate for utilizing the full perspective camera model. This involves estimating camera parameters and establishing a precise, unambiguous 2D-3D relationship.

To do so, we introduce the EPOCH framework, comprising two main components: the pose lifter network (LiftNet) and the pose regressor network (RegNet). LiftNet utilizes the full perspective camera model to precisely estimate the 3D pose in an unsupervised manner. It takes a 2D pose and camera parameters as inputs and produces the corresponding 3D pose estimation. These inputs are obtained from RegNet, which starts from a single image and provides estimates for the 2D pose and camera parameters. RegNet utilizes only 2D pose data as weak supervision. Internally, RegNet predicts a 3D pose, which is then projected to 2D using the estimated camera parameters. This process enables RegNet to establish the unambiguous 2D-3D relationship.

Our experiments show that modeling the lifting as an unsupervised task with a camera in-the-loop results in better generalization to unseen data. We obtain state-of-the-art results for the 3D HPE on the Human3.6M and MPI-INF-3DHP datasets. Our code is available at: [Github link upon acceptance, see supplementary materials].

Figure 1: (a) In human pose estimation, classical approaches perform a direct regression of the 2D/3D joint location directly from an image. If the ground truth is available, the camera parameters can be used/learned to refine the accuracy. (b) Lifting approaches aim at retrieving the depth of each 2D joint to obtain the 3D pose. (c) We propose a novel paradigm, directly estimating the 3D pose and the camera from images. The 2D pose can be calculated by applying the projection of the 3D coordinates to the image space using the camera parameters. (d) Starting from the estimated 2D poses and camera parameters, we perform the lifting to 3D, improving the performances with respect to current approaches.

1 Introduction

There are two main approaches to monocular 3D human pose estimation (HPE) from a single RGB images [18]. One class of algorithms uses a single-stage approach, where the aim is to regress the 3D position of human joints directly from an image [38, 39, 45]. The other class of approaches use two distinct stages, where the first step is to infer 2D poses from monocular RGB images, which is followed by a lifter network that predicts the 3D displacement for each of the 2D joints. Two stage approaches typically outperform single stage approaches [57].

Estimating 3D human poses from a single RGB image is difficult as the problem is inherently ill-posed. For any 2D observation there exist multiple plausible 3D poses that will lead to the same 2D projection [56, 33]. Additionally, collecting reliable ground-truth 3D data is difficult. Annotating 3D ground truth on 2D images inevitably introduces inaccuracies, and collecting actual ground truth requires a complex and expensive controlled environment using multi-view camera systems or additional capture modalities. Even using multiple views, triangulation can lead to errors, as there are inherent ambiguities in the position of the joint under the body surface. While limited datasets providing 3D data are available [17, 19], 2D datasets still provide more data in more general scenarios and environments.

To address the ill-posed nature of the problem, past approaches have relied on different strategies, like fully supervised training using either real [6, 44] or synthetic 3D ground truth [28, 26], weakly supervised training relying on multiple views: either paired [47, 51] or unpaired [50], 2D supervision [46], or video motion consistency [16, 10, 15]. Unsupervised approaches have relied on cycle consistency coupled with a weak perspective camera projection to lift 2D poses to 3D [49, 2]. Relying on a weak perspective camera projection is not ideal because it does not accurately capture the perspective transformation [14], leading to depth inaccuracy and scale ambiguity when projecting a 2D skeleton in the 3D space. Recent works [23] have shown that using fully perspective cameras reduces this ambiguity.

In this paper, we introduce EPOCH, a novel unsupervised framework that effectively addresses the challenges of data scarcity by harnessing unsupervised techniques and mitigates the inherent ill-posed nature of the problem through explicit camera modeling, as shown in Fig. 1. Our approach stands out due to its capability to estimate the full perspective camera parameters leveraging only 2D human poses, without relying on any camera ground truth. We claim that by incorporating the estimated camera into the 3D lifting operation, it is possible to enhance the accuracy and the consistency of 3D unsupervised human pose estimation while generalizing to unseen data. Our method consists of an unsupervised 3D human pose lifter network (LiftNet) and a lightweight capsule-based regressor network (RegNet) that estimates the camera pose and 2D joint positions.

LiftNet performs the 3D lifting from estimated 2D poses and camera parameters. Inspired by [2], we employ a self-supervising cycle-consistent framework. Unlike [49, 2], our approach uses a full perspective camera, allowing us to use a wider range of camera transformations for supervision in our cycle consistency, and improving the accuracy of the model.

We estimate the camera pose and 2D poses used as input for the lifting stage using RegNet, a lightweight capsule-based regressor network that is trained on weakly supervised 2D pose data instead of fully supervised data as [23]. It use contrastive pre-training and heatmap-free joint position regression to estimate the 2D poses as well as the intrinsic (the camera matrix $[K]$) and extrinsic parameters (rotation matrix $[R]$ and translation vector $T$) of a camera based on the standard pinhole camera model. To supervise the camera estimation with the 2D pose data we internally predict a 3D pose that is then projected to 2D with the estimated camera. The internally predicted 3D pose is not quite as accurate as the refined output of LiftNet but helps regularizing the camera estimation.

With no prior about 3D human appearance, both LiftNet and RegNet estimate a single 2D projection which is not enough to guarantee a plausible 3D pose. Thus, we employ Normalizing Flows (NF) to ensure the plausibility of multiple 2D projections of a single 3D estimate. Different from previous approaches [49], our NF is based on simple 1x1 convolutions [20], which can be applied to the full feature representation of the poses, without the need to reduce their dimensionality using the Principal Component Analysis (PCA).

The EPOCH framework is the sequential combination of RegNet and LiftNet, which allows for the direct inference of accurate 3D poses from images. RegNet estimates 3D poses with weak 2D pose supervision, while the camera parameters are estimated without any ground truth camera data. LiftNet predicts 3D poses based on the estimates of 2D poses and camera poses. We argue that RegNet is weakly supervised, whereas LiftNet is fully unsupervised as it relies solely on estimates, without any ground truth data. This reasoning applies to both poses and cameras.

The novelty of our work can be summarized as:

• We define an innovative EPOCH framework to address the challenges of data scarcity and the ill-posed nature of 3D HPE problem by harnessing the full camera perspective projection, enabling the direct lifting of accurate 3D poses from input images.

• We present LiftNet, a novel 3D unsupervised HPE framework that leverages perspective camera projection to improve the accuracy of 2D pose lifting.

• We introduce an original capsule-based regressor, called RegNet, which jointly estimates 3D joint positions and camera parameters. Final 2D joints are computed using the estimated camera by perspective projection.

• We adopt a lightweight Normalizing Flows [20] model to enforce anthropomorphic constraints. Our NF accepts the entire 2D skeleton as input without the need for dimensionality reduction using PCA.

• We obtain state-of-the-art results on both 3D HPE direct regression and 3D unsupervised HPE on the common benchmark datasets Human3.6M and MPI-INF-3DHP.

2 Related work

3D human pose estimation (HPE) from monocular 2D images has been extensively researched through supervised, weakly-supervised, and unsupervised approaches [57, 32]. This section gives an overview of the different methods, also focusing on the challenging in-the-wild approaches.

Fully supervised approaches. In this paradigm the 3D ground truth is readily available. Gathering such data requires collecting vast datasets such as Human3.6M [17], 3DPW [48], MPI-INF-3DHP [36] and CMU Panoptic [19]. In supervised methods, two primary strategies exist: direct regression of 3D coordinates from the image [38, 39, 45] (Fig. 1(a)), or 2D pose estimation followed by lifting to 3D [52, 53, 58, 5] (Fig. 1(b)). Direct regression proves more challenging because it involves the simultaneous estimation of 3D coordinates for each joint, often leading to inferior results compared to lifting-based techniques [57]. Supervised networks achieve the best results on multiple datasets [58, 5], but often struggle to generalize to different scenarios like out-of-distribution poses, challenging camera angles and in-the-wild pose estimation [57].

Weakly-supervised approaches rely on the lifting framework [47, 51, 50, 46, 15, 55, 30, 27, 8] using various supervision signals without directly accessing the 3D ground truth paired with the corresponding 2D image. For instance, multiple paired or unpaired views of the same subject provide a supervision signal, through the consistency of the estimated 3D pose seen from different viewpoints [47, 51, 50, 42, 27, 24]. In monocular approaches, temporally correlated 2D poses can be estimated from an input video and used as a supervision signal for a frame-specific 3D pose estimation [15, 55, 30]. In-the-wild approaches have mostly relied on 2D pose as ground truths to supervise intermediate 3D estimates [54, 12]. Other approaches perform monocular 3D pose estimation using only 2D pose supervision [27, 9, 2, 50].

Following this line of work, our regressor network (RegNet) is a novel weakly-supervised approach that uses 2D poses for supervision, computing the 3D to 2D projection via estimated camera parameters. Moreover, our approach is the first that jointly estimates the full perspective camera parameters without relying on any ground truth camera.

Unsupervised approaches. Unsupervised approaches usually rely on the lifting paradigm, employing multiple different signals to regularise their 3D predictions. In [2], the authors proposed an unsupervised lifting network grounded in closure and invariance properties, incorporating a geometric self-consistency loss. The closure property for a lifted 3D skeleton means that, after random rotation and re-projection, the resulting 2D skeleton will lie within the distribution of valid 2D poses. The invariance propriety means that, when changing the viewpoint of 2D projection from a 3D skeleton, the re-lifted 3D skeleton should be the same. Following a similar concept [49] introduces a weak camera projection to model the lift-reproject-lift process. The weak camera projection is coupled with the elevation estimation of the camera, providing an approximation of the full camera perspective model. Moreover, they introduce the use of Normalizing Flows (NFs) [7], which are used to ensure the closure property more accurately than GAN-based methods [2]. In [13] a similar framework is extended to a multi-person scenario, where relative positions are also used as supervision signals.

Our lifter network (LiftNet) follows this line of work while introducing the following novelties: (i) we employ a full perspective camera model for the projection, making it much more accurate and robust to varying focal lengths, (ii) we drop the need for an additional elevation prediction branch in the lifting network [49], (iii) we avoid applying the PCA to reduce data dimensionality by training a normalizing flow based on Glow [20] instead of RealNVP, (iv) we add a geometric constraint on unnatural joints folds.

3 Method

3.1 Preliminaries

Camera model. While many prior works for 3D human pose estimation rely on simplified weak perspective camera models, we use a full perspective camera model, consisting of intrinsic parameters K (focal length and center of projection) and extrinsic parameters R and T (rotation and translation of the camera respectively). We transform a 3D point $(X,Y,Z)$ into camera coordinates by multiplying with the extrinsic and intrinsic matrices, and get the final image space coordinates $I=[u/w,v/w]$.

$$\begin{bmatrix}u\\ v\\ w\end{bmatrix}_{2D}={[\textbf{K}]}{[\textbf{R}|\textbf{T}]}\begin{bmatrix}X\\ Y\\ Z\\ 1\end{bmatrix}_{3D}$$ (1)

To invert the projection given image coordinates $\tilde{I}$, we need to estimate the unknown depth $w$ of the point. Similar to [31] we estimate the intrinsic parameters directly from the full image size and the bounding box crop using a model $\Psi$. While this is only an approximation of the real field-of-view of the camera, prior work [21] has shown this to be a good approximation for most real life cameras used. The estimated intrinsic camera matrix $[K]$ includes the focal length $f=(f_{w},f_{h})$ and the principal point $c=(c_{w},c_{h})$. Moreover, we estimate a scaling factor $s=(s_{w},s_{h})$ for the regularisation of the image size and skeleton height. See Supplementary Materials for further details on model $\Psi$.

Normalizing flows loss. Normalizing flows (NFs) are a class of generative neural networks capable of map** a complex distribution into a simpler one using invertible functions [22]. They are trained to learn the probability density function (PDF) of a given dataset relying on an invertible function $f$. See Supplementary material for further details about the modeling of this function. When presented with a novel sample, NFs can estimate the likelihood (plausibility) that the given sample belongs to the learned dataset distribution.

In [49], the learnable function $f$ is based on RealNVP [7] which is not suitable for high dimensional data like 2D poses, necessitating a PCA reduction of the input vector $x$ for an optimal convergence during training. In contrast, our NFs are based on the Glow framework [20] relying on 1x1 convolutions. Small and fast convolutions allow the use of the full 2D joints’ coordinates without the need for a feature reduction as well as reducing the computational costs.

During the training of our architecture, the NFs are used to verify whether multiple projections of 3D poses are all plausible 2D poses without relying on multi-view data. To achieve this, we define the normalizing flow loss $\mathcal{L}_{NF}$, as the negative log-likelihood of the PDF:

$$\mathcal{L}_{NF}(x)=-log(p_{x}(\mathbf{x}))$$ (2)

Figure 2: In (I), we define two vectors, denoted as $A$ and $B$, connecting the spine and the hip joints. The cross product of these vectors yields the normal vector $N$, which aligns with the walking direction. In (II) and (III), we show the outcome of the dot product between $N$ and the proximal $p_{l}$ and distal $d_{l}$ components, resulting in their projections $D_{l}$ and $P_{l}$. In (II), $\mathcal{L}_{limbs}$ gives an output of 0, indicating a anthropomorphically complaint prediction. In (III), $\mathcal{L}_{limbs}$ returns a positive value, signaling the need for further correction.

Anthropomorphic constraints. In supervised 3D HPE, the neural network has explicit access to the 3D ground truth to learn the appearance of a 3D human pose. In unsupervised or semi-supervised settings, we introduce regularising losses to ensure that the estimated 3D poses respect anthropomorphic constraints, such as proportional bone lengths and articulation angle limits.

As in [49], we use a bones ratio loss $\mathcal{L}_{bone}(y)$ to ensure that the ratio between bones lengths of 3D pose $y\in\mathbb{R}^{3J}$ are respected. This loss leverages the observed nearly constant ratio between bones across different individuals [40], without fixing the bone length to a pre-defined value.

Additionally, we define a novel $\mathcal{L}_{limbs}$ loss which ensures that joints like knees and elbows do not bend in unrealistic manners (e.g. facing backward with respect to the normal walking direction). It is defined as:

$$\ \mathcal{L}_{limbs}(y)=\frac{1}{L}\sum_{l}^{L}{max(0,P_{l}-D_{l})}$$ (3)

where $L$ represents the number of limbs, $P_{l}=N\cdot p_{l}$ and $D_{l}=N\cdot d_{l}$ denote the normal components of the proximal ($p_{l}$) and distal ($d_{l}$) components of each limb, and $N=A\times B$ represents the normal vector for the plane defined by the hips and spine joints. In Fig. 2 we show a visual representation of $\mathcal{L}_{limbs}$ to better convey the intuitive reasoning behind its mathematical formulation.

3.2 Pose Lifter Network (LiftNet)

LiftNet is our lifter module which introduces the paradigm shown in Fig. 1(d). The overall detailed architecture is shown in Fig. 3.

Figure 3: LiftNet architecture. The red ($2D\rightarrow 3D$), orange ($\circlearrowleft$ and $\circlearrowright$) and yellow ($3D\rightarrow 2D$) blocks describe the Lift, Rotate, Project operations respectively. The symbol $x$ denotes a 2D pose, $y$ denotes a 3D pose. The decorator $\,\hat{}\,$ symbolizes a prediction in the forward pass while $\,\widetilde{}\,$ marks a prediction in the backward pass. The subscript _r stands for rotated. The solid arrows describe the flow of the network, while the dashed arrows connect each intermediate datum to its loss.

LiftNet aims at retrieving the 3D pose $y\in\mathbb{R}^{3J}$, starting from a 2D pose $\hat{x}\in\mathbb{R}^{2J}$ and its estimated camera parameters $[K]$ and $[R|T]$. As shown in Fig. 3, our architecture consists of a cycle consistency structure which can be split into two symmetric branches: a forward branch ($\hat{x}\rightarrow Lift\rightarrow\hat{y}\rightarrow Rotate\rightarrow\hat{y}_{r}\rightarrow Project\rightarrow\hat{x}_{r}$) and a backward branch ($\hat{x}_{r}\rightarrow Lift\rightarrow\widetilde{y}_{r}\rightarrow InverseRotate\rightarrow\widetilde{y}\rightarrow Project\rightarrow\widetilde{x}$). Each step and its input/output are described in Alg. 1. The lift operation is performed by a lifter network while the projection is a mathematical operation. Both of these operations rely on the full perspective model using camera parameters $[K][R|t]$. All the losses provide self-supervision to the cycle consistency, which does not access either 2D or 3D ground truths, making it a fully unsupervised process.

Algorithm 1 Cycle consistency. Both $2D\rightarrow 3D$ and $3D\rightarrow 2D$ rely on the camera parameters $[K][R|t]$ to be solved.

$\hat{x}$, $[K][R|t]$

Forward branch

1. Lift: $2D\rightarrow 3D$: $\hat{x}\rightarrow\hat{y}$

2. Rotate: $\hat{y}\rightarrow\hat{y}_{r}$

3. Project: $3D\rightarrow 2D$: $\hat{y}_{r}\rightarrow\hat{x}_{r}$

Backward branch

4. Lift: $2D\rightarrow 3D$: $\hat{x}_{r}\rightarrow\tilde{y}_{r}$

5. InverseRotate: $\tilde{y}_{r}\rightarrow\tilde{y}$

6. Project: $3D\rightarrow 2D$: $\tilde{y}\rightarrow\tilde{x}$

Differently from previous approaches using the weak camera model [49], our lifter leverages the full perspective camera model to recover the 3rd dimension $w$ for each input 2D pose $\hat{x}$. Using the estimated $w$ we can solve the inverse of the projection of (Eq. 1) and recover the 3D joint positions. This Lift operation is symbolized as $2D\rightarrow 3D$.

Given a 3D poses $\hat{y}$ we can perform the Project operation symbolized as $3D\rightarrow 2D$. That means computing the 2D pose $\hat{x}$ using the full camera projection in Eq. 1.

Inspired by previous approaches [35, 49], our lifter network consists of a simple MLP structure. The MLP receives as input a 2D pose $x$ concatenated with the flattened version the extrinsic parameters $[R|t]$ (12 total values), the intrinsic parameters $f=(f_{w},f_{h})$, $c=(c_{w},c_{h})$ and the scaling factor $s=(s_{w},s_{h})$, resulting in a vector of size $2J+18$. The input vector is fed (I) to a linear layer to obtain an embedded vector of size $dim_{l}$, (II) to 3 residual blocks each containing 2 fully connected layers, (III) to a linear layer to obtain the output vector of size $J$ representing the depth parameter $w$ for each joint. The vector is concatenated with the input $\hat{x}$ resulting in the estimated 3D pose $\hat{y}$.

To train the LiftNet, we minimize the following loss:

$$\mathcal{L}_{lift}=\mathcal{L}_{2D}(\hat{x},\tilde{x}) +\mathcal{L}_{3D}(\hat{y}_{r},\tilde{y}_{r})+\mathcal{L}_{NF}(\tilde{x}_{r})+\mathcal{L}_{bone}(\hat{y})+\mathcal{L}_{limbs}(\hat{y})+\mathcal{L}_{def}$$ (4)

where:

• $\mathcal{L}_{2D}(\hat{x},\tilde{x})=||\hat{x}-\tilde{x}||_{1}$ is the norm-1 distance between the initial prediction $\hat{x}$ and its version after the cycle consistency loop $\tilde{x}$;

• $\mathcal{L}_{3D}(\hat{y}_{r},\tilde{y}_{r})=||\hat{y}-\tilde{y}||_{2}$ is the norm-2 distance between the 3D pose $\hat{y}$ and its version after it has been projected and lifted $\tilde{y}$;

• $\mathcal{L}_{NF}(\tilde{x}_{r})$, $\mathcal{L}_{bone}(\hat{y})$ and $\mathcal{L}_{limbs}(\hat{y})$ are the losses defined in Sec. 3.1;

• $\mathcal{L}_{def}$ is a deformation loss computed between two poses $y_{a}$ and $y_{b}$ belonging to the same batch defined as

  $$\mathcal{L}_{def}=||(\hat{y_{a}}-\hat{y_{b}})-(\tilde{y_{a}}-\tilde{y_{b}})||_{2}$$ (5)

  which ensures that 2 poses from the same batch have not been deformed in completely different manners by the same Project and Lift operations, providing a supervision similar to the temporal consistency defined in [54], but without relying on temporally-related data.

Figure 4: RegNet architecture. The $W\times H$ input image is fed to (a) a contrastive-pretrained encoder and a separate module $\Psi$ that estimates the intrinsic parameters. The output features are then concatenated and (b) fed into our attention-based capsule decoder. The outputs are three separate capsule vectors, representing an estimation of the 3D pose $\hat{y}$, of the camera $[K][R|t]$ and a joint presence vector $\Sigma$. (c) Each of the outputs needs to be further processed before the loss computation. A copy of $\hat{y}$ is randomly rotated around the vertical axis, obtaining $\hat{y}_{r}$. $\hat{y}$ and $\hat{y}_{r}$ are projected into the camera plane and $\Sigma$ goes through a sigmoid activation function. (d) $\hat{y}$, $\hat{x}_{r}$, $\hat{x}$ and $\hat{\sigma}$ are fed to the loss functions.

3.3 Pose Regressor Network (RegNet)

RegNet is our direct regression module (Fig. 1(c)) which is used to estimate the camera pose and initial 2D pose used for the lifting stage. The overall detailed architecture is shown in Fig. 4. The input to RegNet is a single square image $I$ of size $W\times H$ pixels, roughly centered on the pelvis similar to [49, 2, 54]. The objective is to retrieve the 2D pose $x\in\mathbb{R}^{3J}$. Additionally, it estimates the intrinsic camera parameters $K$ consisting of focal length $f=(f_{w},f_{h})$, principal point $c=(c_{w},c_{h})$, and a scaling factor $s=(s_{w},s_{h})$, as well as the extrinsic camera parameters $R$ and $t$. The intrinsic camera parameters are estimated directly from the image size as shown in [31].

RegNet consists of an encoder-decoder architecture, where the encoder is pre-trained using contrastive learning and the decoder is based on capsules. The output of the decoder yields both 3D poses and camera parameters, which are then used to compute the 2D pose $x\in\mathbb{R}^{2J}$, with each joint represented as $[u/w,v/w]$ on the image plane $I$. It is worth noting that we employ 2D poses for the loss computation, thus introducing a form of weak supervision to the 3D pose estimation process.

Contrastive Encoder. Our encoder employs a ResNet50 pre-trained using contrastive learning on ImageNet as in [1]. Despite its general image focus, the use of contrastive learning allows for faster convergence and better generalization, outperforming supervised pre-training methods [3].

The output vector of the encoder is concatenated with the intrinsic parameters $[K]$ that are estimated directly from the full image. The vector resulting from the concatenation of size $dim$ is given as input to the decoder.

Capsule-based decoder. In [25], they first showed how a Soft Attention mechanism can be effectively used to split a feature vector in different capsule features. In [43], they perform an equivalent operation with a fully connected and a Softmax layer. Inspired by [43], we design our capsule-based decoder using a Conv2D fully connected layer to transform the latent space vector of size $dim$ to a vector of size $9\times J$.

Given $J$ values used for the attention mechanism, the remaining $8\times J$ values are divided in the following capsules:

• $\hat{y}\in\mathbb{R}^{3J}$, representing the estimated 3D pose;

• $\Gamma\in\mathbb{R}^{3J}$ from which we can compute $[R|t]$ representing the extrinsic parameters, namely the estimated rotation matrix $R$ and translation vector $t$ with respect to the input image’s viewpoint. The mathematical calculations to derive $[R|t]$ from $\Gamma$ are reported in the Supplementary Materials. To ensure invertibility, an orthogonality constraint is enforced on the rotation matrix $[R]$. The extrinsic parameters are combined with the intrinsic camera matrix $[K]$ computed by $\Psi$ at the encoding stage to obtain the full perspective camera model descriptor $[K][R|t]$;

• $\Sigma\in\mathbb{R}^{2J}$, a vector indicating the presence of each joint. Low values indicate uncertain detection of joints, often due to occlusions or joints being outside the image space.

Outputs. To ensure that the estimated 3D pose is plausible from multiple viewpoints, we require both the 2D pose $\hat{x}$ of the original image $I$, as well as the 2D pose $\hat{x}_{r}$ corresponding to the same pose from a different viewpoint.

For $\hat{x}$, we perform the Project operation as used in the lifting architecture to obtain 2D pose $\hat{x}$ using the full camera projection in Eq. 1 given the 3D pose $\hat{y}$.

Similar to the lifting stage, we observed that we can increase the accuracy of the estimated 3D poses by using a Normalizing Flow loss to ensure the plausibility of multiple 2D projections of the 3D pose seen from different viewpoints. To this end we also calculate a rotated projection $\hat{x}_{r}$, where we rotate the model around the vertical world axis. The Rotate operation is applied randomly to each 3D pose in the range of $[$$,$$]$ to simulate a viewpoint change, before projecting the rotated 3D pose $\hat{y}_{r}$ to obtain $\hat{x}_{r}$.

Regression of the joints’ position can be combined with an uncertainty measure and leads to better results when compared to direct regression of joints and heatmap-based methods [29]. Moreover, the regression is more computation and memory efficient compared to heatmaps. To employ a similar method, we estimate the deviation $\hat{\sigma}$ of the predicted joint’s position from the ground truth by applying a sigmoid function on the estimated presence capsule $\Sigma$.

Losses. To train RegNet, we need to minimize the following loss:

$$\mathcal{L}_{reg}=\mathcal{L}_{bone}(\hat{y})+\mathcal{L}_{limbs}(\hat{y}) +\mathcal{L}_{NF}(\hat{x}_{r})+\mathcal{L}_{RLE}(\hat{x},\hat{\sigma})$$ (6)

where:

• $\mathcal{L}_{NF}(\tilde{x}_{r})$, $\mathcal{L}_{bone}(\hat{y})$ and $\mathcal{L}_{limbs}(\hat{y})$ are the losses defined in Sec. 3.1;

• $\mathcal{L}_{RLE}(x,I)$ is the residual log-likelihood estimation loss defined as,

  $$\mathcal{L}_{RLE}(x,I) =-logP_{\Theta,\phi}(x,I)|_{x=\hat{x}}=-logP_{\phi}(\hat{x})+log(\hat{\sigma})$$ (7)

  As in [29], this loss aims at estimating the joints’ position $\hat{x}$ via direct regression coupled with the learned error distribution $\hat{\sigma}$. We refer to the original work [29] for the full mathematical explanation of the loss.

For further details on loss balancing during training see the Supplementary Materials.

4 Results

We perform our experiments on the common Human3.6M [17] and MPI-INF-3DHP [36] datasets . We follow standard test protocols for both datasets [49]. We also report extensive ablation studies and qualitative results on the unseen 3DPW dataset [48] to demonstrate generalization.

Implementation details. The input images are of size $H=224$ and $W=224$. The skeleton model has $J=17$ joints. RegNet is trained for $45$ epochs using the optimizer AdamW [34], $dim=2048+6$, learning rate $1e-3$, and weight decay $1e-4$. LiftNet is trained for $100$ epochs using the optimizer AdamW, learning rate $2e-4$, and weight decay $1e-5$. Both RegNet and LiftNet are trained with batch size $256$ and bfloat16 precision on single a NVIDIA RTX 3090. Inference runs on the same GPU at $\approx 45$ fps.

Metrics. We adopt the standard mean per joint position error (MPJPE) in two common forms for the 3D HPE unsupervised settings: the PA-MPJPE where reconstructed 3D pose is Procrustes aligned and the N-MPJPE where the 3D pose is normalized the same scale as the ground truth [41]. As in [49], for the MPI-INF-3DHP dataset we report the scale normalized percentage of correct key points (N-PCK) predicted within $150$ mm to the original position and its corresponding area under curve (AUC).

4.1 Quantitative results

Supervision	Method	PA-MPJPE$\downarrow$	N-MPJPE $\downarrow$
Full-3D	Pavalkos [38]	41.8	-
10%-3D	Kundu [28]	49.6	59.4
	Kundu [26]	48.2	57.6
	Gong [11]	39.1	50.2
Multi-view	Rhodin [41]	98.2	122.6
	Kundu [27]	85.8	-
	Usman [47]	44.0	55.0
Full-2D	Fish [9]	97.2	-
	Kundu [27]	62.4	-
	Ours (RegNet)	45.4	69.9

Table 1: Quantitative results for direct regression from images on Human3.6M.

Supervision	Method	PA-MPJPE$\downarrow$	N-MPJPE$\downarrow$
Full	Martinez [35]	37.1	45.5
Weak	Fish [9]	79.0	-
	Wandt [50]	38.2	50.9
	Drover [8]	38.2	-
	Kundu [28]	62.4	-
Multi-view	Kocabas [24]	47.9	54.9
	Wandt [51]	51.4	65.9
Unsupervised	Chen [2]	58.0	-
	Yu [54]	42.0	85.3
	Wandt [49]	36.7	64.0
	Ours (LiftNet)	30.7	50.8

Table 2: Quantitative results for lifting from 2D ground truth on Human3.6M.

Method	Backbone	PA-MPJPE$\downarrow$	N-MPJPE$\downarrow$
Chen [2]	SH [37]	68.0	-
Wandt [50]	SH [37]	65.1	89.9
Kundu [27]	ResNet-50	63.8	-
Kundu [27]	ResNet-50	62.4	-
Yu [54]	CPN [4]	52.3	92.4
Wandt [49]	CPN [4]	50.2	74.4
Ours (LiftNet)	Ours (RegNet)	43.8	67.1

Table 3: Quantitative results for lifting from 2D predictions on Human3.6M.

Supervision	Method	2D Input	PA-MPJPE$\downarrow$	N-PCK$\uparrow$	AUC$\uparrow$
Weak	Kundu [27]	GT	93.9	84.6	60.8
Unsup.	Yu [54]	GT	-	86.2	51.7
	Wandt [49]	GT	54.0	86.0	50.1
	Ours (LiftNet)	Pred	46.8	93.6	61.3
	Ours (LiftNet)	GT	33.6	97.6	69.5
Unseen data	Ours (LiftNet)	Pred	57.3	77.5	46.4

Table 4: Quantitative results for 3D HPE on MPI-INF-3DHP.

Direct regression from images. In Tab. 2, we report the results of weakly-supervised direct regression of 3D pose from images on Human3.6M dataset. Weakly-supervised approaches include ones using only a small portion of 3D annotated data (10%-3D) or multi-view supervision. Both approaches rely on 3D spatial information for their supervision, leading to results close to the baseline fully supervised approach [38]. Among approaches using only 2D information, our RegNet outperforms others, obtaining results in line with baseline 3D supervised. In contrast to the previous approaches, RegNet also performs the unsupervised estimation of the full perspective camera parameters $[K][R|t]$.

Lifting from 2D ground truth (GT). In Tab. 2, we report the results of lifting GT 2D pose to 3D on Human3.6M dataset. We compare against baseline fully supervised approaches [35], multi-view supervised approaches [24, 51], and weakly supervised approaches adopting different strategies like domain adaptation [28], 2D GT poses [8, 9] and partial 3D GT [50]. Our unsupervised LiftNet outperforms all previous weakly supervised approaches, also obtaining the best results among unsupervised methods. We also outperform [49] which adopts a weak perspective camera modeling combined with elevation estimation, demonstrating the efficacy of our LiftNet that explicitly leverages the full perspective camera model.

Lifting from 2D predictions. In Tab. 4, we report the results of lifting 2D pose predictions to 3D on the Human3.6M dataset. In contrast to other unsupervised approaches that use either ResNet50 [27], Stacked Hourglass (SH) [2, 50] or Cascaded Pyramid Network (CPN) [4, 49] as the backbone to extract 2D predictions to be lifted, we use RegNet which also provides an unsupervised estimation of the camera parameters. The full EPOCH approach combining both RegNet and LiftNet outperforms all other methods, demonstrating its efficacy as an end-to-end approach to estimate 3D poses from images.

Generalising to unseen data. In Tab. 4, we report the results of 3D HPE on MPI-INF-3DHP dataset. When trained on MPI-INF-3DHP, LiftNet outperforms both weakly-supervised [27] and unsupervised [49, 54] lifting methods starting from either the ground truth or from poses predicted by RegNet. When not trained on MPI-INF-3DHP (last row), EPOCH trained only on Human3.6M achieves results comparable to fine-tuned approaches, proving its ability to generalize to unseen data.

Ablation studies. In Tab. 6, we report the results of the ablation study for RegNet. First, we show how using a backbone trained with supervised learning (S) leads to poorer features (first row), leading to performance degradation compared to the full model trained with contrastive learning (C) (last row). Next, we ablate different losses, showing how $\mathcal{L}_{NF}$ is the loss that causes the biggest drop in performances since it ensures the network does not estimate 3D poses that are plausible only from a single viewpoint. $\mathcal{L}_{bone}$ and $\mathcal{L}_{limbs}$ have similar effect on the results as they both enforce anthropomorphic constraints.

In Tab. 6, we report the results of the ablation study for LiftNet. We perform an ablation study on the camera modeling, showing how using a weak perspective camera leads to performance degradation due to its less precise modeling of the 2D/3D relation than the full perspective camera. Moreover, we proceed to study the effect of each loss on the performances. As for RegNet, $\mathcal{L}_{NF}$ causes the biggest drop in performances. The deformation $\mathcal{L}_{def}$, and the two anthropomorphic constraints $\mathcal{L}_{bone}$ and $\mathcal{L}_{limbs}$ have similar effects on the numerical results, as they are all enforcing comparable constraints on the deformation and proportions of 3D poses. $\mathcal{L}_{2D}$ and $\mathcal{L}_{3D}$ all provide regularisation between different stages of the cycle consistency, so removing one of them has a comparable effect on the performance’s degradation, since LiftNet is still regularised by the remaining ones.

Backbone	$\mathcal{L}_{NF}$	$\mathcal{L}_{bone}$	$\mathcal{L}_{limbs}$	PA-MPJPE$\downarrow$	MPJPE$\downarrow$
S	✓	✓	✓	60.1	82.7
	✗	✓	✓	239.2	342.8
	✓	✗	✓	49.1	71.4
	✓	✓	✗	48.3	70.9
C	✓	✓	✓	45.4	69.9

Table 5: Ablation study for RegNet. The last row corresponds to our full model. S = Supervised. C = Contrastive.

Camera model	$\mathcal{L}_{NF}$	$\mathcal{L}_{bone}$	$\mathcal{L}_{def}$	$\mathcal{L}_{limbs}$	$\mathcal{L}_{2D}$	$\mathcal{L}_{3D}$	PA-MPJPE$\downarrow$	MPJPE$\downarrow$
W	✓	✓	✓	✓	✓	✓	47.7	84.9
	✗	✓	✓	✓	✓	✓	232.6	293.0
	✓	✗	✓	✓	✓	✓	41.5	56.4
	✓	✓	✗	✓	✓	✓	41.3	55.8
	✓	✓	✓	✗	✓	✓	40.9	54.9
	✓	✓	✓	✓	✗	✓	40.1	54.3
	✓	✓	✓	✓	✓	✗	39.5	53.4
F	✓	✓	✓	✓	✓	✓	30.7	50.8

Table 6: Ablation study for LiftNet. The last row correspond to our complete model. W = Weak perspective camera. F = Full perspective camera.

Figure 5: EPOCH qualitative results on MPI-INF-3DHP [36] (columns 1, 2, 3, 4), 3DPW [48] (columns 5, 6). Rows: input images, RegNet output, LiftNet output (front and side view). Our method can generalize to unseen in-the-wild data (3DPW) even if only trained on Human3.6M data.

4.2 Qualitative results

Fig. 5 shows our qualitative results on challenging poses for Human3.6M and MPI-INF-3DHP. Even in the presence of occlusions and rare poses (e.g. sitting on a chair, lying on the floor with crossed legs), both RegNet and LiftNet obtain visually plausible 3D poses. Moreover, we display results on 3DPW which is unseen at training time. Even if the scale of the 3D pose is different, we still obtain plausible poses from challenging images, demonstrating the ability of our EPOCH approach to generalize to unseen scenarios.

5 Conclusions

In this paper, we presented EPOCH, a novel framework that jointly estimates the 3D pose of cameras and humans consisting of LiftNet and RegNet. LiftNet performs the unsupervised 3D lifting starting from estimations of both 2D poses and camera parameters. To address the unavailability of camera parameters in real world scenarios, we design RegNet, a novel human pose regressor that can jointly estimate 2D and 3D poses as well as perspective camera parameters using weak 2D pose supervision. We show that an estimated full perspective camera allows us to substantially improve the unsupervised 3D human pose estimation accuracy and consistency over state-of-the-art results. By estimating the camera only from 2D poses without any 3D or camera ground truth, we can generalise to unseen data, making a step forward towards fully unsupervised 3D HPE in-the-wild. Supplementary Materials

6 Code

The code will be made available in a Github repository upon acceptance.

7 Details about the Normalizing Flows’ invertible function

As in [49], we design a NF to estimate the plausibility of 2D poses. We train the NF to map each pose $x\in\mathbb{R}^{2J}$ onto a sample point $z$ of a Gaussian Distribution $p_{z}(z)$. Given a 2D pose $x$ and an invertible function $f$ such that $\mathbf{f}(z)=x$, the PDF of $x$ can be computed as

$$p_{x}(\mathbf{x})=p_{z}(\mathbf{f}^{-1}(x))\left|\det\left(\frac{\partial\mathbf{f}^{-1}}{\partial\mathbf{x}}\right)\right|,$$ (8)

where $\frac{\partial\mathbf{f}^{-1}}{\partial\mathbf{x}}$ is the Jacobian matrix of the inverse transformation. The NF is trained offline to learn the PDF $p_{z}(\mathbf{f}^{-1}(x))$ of the 2D poses in a chosen training dataset. Once trained, the probability $p_{x}(\mathbf{x})$ can be computed for a new sample $x$ and used as a measure of its plausibility to lie within the distribution of the learned dataset.

8 Camera matrix notation

In the paper we refer to the intrinsic $[K]$ and extrinsic $[R|t]$ camera matrices as detailed below.

The intrinsic matrix, representing the internal parameters of a camera, is defined as:

$$[K]=\begin{bmatrix}f_{w}&sk&c_{w}\\ 0&f_{h}&c_{h}\\ 0&0&1\end{bmatrix}$$

Where:

• $f_{w}$ and $f_{h}$ are the focal lengths in the x and y directions.

• $sk$ represents skew, which is usually zero in most cameras. In the paper we assume $sk=0$ for all cameras.

• $(c_{w},c_{h})$ is the principal point, the optical center of the image.

The extrinsic matrix, denoting the rotation and translation of the camera with respect to the world coordinate system, can be expressed as:

$$[R|t]=\left[\begin{array}[]{ccc|c}r_{11}&r_{12}&r_{13}&t_{X}\\ r_{21}&r_{22}&r_{23}&t_{Y}\\ r_{31}&r_{32}&r_{33}&t_{Z}\end{array}\right]$$

Where:

• $[R]$ represents the rotation matrix.

• $t$ represents the translation vector.

9 Mathematical derivation of the intrinsic parameters from the image

The scaling parameters $s_{w}$ and $s_{h}$ in the paper are computed as follows:

$$s_{w}=\frac{W}{W_{BB}*\mu_{h}}$$

$$s_{h}=\frac{H}{H_{BB}*\mu_{h}}$$

where $\mu_{h}$ is the mean length of the vector from the root joint to the head joint, as in [49]. $W_{BB},H_{BB}$ are the width and height of the bounding box in pixel prior scaling it to $224\times 224$.

Taking inspiration from [31], we estimate the focal length starting from the uncropped input image width $W_{full}$ and height $H_{full}$ as follows:

$$f=\sqrt{W_{full}^{2}+H_{full}^{2}}$$

and scale it along each axis

$$f_{w}=f*s_{w}$$

$$f_{h}=f*s_{h}$$

The principal point is computed as:

$$c_{w}=\Big{(}C_{w}-LEFT-\frac{W}{2}\Big{)}*s_{w}$$

$$c_{h}=\Big{(}C_{h}-TOP-\frac{H}{2}\Big{)}*s_{h}$$

where $C_{w}=\frac{W_{full}}{2}$, $C_{h}=\frac{H_{full}}{2}$. $LEFT$ and $TOP$ are the pixel coordinates of the top-left corner of the unscaled bounding box in the full-size image.

10 Mathematical derivation of the extrinsic parameters from the capsule

When predicting capsules, we estimate $\Gamma\in\mathbb{R}^{3J}$ from which we can compute the rotation matrix $[R]$ and translation vector $t$ with respect to the input image’s viewpoint.

Starting from $\Gamma$ we compute the average vector $\bar{\Gamma}\in\mathbb{R}^{3}$. $\bar{\Gamma}$ is expressed as $[\theta_{X},\theta_{Y},w_{p}]$, where $\theta_{X}$ and $\theta_{Y}$ are the euler rotation angles in the $X$ and $Y$ world-space axis and $w_{p}$ is the distance from the pelvis to the camera along the Z axis. We discard the rotation along the $Z$ axis because that would lead to infinite configurations of the predicted camera and 3D pose. Thus, we set $\theta_{Z}=0$.

Starting from $[\theta_{X},\theta_{Y},0]$, we can derive the rotation matrix as:

$$[R]=\left[\begin{array}[]{ccc}1&0&0\\ 0&\cos(\theta_{X})&-\sin(\theta_{X})\\ 0&\sin(\theta_{X})&\cos(\theta_{X})\end{array}\right] \left[\begin{array}[]{ccc}\cos(\theta_{Y})&0&\sin(\theta_{Y})\\ 0&1&0\\ -\sin(\theta_{Y})&0&\cos(\theta_{Y})\end{array}\right]$$

Starting from $w_{p}$, we can derive the translation vector. We define a setting in which the pelvis is centered in $(X,Y,Z)=(0,0,0)$ in world coordinates and in $(u,v)=(0,0)$ on the image plane. Thus we can express the relation between the pelvis in 2D and 3D as:

$$\begin{bmatrix}0\\ 0\\ w_{p}\end{bmatrix}=\begin{bmatrix}f_{w}&0&c_{w}\\ 0&f_{h}&c_{h}\\ 0&0&1\end{bmatrix}\left[\begin{array}[]{ccc|c}r_{11}&r_{12}&r_{13}&t_{X}\\ r_{21}&r_{22}&r_{23}&t_{Y}\\ r_{31}&r_{32}&r_{33}&t_{Z}\end{array}\right]\begin{bmatrix}0\\ 0\\ 0\\ 1\end{bmatrix}$$

from which we can obtain

$$\begin{bmatrix}0\\ 0\\ w_{p}\end{bmatrix}=\begin{bmatrix}f_{w}*t_{X}+c_{w}*t_{Z}\\ f_{h}*t_{Y}+c_{h}*t_{Z}\\ t_{Z}\end{bmatrix}$$

thus deriving

$$\begin{bmatrix}t_{X}\\ t_{Y}\\ t_{Z}\end{bmatrix}=\begin{bmatrix}-\frac{c_{w}*w_{p}}{f_{w}}\\ -\frac{c_{h}*w_{p}}{f_{h}}\\ w_{p}\end{bmatrix}$$

11 Optimization of Loss Function Weights

In RegNet and LiftNet, we encounter a discrepancy in the scale of different components of the loss function. Specifically, certain losses such as $\mathcal{L}_{RLE}$ and $\mathcal{L}_{NF}$ are formulated as negative-log-likelihoods, resulting in predominantly highly negative values. In contrast, other components of the loss function yield positive values close to zero. To facilitate a more stable and efficient optimization process, we implement a scaling mechanism for $\mathcal{L}_{RLE}$ and $\mathcal{L}_{NF}$ to align their value ranges with the other loss components. More in detail, we shift the loss function vertically by its expected lowest value and divide it by its expected vertical span, to make the scaled loss fit the interval $[0,1]$.

Additionally, we introduce a non-trainable hyperparameter $\lambda$, set to $10$, to strategically weight certain loss components more heavily in the overall loss function. This approach allows for greater emphasis on key aspects of the learning objective.

For RegNet, we apply this weighting strategy to $\mathcal{L}_{RLE}$, recognizing its critical role in the network’s performance. In the case of LiftNet, the weights are applied to $\mathcal{L}_{2D}$ and $\mathcal{L}_{bone}$. This ensures the integrity of the pose cycle consistency and maintains the correct proportions of the 3D skeletal model in terms of bone lengths.

In both RegNet and LiftNet, $\mathcal{L}_{limbs}$ is treated as a regularization term. Accordingly, it is scaled down by a factor of $0.1$, reflecting its role in preventing overfitting and ensuring generalization, rather than directly driving the primary learning objective.

12 Additional qualitative results

In Fig. 6 we show additional qualitative results for LiftNet. Our method shows promising results even when the original annotation $x$ is not correct (see Fig. 7), since the estimated camera ($[K][R|t]$) and the estimated 2D pose ($\hat{x}$) that RegNet gives as output are robust enough to provide a good pseudo-ground truth.

Figure 6: Intermediate results of our LiftNet architecture on MPI-INF-3DHP [36], as detailed in Fig. 3 in the main paper.

Figure 7: Additional LiftNet intermediate results on MPI-INF-3DHP [36]. In this particular example, our 2D joint predictions $\hat{x}$ coming from the RegNet correct an annotation error (right and left legs are flipped) in the original annotation $x$, leading to a possibly better 3D annotation as well.