EPro-PnP: Generalized End-to-End Probabilistic
Perspective-n-Points for Monocular
Object Pose Estimation

In general, geometry-based methods exploit the points, edges or other types of representation that are subject to the projection constraints under the perspective camera. Then, the pose can be solved by optimization. A large body of work utilizes point representation, which can be categorized into sparse keypoints and dense correspondences. BB8 [15] and RTM3D [19] locate the corners of the 3D bounding box as keypoints, while PVNet [13] defines the keypoints by farthest point sampling and Deep MANTA [20] by handcrafted templates. On the other hand, dense correspondence methods [17, 18, 8, 28, 16] predict pixel-wise 3D coordinates within a cropped 2D region. Most existing geometry-based methods follow a two-stage strategy, where the intermediate representations (i.e., 2D-3D correspondences) are learned with a surrogate loss function, which is sub-optimal compared to end-to-end learning.

To mitigate the limitation of surrogate correspondence learning, end-to-end approaches have been proposed to backpropagate the gradient from pose to intermediate representation. Using implicit differentiation w.r.t. the optimal pose or its approximations, Brachmann and Rother [10] propose a dense correspondence network where 3D points are learnable, BPnP [12] predicts 2D keypoint locations, and BlindPnP [11] learns the corresponding weight matrix given a set of unordered 2D/3D points. The above methods are all coupled with surrogate regularization loss, otherwise convergence is not guaranteed due to numerical instability [10] and the non-differentiable nature of the optimal pose. Under the probabilistic framework, these methods can be regarded as a Laplace approximation approach (Section 3.1).

Beyond point correspondence, RePOSE [7] proposes a feature-metric correspondence network trained by backpropagating the PnP solver (e.g. Levenberg-Marquardt), but it is insufficient under pose ambiguity although it can be leveraged as a local regularization technique in our framework (Section 3.4).

Probabilistic methods account for uncertainty in the model and the data, known respectively as epistemic and aleatoric uncertainty [29]. The latter involves interpreting the prediction as learnable probabilistic distributions. Discrete categorical distribution via Softmax has been widely adopted as a smooth approximation of one-hot $\operatorname*{arg\,max}$ for end-to-end classification. This inspired works such as DSAC [9], a smooth RANSAC with a finite hypothesis pool. Meanwhile, tractable parametric distributions (e.g., normal distribution) are often used in predicting continuous variables [30, 31, 29, 32, 33, 28], and mixture distributions can be employed to further capture ambiguity [34, 35, 36], e.g., ambiguous 6DoF pose [37]. In this paper, we propose yet a unique contribution: backpropagating a complicated continuous distribution derived from a nested optimization layer (the PnP layer) approximated by importance sampling, essentially making it a continuous counterpart of Softmax.

Given an object proposal, our goal is to predict a set $X=\mathopen{}\mathclose{{}\left\{x^{\text{3D}}_{i},x^{\text{2D}}_{i},w^{\text{2D}}_{i}}\right\}_{i=1}^{N}$ of $N$ corresponding points, with 3D object coordinates $x^{\text{3D}}_{i}\in\mathbb{R}^{3}$, 2D image coordinates $x^{\text{2D}}_{i}\in\mathbb{R}^{2}$, and 2D weights $w^{\text{2D}}_{i}\in\mathbb{R}^{2}_{+}$, from which a weighted PnP problem can be formulated to estimate the object pose relative to the camera.

The essence of a PnP layer is searching for an optimal pose $y$ (expanded as rotation matrix $R$ and translation vector $t$) that minimizes the cumulative squared weighted reprojection error:

where $\pi(\cdot)$ is the projection function with camera intrinsics involved, $\circ$ stands for element-wise product, and $f_{i}(y)$ compactly denotes the weighted reprojection error.

Eq. (1) formulates a non-linear least squares problem that may have non-unique solutions, i.e., pose ambiguity [21, 22]. Previous work [10, 12, 11] only backpropagates through a local solution $y^{\ast}$, which is inherently unstable and non-differentiable. To construct a differentiable alternative for end-to-end learning, we model the PnP output as a distribution of pose, which guarantees differentiable probability density. The cumulative error is considered to be the negative logarithm of the likelihood function $p(X|y)$ defined as:

With an additional prior pose distribution $p(y)$, we can derive the posterior pose $p(y|X)$ via the Bayes theorem. Using an uniform prior in the domain $Y$, the posterior density is simplified to the normalized likelihood:

Eq. (3) can be interpreted as a continuous counterpart of categorical Softmax.

In this section, we introduce a GPU-friendly efficient Monte Carlo approach to the integration in the proposed loss function, based on the Adaptive Multiple Importance Sampling (AMIS) algorithm [23].

Considering $q(y)$ to be the probability density function of a proposal distribution that approximates the shape of the integrand $\exp-\frac{1}{2}\sum_{i=1}^{N}\mathopen{}\mathclose{{}\left\|f_{i}(y)}\right\|^{2}$, and $y_{j}$ to be one of the $K$ samples drawn from $q(y)$, the estimation of the second term $\mathcal{L}_{\text{pred}}$ in Eq. (5) is thus:

where $v_{j}$ compactly denotes the importance weight at $y_{j}$. Eq. (6) gives the vanilla importance sampling, where the choice of proposal $q(y)$ strongly affects the numerical stability. The AMIS algorithm is a better alternative as it iteratively adapts the proposal to the integrand.

In brief, AMIS utilizes the sampled importance weights from past iterations to estimate the new proposal. Then, all previous samples are re-weighted as being homogeneously sampled from a mixture of the overall sum of proposals. [23] Initial proposal can be determined by the mode and covariance of the predicted pose distribution (see supplementary for details). A pseudo-code is given below.

In this paper, we empirically set the AMIS iteration count $T$ to 4, and the number of samples per iteration $K^{\prime}$ to 128 for 6DoF pose and 32 for 4DoF pose (1D yaw-only orientation). These hyperparameters can be adjusted to balance computation and accuracy.

Although backpropagation can be simply implemented with automatic differentiation packages, here we analyze the gradients of the loss function for an intuitive understanding of the learning process. In general, the gradients of the loss function defined in Eq. (5) is:

where the first term is the gradient of reprojection errors at target pose, and the second term is the expected gradient of reprojection errors over predicted pose distribution, which is approximated by backpropagating the importance weights in the Monte Carlo pose loss.

In practice, we observe that the KL divergence loss has two limitations:

• •

  While the KL divergence is a good metric for the probabilistic distribution, existing evaluation protocols are all based on the point estimate of pose $y^{\ast}$. Therefore, for inference it is still required to locate a mode $y^{\ast}$ of the posterior $p(y|X)$ by solving the PnP problem in Eq. (1), which could be sub-optimal if trained solely with the KL loss.

• •

  The 2D-3D correspondences are underdetermined if we only impose the KL loss when training the network. Learning these entangled elements could be difficult if the network architecture is not designed carefully with preferable inductive bias.

The above limitations can be mitigated by an additional regularization loss on $y^{\ast}$ that backpropagates through the Gauss-Newton (GN) least squares solver or its variants [7]. We call it the derivative regularization loss, since GN is a derivative-based optimizer, and the loss therefore acts on the derivatives of the log-density $\log{p(y|X)}$ to direct the GN increment $\Delta y$ towards the true pose $y_{\text{gt}}$.

To employ the regularization during training, a detached solution $y^{\ast}$ is obtained first. Then, at $y^{\ast}$, a final GN increment is evaluated (which ideally equals 0 if $y^{\ast}$ has already converged to the local optimum):

where $F(y^{\ast})=\mathopen{}\mathclose{{}\left[f_{1}^{\text{T}}(y^{\ast}),f_{2}^{\text{T}}(y^{\ast}),\cdots,f_{N}^{\text{T}}(y^{\ast})}\right]^{\text{T}}$ is the flattened weighted reprojection errors of all points, $J=\mathop{}\!\mathrm{\partial}{F(y)}/\mathop{}\!\mathrm{\partial}{y^{\text{T}}}\negmedspace\bigm{|}_{y=y^{\ast}}$ is the Jacobian matrix, $J^{\text{T}}F(y)$ equals the gradient $g$ of the negative log-likelihood (NLL) w.r.t. object pose, i.e., $\mathop{}\!\mathrm{\partial}{\frac{1}{2}\sum_{i=1}^{N}\mathopen{}\mathclose{{}\left\|f_{i}(y)}\right\|^{2}}/\mathop{}\!\mathrm{\partial}{y}$, and $J^{\text{T}}J$ is an approximation of the Hessian matrix $H=\mathop{}\!\mathrm{\partial}{g}/\mathop{}\!\mathrm{\partial}{y^{\text{T}}}$. We therefore design the regularization loss as follows:

where $l(\cdot,\cdot)$ is a distance metric for pose. We adopt smooth L1 for position and cosine similarity for orientation (see supplementary materials for details). Note that the gradient is only backpropagated through $\Delta y$, which is analytically differentiable w.r.t. the 2D-3D correspondences.

This loss not only addresses the first limitation by moving $y^{\ast}$ towards $y_{\text{gt}}$, but also partially disentangles the 2D-3D correspondences. To analyze the effect of the loss on the correspondences, we consider a local approximation of Eq. (10), assuming equal weights for position and orientation:

Note that $(J^{\text{T}}J)^{-1}J^{\text{T}}$ is also the pseudo inverse of the matrix $J$, which can be denoted by $J^{+}$ for brevity. Then, taking the first-order approximation $F(y^{\ast})=F(y_{\text{gt}})+J\mathopen{}\mathclose{{}\left(y^{\ast}-y_{\text{gt}}}\right)$, the loss can be approximated into:

This indicates that the derivative regularization loss is analogous to the reprojection-based surrogate loss $\mathopen{}\mathclose{{}\left\|F(y_{\text{gt}})}\right\|^{2}$ (Section 3.1.2). Although the extra weighting matrix $J^{+}$ makes the individual elements in the reprojection vector $F(y_{\text{gt}})$ underdetermined, over multiple samples and mini-batches there remains a tendency of independently minimizing each of the elements, i.e., minimizing the reprojection error of each correspondence. Thus, it helps to overcome the potential training difficulties associated with the KL loss.

The regularization loss can also serve as an independent objective for training pose estimators, akin to RePOSE [7]. However, since we observe that this objective alone is not effective in addressing pose ambiguity, it is treated as a secondary regularization in this study.

Following [28], we compute a dynamic loss weight for $\mathcal{L}_{\text{KL}}$ so that the magnitude of its gradients is consistent regardless of the entropy of the distribution. This is implemented by computing the exponential moving average (EMA) of the 1-norm of the sum of weights $\mathopen{}\mathclose{{}\left\|\sum_{i=1}^{N}{w^{\text{2D}}_{i}}}\right\|_{1}$, and using the reciprocal of the EMA value as the dynamic loss weight for $\mathcal{L}_{\text{KL}}$. Intuitively, this cancels out the effect of the magnitude of $w^{\text{2D}}_{i}$ on the loss gradients w.r.t. $x^{\text{2D}}_{i}$ and $x^{\text{3D}}_{i}$.

For the PnP formulation in Eq. (1), the plain L2 reprojection errors $\mathopen{}\mathclose{{}\left\|f_{i}(y)}\right\|^{2}$ are sensitive to outliers, which limits the model’s expressiveness in representing multi-modal distributions that characterizes ambiguity. Therefore, we robustify the reprojection errors using the Huber kernel $\rho(\cdot)$, yielding an alternative formulation:

The Huber kernel with threshold $\delta$ is defined as:

To robustify the weighted reprojection errors of various scales, we adopt an adaptive threshold $\delta$ defined as a function of the weights ${w^{\text{2D}}_{i}}$ and 2D coordinates ${x^{\text{2D}}_{i}}$:

with the relative threshold $\delta_{\text{rel}}$ as a hyperparameter, and the mean vectors $\bar{w}^{\text{2D}}=\frac{1}{N}\sum_{i=1}^{N}{w_{i}^{\text{2D}}},\,\bar{x}^{\text{2D}}=\frac{1}{N}\sum_{i=1}^{N}{x_{i}^{\text{2D}}}$.

Accordingly, the reprojection errors $F(y)$ and Jacobian matrix $J$ in Eq. (9) have to be rescaled (see supplementary).

Since the LM solver only finds a local solution, initialization plays a determinant role in dealing with ambiguity. We implement a random sampling algorithm analogous to RANSAC, to search for the global optimum efficiently.

Given the $N$-point correspondence set $X=\mathopen{}\mathclose{{}\left\{x^{\text{3D}}_{i},x^{\text{2D}}_{i},w^{\text{2D}}_{i}}\right\}_{i=1}^{N}$, we generate $M$ subsets consisting of $n$ corresponding points each ($3\leq n<N$), by repeatedly sub-sampling $n$ indices without replacement from a multinomial distribution, whose probability mass function $p(i)$ is defined by the corresponding weights:

From each subset, a pose hypothesis can be solved via the LM algorithm within very few iterations (e.g. 3 iterations). This is implemented as a batch operation on GPU, and is rather efficient for small subsets. We take the hypothesis with maximum log-likelihood $\log{p(X|y)}$ as the initial point, starting from which subsequent LM iterations are computed on the full set $X$.

The original CDPN feeds cropped image regions within the detected 2D boxes into the pose estimation network, to which two decoupled heads are appended for rotation and translation respectively. The rotation head is PnP-based while the translation head uses explicit center and depth regression. This paper discards the translation head to focus entirely on PnP, and modifies only the last layer of the rotation head for strict comparison to the baseline.

As shown in Fig. 4, apart from the standard 3D coordinate map, the network predicts a 2-channel weight map (originally it is a single channel segmentation mask). We find it necessary to predict a global scale $w^{\text{2D}}_{\text{S}}$ separately, and apply it to the normalized weights ${w^{\text{2D}}_{\text{N}}}_{i}$ that satisfies $\sum_{i=1}^{N}{{w^{\text{2D}}_{\text{N}}}_{i}}=[1,1]^{\text{T}}$. Intuitively, the global scale controls the entropy of the pose distribution $p(y|X)$ as it scales the entire log-likelihood, while the normalized weights determines the relative importance of each correspondence. This helps to overcome the entangling effect of the KL loss mentioned in Section 3.4. Inspired by the attention mechanism [24], the normalized weights are activated via spatial Softmax, focusing on important regions in the image. The global scale is usually inversely proportional to the 2D size of the object due to the uncertainty in reprojection, and is hard-coded as such in this network.

The original CDPN imposes masked coordinate regression loss[18] to learn the dense correspondences, using the ground truth object 3D models to render the target masks and 3D coordinate maps. With EPro-PnP, however, this extra geometry supervision is optional, as we demonstrate that the entire network can be trained solely by the KL loss $\mathcal{L}_{\text{KL}}$ and/or the derivative regularization loss $\mathcal{L}_{\text{reg}}$. To reduce the Monte Carlo overhead, 512 points are randomly sampled from the 64×64 dense points to compute $\mathcal{L}_{\text{KL}}$.

As in CDPN, we use the LineMOD [6] 6DoF pose estimation dataset to conduct our experiments. The dataset consists of 13 sequences, each containing about 1.2K images annotated with 6DoF poses of a single object. Following [36], the images are split into the training and testing sets, with about 200 images per object for training. For data augmentation, we use the same synthetic data as in CDPN [18].

We use two common metrics for evaluation: ADD(-S) and $n\text{\textdegree},n\,\text{cm}$. The ADD measures whether the average deviation of the transformed model points is less than a certain fraction of the object’s diameter (e.g., ADD-0.1d). For symmetric objects, ADD-S computes the average distance to the closest model point. $n\text{\textdegree},n\,\text{cm}$ measures the accuracy of pose based on angular/positional error thresholds. All metrics are presented as percentages.

Despite that some objects in the dataset are nearly rotational symmetric, we observe that our model has no trouble identifying their exact orientations. Therefore, the presented results shall be closer to the scenario without pose ambiguity.

For strict comparison, general settings are kept the same as in CDPN [18] (with ResNet-34 [40] as backbone). As shown in Table I, the original CDPN-Full (A0) trains the network in 3 stages totaling 480 epochs using RMSprop. With the translation head removed, we only train the rotation head in a single stage of 160 epochs (A1), which greatly impacts the pose accuracy (45.75 vs. 63.21). Additionally, we improve the baseline by using the LM solver with Huber kernel at test time, and increase the batch size to 32 for less training wall time (A2). Instead of using the advanced initialization technique in Section 4.3, we adopt the simple EPnP [41] initialization without RANSAC.

As shown in Table II, we conduct ablation studies to reveal the contributions of the Monte Carlo KL loss $\mathcal{L}_{\text{KL}}$, the derivative regularization loss $\mathcal{L}_{\text{reg}}$, the original coordinate regression loss $\mathcal{L}_{\text{crd}}$ in CDPN [18], and initializing the model with pretrained weights from A1.

As shown in Table III, when the coordinate regression loss is removed, i.e., object 3D models are unavailable, both implicit differentiation and reprojection loss fail to learn the pose properly. Yet EPro-PnP manages to learn the 3D coordinates and weights from scratch. This validates that EPro-PnP can be used as a general pose estimator without relying on geometric prior.

As shown in Table IV, although we base EPro-PnP on the older baseline CDPN [18], the results are better than some of the more advanced methods, e.g., the pose refiner RePOSE [7] that adds extra overhead to the PnP-based initial estimator PVNet [13]. Among all these entries, EPro-PnP is the most straightforward as it simply solves the PnP problem itself, without refinement network [7, 8, 42], explicit depth prediction [18, 43], or multiple representations [14].

Moreover, removing the translation head (depth prediction) from the original CDPN-Full results in far fewer parameters in our model (from 113M to 27M) , and the overall inference speed is more than twice as fast as CDPN-Full (including dataloading, measured at a batch size of 32), even though we introduce the iterative LM solver. Furthermore, faster inference is possible if the number of points $N=64\times 64$ is reduced to an optimal level.

As illustrated in Figure 5, the weight maps predicted by the model trained with the KL loss (B0) tend to be more focused on important parts of the objects (e.g., the head and handle of the watering can), while those with the derivative regularization loss (B1) are more evenly spread out. Combining the two loss functions (B2) leads to more reasonable weighting, and more details in the object geometry (represented by $x^{\text{3D}}$). With additional geometry pretraining and supervision (B6), the model outputs sharper correspondence maps, which contribute to higher pose accuracy and lower entropy of the probabilistic pose.

As shown in Figure 6, the deformable correspondence network is an extension of the FCOS3D [44] framework. The original FCOS3D is a one-stage detector that directly regresses the center offset, depth, and yaw orientation of multiple objects for 4DoF pose estimation. In our adaptation, the outputs of the multi-level FCOS head [45] are modified to generate object queries instead of directly predicting the pose. Inspired by Deformable DETR [27], the appearance and position of a query is disentangled into the object embedding vector and the reference point. Moreover, to better distinguish objects of different classes, we learn a set of class embedding vectors, one of which will be selected according to the object label to be aggregated into the object embedding vector via addition (not shown in Figure 6 for brevity).

With the object queries, a multi-head deformable attention layer [27] is adopted to sample the key-value pairs from interpolated dense feature map, with the value projected into point-wise features (point feat), and meanwhile aggregated into the object-level features (obj feat).

The point features are passed into a subnet that predicts the 3D points and corresponding weights (normalized by Softmax). Following MonoRUn [28], the 3D points are set in the normalized object coordinate (NOC) space to handle categorical objects of various sizes.

The object features are responsible for predicting the object-level properties: (a) the 3D score (i.e., 3D localization confidence), (b) the global weight scale, (c) the 3D box size for recovering the absolute scale of the 3D points, and (d) other optional properties (velocity, attribute) required by the nuScenes benchmark [4].

We evaluate the deformable correspondence network on the nuScenes 3D object detection benchmark [4], which provides a large scale of data collected in 1000 scenes. Each scene contains 40 keyframes, annotated with a total of 1.4M 3D bounding boxes from 10 categories. Each keyframe includes 6 RGB images collected from surrounding cameras. The data is split into 700/150/150 scenes for training/validation/testing. The official benchmark evaluates the average precision with true positives judged by 2D center error on the ground plane. The mAP metric is computed by averaging over the thresholds of 0.5, 1, 2, 4 meters. Besides, there are 5 true positive metrics: Average Translation Error (ATE), Average Scale Error (ASE), Average Orientation Error (AOE), Average Velocity Error (AVE) and Average Attribute Error (AAE). Finally, there is a nuScenes detection score (NDS) computed as a weighted average of the above metrics.

An example of the monocular detection result is shown in Figure 7. We observe that the red 2D points (indicating greater $x^{\text{3D}}$ in the X axis) are usually spread right over the objects, which mainly determines the orientation, while the green 2D points (indicating greater $x^{\text{3D}}$ in the Y axis) are off the top and bottom of the objects, which determines the position (mainly the depth). It seems that the network learns to associate object depth to the height of the object’s projection, since the height invariant to 1D orientation in the ground plane.

Figure 8 shows that the flexibility of EPro-PnP allows predicting multimodal distributions with strong expressive power, successfully capturing the orientation ambiguity without discrete multi-bin classification [44, 57] or complicated mixture model [37].

The average inference time per frame (comprising a batch of 6 surrounding 1600×672 images, without TTA) is shown in Table VII, measured on RTX 3090 GPU and Core i9-10920X CPU. On average, the batch PnP solver takes 26 ms/46 ms processing 655.3 objects per frame, before non-maximum suppression (NMS).