DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Image

Abstract

Reconstructing 3D hand-face interactions with deformations from a single image is a challenging yet crucial task with broad applications in AR, VR, and gaming. The challenges stem from self-occlusions during single-view hand-face interactions, diverse spatial relationships between hands and face, complex deformations, and the ambiguity of the single-view setting. The first and only method for hand-face interaction recovery, Decaf [97], introduces a global fitting optimization guided by contact and deformation estimation networks trained on studio-collected data with 3D annotations. However, Decaf suffers from a time-consuming optimization process and limited generalization capability due to its reliance on 3D annotations of hand-face interaction data. To address these issues, we present DICE, the first end-to-end method for Deformation-aware hand-face Interaction reCovEry from a single image. DICE estimates the poses of hands and faces, contacts, and deformations simultaneously using a Transformer-based architecture. It features disentangling the regression of local deformation fields and global mesh vertex locations into two network branches, enhancing deformation and contact estimation for precise and robust hand-face mesh recovery. To improve generalizability, we propose a weakly-supervised training approach that augments the training set using in-the-wild images without 3D ground-truth annotations, employing the depths of 2D keypoints estimated by off-the-shelf models and adversarial priors of poses for supervision. Our experiments demonstrate that DICE achieves state-of-the-art performance on a standard benchmark and in-the-wild data in terms of accuracy and physical plausibility. Additionally, our method operates at an interactive rate (20 fps) on an Nvidia 4090 GPU, whereas Decaf requires more than 15 seconds for a single image. Our code will be publicly available upon publication.

1 Introduction

Hand-face interaction is a common behavior observed up to 800 times per day across all ages and genders [98]. Therefore, faithfully recovering hand-face interactions with plausible deformations is an important task given its wide applications in AR/VR [87, 36, 113], character animation [88, 136], and human behavior analysis [66, 29, 74]. Given the speed requirement of downstream applications like AR/VR, fast and accurate 3D reconstruction of hand-face interactions is highly desirable. However, several challenges make monocular hand-face deformation and interaction recovery a cumbersome task: 1) the self-occlusion involved in hand-face interaction, 2) the diversity of hand and face poses, contacts, and deformations, and 3) ambiguity in the single-view setting. Most existing methods [91, 75] only reconstruct the hand [92] and face [57] meshes, or unified as a whole body [67, 82] with the body parts, without capturing contacts and deformations. A seminal advance, Decaf [97], recovers hand-face interactions with deformations and contacts taken into account. However, it requires time-consuming optimization, which takes more than 15 seconds per image, rendering it unsuitable for interactive applications. The iterative fitting process of Decaf relies on an accurate estimation of hand and face keypoints and contacts on the hand and face surfaces, which could fail when significant occlusion is present in the image; See Fig. 8 in the Appendix D.1. Additionally, Decaf cannot scale up their training to fruitful hand-face interaction data in the wild, as they require 3D ground-truth annotations, i.e., contact labels and deformations.

\begin{overpic}[width=433.62pt]{Figs/fig_teaser.pdf} \end{overpic}

Figure 1: Our method, DICE, is the first end-to-end approach that captures hand-face interaction and deformation from a monocular image. (a) Decaf validation dataset. (b) In-the-wild images. (c) Use-cases in VR.

To tackle the issues above, we present DICE, the first end-to-end approach for Deformation-aware hand-face Interaction reCovEry from a monocular image. We use a Transformer-based model with the attention mechanism to effectively capture hand-face relationships. Motivated by the global nature of the pose and shape of hand and face and the local nature of the deformation field and contact probabilities, we further propose disentangling the regression of deformation from the pose and shape of hand and face represented by mesh vertex positions into two network branches, which enhances the estimation of deformations and contacts while resulting in accurate and robust hand and face mesh recovery. Instead of directly regressing hand and face parameters, we learn an intermediate non-parametric mesh representation. We then use this representation to regress the pose and shape parameters of hand and face with a neural inverse-kinematics network. Compared with directly regressing the pose and shape parameters which learns the abstract parameters is a highly non-linear process and suffers from image-model misalignment, predicting vertex positions in Euclidean space and then applying inverse-kinematics enhances the reconstruction accuracy [53, 52, 51]. Consequently, our model achieves higher reconstruction accuracy than all previous regression and optimization-based methods. It also reaps the benefits of an animatable parametric hand and face representation that could be readily used by downstream applications.

Meanwhile, despite containing rich annotations, the existing benchmark dataset [97] collected in a studio is still limited in the diversity of hand motions, facial expressions, and appearances. Training a model only on such a dataset limits its generalization capability when applied to in-the-wild data. To achieve robust and generalizable hand-face interaction and deformation recovery, we introduce a weak-supervision training pipeline that utilizes in-the-wild images without the reliance on 3D annotations.

In addition to the 2D keypoint supervision for in-the-wild images, we propose a novel depth supervision pipeline. This pipeline leverages the robust depth prior from a diffusion-based monocular depth estimation model [44], which provides essential geometric information for accurate mesh recovery and captures spatial relationships critical for contact state and deformation estimation. To improve our model’s robustness, we further employ pose priors of the hand and face by introducing hand and face parameter discriminators that learn rich hand and face motion priors from multiple datasets on hand or face separately [79, 140]. By incorporating a small set of real-world images alongside the Decaf dataset and leveraging our weak-supervision pipeline, we markedly enhance the accuracy and generalization capacity of our model.

As a result, our method achieves superior performance in terms of accuracy, physical plausibility, inference speed, and generalizability. It surpasses all previous methods in accuracy on both standard benchmarks and challenging in-the-wild images. Fig. 1 visualizes some results of our method. We conduct extensive experiments to validate our method. In summary, our contribution is three-fold:

• We propose DICE, the first end-to-end learning-based approach that accurately recovers hand-face interactions and deformations from a single image.

• We propose a novel weak-supervised training scheme with depth supervision on keypoints to augment the Decaf data distribution with a diverse real-world data distribution, significantly improving the generalization ability.

• DICE achieves superior reconstruction quality compared to baseline methods while running at an interactive rate (20 fps).

2 Related Work

Extensive efforts have been made to recover meshes from monocular images, including human bodies [2, 71, 53, 5, 14, 116, 112, 111, 64, 43, 4, 128, 25, 59, 110, 18, 11, 40, 63], hands [93, 73, 72, 70, 76, 81, 121, 120, 54, 126], and faces [24, 26, 115, 16, 139, 7, 134, 78, 35, 8, 48, 50]. This also includes recovering the surrounding environments [12, 39, 32, 33, 135, 58, 133, 96, 69, 114] and interacting objects [119, 129, 85, 104, 30, 100, 129, 27, 86, 34, 123, 9, 10, 65, 15] while reconstructing the mesh. The acquired versatile behaviors play a crucial role in various applications, including motion generation [101, 83, 80, 28, 108, 117, 118, 61, 138, 105, 84, 17, 106], augmented reality (AR), virtual reality (VR), and human behavior analysis [130, 122, 132, 131, 29, 66]. In the following, we mainly review the related works on hand, face and full-body mesh recovery.

3D Interacting Hands Recovery. Recent advancements have markedly enhanced the capture and recovery of 3D hand interactions. Early studies have achieved reconstruction of 3D hand-hand interactions utilizing a fitting framework, employing resources such as RGBD sequences [77], hand segmentation maps [74], and dense matching maps [107]. The introduction of large-scale datasets for interacting hands [73, 72] has motivated the development of regression-based approaches. Notably, these include regressing 3D interacting hand directly from monocular RGB images [93, 70, 127, 55, 141]. Additionally, research has extended to recovering interactions between hands and various objects in the environment, including rigid [6, 27, 65, 100, 20, 125, 124, 13], articulated [21], and deformable [103] objects. Following [97], our work distinguishes itself by introducing hand interactions with a deformable face, characterized by its non-uniform stiffness—a significant difference from conventional deformable models. This innovation presents unique challenges in accurately modeling interactions.

3D Human Face Recovery. Research in human face recovery encompasses both optimization-based [1, 102] and regression-based [26, 94] methodologies. Beyond mere geometry reconstruction, recent approaches have evolved to incorporate training networks with the integration of differentiable renderers [24, 139, 137, 109, 11]. These methods estimate variables such as lighting, albedo, and normals to generate facial images and compare them with the monocular input. However, a significant limitation in much of the existing literature is the neglect of the face’s deformable nature and hand-face interactions. Decaf [97] represents a pivotal development in this area, attempting to model the complex mimicry of musculature and the underlying skull anatomy through optimization techniques. In contrast, our work introduces a regression-based, end-to-end method for efficient problem-solving, setting a new benchmark in the field.

3D Full-Body Recovery. The task of monocular human pose and shape estimation involves reconstructing a 3D human body from a single-color image. Optimization-based approaches [2, 82, 95, 90] employ the SMPL model [67], fitting it to 2D keypoints detected within the image. Conversely, regression-based methods [53, 49, 45, 43, 23, 22, 62, 5, 25] leverage deep neural networks to directly infer the pose and shape parameters of the SMPL model. Hybrid methods [46] integrate both optimization and regression techniques, enhancing 3D model supervision. Distinct from these approaches, we follow parametric methods [53, 5, 43, 2] due to its flexibility for animation purposes. Unlike most research in this domain, which primarily concentrates on the main body with only rough estimations of hands and face, our methodology uniquely accounts for detailed interactions between these components.

3 Method

\begin{overpic}[width=433.62pt]{Figs/fig_pipeline_keypoints.pdf} \end{overpic}

Figure 2: Overview of the proposed DICE framework. The input image is first fed to a CNN to extract a feature map, which is then passed to the Transformer-based encoders for mesh and interaction, i.e., MeshNet and InteractionNet. The MeshNet extracts hand and face mesh features, which is then used by the Inverse Kinematics models (IKNets) to predict pose and shape parameters that drive FLAME [57] and MANO [92] models. The InteractionNet predicts per-vertex hand-face contact probabilities and face deform fields from the feature map, the latter is applied to the face mesh output by the FLAME model. To improve the generalization capability, we introduce a Weakly-Supervised Training Scheme using off-the-shelf 2D keypoint detection models [68, 3] and depth estimation models [44] to provide depth supervision on keypoints. In addition, we use head and hand discriminators to constrain the distribution of parameters regressed by IKNets.

Problem Formulation. Following Decaf [97], we adopt the FLAME [57] and MANO [92] parametric models for hand and face. Given a single RGB image $\mathbf{I}\in\mathbb{R}^{224\times 224\times 3}$, the objective of this task is to reconstruct the vertices of hand mesh $\mathbf{V}_{H}\in\mathbb{R}^{778\times 3}$ and face mesh $\mathbf{V}_{F}\in\mathbb{R}^{5023\times 3}$, along with capturing the face deformation vectors $\mathbf{D}\in\mathbb{R}^{5023\times 3}$ caused by hand-face interaction and its non-rigid nature, and per-vertex contact probabilities of hand $\mathbf{C}_{H}\in\mathbb{R}^{778}$ and face ${\mathbf{C}_{F}}\in\mathbb{R}^{5023}$.

3.1 Transformer-based Hand-Face Interaction Recovery

Our model incorporates a two-branch Transformer architecture and integrates inverse-kinematic models, specifically MeshNet, InteractionNet, and IKNets. A differentiable renderer [89] is used to compute depth maps from the predicted mesh for depth supervision, and the hand and face discriminators are used as priors for constraining the hand and face poses; See Fig. 2 for an overview.

Given a monocular RGB image $\mathbf{I}$, we use a pretrained HRNet-W64 [99] backbone to extract a feature map $\bm{X}_{\text{I}}\in\mathbb{R}^{H\times W\times C}$, where $H,W$ are the spatial dimension and $C$ is the channel dimension. Following [63, 64], we flatten the image feature maps and upsample the $H\times W$ feature maps to $N$ feature maps, one for each keypoint and coarse vertex of both hand and face. We then concatenate the $\mathbf{F}^{\prime}\in\mathbb{R}^{N\times C}$-dimensional feature maps with a $N\times 3$-dimensional downsampled hand and face vertices and keypoints of the mean pose, as mesh vertices and joints queries for the transformer, resulting in a feature map $\mathbf{F}\in\mathbb{R}^{N\times(C+3)}$. The mesh vertices and joints also serve as the positional encoding. To effectively model the interaction between the vertices, we mask the image feature maps corresponding to a random subset of vertices.

We propose to use two separate branches, MeshNet and InteractionNet, splitting the regression of mesh vertices and a deformation field for their semantic difference: the mesh positions are more global while the deformation vectors and contact states are relatively local. Specifically, the network is followed by two progressively downsampling transformers: MeshNet, which takes the feature map $\mathbf{F}$ as input and regresses the rough vertex positions of hand $\mathbf{V}_{H}^{\prime}$ and face, $\mathbf{V}_{F}^{\prime}$; and InteractionNet, which first downsamples the feature map $\mathbf{F}$, then uses it to predict the 3D deformation field $\mathbf{D}$ at each face vertex along with the contact labels for each hand and face vertices, $\mathbf{C}_{H}$ and ${\mathbf{C}_{F}}$. Note the contacts and deformations are regressed in the same encoder to model their close relationship: the contacts cause the deformations. We validate our design in Sec. 4.4.

Next, instead of directly regressing the hand and face vertices, we regress the pose and shape of the parametric hand and face models, making the output readily animatable for downstream applications. This is achieved by a neural inverse kinematics model, named IKNet, similar to Kolotouros et al. [47]. Our IKNet takes roughly estimated hand and face mesh vertices $\mathbf{V}_{H}^{\prime}$ and $\mathbf{V}_{F}^{\prime}$ as inputs and predict hand and face pose, shape and expression parameters $(\theta_{\text{h}},\beta_{\text{h}})$, $(\theta_{\text{f-pose}},\beta_{\text{f}},\theta_{\text{f-exp}})$, along with the position and orientation for hand [57] and face [92], respectively. We use the predicted parameters to first obtain the hand mesh and undeformed face mesh $\mathbf{V}_{H}$, $\mathbf{V}_{F}^{*}$. Then, we apply the deformation $\mathbf{D}$ predicted by the InteractionNet on $\mathbf{V}_{F}^{*}$ to get the final deformed face $\mathbf{V}_{F}$. Regressing parameters offers several advantages: first, it enables readily animatable meshes; second, compared to non-parametric regression methods, where meshes typically contain artifacts such as spikes [63, 11, 64], the mesh quality is significantly improved; third, the compact parameter space facilitates a more effective discriminator, which will be discussed in the following section.

3.2 Weakly-Supervised Training Scheme

Although the aforementioned benchmark, Decaf [97], accurately captures hand, face, self-contact, and deformations, it consists of eight subjects and is recorded in a controlled environment with green screens. Training a model only with Decaf limits its generalization capability to in-the-wild images that have far more complex and diverse human identities, hand poses, and face poses.

To further enhance the generalization capability of our model, we train our model with $500$ diverse in-the-wild images of hand-face interaction collected from the internet without the reliance on the 3D ground truth annotations. First, we use 2D hand-and-face keypoints detected by [68] and [3].

Then, we propose to use Marigold [44], a diffusion-based monocular depth estimator pre-trained on a large number of images to generate 2D affine-invariant depth maps for supervision in the direction of depth (see Eq. 4). The depth supervision provides a strong depth prior, which guides the spatial relationship between hand and face meshes, promoting accurate modeling of hand-face interaction. For supervision, we use a differentiable rasterizer [89] to compute a depth map from the predicted hand and face meshes and supervise the network using a depth loss calculated between the depth values of hand and face keypoints and corresponding points on the predicted depth map. The introduced weak-supervision pipeline significantly enhances our model’s generalization capability and robustness, which we investigate in Sec. 4.4. In our experiment, we found that when training the model with only a small dataset of 500 images, we could significantly improve the model’s accuracy and generalization capability. Moreover, we train adversarial priors on the hand and face parameter space on multiple hand and face pose datasets: the face-only RenderMe-360 [79], the hand-only FreiHand [140], and Decaf [97]. This ensures the plausibility of generated face and hand poses and shapes while allowing for flexible poses and shapes beyond the Decaf data distribution to handle in-the-wild cases.

3.3 Loss Functions

Mesh losses $\mathcal{L_{\text{mesh}}}$: For richly annotated data in Decaf [97], we employ $L_{1}$ loss for 3D keypoints, 3D vertices, and 2D reprojected keypoints against their respective ground-truths, following common practice in human- and hand-mesh recovery [63, 11, 18]. We further apply a $L_{1}$ loss $\mathcal{L}_{\text{params}}$ on the estimated hand and face pose, shape, and facial expression against the ground-truth parameters. For in-the-wild data, only the 2D reprojected keypoints are supervised, as this is the only type with corresponding ground truth.

Interaction losses $\mathcal{L_{\text{interaction}}}$: For data in Decaf [97], we impose Chamfer Distance losses to enforce touch for predicted contact vertices and discourage collision. We also have a binary cross-entropy loss to supervise contact labels and a deform loss with adaptive weighting to supervise deform vectors, similar to [97]. For in-the-wild data, we also impose touch and collision losses since they do not require annotations.

Adversarial loss $\mathcal{L}_{\text{adv}}$ are applied to the predicted hand and face parameters for in-the-wild data to constrain their parameter space, and for decaf data to facilitate the training of the discriminators. The adversarial loss is given by:

$$\mathcal{L}_{\text{adv}}(E)=\mathbb{E}_{\theta_{F}\sim p_{E}}\left[\log\left(1-D_{F}(E(I))\right)\right]+\mathbb{E}_{\theta_{H}\sim p_{E}}\left[\log\left(1-D_{H}(E(I))\right)\right].$$ (1)

The losses for the hand and face discriminators are given by:

$$\mathcal{L}_{\text{adv}}(D_{F})=-\left(\mathbb{E}_{\theta_{F}\sim p_{E}}\left[\log\left(1-D_{F}(E(I))\right)\right]+\mathbb{E}_{\theta_{F}\sim p_{\text{data}}}\left[\log\left(D_{F}(\theta)\right)\right]\right)$$ (2)

and

$$\mathcal{L}_{\text{adv}}(D_{H})=-\left(\mathbb{E}_{\theta_{H}\sim p_{E}}\left[\log\left(1-D_{H}(E(I))\right)\right]+\mathbb{E}_{\theta_{H}\sim p_{\text{data}}}\left[\log\left(D_{H}(\theta)\right)\right]\right),$$ (3)

where $E$ jointly denotes the image backbone, the mesh encoder and the parameter regressor, $\theta_{F}=\text{concat}(\theta_{\text{face-shape}},\theta_{\text{face-jaw}},\theta_{\text{face-exp}})$, $\theta_{H}=\text{concat}(\theta_{\text{hand-shape}},\theta_{\text{hand-pose}})$.

Depth loss $\mathcal{L}_{\text{depth}}$: To provide pseudo-3D hand and face keypoints supervision for in-the-wild data, we use a modified SILog Loss [19], an affine-invariant depth loss as our depth supervision $\mathcal{L}_{\text{depth}}$. Formally, let $\hat{K}_{D}$ denote the pseudo-ground-truth affine-invariant depth of the face and hand keypoints, and $K_{D}$ denote the rendered depth for the keypoints,

$$\mathcal{L}_{\text{depth}}=\left[\mathbf{Var}\left(\log(K_{D}+\varepsilon)-\log(\hat{K}_{D}+\varepsilon)\right)\right]^{1/2},$$ (4)

where $\mathbf{Var}$ is the standard variance operator and $\varepsilon=10^{-7}$.

Overall, our loss for the mesh and interaction networks is formulated by

$$\mathcal{L}=\lambda_{\text{mesh}}\mathcal{L_{\text{mesh}}}+\lambda_{\text{interaction}}\mathcal{L_{\text{interaction}}}+\lambda_{\text{adv}}\mathcal{L_{\text{adv}}}+\lambda_{\text{depth}}\mathcal{L_{\text{depth}}},$$ (5)

where $\lambda_{\text{mesh}}=12.5,\lambda_{\text{interaction}}=5,\lambda_{\text{depth}}=2.5,\lambda_{\text{adv}}=1$ for all the experiments in the paper; See more details in Appendix C.

4 Experimental Results

4.1 Datasets and Metrics

Datasets

We employ Decaf [97] for reconstructing 3D face and hand interactions with deformations, along with the in-the-wild dataset we collected with 500 images. We use the hand and face shape, pose, and expression data from Decaf [97], RenderMe-360 [79], and FreiHand [140] for training the adversarial priors. We use the training set of the aforementioned datasets for network training. We utilize the Decaf test set for quantitative evaluation, and additionally, we visualize in-the-wild images from the test set for qualitative evaluation.

Metrics.

We adopt commonly-used metrics for mesh recovery accuracy following [43, 63, 18, 11]:
$\bullet$ Mean Per-Joint Position Error (MPJPE): the average Euclidean distance between predicted keypoints and ground-truth keypoints.
$\bullet$ PAMPJPE: MPJPE after Procrustes Analysis (PA) alignment.
$\bullet$ Per Vertex Error: per vertex error (PVE) with translation.

Following Decaf [97], we use the plausibility metrics mentioned below:
$\bullet$ Collision Distance (Col. Dist.): the average collision distances over vertices and frames;
$\bullet$ Non-Collision Ratio (Non. Col.): the proportion of frames without hand-face collisions;
$\bullet$ Touchness Ratio: the ratio of hand-face contacts among ground truth contacting frames;
$\bullet$ F-Score: the harmonic mean of Non-Collision Ratio and Touchness Ratio.

4.2 Implementation Details

We train the MeshNet, InteractionNet, and IKNet along with the face and hand discriminators with three identical AdamW optimizers with a learning rate of $6\times 10^{-4}$, and a learning rate decay of $1\times 10^{-4}$, optimizing in an alternating manner. Our batch size is set to $16$ during the training stage. Each training task takes $40$ epochs using $48$ hours. The model is trained and evaluated on 8 Nvidia A6000 GPUs with an AMD 128-core CPU. For a fair comparison, all baseline models used the settings in their original papers. Inference times are calculated on a single Nvidia A6000 GPU.

4.3 Performance on Hand-Face Interaction and Deformation Recovery

In addition to baselines considered in Decaf [97], we compare our method with a representative work in human body/hand mesh recovery, METRO, an end-to-end transformer-based model. For a fair comparison, we compare our method with a modified version of METRO [63] for predicting hand and face meshes, with extra output heads added to predict contact and deformation.

\begin{overpic}[width=433.62pt]{Figs/vis_contact.pdf} \end{overpic}

Figure 3: Qualitative results of hand-face interaction, deformation, and contact recovery by DICE on Decaf and in-the-wild images. In contact visualizations, a deeper color indicates a higher contact probability.

\begin{overpic}[width=325.215pt]{Figs/fig_compare_all} \end{overpic}

Figure 4: Qualitative comparsion of DICE, Decaf [97], PIXIE [23], METRO*[64] on Decaf validation set and in-the-wild images. Our method achieves superior reconstruction accuracy and plausibility in the Decaf [97] dataset, while generalizing well to difficult in-the-wild actions unseen in Decaf.

4.3.1 Quantitative Evaluations

Table 1: Comparison of hand-face interaction and deformation recovery on Decaf.

[b] Methods Type 3D Reconstruction Error Physics Plausibility Metrics Running Time (per image; s)$\downarrow$ PVE‡$\downarrow$ MPJPE$\downarrow$ PAMPJPE$\downarrow$ Col. Dist. $\downarrow$ Non. Col. $\uparrow$ Touchness $\uparrow$ F-Score $\uparrow$ Decaf [97] O $9.65$ $-$ $-$ $1.03$ $83.6$ $96.6$ 89.6 $19.59$ Benchmark [68, 57] O $17.7$ $-$ $-$ $19.3$ $64.2$ $73.2$ $68.4$ $16.40$ PIXIE (hand+face) [23] O $26.3$ $-$ $-$ $7.04$ $75.9$ $75.1$ $75.5$ $-$ PIXIE (whole-body) [23] R $39.7$ $-$ $-$ $0.11$ $97.1$ $51.8$ $67.6$ $\underline{\textbf{0.070}}$ METRO* (hand+face) [63] R $11.8$ $15.4$ $11.9$ $0.08$ $80.7$ $54.8$ $65.2$ $0.103$ DICE (Ours) R $\underline{\textbf{8.32}}$ $\underline{\textbf{9.95}}$ $\underline{\textbf{7.27}}$ $0.16$ $66.6$ $79.9$ $\underline{72.7}$ $0.088$

• * parametric version. O and R denote optimization-based and regression-based methods, respectively. ${\ddagger}$ calculated after translating the center of the head to the origin. bold and underline denote the overall best and the best among regression-based approaches, respectively. Note our method operates at an interactive rate (20 fps; 0.049s per image) on an Nvidia 4090 GPU. Here we report the runtime performance on a single A6000 GPU for a fair comparison.

Reconstruction Accuracy In Tab. 1, our method surpasses all baseline methods in terms of reconstruction accuracy, achieving a $7.5\%$ reduction in per-vertex error compared to the current state-of-the-art, Decaf. Note that our method is regression-based and allows inference at an interactive rate, while Decaf [97] uses a cumbersome test-time optimization process, taking more than $700$x more time per image. Decaf also requires using temporal information in successive frames, while our method only uses a single frame. Our method shows a $30\%$ reduction in reconstruction error compared to the modified METRO baseline, and up to $79\%$ reduction compared to other end-to-end baselines.

Plausibility In addition to high accuracy, our method achieves the highest overall physical plausibility (F-Score) among all regression-based methods. Note that Touchness and Non-Collision ratio are complement to each other and are meaningless when considered individually, while F-Score measures the two values as a whole. Our method has a much lower interpenetration distance (Col. Dist.) compared to Benchmark and PIXIE (hand+face), which consider hand and face separately, therefore generating implausible interactions. Note that PIXIE (whole body) and METRO* show lower collision distances with a much lower Touchness than our method, indicating that the reconstructed hands and faces often appear incorrectly as if they are not interacting. On the other hand, our method shows low collision distance with a high Touchness, indicating plausible hand-face interaction reconstruction.

Contact Estimation In Tab. 2, DICE achieves superior contact estimation performance on Decaf dataset, surpassing previous work [97] in F-Score for both face and hand contacts. Here, the F-score provides a comprehensive measure of both the precision and the recall ratio combined. These two metrics are complementary and less meaningful when only considered individually; See Fig. 3 for qualitative results.

Table 2: Comparison of hand-face interaction and deformation recovery on Decaf.

Method	F-score $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	Accuracy$\uparrow$
Decaf (face) [97]	$0.57$	$0.69$	$0.49$	$0.99$
Decaf (hand) [97]	$0.47$	$0.62$	$0.39$	$0.98$
DICE (face)	0.61	$0.64$	$0.57$	$1.00$
DICE (hand)	0.50	$0.55$	$0.45$	$0.98$

4.3.2 Qualitative Evaluations

As discussed in Sec. 3.2, the Decaf [97] dataset is collected in an indoor environment with a green screen, which doesn’t reflect the complex environment where real-world hand-face interactions occur. Therefore, a model only trained with the Decaf dataset might have generalization issues when tested on in-the-wild data. Fig. 4 confirms this result by our model’s superior generalization performance on in-the-wild data with unseen identity and pose. As shown in Fig. 3, our method faithfully reconstructs hand-face interaction and deformation and accurately labels the area of contact.

4.4 Ablation Study

In-the-wild data As shown in Tab. 3, adding weak-supervision training and in-the-wild data for DICE training improves all reconstruction error metrics (PVE*, MPJPE, PAMPJPE) while maintaining a high plausibility (F-Score). This is because the limited pose and identity distribution of the Decaf training dataset may cause the model to overfit, and the inclusion of in-the-wild images out of the Decaf data distribution effectively improves the generalization capability of DICE.

Depth Supervision Although depth supervision is only applied to in-the-wild data, as shown in Tab. 3, removing it also significantly affects performance on the Decaf validation set. Without depth loss, wrong predictions in depth are not penalized for in-the-wild data, introducing noise in the training process, and resulting in erroneous depth predictions in the Decaf dataset. As shown in Appendix Fig. 7, the absence of depth supervision introduces ambiguity in the z-direction, resulting in artifacts such as self-collision.

Adversarial Prior The adversarial prior incorporates diverse but realistic pose and shape distribution beyond Decaf [97], ensuring the reality of regressed mesh while allowing for generalization. As shown in Tab. 3, introducing adversarial supervision improves the accuracy and physical plausibility.

Parameter Supervision Supervising parameters directly, in addition to the indirect supervision of parameters by the mesh losses, improves both plausibility and accuracy. This is because direct parameter supervision eliminates ambiguity, without which the network may resort to other parameter combinations that produce incorrect meshes similar in terms of Euclidean distance.

Intermediate Supervision Removing the constraint that $\mathbf{V}_{F}^{\prime},\mathbf{V}_{H}^{\prime}$ being rough meshes and treating them only as feature maps results in a substantial drop in accuracy and a slight drop in plausibility (F-Score). Also, note that there is a sharp increase in collision distance, which is attributed to the spatial inaccuracy of the final output mesh. This indicates that using an intermediate mesh feature instead of an ordinary feature map increases the spatial accuracy of the output meshes, which also benefits plausibility.

Network Design In Tab. 3, adopting the two-branch architecture, which separates deformation and interaction estimation from mesh vertices regression, improves both accuracy and plausibility.

4.5 Limitations and Future Works

While our method achieves SotA accuracy on the Decaf [97] dataset and generalizes well to unseen scenes and in-the-wild cases, we still have failure cases when the hand-pose interactions are extremely challenging and have severe occlusions (see Appendix D.2). Moreover, while our method effectively recovers hand and face meshes with visually plausible deformations, there remains room for improvement in deformation accuracy and physical plausibility. In the future, physics-based simulation [37, 56, 38, 31, 60, 41] can be used as a stronger prior, producing more physically accurate estimations. In this paper, although we found using 500 in-the-wild images significantly improves the model’s generalization ability, scaling up to a larger amount of in-the-wild data, on the order of millions or billions, would further enhance performance, which we will study in future work.

Table 3: Comparison of hand-face interaction and deformation recovery on Decaf. Bold denotes the best result.

[b] Methods 3D Reconstruction Error Physics Plausibility Metrics PVE*$\downarrow$ MPJPE$\downarrow$ PAMPJPE$\downarrow$ Col. Dist. $\downarrow$ Non. Col. $\uparrow$ Touchness $\uparrow$ F-Score $\uparrow$ DICE (single branch) $9.29$ $11.6$ $8.51$ $0.04$ $87.4$ $57.4$ $69.3$ DICE (w.o. in-the-wild data) $8.93$ $11.0$ $7.50$ $0.11$ $74.6$ $71.9$ 73.3 DICE (w.o. $\mathcal{L}_{\text{depth}}$) $15.6$ $19.5$ $13.7$ $0.64$ $58.6$ $71.1$ $64.2$ DICE (w.o. $\mathcal{L}_{\text{params}}$) $10.3$ $12.8$ $10.4$ $0.08$ $80.9$ $53.9$ $64.7$ DICE (w.o. $\mathcal{L}_{\text{adv}}$) $11.1$ $14.2$ $10.4$ $0.05$ $82.1$ $60.7$ $69.8$ DICE (Full) 8.32 9.95 7.27 $0.16$ $66.6$ $79.9$ $72.7$

5 Conclusion

In this work, we present DICE, the first end-to-end approach for reconstructing 3D hand and face interaction with deformation from monocular images. Our approach features a two-branch transformer structure, MeshNet and InteractionNet, to model local deform field and global mesh geometry. An inverse-kinematic model, IKNet, is used to output the animatable parametric hand and face meshes. We also proposed a novel weak-supervision training pipeline, using a small amount of in-the-wild images and supervising with a depth prior and an adversarial loss to provide pose priors. Benefitting from our network design and training scheme, DICE demonstrates state-of-the-art accuracy and plausibility, compared with all previous methods. Meanwhile, our method achieves a fast inference speed (20 fps), allowing for more downstream interactive applications. In addition to strong performance on the standard benchmark, DICE also achieves superior generalization performance on in-the-wild data.

Appendix A Implementation Details

A.1 CNN Backbone

The CNN backbone used in our framework is an HRNet-W64 [99], initialized with ImageNet-pretrained weights. The weights of the backbone would be updated during training. We extract a ($49\times H$)-dim feature map from this network and upsamples it to a ($N\times H$)-dim feature map, where $N=N_{h_{k}}+N_{f_{k}}+N_{h_{v}}+N_{h_{v}}$, the total number of head and hand keypoints $N_{h_{k}},N_{f_{k}}$ and vertices $N_{h_{v}},N_{f_{v}}$. Then, we concatenate the keypoints and the vertices corresponding to the head and hand mean pose as keypoints and vertex queries, resulting in a ($(N+3)\times H$)-dim feature map. Random masking of keypoints and vertex queries of rate $30\%$ is applied, following [63].

A.2 MeshNet and InteractionNet

Our MeshNet and InteractionNet have similar progressive downsampling transformer encoder structures, see Fig. 5 for an illustration. The MeshNet has three component transformer encoders with decreasing feature dimensions. The InteractionNet starts with a fully connected layer that downsamples the feature dimension, followed by two transformer encoders. Each transformer encoder has a Multi-Head Attention module consisting of 4 layers and 4 attention heads. In addition to head and hand mesh features, MeshNet also regresses head and hand keypoints, which are only for supervision and not used by any downstream components.

\begin{overpic}[width=433.62pt]{Figs/ablation_transformer_encoder} \end{overpic}

Figure 5: Structural details of the MeshNet and InteractionNet. (a) MeshNet; (b) InteractionNet; (c) Internal structure of a Transformer Encoder block.

A.3 IKNet

Our IKNets take in rough mesh features $\mathbf{V}_{F}^{\prime},\mathbf{V}_{H}^{\prime}$ and output the pose and shape parameters $(\theta,\beta)$, as well as the global rotation and translation $(R,T)$. They feature a Multi-Layer Perceptron (MLP) structure, each consisting of five MLP Blocks and a final fully connected layer. Each MLP Block contains a fully connected layer, followed by a batch normalization layer [42] and a ReLU activation layer. There are two skip-connections, connecting the output of the first block with the input of the third block, and the output of the third block with the input of the final fully connected layer. See Fig. 6 for an illustration. The hand and head IKNets have the same structure, differing only in their input and output dimensions. The hidden dimensions of the two IKNets are 1024.

\begin{overpic}[width=173.44534pt]{Figs/ablation_ik_net} \end{overpic}

Figure 6: Structural details of the IKNet.

A.4 Training and Testing Details

To be consistent with the training setting of Decaf [97], in the Decaf dataset, we use all eight camera views and the subjects S2, S4, S5, S7, and S8 in the training data split for training. For testing, we use only the front view (view 108) and the subjects S1, S3, and S6 in the testing data split. The low, mid, and high-resolution head mesh consists of $559$, $1675$, and $5023$ vertices, respectively. The low and high-resolution hand mesh consists of $195$ and $778$ vertices, respectively. We use the middle-resolution head mesh and the high-resolution hand mesh as the inputs of head and hand IKNets.

Appendix B More Qualitative Comparisons

We demonstrate qualitatively the effect of the absence of the depth loss in Fig. 7. When trained without depth loss, the network is only supervised with 2D information on in-the-wild data, without any constraints in the z-direction. As a result, artifacts such as self-penetration frequently occur in this case. The introduction of depth loss eliminates this ambiguity, allowing the correct relative positioning of hand and face.

\begin{overpic}[width=303.53267pt]{Figs/fig_depth_ablation} \end{overpic}

Figure 7: Qualitative demonstration of the effects of the depth loss. The model generalizes poorly in the z-direction when trained without depth supervision.

Appendix C Addition details on Losses

Here, we provide the details of the mesh losses and the interaction losses. The details of the adversarial loss and the depth loss are already mentioned in the main paper.

C.1 Mesh losses

The mesh loss $\mathcal{L}_{\text{mesh}}$ consists of four components.

$$\mathcal{L}_{\text{mesh}}=\mathcal{L}_{\text{reproj}}+4\mathcal{L}_{\text{vert}}+2\mathcal{L}_{\text{key}}+2\mathcal{L}_{\text{params}}.$$ (6)

Vertices Loss $L_{1}$ loss is used for predicted rough 3D face and hand vertices ${\mathbf{V}_{f}^{\prime}}$, ${\mathbf{V}_{h}^{\prime}}$, FLAME-regressed undeformed 3D face vertices ${{\mathbf{V}_{f}^{*}}}$ and MANO-regressed 3D hand vertices ${{\mathbf{V}_{h}}}$ against the ground-truth 3D undeformed face vertices $\mathbf{\hat{V}}_{f}$ and 3D hand vertices $\mathbf{\hat{V}}_{h}$.

$$\mathcal{L}_{\text{vert}}=\lambda_{h}(\mu_{\text{nonpara}}\|\mathbf{V}_{h}^{\prime}-\mathbf{\hat{V}}_{h}\|_{1}+\|\mathbf{V}_{h}-\mathbf{\hat{V}}_{h}\|_{1})+\lambda_{f}(\mu_{\text{nonpara}}\|\mathbf{V}_{f}^{\prime}-\mathbf{\hat{V}}_{h}\|_{1}+\|\mathbf{V}_{f}^{*}-\mathbf{\hat{V}}_{f}\|_{1}),$$ (7)

where $\lambda_{h},\lambda_{f}$ are empirically set to $3$ and $1$ respectively. $\mu_{\text{nonpara}}$ is set to $4$ to emphasize the supervision on the more complex non-parametric mesh features.
Keypoints Loss We use $L_{1}$ loss for predicted rough 3D face and hand keypoints $\mathbf{K}_{f}^{\prime}$, $\mathbf{K}_{h}^{\prime}$, 3D face and hand keypoints extracted from rough mesh ${\mathbf{K}_{f_{\text{mesh}}}},{\mathbf{K}_{h_{\text{mesh}}}}$, FLAME-regressed 3D face keypoints ${\mathbf{K}_{f}}$ and MANO-regressed 3D hand keypoints ${\mathbf{K}_{h}}$ against the ground-truth 3D undeformed face keypoints ${\mathbf{\hat{K}}_{f}}$ and 3D hand keypoints ${\mathbf{\hat{K}}_{f}}$.

$$\mathcal{L}_{\text{key}}=\mu_{\text{nonpara}}(\|\mathbf{K}_{h}^{\prime}-\mathbf{\hat{K}}_{h}\|_{1}+\|{\mathbf{K}_{h_{\text{mesh}}}}-\mathbf{\hat{K}}_{h}\|_{1}+\|\mathbf{K}_{f}^{\prime}-\mathbf{\hat{K}}_{f}\|_{1}+\|{\mathbf{K}_{f_{\text{mesh}}}}-\mathbf{\hat{K}}_{f}\|_{1})$$ (8)

$$+\|{\mathbf{K}_{f}}-{\mathbf{\hat{K}}_{f}}\|_{1}+\|{\mathbf{K}_{h}}-{\mathbf{\hat{K}}_{h}}\|_{1},$$ (9)

where $\mu_{\text{nonpara}}$ is empirically set to $4$, to put more weight on the non-parametric mesh with high degrees of freedom.
Reprojection loss $L_{1}$ loss is used for reprojected rough 3D face and hand keypoints $\mathbf{K}_{f}^{\prime}$, $\mathbf{K}_{h}^{\prime}$, 3D face and hand keypoints extracted from rough mesh ${\mathbf{K}_{f_{\text{mesh}}}},{\mathbf{K}_{h_{\text{mesh}}}}$, FLAME-regressed 3D face keypoints ${\mathbf{\hat{K}}_{f}}$ and MANO-regressed 3D hand keypoints ${\mathbf{\hat{K}}_{h}}$ against the ground-truth face and hand 2D keypoints $\mathbf{\hat{K}}_{f_{\text{2D}}},\mathbf{\hat{K}}_{h_{\text{2D}}}$.

$$\mathcal{L}_{\text{reproj}}=\lambda_{h}(\|\Pi(\mathbf{K}_{h}^{\prime})-\mathbf{\hat{K}}_{h_{\text{2D}}}\|_{1}+\|\Pi({\mathbf{K}_{h_{\text{mesh}}}})-\mathbf{\hat{K}}_{h_{\text{2D}}}\|_{1}+\|\Pi({\mathbf{K}_{h}})-\mathbf{\hat{K}}_{h_{\text{2D}}}\|_{1})$$ (10)

$$+\lambda_{f}(\|\Pi(\mathbf{K}_{f}^{\prime})-\mathbf{\hat{K}}_{f_{\text{2D}}}\|_{1}+\|\Pi({\mathbf{K}_{f_{\text{mesh}}}})-\mathbf{\hat{K}}_{f_{\text{2D}}}\|_{1}+\|\Pi({\mathbf{K}_{f}})-\mathbf{\hat{K}}_{f_{\text{2D}}}\|_{1}),$$ (11)

where $\Pi$ is the learned camera projection function. $\lambda_{h},\lambda_{f}$ are set to $4$ and $1$ respectively.

Parameter loss We apply $L_{1}$ loss on the regressed hand and face pose, shape, and facial expression parameters against their respective ground truths.

$$\mathcal{L}_{\text{face-params}}=(\|\beta_{\text{f}}-\hat{\beta}_{\text{f}}\|_{1}+\|\theta_{\text{f-exp}}-\hat{\theta}_{\text{f-exp}}\|_{1}+\|\theta_{\text{f-pose}}-\hat{\theta}_{\text{f-pose}}\|_{1})/3$$ (12)

$$\mathcal{L}_{\text{hand-params}}=(\|\beta_{\text{h}}-\hat{\beta}_{\text{h}}\|_{1}+\|\theta_{\text{h}}-\hat{\theta}_{\text{h}}\|_{1})/2$$ (13)

$$\mathcal{L}_{\text{params}}=\mathcal{L}_{\text{face-params}}+\mathcal{L}_{\text{hand-params}}$$ (14)

C.2 Interaction losses

The interaction loss $\mathcal{L}_{\text{interaction}}$ consists of four components:

$$\mathcal{L}_{\text{interaction}}=0.2\mathcal{L}_{\text{touch}}+0.6\mathcal{L}_{\text{contact}}+\mathcal{L}_{\text{collision}}+6\mathcal{L}_{\text{deform}}.$$ (15)

Deformation loss Due to the human anatomy, some vertices on the face are more easily deformed than other vertices. Therefore, we impose an adaptive weighting on each vertex and use square loss to penalize large deformation. We also have a regularization term to penalize extremely large deformations.

$$\mathcal{L}_{\text{deform}}=\sum_{i\in\mathcal{I}}(1+\mu\|\hat{d_{i}}\|_{2})\|\hat{d_{i}}-d_{i}\|_{2}^{2}+\lambda\sum_{i\in\mathcal{L}}\|d_{i}\|,$$ (16)

where $\mathcal{I}$ is the set of indices of face vertices, $d_{i}$, $\hat{d_{i}}$ are the predicted and ground truth deformation vector for index $i$, and $\mathcal{L}=\{i\in\mathcal{I}:\|d_{i}\|_{2}>3cm\}$ the vertices of large deformations. $\mu$ and $\lambda$ are empirically set to be $5000$, $100$ respectively.

Touch loss Let $\mathbf{V}_{F_{C}}$ and $\mathbf{V}_{H_{C}}$ denote the set of face and hand vertices that are predicted by the model to have contact probability greater than $0.5$.

$$\mathcal{L}_{\text{touch}}=\text{CD}(\mathbf{V}_{F_{C}},\mathbf{V}_{H_{C}})+\text{CD}(\mathbf{V}_{H_{C}},\mathbf{V}_{F_{C}}),$$ (17)

where $\text{CD}(X,Y)$ gives the mean Chamfer Distance (CD) between each point in $X$ to the closest point in $Y$.

Collision loss Let $\mathbf{V}_{H_{\text{Col}}}$ denote the set of hand vertices that penetrates the face surface, $\mathbf{V}_{F}$ and $\mathbf{D}_{F}$ denote the predicted face mesh vertices and deformations.

$$\mathcal{L}_{\text{collision}}=\text{CD}(\mathbf{V}_{H_{\text{Col}}},\mathbf{V}_{F}-\mathbf{D}_{F}).$$ (18)

Contact loss Let $\mathbf{C}_{H}$ and $\mathbf{C}_{F}$ denote the predicted hand and face contact probabilities and ${\mathbf{\hat{C}}_{H}}$, ${\mathbf{\hat{C}}_{F}}$ denote the ground-truth contact labels.

$$\mathcal{L}_{\text{contact}}=\text{BCE}(\mathbf{C}_{H},{\mathbf{\hat{C}}_{H}})+\text{BCE}({\mathbf{C}_{F}},{\mathbf{\hat{C}}_{F}}),$$ (19)

where BCE denotes the binary cross-entropy loss.

Appendix D More Discussions

D.1 Performance under Challenging Occlusion.

\begin{overpic}[width=433.62pt]{Figs/ablation_decaf_inaccurate.pdf} \end{overpic}

Figure 8: Examples of failed keypoint estimation in case of large self-occlusion. (a) input image; (b) inaccurate keypoint estimation by the same keypoint estimators used in Decaf [68, 3]; (c) reconstructed hand-face interaction by our method. (d) reconstructed hand-face interaction by Decaf.

As seen in Fig. 8, our end-to-end DICE method is robust under challenging self-occlusion cases, such as the hand covering more than half of the face. On the other hand, Decaf [97], which requires an initial keypoint prediction for test-time optimization, performs poorly in this situation.

D.2 Failure Cases

In Fig. 9, we demonstrate the failure cases of our method. As shown in Fig. 9 (a), when there is a complex interaction between the hand and face, such as the presence of a cleaning sponge, there is a drop in the reconstruction accuracy of the hand mesh recovery. Also, as in Fig. 9 (b), When the face completely occludes the hand, a highly challenging scenario unseen in the training data, our model could not faithfully reconstruct the hand position.

\begin{overpic}[width=433.62pt]{Figs/ablation_failure_cases.pdf} \end{overpic}

Figure 9: Samples of failure cases.