Real3D-Portrait: One-shot Realistic 3D
Talking Portrait Synthesis

When designing the network structure of the I2P model, we notice two main challenges for the network: (1) it should map the input image to a canonical tri-plane, which requires a coordinate transform from pixel coordinate to world coordinate. (2) It should extract rich appearance features from the source image to guarantee the fidelity of the rendered image. To this end, as shown in Figure 2(a), we design a hybrid network consisting of a ViT and a VGG branches. The ViT branch comprises a stack of SegFormer blocks, which executes attention among patches and could efficiently handle the pixel-to-world canonicalization process. Since ViT cannot maintain high-frequency texture due to the patch embedding operation, as a complementary, we design a VGG branch, which is simply a stack of convolution layers to extract high-frequency appearance features. The two branches’ output information is fused via concatenation and further processed with shallow convolution layers to produce the final tri-plane representation. Note that we remove all normalization in the VGG branch to keep the identity-specific appearance-related bias in all hidden layers.

The talking face dataset typically lacks multi-view frames, which are necessary for the model to learn 3D prior knowledge. To improve the generalizability of 3D face reconstruction, inspired by the multi-view reconstruction task proposed by Trevithick et al. (2023), we first pre-train the I2P model on a multi-view image dataset synthesized by EG3D (Chan et al., 2022), a 3D GAN for generating human face. We illustrate the pre-training process in Appendix B.1.

We use the projected normalized coordinate code (PNCC) (Zhu et al., 2016; Kim et al., 2018; Li et al., 2023a) as the motion representation, which is a pose/appearance-agnostic feature map that possesses fine-grained facial expression information based on a 3DMM face. Specifically, given a pair of identity code $\mathbf{i}$ and expression code $\mathbf{e}$, we could obtain the PNCC by rasterizing the 3DMM face mesh at the canonical pose through Z-Buffer (Phong, 1998) algorithm with NCC (Zhu et al., 2016) as its colormap. We provide details of calculating PNCC in Appendix B.2.

Thanks to the identity-expression decomposition of 3DMM, we can utilize PNCC to achieve identity-agnostic motion-conditioned face animation. During training, we first fit the 3DMM parameters of the training video to obtain the ground truth PNCC. During inference, we could construct the driving PNCC by:

where $\mathbf{i}_{\text{src}}$ is the identity coefficient of the source image, and $\mathbf{e}_{\text{drv}}$ is the expression coefficient that is either extracted from a driving video or predicted by an audio-to-motion model (Sec. 3.4).

Once the motion representation is decided, the second question is how to inject the motion condition into the 3D representation to control the facial expression. We do not choose the deformation field as previous works do since it typically results in bad quality of the predicted mesh (Li et al., 2023a). Instead, given a well-trained I2P model to produce a 3D representation that possesses accurate geometry/texture information, we propose to learn a light-weight Motion Adapter (MA) to predict a residual motion diff-plane $\mathbf{P}_{\text{diff}}$ that only edits the minimal geometry change of the canonical tri-plane $\mathbf{P}_{\text{cano}}$ based on the different motion condition. As for the network structure of the proposed MA, as shown in Fig. 2(b), we adopt a shallow SegFormer (Xie et al., 2021) to enjoy its high efficiency and strong ability to achieve cross-coordinate transform brought by the attention over feature map patches. To be specific, the process of animating the source image $\mathbf{I}_{\text{src}}$ given the input motion condition $\mathbf{PNCC}_{\text{drv}}$ and camera pose $\mathbf{cam}$ can be expressed as:

where VR and SR are a vanilla volume renderer and super-resolution module, whose structures are shown in Fig. 5; I2P and MA are the proposed image-to-plane model and motion adapter, respectively; $\mathbf{PNCC}_{\text{drv}}$ are the driving motion representation defined in Eq. 1 and $\mathbf{PNCC}_{\text{src}}$ is the PNCC extracted from the source image. Note that the input of MA is the concatenation of $\mathbf{PNCC}_{\text{src}}$ and $\mathbf{PNCC}_{\text{drv}}$. It is since the tri-plane predicted by the I2P model is of the source expression and the MA should be aware of both the source/driving expression to correctly map the 3D representation of the source expression into the target expression.

We illustrate the training process at this stage in Fig. 3. Apart from learning the motion adapter from scratch, we also fine-tuned the I2P model and the VR/SR module from the pre-trained weights in Sec. 3.1. We train the model on a large-scale and high-fidelity talking face video dataset (Zhu et al., 2022). To construct the training data pair, we randomly select two frames from a video and define them as source image $\mathbf{I}_{\text{src}}$ and target image $\mathbf{I}_{\text{tgt}}$. Since camera pose is only highly correlated with the movement of the head part, we only consider the head region at this stage, so all images are preprocessed with a face parsing model to extract the head segment. The source head image is fed into the I2P model to reconstruct a canonical tri-plane $\mathbf{P}_{\text{cano}}$, and the $\mathbf{PNCC}_{\text{tgt}}$ extracted from $\mathbf{I}_{\text{tgt}}$ is fed into the motion adapter MA to obtain the residual motion diff-plane $\mathbf{P}_{\text{diff}}$. Then we could obtain the predicted image $\mathbf{I^{\prime}}_{\text{tgt}}$ following Eq. 2. The training loss is as follows:

where the first two terms are L1 loss and VGG19/VGGFace-based (Simonyan & Zisserman, 2014; Parkhi et al., 2015) perceptual loss; $\mathcal{L}_{\text{DualAdv}}$ is the dual adversarial loss proposed by (Chan et al., 2022) to improve the image fidelity and consistency; and $\mathcal{L}_{\text{Lap}}$ is our proposed Laplacian loss over the motion diff-planes of adjacent frames, which acts as a regularization term to eliminate temporal jittering. To be intuitive, given PNCCs from frame $\{t-1,t,t+1\}$, we expect the diff-plane $\mathbf{P}_{\text{diff}}$ of frame ${t}$ to be the average of that of ${t-1}$ and ${t+1}$:

As shown in Fig. 6 of Appendix 7, the HTB-SR model consists of an SR branch, a Torso branch, and a Background branch. (1) The SR branch shares a similar structure with the vanilla SR module used in Sec. 3.2 so we could initialize its weights from the pre-trained model. (2) As for the Torso branch, since the movement of the torso is of small amplitude and often translational, we propose to model the torso part with a 2D war**-based renderer that is similar to (Wang et al., 2021), which is computationally efficient and proven robust in various scenes (Siarohin et al., 2019). As shown in Fig. 6(b), the torso segments are warped by dense flows conditioned on predefined key points to predict the torso feature map of the target image. Note that instead of learning implicit key points in an unsupervised manner as (Wang et al., 2021), we select several key points in the reconstructed 3DMM face vertex as the driving condition of the torso branch, which improves the temporal stability of the predicted torso. We provide details of the torso branch in Appendix 7. (3) As for the Background branch, the biggest challenge is to fill the pixels that were occupied by the foreground (i.e., the person) in the source image. To this end, as shown in Fig. 6(c), we first adopt a K-nearest-neighbor (KNN)-based inpainting method ²²2We can also use state-of-the-art neural network-based inpainting methods to obtain a more realistic background image. But since that is not the focus of this paper, we simply used the naive KNN-based method. to preprocess the background segment of the source image. Then, we use shallow convolution layers to extract texture features from the inpainted background. More details about the background branch can be found in Appendix B.4. (4) As for fusing the three feature maps of head, torso, and background segments into a composite image, we found a direct channel-wise concatenation leads to hollow artifacts and blurry results in the boundary region (shown in Fig. 15). We suppose that the problem is caused by the unlimited information propagation among these three segments’ feature maps and handle this problem with a alpha-blending-style fusion mechanism. To be specific, we first obtain the head mask $\mathbf{M}_{\text{head}}$ and torso mask $\mathbf{M}_{\text{torso}}$, then integrate the three segments with the awareness of occlusion:

where $\mathbf{F}$ denotes the extracted feature map, $\mathbf{M}_{\text{person}}$ is the person mask obtained by bitwise-or operation to $\mathbf{M}_{\text{head}}$ and $\mathbf{M}_{\text{torso}}$. Details of obtaining $\mathbf{M}_{\text{head}}$ and $\mathbf{M}_{\text{torso}}$ can be found in Appendix B.5.

As shown in Fig. 7, we load the pre-trained I2P model, motion adapter, and volume renderer from Sec. 3.2 and replace the SR module with our HTB-SR model. Only the HTB-SR model is updated at this stage, and all other parameters are frozen. The training objective is similar to Eq. 3. The difference is that the GT and predicted images are full images with head/torso/background parts instead of only the head segment.

Once the four-stage training process is done, no further training is required for a new identity. During inference, as shown in Fig. 1, we first fit the 3DMM parameters of the source image to obtain the source identity code $\mathbf{i_{\text{src}}}$. As for audio-driven scenarios, we obtain the expression sequence $\mathbf{e_{\text{drv}}}$ corresponding to the input audio using the A2M model; as for the video-driven scenarios, we fit 3DMM on the reference video to obtain the $\mathbf{e_{\text{drv}}}$. Then, we could obtain the driving PNCC following Eq. 1 and render the final images following Eq. 2 and Eq. 5.

In the video-driven (VD) scenario, the driving motion condition and head pose are obtained from a reference video. Under the same-identity setting, we use the first frame of the reference video as the source image; otherwise, the source image is of a different identity. As for the same-identity setting, we evaluate PSNR, SSIM, cosine similarity of the identity embedding (CSIM) by Deng et al. (2019a), average expression distance (AED) and average pose distance (APD) based on (Deng et al., 2019b), average keypoint distance (AKD) based on (Bulat & Tzimiropoulos, 2017), as well as LPIPS, L1 and FID between the reenacted and ground truth frames. As for the cross-identity setting, since there is no ground truth, we evaluate the results based on the CSIM, AED, APD, and FID metrics. The results are shown in Table 1. Firstly, our video-driven Real3D-Portrait performs best in terms of L1, PSNR SSIM, LPIPS, and FID, hence achieving the best image quality. Secondly, our Real3D-Portrait gets the highest CSIM, which denotes that it preserves the identity of the source image. Finally, our method achieves the best AED, APD, and AKD, demonstrating that ours could accurately animate the 3D avatar given the input condition.

In the audio-driven (AD) setting, the driving motion conditions are predicted from the input audio. Similar to the cross-identity reenactment, since there are no ground truth samples, we use CSIM to measure the identity preservation, FID to measure the image quality, and AED and Sync score (Prajwal et al., 2020) to measure the audio-lip synchronization. The results are shown in Table 2. When compared with MakeItTalk and PC-AVS, two one-shot 2D methods, our method shows significantly better identity similarity (CSIM), image quality (FID), and lip-synchronization (AED and Sync score). When compared with RAD-NeRF, a person-specific 3D method that over-fits an individual model on the tested identity’s 3-minute-long video, apart from achieving better lip synchronization, our method remarkably shows comparable image quality and identity preserving thanks to the well-trained large I2P model and motion adapter. To summarize, the experiment demonstrates that our one-shot Real3D-Portrait outperforms other one-shot baselines and could perform closely to the SOTA person-specific over-fitting method.

To make a clear comparison among all tested methods, we provide demo videos at https://real3dportrait.github.io. We also provide more visualization results in Appendix D.3. Specifically, (1) we showcase the overall qualitative comparison of our Real3D-Portrait and other VD/AD baselines in Fig. 9 and Fig.LABEL:fig:qual_aud. We also provide examples to show that: (2) how PNCC animates the 3D avatar in Fig. 10; (3) we achieve natural torso movement under large head poses in Fig. 11; (4) we support switchable background in Fig. 12; (5) our generic audio-to-motion model predicts synchronized lip motion in Fig. 13.

We conduct user studies to test the quality of generated samples. Specifically, we sample 10 audio clips from different languages and ten different identities for all methods to generate the videos and then involve 20 attendees for user studies. We adopt the Mean Opinion score (MOS) rating protocol for evaluation, which is scaled from 1 to 5. Following (Chen et al., 2020) The attendees are required to rate the videos from three aspects: (1) identity preserving; (2) visual quality (including image fidelity and temporal smoothness); (3) lip synchronization. Detailed user study settings are in Appendix D.2. We compute the average score for each method, and the results are shown in Table 3. We have the following observations: 1) Real3D-Portrait has better identity-preserving power and visual quality than previous one-shot methods and performs closely to person-specific methods (RAD-NeRF). 2) As for the lip-synchronization, Real3D-Portrait shows obvious superiority over the person-specific audio-driven method RAD-NeRF thanks to the powerful generic audio-to-motion model. Besides, among the video-driven methods that use GT motion as input, ours achieves the best lip synchronization, showing the effectiveness of our motion adapter to animate the avatar given the input motion accurately.

We test four settings on the I2P and motion adapter: (1) w/o pre-training, which does not pre-train the I2P model on the multi-view image dataset in Sec.3.1; (2) w/o fine-tuning, which fix the pre-trained I2P model when training on the video dataset in Sec.3.2; (3) small/large I2P model size, which tries I2P backbone of difference scales of 40M and 200M parameters (note that the default setting is 87M parameters); (4) w/o Lap loss, which removes the laplacian loss in Eq.4. We show the result in Table 4. As shown in line 1 and Fig. 14 of Appendix. D.4, without the pre-training process, the identity similarity, image quality, and expression accuracy drop significantly. Besides, as shown in line 2, fine-tuning is also necessary to achieve a low AED. We suspect it is because the pre-trained I2P only learns to reconstruct the 3D avatar with the source expression, and it needs further updates to support face animation given the target expression. Besides, there is a domain gap between the image dataset and the video dataset, hence fine-tuning is necessary to achieve better visual quality. As for the I2P model scale, as shown in line 3 and 4, we found that 87M achieves significantly better image quality than 50M, while the performance difference between default settings to 200M is not obvious. In line 5, we found laplacian loss is necessary to improve motion-conditioned head animation.

We test three settings on the HTB-SR model: (1): w/ unsup. KP., which is similar to Face-vid2vid that jointly learns a predictor to extract unsupervised driving key points from the predicted head image; (2) w/ concat, which replaces the proposed alpha-blending-style fusion module in Eq.5 with a naive channel-wise concatenation of the head/torso/background feature maps; (3) w/o inpaint, which removes the KNN-based inpainting of the background image. The results are shown in Table 4. In line 1, we can observe that unsupervised key points lead to worse visual quality due to the instability of the extra predictor network. In line 2, we find that alpha-blending-style fusion is necessary to obtain a good identity preserving and image quality and eliminate the artifacts shown in Fig. 15 of Appendix. D.4. In line 3, we find removing the background inpainting process results in worse image quality.