Fiducial Focus Augmentation for Facial Landmark Detection

Abstract

Deep learning methods have led to significant improvements in the performance on the facial landmark detection (FLD) task. However, detecting landmarks in challenging settings, such as head pose changes, exaggerated expressions, or uneven illumination, continue to remain a challenge due to high variability and insufficient samples. This inadequacy can be attributed to the model’s inability to effectively acquire appropriate facial structure information from the input images. To address this, we propose a novel image augmentation technique specifically designed for the FLD task to enhance the model’s understanding of facial structures. To effectively utilize the newly proposed augmentation technique, we employ a Siamese architecture-based training mechanism with a Deep Canonical Correlation Analysis (DCCA)-based loss to achieve collective learning of high-level feature representations from two different views of the input images. Furthermore, we employ a Transformer + CNN-based network with a custom hourglass module as the robust backbone for the Siamese framework. Extensive experiments show that our approach outperforms multiple state-of-the-art approaches across various benchmark datasets.

1 Introduction

Facial Landmark Detection (FLD) aims to detect coordinates of the predefined landmarks on given facial image. The rich geometric information provided by landmarks with distinct semantic significance, such as eye corner, nose tip, or jawline, can be helpful in various tasks like 3D face reconstruction [Kittler et al.(2016)Kittler, Huber, Feng, Hu, and Christmas, Koppen et al.(2018)Koppen, Feng, Kittler, Awais, Christmas, Wu, and Yin, Roth et al.(2016)Roth, Tong, and Liu], face identification [Masi et al.(2016)Masi, Rawls, Medioni, and Natarajan, Taigman et al.(2014)Taigman, Yang, Ranzato, and Wolf, Yang et al.(2017)Yang, Ren, Zhang, Chen, Wen, Li, and Hua], emotion recognition [Fabian Benitez-Quiroz et al.(2016)Fabian Benitez-Quiroz, Srinivasan, and Martinez, Li et al.(2017)Li, Deng, and Du, Walecki et al.(2016)Walecki, Rudovic, Pavlovic, and Pantic], and face morphing [Hassner et al.(2015)Hassner, Harel, Paz, and Enbar]. Several FLD algorithms, based either on coordinate regression [Sun et al.(2013)Sun, Wang, and Tang, Toshev and Szegedy(2014), Trigeorgis et al.(2016)Trigeorgis, Snape, Nicolaou, Antonakos, and Zafeiriou, Lv et al.(2017)Lv, Shao, Xing, Cheng, and Zhou, Zhang et al.(2014)Zhang, Shan, Kan, and Chen, Zhou et al.(2013a)Zhou, Fan, Cao, Jiang, and Yin] or heatmap regression [Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen, Bulat et al.(2021)Bulat, Sanchez, and Tzimiropoulos, Huang et al.(2020)Huang, Deng, Shen, Zhang, and Ye, Li et al.(2022)Li, Guo, Rhee, Han, and Han, Wen et al.(2022)Wen, Ding, Yao, Wang, and Qian, Lan et al.(2021)Lan, Hu, Chen, Xue, and Cheng], have emerged in recent years with promising performance on various datasets. However, landmark detection still remains challenging task due the high variability in poses, lighting and expressions. Despite the various existing FLD methodologies, none have focused on robust image augmentation techniques to solve these challenges. This study illustrates that meticulously designed image augmentations can considerably enhance the FLD performance.

Figure 1: Illustration of the proposed Fiducial Focus Augmentation (FiFA). In row (a), 5$\times$5 black patches are created around the landmark joints (along with other standard augmentations) in the initial epochs and reduced over the epochs. Rows (b) and (c) show corresponding GradCAM-based saliency maps of the network’s last layer with and without FiFA, respectively. It is clearly seen that activations are more prominent around the desired landmarks when FiFA is used as additional augmentation.

But why do sophisticated deep neural network (DNN) architectures struggle to detect landmarks accurately in challenging scenarios? The reason is that the DNN is unable to learn the facial structure information as accurately as required. If a DNN model can accurately capture features that extract a facial structure, it can predict the landmarks more accurately even from obscured facial regions, like occluded areas. To learn facial structures effectively, we propose new augmentation technique called Fiducial Focus Augmentation (FiFA), which leverages the ground truth landmark coordinates as an inductive bias for facial structure. To this end, we introduce $n\times n$ black patches around the landmark locations in the training images, gradually reducing them over the epoch and then removing completely for the rest of the training, as illustrated in Fig 1. Since the patches cover key semantic regions of the face, e.g., eyes, nose, lips and jawline, when the model learns to predict these patches, it is able to learn the entire facial structure significantly better, as compared to an architecture without this inductive bias. One could view this augmentation technique as similar to Curriculum Learning (CL) [Hacohen and Weinshall(2019)], a strategy that trains a machine learning model from simpler data to more difficult data, mimicking the meaningful order found in human-designed learning curricula.

Drawing inspiration from [Bulat et al.(2021)Bulat, Sanchez, and Tzimiropoulos], we leverage the Siamese architecture to acquire a comprehensive understanding of reliable landmark predictions across various image augmentations. However, our method employs Deep Canonical Correlation Analysis (DCCA) [Andrew et al.(2013)Andrew, Arora, Bilmes, and Livescu] as loss function in Siamese architecture to amplify the efficacy of the learning process between distinctively augmented views. This loss function assists in the extraction of features that are correlated across views, while simultaneously eliminating uncorrelated noise. To design a robust backbone for the Siamese architecture, we adopt Vision Transformer (ViT) [Dosovitskiy et al.(2020)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.]. We further improved its performance and efficiency by incorporating a Convolutional Neural Network (CNN)-based hourglass module in-between the transformer layers of the ViT. Modern CNNs are usually considered to be shift-invariant; we hence use an Anti-aliased CNN [Zhang(2019)] inside the hourglass module to leverage this benefit. We summarize the contributions of this paper as follows.

• To the best of our knowledge, this is the first effort in literature to propose a new patch-based augmentation technique for FLD task to learn facial semantic structures effectively.

• We employ a Siamese-based training scheme utilising DCCA loss between feature representations of two different views of the same image, that enforces consistent predictions of the landmark for the two views. To incorporate virtues of both a Transformer and a CNN, we design a robust Transformer + CNN-based backbone in our proposed framework.

• We performed extensive experiments on various benchmark datasets showing significant improvements over prior work. We also conducted ablation studies on our framework components and additional empirical analysis to study the usefulness of the proposed method.

2 Related Works

Earlier efforts on FLD task, especially those in recent years, can broadly be categorized into network architecture enhancements for heatmap generation and loss function improvements.

Network architecture enhancements: Coordinate regression-based methods [Sun et al.(2013)Sun, Wang, and Tang, Toshev and Szegedy(2014), Trigeorgis et al.(2016)Trigeorgis, Snape, Nicolaou, Antonakos, and Zafeiriou, Lv et al.(2017)Lv, Shao, Xing, Cheng, and Zhou, Zhang et al.(2014)Zhang, Shan, Kan, and Chen, Zhou et al.(2013a)Zhou, Fan, Cao, Jiang, and Yin] directly perform regression on landmark coordinate vectors through a fully connected output layer that disregards the spatial correlations of features and results in limited accuracy of landmark detection. On the other hand, heatmap regression-based methods [Bulat et al.(2021)Bulat, Sanchez, and Tzimiropoulos, Bulat and Tzimiropoulos(2017), Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen, Huang et al.(2021)Huang, Yang, Li, Kim, and Wei, Huang et al.(2020)Huang, Deng, Shen, Zhang, and Ye, Lan et al.(2021)Lan, Hu, Chen, Xue, and Cheng, Xia et al.(2022)Xia, Qu, Huang, Zhang, Wang, and Xu, Wen et al.(2022)Wen, Ding, Yao, Wang, and Qian, Li et al.(2022)Li, Guo, Rhee, Han, and Han] predict landmark coordinates by creating heatmaps. By doing so, they effectively maintain the original spatial relationships between pixels and achieve promising landmark detection accuracy. Therefore, heatmap regression has become the de facto choice for the FLD task in modern times. In [Bulat and Tzimiropoulos(2017)], Bulat et al. proposed an encoder-decoder based framework with heatmap regression for FLD. Their network incorporates hourglass and hierarchical blocks. Several research works [Sun et al.(2019)Sun, Zhao, Jiang, Cheng, Xiao, Liu, Mu, Wang, Liu, and Wang, Wang et al.(2020)Wang, Sun, Cheng, Jiang, Deng, Zhao, Liu, Mu, Tan, Wang, et al., Xiao et al.(2018)Xiao, Wu, and Wei] have been published based on the ResNet [He et al.(2016)He, Zhang, Ren, and Sun] architecture and modify their network for dense pixel-wise landmark predictions. Recently, the Vision Transformer (ViT) [Dosovitskiy et al.(2020)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.] has been incorporated in FLD task by Zhang et al. [Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen] and has produced remarkable results. In our proposed framework, we also use ViT as the backbone network and improve its performance by introducing CNN layers in between transformer layers. This allows us to combine the best of both designs.

Loss function improvements: A pixel-wise $L2$ or $L1$ loss is the conventional loss generally applied to heatmap regression-based methods [Zhou et al.(2013b)Zhou, Fan, Cao, Jiang, and Yin, Deng et al.(2019)Deng, Trigeorgis, Zhou, and Zafeiriou, Dong et al.(2018)Dong, Yan, Ouyang, and Yang, Newell et al.(2016)Newell, Yang, and Deng, Wei et al.(2016)Wei, Ramakrishna, Kanade, and Sheikh]. To emphasize the importance of tiny and medium range errors during the training process, Feng et al. [Feng et al.(2018)Feng, Kittler, Awais, Huber, and Wu] introduced the Wing loss, which modifies the L1 loss by using a logarithmic function to amplify the impact of errors within a specific range. Additionally, Wang et al. [Wang et al.(2019)Wang, Bo, and Fuxin] developed the Adaptive Wing Loss, which can adjust its curvature based on the ground truth pixels. In [Kumar et al.(2020)Kumar, Marks, Mou, Wang, Jones, Cherian, Koike-Akino, Liu, and Feng], Kumar et al. proposed the LUVLi loss that optimizes the position of the keypoints, the uncertainty, and the likelihood of visibility. Recently, the authors from [Huang et al.(2020)Huang, Deng, Shen, Zhang, and Ye] proposed the Focal Wing Loss, which is used to mine and emphasize difficult samples under in-the-wild conditions.

In this work, we use the standard Binary Cross Entropy (BCE) and $L2$ losses for heatmap and coordinate regression, respectively. We however employ the DCCA loss [Andrew et al.(2013)Andrew, Arora, Bilmes, and Livescu] which suits our framework and has never been used before for the FLD task. These simple losses help the proposed framework set a new benchmark. Our study of literature revealed that well-designed image augmentations are largely ignored for the FLD task. This paper attends to this very issue and introduces a new augmentation technique called FiFA that accounts for our impressive results.

Figure 2: An overview of the proposed Siamese-based framework. PPE = Patch + Position Embeddings; RB = Residual Block; MHA = Multi-Head Attention, MLP = Multi-Layer Perceptron; CBP = Convolution+BlurPool; BU = Bilinear Upsampling; FFP = FF-Parser.

3 Proposed Framework

3.1 Problem Statement & Notations

Given an input image $I$, FLD aims to detect $\{x,y\}\in\mathbb{R}^{k\times 2}$, the coordinates of $K$ predefined landmarks. To this end, we propose a heatmap-based approach to regress the facial landmarks. During training, it encodes the target ground truth coordinates as a series of $k$ heatmaps with a 2D Gaussian curve centered on them:

$$\Psi_{i,j,k}=\frac{1}{2\pi\sigma^{2}}e^{-\frac{1}{2\sigma^{2}}\left[(i-\bar{x}_{k})^{2}+(j-\bar{y}_{k})^{2}\right]}$$ (1)

where $x_{k}$ and $y_{k}$ are the spatial coordinates of the $k^{th}$ point, while $\bar{x}_{k}$ and $\bar{y}_{k}$ are their scaled, quantized version obtained by scaling factor $s$ and rounding operator $\lfloor\cdot\rceil$, i.e.

$$(\bar{x}_{k},\bar{y}_{k})=(\lfloor\frac{1}{s}x_{k}\rceil,\lfloor\frac{1}{s}y_{k}\rceil)$$ (2)

As shown in Eq. (1), we use a Gaussian with variance $\sigma$ around each coordinate from $\{x,y\}$ to generate the corresponding heatmap $\mathbb{H}\in\mathbb{R}^{k\times W\times H}$. Finally, the pixels with maximum intensity of the heatmap $\mathbb{H}$ are selected to get the final $K$ landmarks in the FLD task.

To attain precise facial landmarks, we propose a novel augmentation technique called Fiducial Focus Augmentation (FiFA) that helps the network to learn facial structures in the provided images, along with a Siamese network with a robust backbone and the DCCA loss to ensure consistent predictions between different augmented views. Detailed explanations of these modules are provided in the subsequent subsections.

3.2 Fiducial Focus Augmentation

We seek to explore the potential of carefully designed image augmentations for the FLD task in this section. To this end, we propose an augmentation $f_{A}$ for input training images, where $f_{A}=f_{A_{2}}\circ f_{A_{1}}$. Here, $f_{A_{1}}$ can be any standard image augmentations used in the FLD task [Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen, Bulat et al.(2021)Bulat, Sanchez, and Tzimiropoulos, Huang et al.(2020)Huang, Deng, Shen, Zhang, and Ye, Xia et al.(2022)Xia, Qu, Huang, Zhang, Wang, and Xu, Wen et al.(2022)Wen, Ding, Yao, Wang, and Qian] and $f_{A_{2}}$ is the proposed Fiducial Focus Augmentation (FiFA).

First, we take the original input image $I$ and apply standard image augmentation $f_{A_{1}}$ to get the augmented image ($I^{\prime}$). Mathematically, this can be expressed as:

$$I^{\prime}=f_{A_{1}}\otimes I.$$ (3)

To get the final augmented image $I^{\prime\prime}$, $I^{\prime}$ is passed through the proposed augmentation operation i.e., $f_{A_{2}}$ (as descibed in Alg 1), i.e.

$$I^{\prime\prime}=f_{A_{2}}\otimes I^{\prime}=\hat{I}\otimes I^{\prime}=\hat{I}\otimes(f_{A_{1}}\otimes I).$$ (4)

Here, we aim to incorporate the available facial structure ground truth information into the augmented image, $I^{\prime}$ in order to aptly utilize the underlying facial structure. To achieve this, we construct black square patches of dimensions $h_{f}\times w_{f}$, where $h_{f},w_{f}\in\{1,\cdots,n\}$ while retaining the landmarks as the intersection points of the two diagonals of the square patches (see Figure 1 (a)). These patches comprise of four coordinates which can be expressed as:

$$\{(x_{i}-w_{f},y_{i}+h_{f}),(x_{i}+w_{f},y_{i}+h_{f}),(x_{i}+w_{f},y_{i}-h_{f}),(x_{i}-w_{f},y_{i}-h_{f})\}\ \forall\ \{x_{i},y_{i}\}\in L.$$ (5)

Here, we start with a bigger patch size of $n\times n$ for a certain number of epoch intervals $\mathcal{E}$. After every such interval, we reduce the patch size by 1 pixel and eventually, these patches are removed from the images and rest of the training goes on with augmentation $f_{A_{1}}$ only. So the final augmented image is (where $T_{n}$ is the total number of epochs):

$$\begin{cases}I^{\prime\prime}&\text{when epoch no. $\leq n\cdot\mathcal{E}$}\\ I^{\prime}&\text{when $n\cdot\mathcal{E}$ < epoch no. $\leq T_{n}$}.\end{cases}$$ (6)

Initialize:   $I^{\prime}$: Augmented Image, where $I^{\prime}=f_{A_{1}}\otimes I$,

     $L_{n}$: Number of landmarks in $I$,

       $L$: Set of $L_{n}$ landmarks, where $L=\{(x_{i},y_{i})\}$, $i\in\{1,...,L_{n}\}$,

     $h_{f},w_{f}$: Height and width of the patches $(S)$, where $h_{f},w_{f}\in\{n,...,1\}$,

     $I_{in}$: Pixel intensity of $S$, where $I_{in}=(0,0,0)$,

       $\mathcal{E}$: Epoch interval, where $\mathcal{E}\in\{1,...,n\}\land n<$ Total number of epochs $(T_{n})$

     $T_{n}:$ Total number of epochs $=\sum_{i=1}^{n}\mathcal{E}_{i}+w$, where $w\in\mathbb{W}$

Procedure:

  for $i$ in range $T_{n}$ do

    $k\leftarrow\lfloor i/\mathcal{E}\rfloor$

    $h_{f},w_{f}\leftarrow|n-k|$

    for $j$ in range $L_{n}$ do

      $C\leftarrow\{(x_{j}-w_{f}/2,y_{j}+h_{f}/2),(x_{j}+w_{f}/2,y_{j}+h_{f}/2),(x_{j}+w_{f}/2,y_{j}-h_{f}/2),(x_{j}-w_{f}/2,y_{j}-h_{f}/2)\}$

      Create patch $S$ with $C$ of $I_{in}$

      $I^{\prime}\leftarrow S\otimes I^{\prime}$

    end for

  end for

  $\hat{I}\leftarrow I^{\prime}$

  return $\hat{I}$

Algorithm 1 Fiducial Focus Augmentation $(f_{A_{2}})$

The proposed FiFA helps the backbone network learn the underlying facial structure and address difficult test samples, since the patches cover the entire face uniformly over the different joints (eyes, lips, nose and jawline). At the beginning of training, the model is exposed to larger patches as low-confidence regions to concentrate on the joints and eventually, as the model learns progressively with each epoch, smaller patches are introduced as high-confidence regions around the joints. When the patches are removed completely, the model tries to predict the joints with the inductive bias provided by earlier training steps in our augmentation process. Since the patches can be used with any facial variations (such as pose or expression), their integration into the images as augmentations enables the model to learn the inherent facial structures.

3.3 Matching Two Views

Earlier work on the task of FLD has seen limited exploration of Siamese architecture-based training, with the exception of [Bulat et al.(2021)Bulat, Sanchez, and Tzimiropoulos]. In this paper, we propose a Siamese architecture-based framework as illustrated in Fig. 2. The network $f$ takes the two input images $I^{\prime}$ and $I^{\prime\prime}$ generated using two different augmentations $f_{A_{1}}$ and $f_{A}$. This training scheme using augmentations holds a notable advantage, as CNNs may not be invariant under arbitrary affine transformations. Therefore, even minor variations within the input space may produce significant changes in the output. By optimizing jointly using the Siamese architecture and combining the two predictions, we enhance the robustness and consistency of the predictions (under such variations).

To maximize the correlation between two different augmented views, we employ the Deep Canonical Correlation Analysis (DCCA) loss [Andrew et al.(2013)Andrew, Arora, Bilmes, and Livescu] between the high-level representation map**s $f_{1}(I^{\prime})$ and $f_{2}(I^{\prime\prime})$, where $f_{1}=f_{2}=f$. The correlation between these two map**s can be expressed as below:

$$corr(f_{1}(I^{\prime}),f_{2}(I^{\prime\prime}))=\frac{cov(f_{1}(I^{\prime}),f_{2}(I^{\prime\prime}))}{\sqrt{var(f_{1}(I^{\prime}))\cdot var(f_{2}(I^{\prime\prime}))}}.$$ (7)

The DCCA loss (i.e., $\mathcal{L}_{DCCA}$) is then computed as:

$$\mathcal{L}_{DCCA}=-corr(f_{1}(I^{\prime}),f_{2}(I^{\prime\prime})).$$ (8)

The use of DCCA loss presents three key advantages: (i) correlated representations partially reconstruct the information in the second view, when it is unavailable; (ii) it has potential to eliminate noise that is uncorrelated across the two views; and (iii) if $f_{1},f_{2}$ capture features that are correlated across the views, they may represent latent aspects of the face. This, in turn helps the backbone network in capturing the facial structure in the images.

3.4 Architectural Details

In the proposed framework, we employ a transformer-based architecture (a pre-trained ViT-B/16 [Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen] consisting of 12 layers and a width of 768) as a backbone. To enhance its performance further, we incorporated three custom CNN-based hourglass modules after every four layers of the transformer network. The purpose of this module is to introduce desirable properties of CNNs, such as shift, scale, and distortion invariance, into the ViT architecture, while still retaining the characteristics of transformers, i.e., dynamic attention, global context, and better generalization. This results in a robust backbone network (Transformer + CNN) which learns facial structures effectively.

The utilization of pooling layers in CNNs often provides a certain degree of shift invariance in the model. However, in our task, it is imperative to avoid the loss of structural information caused by pooling layers. We therefore adopt the Anti-aliased CNN [Zhang(2019)] into our hourglass modules, hereafter known as Anti-aliased Hourglass. The combination of these components significantly enhances the caliber of our network towards high-quality heatmap generation. Nevertheless, the upsampling + concatenation (U+A) operation in the hourglass modules may introduce some high-frequency noise. To mitigate this negative impact and filter the features in the Fourier space, we integrate a FF-Parser layer [Wu et al.(2022)Wu, Fang, Zhang, Yang, and Xu] after each U+A operation in the hourglass modules. We provide ablation studies on these components in our results to demonstrate their usefulness.

4 Experiments and Results

This section discusses the implementation details, comparison with SOTA methods on benchmark datasets and ablation analysis of the introduced components of the proposed method.

Implementation Details: The proposed method is trained/tested on the various benchmark datasets, i.e., WFLW [WFL()], 300W [Sagonas et al.(2016)Sagonas, Antonakos, Tzimiropoulos, Zafeiriou, and Pantic], COFW [Burgos-Artizzu et al.(2013)Burgos-Artizzu, Perona, and Dollár] and AFLW [Köstinger et al.(2011)Köstinger, Wohlhart, Roth, and Bischof]. Details of these datasets are discussed in the Supplementary material. During the training phase, the input image is cropped and resized to $512\times 512$. The output feature map size of every hourglass module is set to $128\times 128$, which is $4\times$ smaller than the input image size. The ground truth heatmaps are generated by a Gaussian with $\sigma=1.5$ and radius $r=5$. During training process, we used AdamW [Loshchilov and Hutter(2017)] to optimize our network with the initial learning rate of $1\times 10^{-4}$ and trained for 250 epochs. Apart from the proposed augmentation (FiFA), other standard data augmentations ($f_{A_{1}}$) are employed at training time, such as random masking, bilinear interpolation, random occlusion, random gray, random gamma, random blur, noise fusion. For effective learning, along with the DCCA loss (i.e., $\mathcal{L}_{DCCA}$), we also employ the standard BCE loss (i.e., $\mathcal{L}_{BCE}$) and mean absolute error loss (i.e., $\mathcal{L}_{L1}$) for heatmap and coordinate regression, respectively with equal weights (i.e., 1.0). For evaluation, we used the standard evaluation metrics i.e., Normalized Mean Error ($NME$) variants (i.e., $NME_{ic}$, $NME_{box}$, $NME_{diag}$), Failure Rate ($FR^{10}_{ic}$), Area Under the Curve ($AUC_{box}$). Detailed definitions of these metrics have been discussed in the Supplementary material. For comparison, we choose recent baselines such as FaRL [Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen], ADNet [Huang et al.(2021)Huang, Yang, Li, Kim, and Wei], SH-FAN [Bulat et al.(2021)Bulat, Sanchez, and Tzimiropoulos], PropNet [Huang et al.(2020)Huang, Deng, Shen, Zhang, and Ye], HIH [Lan et al.(2021)Lan, Hu, Chen, Xue, and Cheng], SLPT [Xia et al.(2022)Xia, Qu, Huang, Zhang, Wang, and Xu], PicassoNet [Wen et al.(2022)Wen, Ding, Yao, Wang, and Qian] and DTLD [Li et al.(2022)Li, Guo, Rhee, Han, and Han]. All the experiments were implemented using PyTorch and the network was trained on 4 GPUs (40GB NVIDIA A100), with batch size 5 per GPU.

4.1 Result Analysis

Table 1: Comparison against the state-of-the-art on COFW, 300W and AFLW dataset. Best result is bolded and second best result is underlined.

Method Remarks COFW 300W AFLW $NME_{ic}\downarrow$ $FR^{10}_{ic}\downarrow$ $NME_{ic}\downarrow$ $NME_{diag}\downarrow$ $NME_{box}\downarrow$ $AUC_{box}\uparrow$ Full Common Challenge Full Frontal Full Full FaRL [Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen] CVPR ’22 3.11 0.12 2.93 2.56 4.45 0.94 0.82 1.33 81.3 ADNet [Huang et al.(2021)Huang, Yang, Li, Kim, and Wei] ICCV ’21 4.68 0.59 2.93 2.53 4.58 — — — — SH-FAN [Bulat et al.(2021)Bulat, Sanchez, and Tzimiropoulos] BMVC ’21 3.02 0.00 2.94 2.61 4.13 1.31 1.12 2.14 70.0 PropNet [Huang et al.(2020)Huang, Deng, Shen, Zhang, and Ye] CVPR ’20 3.71 0.20 2.93 2.67 3.99 — — — — HIH [Lan et al.(2021)Lan, Hu, Chen, Xue, and Cheng] ICCVW ’21 3.21 0.00 3.09 2.65 4.89 — — — — SLPT [Xia et al.(2022)Xia, Qu, Huang, Zhang, Wang, and Xu] CVPR ’22 3.32 0.59 3.17 2.75 4.90 — — — — DTLD [Li et al.(2022)Li, Guo, Rhee, Han, and Han] CVPR ’22 3.02 — 2.96 2.60 4.48 1.37 — — — PicassoNet [Wen et al.(2022)Wen, Ding, Yao, Wang, and Qian] TNNLS ’22 — — 3.58 3.03 5.81 1.59 1.30 — — FiFA (Ours) — 2.96 0.00 2.89 2.51 4.47 0.92 0.80 1.31 81.8

Table 2: Comparison against the state-of-the-art on WFLW testset. Best result is bolded and second best result is underlined.

Metric Models Remarks Fullset Subset Pose Expression Illumination Make Up Occlusion Blur $NME_{\text{ic}}$(%)$\downarrow$ FaRL [Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen] CVPR’22 3.99 6.61 4.18 3.90 3.84 4.71 4.53 ADNet [Huang et al.(2021)Huang, Yang, Li, Kim, and Wei] ICCV’21 4.14 6.96 4.38 4.09 4.05 5.06 4.79 SH-FAN [Bulat et al.(2021)Bulat, Sanchez, and Tzimiropoulos] BMVC’21 3.72 — — — — — — PropNet [Huang et al.(2020)Huang, Deng, Shen, Zhang, and Ye] CVPR’20 4.05 6.92 3.87 4.07 3.76 4.58 4.36 HIH [Lan et al.(2021)Lan, Hu, Chen, Xue, and Cheng] ICCVW’21 4.08 6.87 4.06 4.34 3.85 4.85 4.66 SLPT [Xia et al.(2022)Xia, Qu, Huang, Zhang, Wang, and Xu] CVPR’22 4.14 6.96 4.45 4.05 4.00 5.06 4.79 DTLD [Li et al.(2022)Li, Guo, Rhee, Han, and Han] CVPR’22 4.05 — — — — — — PicassoNet [Wen et al.(2022)Wen, Ding, Yao, Wang, and Qian] TNNLS’22 4.82 8.61 5.14 4.73 4.68 5.91 5.56 FiFA (Ours) — 3.89 6.47 4.09 3.80 3.76 4.63 4.43 $FR_{\text{ic}}^{10}$(%)$\downarrow$ FaRL [Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen] CVPR’22 1.76 — — — — — — ADNet [Huang et al.(2021)Huang, Yang, Li, Kim, and Wei] ICCV’21 2.72 12.72 2.15 2.44 1.94 5.79 3.54 SH-FAN [Bulat et al.(2021)Bulat, Sanchez, and Tzimiropoulos] BMVC’21 1.55 — — — — — — PropNet [Huang et al.(2020)Huang, Deng, Shen, Zhang, and Ye] CVPR’20 2.96 12.58 2.55 2.44 1.46 5.16 3.75 HIH [Lan et al.(2021)Lan, Hu, Chen, Xue, and Cheng] ICCVW’21 2.60 12.88 1.27 2.43 1.45 5.16 3.10 SLPT [Xia et al.(2022)Xia, Qu, Huang, Zhang, Wang, and Xu] CVPR’22 2.76 12.72 2.23 1.86 3.40 5.98 3.88 DTLD [Li et al.(2022)Li, Guo, Rhee, Han, and Han] CVPR’22 2.68 — — — — — — PicassoNet [Wen et al.(2022)Wen, Ding, Yao, Wang, and Qian] TNNLS’22 5.64 25.46 5.10 4.30 5.34 10.59 7.12 FiFA (Ours) — 1.60 7.05 1.27 1.43 1.45 3.39 1.94

Comparison on COFW: In Table 1, we presents a comparison of the proposed FiFA approach with existing SOTA methods on the COFW testset, which is a well-known benchmark for heavy occlusion and a wide range of head pose variation. It is noteworthy that the proposed FiFA model outperforms the existing SOTA methods. The leading $NME_{ic}$ and $0\%$ $FR_{ic}^{10}$ demonstrate its robustness against extreme situations.

Comparison on 300W: On the 300W dataset, our approach exhibits superior performance in comparison to SOTA methods in terms of $NME_{ic}$, and is given in Table 1. In challenge-set, the proposed approach performs slightly lower than PropNet [Huang et al.(2020)Huang, Deng, Shen, Zhang, and Ye] and SH-FAN [Bulat et al.(2021)Bulat, Sanchez, and Tzimiropoulos] methods. However, it has achieved SOTA results in other scenarios (i.e., full-set and common-set), which suggests that our method makes plausible predictions even in deplorable situations.

Comparison on AFLW: The results on AFLW testset are presented in Table 1. Adhering to the evaluation protocol adopted in [Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen], we report comparisons in terms of $NME_{diag}$, $NME_{box}$ and $AUC_{box}^{7}$. This table clearly indicates that our approach has outperformed the SOTA results, despite the fact that the dataset is almost saturated.

Figure 3: Qualitative results on WFLW testset. Landmarks shown in green are produced by our method, while the ones in red by the state-of-the-art approach of [Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen].

Comparison on WFLW: In Table 2, we compare results in terms of $NME_{ic}$, and $FR_{ic}^{10}$. Here, it is observed that the proposed FiFA approach obtains better $NME_{ic}$ for Pose, Illumination and Make Up subsets. Additionally, in comparison on $FR_{ic}^{10}$, the proposed approach achieves higher performance in all subsets i.e., Pose, Expression, Illumination, Make Up, Occlusion, Blur by 44%, 41%, 23%, 1%, 34%, 37.4%, respectively over the previous best performing SOTA methods. These results show that our method improves the accuracy in challenging scenarios while also reducing the overall failure ratio for difficult images. Moreover, Fig. 3 visually conveys that the proposed approach delivers significantly more precise landmarks in challenging scenarios.

4.2 Ablation Studies & Analysis

This section presents the ablation analysis carried out to establish the efficacy of the proposed framework. To ensure fair comparison, all experiments were performed on COFW dataset.

Effects of method’s components: Herein, we investigate the impact of each component of the proposed framework. The results, presented in Table 5, reveal that the baseline network, i.e., Vanilla backbone (ViT-B/16), attains an $NME_{ic}$ of 3.11 when trained solely with standard augmentations, i.e., $f_{A_{1}}$. When anti-aliased CNN-based hourglass modules are incorporated into baseline, an improvement in $NME_{ic}$ to 3.07 is observed. By employing the proposed augmentation, $f_{A_{2}}$, on the input images during training, a remarkable performance boost is achieved, with an $NME_{ic}$ of 3.00. The highest $NME_{ic}$ of 2.96 is attained when incorporating the Siamese training approach with DCCA loss on both $f_{A_{1}}$ and $f_{A_{2}}$ augmented images. This finding demonstrates that training the backbone with proposed components gives best performance in results.

Table 3: Effect of method’s components on COFW.

Table 4: Effect of patch sizes in FiFA on COFW.

Table 5: Effect of FiFA over standard augmentations on COFW. BI = Bilinear Interpolation; RM = Random Masking; RO = Random Occlusion; RGr = Random Gray; RGm = Random Gamma; RB = Random Blur; NF = noise fusion.

Method $NME_{ic}$(%)$\downarrow$ Vanilla backbone (ViT-B/16) 3.11 + anti-aliased CNN-based hourglass 3.07 + Fiducial Focus Augmentation 3.00 + Siamese training (w DCCA) 2.96

FiFA patch progression $NME_{ic}(\%)\downarrow$ 3$\times$3 $\rightarrow\cdot\cdot\cdot\rightarrow 1\times 1\rightarrow$ no patch 3.05 4$\times$4 $\rightarrow\cdot\cdot\cdot\rightarrow 1\times 1\rightarrow$ no patch 3.00 5$\times$5 $\rightarrow\cdot\cdot\cdot\rightarrow 1\times 1\rightarrow$ no patch 2.96 6$\times$6 $\rightarrow\cdot\cdot\cdot\rightarrow 1\times 1\rightarrow$ no patch 2.99 7$\times$7 $\rightarrow\cdot\cdot\cdot\rightarrow 1\times 1\rightarrow$ no patch 3.02

Augmentations $NME_{ic}$(%)$\downarrow$ RM + RO 3.15  + FiFA 3.08 RM + {RO, RGr} 3.12  + FiFA 3.07 RM + {RO, RGr, RGm} 3.10  + FiFA 3.04 RM + {RO, RGr, RGm, RB} 3.10  + FiFA 3.04 RM + BI + {RO, RGr, RGm, RB} 3.08  + FiFA 3.03 RM + BI + NF + {RO, RGr, RGm, RB} 3.07  + FiFA 3.00

Table 4: Effect of patch sizes in FiFA on COFW.

Table 5: Effect of FiFA over standard augmentations on COFW. BI = Bilinear Interpolation; RM = Random Masking; RO = Random Occlusion; RGr = Random Gray; RGm = Random Gamma; RB = Random Blur; NF = noise fusion.

Effects of fiducial mask sizes: We have conducted a series of experiments to determine the optimal initial patch size for the proposed FiFA. As shown in Table 5, a patch size of $5\times 5$ yields the best $NME_{ic}$ of 2.96, while deviating from this size leads to a deterioration in performance. This can be attributed to the fact that during the initial stages of training, when the network weights are not yet sufficiently tuned, a patch size that is either too large or too small will result in a confidence region that is either too broad or too narrow for the network to focus on the landmarks. This, in turn, has an adverse effect on the learning process and ultimately on the performance of the network.

Effect of FiFA over standard augmentations: Several experiments were conducted to prove the effectiveness of our proposed FiFA over other standard augmentations. Due to the availability of only one view of augmented images, all these experiments were performed without a Siamese-based training mechanism. Table 5 displays the results obtained in terms of $NME_{ic}$ on the COFW testset. One can notice that the inclusion of our proposed FiFA in standard augmentation techniques leads to a notable improvement in the $NME_{ic}$ value.

Comparison with other losses in Siamese training: We employ DCCA loss [Andrew et al.(2013)Andrew, Arora, Bilmes, and Livescu] in Siamese training to maximize the correlation between different views. To demonstrate the efficacy of DCCA loss, we conducted several experiments with different losses (i.e., L2, L1, Smooth L1, and Wing loss [Feng et al.(2018)Feng, Kittler, Awais, Huber, and Wu]), and the corresponding results are presented in Table 6. One can observe that the DCCA loss helps to obtain better $NME_{ic}$, exhibiting a 3% increase as compared to previous best-performing Wing loss.

Table 6: Effect of different losses in Siamese training on COFW.

Loss L2 L1 Smooth L1 Wing [Feng et al.(2018)Feng, Kittler, Awais, Huber, and Wu] DCCA [Andrew et al.(2013)Andrew, Arora, Bilmes, and Livescu] $NME_{ic}$(%)$\downarrow$ 3.14 3.09 3.11 3.05 2.96

Effectiveness of the proposed components to other SOTA methods: To validate the effectiveness of the proposed components, we conducted a series of experiments wherein the proposed FiFA augmentation and Siamese network based DCCA loss were implemented on other baseline methods such as HRNet [Wang et al.(2020)Wang, Sun, Cheng, Jiang, Deng, Zhao, Liu, Mu, Tan, Wang, et al.], ADNet [Huang et al.(2021)Huang, Yang, Li, Kim, and Wei], SH-FAN backbone [Bulat et al.(2021)Bulat, Sanchez, and Tzimiropoulos], FaRL [Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen], SLPT [Xia et al.(2022)Xia, Qu, Huang, Zhang, Wang, and Xu] and the corresponding results are summarized in Table 7. The proposed FiFA augmentation technique improved the performance of baseline methods. Additionally, the Siamese network based DCCA loss contributed to improve the NME score further. This clearly indicates the generalization capability of our method.

Table 7: Effect of proposed FiFA augmentation technique and Siamese-based DCCA loss on baseline methods on COFW testset.

Methods Remarks Baseline + FiFA + FiFA + Siamese training (w DCCA) HRNet [Wang et al.(2020)Wang, Sun, Cheng, Jiang, Deng, Zhao, Liu, Mu, Tan, Wang, et al.] $ICCV_{21}$ 3.45 3.32 3.28 ADNet [Huang et al.(2021)Huang, Yang, Li, Kim, and Wei] $ICCV_{21}$ 4.68 4.51 4.45 SH-FAN Backbone [Bulat et al.(2021)Bulat, Sanchez, and Tzimiropoulos] $BMVC_{21}$ 3.25 3.12 3.07 FaRL [Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen] $CVPR_{22}$ 3.11 3.04 3.01 SLPT [Xia et al.(2022)Xia, Qu, Huang, Zhang, Wang, and Xu] $CVPR_{22}$ 3.32 3.15 3.10

5 Conclusion & Future Work

In this paper, we successfully proposed a simple yet effective image augmentation technique called Fiducial Focus Augmentation (FiFA) for facial landmark detection task. The integration of FiFA during training significantly enhanced the accuracy of proposed approach on testing benchmarks without extreme modifications to its backbone network and the loss function. Our findings suggest that the employment of FiFA as an image augmentation technique, when used in conjunction with a Siamese-based training with DCCA loss results in state-of-the-art performance. Additionally, we employed an anti-aliased CNN-based hourglass network with ViT as our backbone network to address shift invariance and noise. We performed extensive experimentation and ablation studies to validate the effectiveness of the proposed approach. In future work, FiFA can be studied further to extend it for other face-related tasks.