GRACE: Graph-Regularized Attentive Convolutional Entanglement with Laplacian Smoothing for Robust DeepFake Video Detection ^††thanks: This study was supported in part by the Ministry of Science and Technology (MOST), Taiwan, under grants MOST XXX; and partly by the Higher Education Sprout Project of Ministry of Education (MOE) to the Headquarters of University Advancement at National Cheng Kung University (NCKU). ^††thanks: (Corresponding author: Chih-Chung Hsu.) ^††thanks: C.-C. Hsu, S.-N. Chen, M.-H. Wu, Y.-F. Wang, C.-M. Lee and Y.-S. Chou are with Institute of Data Science and Department of Statistics, National Cheng Kung University, Tainan, Taiwan (R.O.C.), (e-mail:[email protected], [email protected], [email protected], [email protected], [email protected], [email protected].)

The proposed GRACE’s flowchart is illustrated in Fig. 2. First, a face detector extracts facial images from each video frame. A CNN-based backbone network extracts high-level semantic features from the spatial domain of the acquired facial parts, as displayed in the center of 2. Using the extracted spatial features, the spatial and temporal representations at frame $n$ and location $(i,j)$ across all feature maps ${\bm{X}}\in\mathbb{R}^{d\times c}$ can be obtained for the face sequence, where $d=N\times w\times h$, potentially including partially invalid faces. This feature representation captures feature responses at frame $n$ and location $(i,j)$ across all frames, thereby integrating temporal information, as shown in Fig. 2.

To augment the correlation between the spatial and temporal feature representation ${\bm{X}}$ acquired in the previous step, we introduce a novel Feature Entanglement (FE) with sparse constraint, denoted as ${\bm{X}}_{\textrm{FE}}=G_{\textrm{FE}}({\bm{X}})\in\mathbb{R}^{d\times d}$, which carefully embeds both temporal and spatial features into its graph representation by affinity matrix ${\bm{X}}_{\textrm{FE}}$ from original feature ${\bm{X}}$. In highly noisy face sequences, the number of invalid faces could be more than that of valid ones. Therefore, the essential features could be relatively fewer, motivating us to introduce the sparsity constraint into our GRACE to focus on those essential features. Then, to efficiently discern the importance of the graph representation ${\bm{X}}_{\textrm{FE}}$, we introduce the GCN to capture the contextual features between nodes (spatiotemporal features) in ${\bm{X}}$. To further remove the noisy nodes from the original ${\bm{X}}_{\text{FE}}$, Graph Laplacian is judiciously adapted to each layer of the GCN for better performance under noisy face sequences. Finally, a softmax classifier is connected to the outcome of GCN to evaluate the authenticity of the supplied facial parts.

We develop a method inspired by spatiotemporal feature extraction [44][45]. Traditional CNNs serve as the backbone network to obtain the spatial feature representation ${\bm{X}}^{n}\in\mathbb{R}^{c\times h\times w}$ for each frame ${\bm{y}}_{d}^{n}$ of the video. Assuming that the size of the extracted feature map is $c\times h\times w$, the spatial feature representation of a specific video at location $(i,j)$ via the backbone network can be vectorized into ${\bm{x}}_{n,i,j}=[x_{(}n,i,j)^{1},x_{(}n,i,j)^{2},...,x_{(}n,i,j)^{c}]\in\mathbb{R}^{c\times 1}$, where $c$ is the number of channels in the extracted spatial feature map and $i=1,...,w$, $j=1,...,h$. Then, let $d=Nwh$, we create the feature context ${\bm{X}}\in\mathbb{R}^{d\times c}$ based on location-wise feature concatenation, as follows:

where ${\bm{X}}$ represents the spatiotemporal feature.

The efficient extraction of joint spatial and temporal feature representations from ${\bm{X}}$ necessitates addressing potential inefficiencies linked to considerable distances between the $n$-th and $m$-th feature vectors, ${\bm{x}}^{n}$ and ${\bm{x}}^{m}$, especially when $m$ and $n$ are significantly apart. Directing adopting the Transformer [44][46] could lead to large computational complexity since the number of tokens could be large enough to have a long-range correlation.

To address these issues and efficiently exploit joint spatial and temporal feature representations, we propose a novel concept, i.e., FE, which is exactly an affinity matrix, creating a graph representation that encapsulates the relationships between nodes—each node representing entangled spatial and temporal features based on edges. Every element (node) in ${\bm{X}}_{\textrm{FE}}$ intertwines spatial and temporal information, thereby facilitating graph neural networks to learn the relationships among spatiotemporal features more effectively, in which the features of the first and last frames might still have a link. The feature entanglement ${\bm{X}}_{\textrm{FE}}$ is defined as follows:

where $\mathbf{X}_{\text{FE}}\in\mathbb{R}^{d\times d}$ is the affinity matrix obtained through feature entanglement. Consider, for instance, the feature element $\mathbf{X}_{\text{FE}}({1,c})$, which is computed by taking the inner product of the first and the $c$-th row of $\mathbf{X}$. This element encapsulates the correlation between the spatial features at location $(1,1)$ across all frames and the spatial features at location $(w,h)$ in the first frame. By constructing the affinity matrix $\mathbf{X}_{\text{FE}}$ in this manner, we effectively capture the spatiotemporal dependencies within the feature representation. However, learning discriminative features from $\mathbf{X}_{\text{FE}}$ can be challenging, particularly in the presence of highly noisy face sequences, where a significant portion of the facial images may be invalid. To address this issue, we introduce a sparsity constraint as a regularization term in the learning objective, formulated as the $\ell_{1}$ norm of $\mathbf{X}$, denoted by $|\mathbf{X}|_{1}$. By enforcing sparsity on $\mathbf{X}$, we encourage the model to focus on the most informative features, thereby enhancing its robustness to noisy face sequences and improving its effectiveness in DeepFake video detection.

In the previous subsection, we introduced the overall pipeline of our proposed GRACE method. A crucial component of GRACE is the FE with sparse constraint, which aims to efficiently exploit joint spatial and temporal feature representations. In this subsection, we delve into the details of FE and discuss how it addresses the challenges associated with unreliable face sequences in DeepFake detection.

Graph Convolutional Networks (GCNs) have emerged as a powerful tool for processing graph-structured data [47, 48], making them a suitable choice for handling the graph embedding obtained through feature entanglement. However, the noise level in each node can vary depending on the degree of distortion. For highly distorted or noisy nodes (i.e., those with entangled features primarily contributed by invalid facial images), it is beneficial to eliminate them to ensure stable and excellent performance in DeepFake video detection. To mitigate the impact of features from invalid facial images, we judiciously integrate the Graph Laplacian Smoothing Prior (GLSP), a well-established concept in Graph Signal Processing (GSP) [49], as a regularizer into the GCN to filter out highly noisy nodes [50]. It is important to note that GSP and GCN are distinct domains, with GSP focusing on the analysis and processing of signals defined on graphs, while GCN aims to learn representations by exploiting the graph structure. In this study, we ingeniously leverage the properties of GLSP from GSP and seamlessly incorporate it into the GCN framework, enabling end-to-end training without explicitly computing the eigendecomposition of the Graph Laplacian matrix.

Let $\mathbf{X}\in\mathbb{R}^{d\times c}$ denote the SFE feature matrix, where $d=Nwh$ represents the spatiotemporal dimensions, with $N$ being the number of frames, $w$ and $h$ being the width and height of the feature maps, and $c$ represents the feature dimension. This matrix encapsulates the spatiotemporal features extracted from the entire video sequence.

We construct affinity matrix ${\bm{A}}=\mathbf{X}_{\text{FE}}\in\mathbb{R}^{d\times d}$ through feature entanglement, where $\mathbf{A}_{ij}$ represents the edge weight between nodes $i$ and $j$. This matrix captures the similarity between different nodes. The degree matrix $\mathbf{D}\in\mathbb{R}^{d\times d}$ is a diagonal matrix where $\mathbf{D}_{ii}=\sum_{j}\mathbf{A}_{ij}$ represents the sum of edge weights connected to each node. The Graph Laplacian matrix $\mathbf{L}=\mathbf{D}-\mathbf{A}$ captures the topological structure of the graph and the differences between nodes.

To further remove the redundancy among nodes, we apply adaptively thresholding to $\mathbf{A}$ to filter out weak or irrelevant connections. For each sample $i$, we compute the mean value of its feature entanglement matrix $\mathbf{A}$ and keep only the elements that are greater than half of the mean value. The indices and values of these elements are then extracted to form the edge indices and edge weights of a sparse affinity matrix $\mathbf{A}^{(i)}$, as follows:

where $\mathbf{A}^{(i)}_{jk}$ denotes the edge weight between nodes $j$ and $k$ in the adjacency matrix of sample $i$, and $q$ is the factor controlling how strict the node being filtered. In this study, $q=0.5$ for all experiments. This thresholding operation helps to focus on the most important connections and reduces the computational burden of the GCN.

The resulting sparse adjacency matrix $\mathbf{A}^{(i)}$, along with the node features $\mathbf{X}^{(i)}$, serve as the input to the GCN for learning the graph structure and node representations. By combining the FE matrix ${\bm{X}}_{\text{FE}}$ and the sparse adjacency matrix ${\bm{A}}$, our method leverages the advantages of both representations, capturing rich spatiotemporal correlations while focusing on the most informative connections for DeepFake detection.

Graph Convolutional Networks (GCNs) have shown remarkable performance in various tasks by leveraging the power of graph-structured data. The core operation of GCNs in the $l$-th layer can be described as follows:

where $\hat{\mathbf{A}}=\mathbf{A}+\mathbf{I}_{d}$ is the adjacency matrix with self-loops, $\hat{\mathbf{D}}_{ii}=\sum_{j=1}^{d}\hat{\mathbf{A}}_{ij}$ is the corresponding degree matrix, $\mathbf{W}^{(l)}$ is the weight matrix of the $l$-th layer, and $\sigma$ is the activation function. This equation describes the aggregation and transformation of node features based on the graph structure.

However, the performance of GCNs may be compromised when dealing with highly noisy scenarios, limiting their applicability in real-world situations. To address this challenge, we propose the incorporation of Graph Laplacian regularization into GCNs to enhance their robustness and improve their performance in the presence of significant noise.

Given an undirected graph $G=(V,E)$, where $V$ is the set of nodes and $E$ is the set of edges, the Graph Laplacian matrix $\mathbf{L}$ is defined as:

where $\mathbf{D}$ is the degree matrix and $\mathbf{A}$ is the adjacency matrix of the graph $G$. The degree matrix $\mathbf{D}$ is a diagonal matrix, where $\mathbf{D}_{ii}$ equals the degree of node $i$ in $G$. It is worth noting that the Graph Laplacian matrix $\mathbf{L}$ is different from the matrix $\hat{\mathbf{L}}$ used in the GCN propagation rule, which will be discussed later.

As a real symmetric matrix, the Graph Laplacian matrix $\mathbf{L}$ possesses an eigendecomposition:

where $\mathbf{U}$ is the matrix of eigenvectors and $\Lambda$ is the diagonal matrix of eigenvalues. The eigenvalues of $\mathbf{L}$ represent the frequencies of the graph signals, with smaller eigenvalues corresponding to lower frequencies and larger eigenvalues corresponding to higher frequencies. In GSP, the Graph Laplacian matrix is often used as a low-pass filter to smooth signals defined on graphs, effectively suppressing high-frequency noise while preserving low-frequency information. Although we do not explicitly compute the eigendecomposition of $\mathbf{L}$ in our implementation, it is essential to understand that the Graph Laplacian matrix inherently encodes the spectral properties of the graph, which enables effective feature smoothing and noise suppression.

In practice, we calculate the Graph Laplacian matrix $\mathbf{L}$ using the degree matrix $\mathbf{D}$ and the adjacency matrix $\mathbf{A}$, which are obtained from the graph structure. This calculation does not involve the explicit computation of eigenvalues and eigenvectors. However, the resulting matrix $\mathbf{L}$ still possesses the spectral properties that enable effective feature smoothing and noise suppression.

To integrate the Graph Laplacian smoothing prior into the GCN propagation rule, we propose a modified version of the Graph Laplacian matrix that incorporates the graph’s structural information and enables effective feature smoothing and noise suppression. Our proposed Graph Laplacian matrix, denoted as $\hat{\mathbf{L}}$, is defined as follows:

where $\hat{\mathbf{A}}=\mathbf{A}+\mathbf{I}_{d}$ is the adjacency matrix with self-loops, $\hat{\mathbf{D}}$ is the corresponding degree matrix with $\hat{\mathbf{D}}_{ii}=\sum_{j=1}^{d}\hat{\mathbf{A}}_{ij}$, and $\mathbf{I}_{d}$ is the identity matrix.

The matrix $\hat{\mathbf{L}}$ is a normalized version of the Graph Laplacian matrix, which captures the graph’s structure and enables the smoothing of node features. By incorporating self-loops into the adjacency matrix, we ensure that each node’s feature is considered during the smoothing process, enhancing the stability and expressiveness of the learned representations.

To integrate the Graph Laplacian smoothing prior into the GCN propagation rule, we modify the propagation equation as follows:

where $\mathbf{Z}^{(l)}$ represents the node features at layer $l$, $\mathbf{W}^{(l)}$ is the learnable weight matrix, and $\sigma$ is the activation function.

The term $\hat{\mathbf{L}}\mathbf{Z}^{(l)}$ effectively applies the Graph Laplacian smoothing prior to the node features. By multiplying the node features with the normalized Graph Laplacian matrix $\hat{\mathbf{L}}$, we achieve a smoothing effect that takes into account the graph’s structure. This operation allows the model to leverage the connectivity information encoded in the graph to refine the node features and suppress high-frequency noise.

It is important to note that although we do not explicitly compute the eigendecomposition of $\hat{\mathbf{L}}$ in our implementation, the matrix $\hat{\mathbf{L}}$ inherently possesses the spectral properties of the Graph Laplacian. By using $\hat{\mathbf{L}}$ in the GCN propagation rule, we implicitly leverage these spectral properties to achieve effective feature smoothing and noise suppression.

The effectiveness of the Graph Laplacian matrix $\hat{\mathbf{L}}$ as a low-pass filter can be understood by examining its eigendecomposition:

where $\mathbf{U}$ is the matrix of eigenvectors and $\Lambda$ is the diagonal matrix of eigenvalues. The eigenvalues of $\hat{\mathbf{L}}$ represent the frequencies of the graph signals, with smaller eigenvalues corresponding to lower frequencies and larger eigenvalues corresponding to higher frequencies. By multiplying the node features with $\hat{\mathbf{L}}$, we essentially apply a low-pass filter to the graph signals, attenuating the high-frequency components while preserving the low-frequency information. This operation effectively suppresses noise and promotes the smoothness of the node features across the graph. In practice, we compute $\hat{\mathbf{L}}$ using the normalized adjacency matrix $\hat{\mathbf{A}}$ and the corresponding degree matrix $\hat{\mathbf{D}}$, as shown in Equation (6). This computation can be efficiently performed using sparse matrix operations, without the need for explicit eigendecomposition.

To provide a rigorous theoretical analysis of the effectiveness of our proposed Graph Laplacian regularization, we can examine the convergence and generalization properties of the method. Let $\mathbf{Z}^{*}$ denote the optimal node features that minimize the loss function $\mathcal{L}(\mathbf{Z})$. We can show that by incorporating the Graph Laplacian regularization term $\hat{\mathbf{L}}\mathbf{Z}^{(l)}$ into the GCN propagation rule, the learned node features $\mathbf{Z}^{(l)}$ converge to $\mathbf{Z}^{*}$ under mild assumptions on the graph structure and the loss function. Specifically, if the graph is connected and the loss function is convex and smooth, the iterative updates of the node features using Equation (8) will converge to the optimal solution $\mathbf{Z}^{*}$ (see Theorem 1). Furthermore, the generalization error of the learned node features can be bounded by the graph Laplacian regularization term, indicating that the proposed method effectively controls the model complexity and prevents overfitting to noisy or irrelevant features.

By incorporating the Graph Laplacian smoothing prior into the GCN, our method effectively addresses the challenges posed by noisy scenarios. The smoothing operation helps to mitigate the impact of noisy or irrelevant features, enhancing the robustness and generalization ability of the learned representations.

Finally, the output features can be obtained by passing the final layer’s features $\mathbf{Z}^{(L)}$ through a fully connected layer (FC):

where ${\bm{W}}_{\text{out}}\in\mathbb{R}^{g_{\text{dim}}\times n_{\text{out}}}$ denotes the weight of the FC, $n_{\text{out}}$ and $g_{\text{dim}}$ stand for the number of neurons of the FC and the embedding dimension of GCN. Finally, the predicted result could be done via

where ${\bm{W}}_{\text{cls}}\in\mathbb{R}^{n_{\text{out}}\times n_{\text{cls}}}$ denotes the weight of the FC, $n_{\text{cls}}$ stands for the number of classes.

During the training phase, the cross-entropy loss is employed to optimize the model parameters:

where $\mathbf{Y}$ is the one-hot encoded ground-truth label, and $\alpha$ stands for weight of the sparsity constraint, where $\alpha=1e^{-5}$ in this study for all experiments.

By incorporating Graph Laplacian regularization into the GCN, our proposed method effectively addresses the challenges posed by noisy face sequences in DeepFake video detection. The Graph Laplacian smoothing helps to filter out noisy nodes and enhances the robustness and stability of the model. The combination of feature entanglement, sparse regularization, and Graph Laplacian regularization enables our method to make the most of the available valid information, suppress the influence of noisy features, and improve the correlation between relevant features. As a result, our approach achieves high-performance and robust DeepFake video detection, even in the presence of low-quality and noisy face sequences commonly encountered in real-world scenarios.

In summary, our proposed GCN with Graph Laplacian regularization effectively leverages the spectral properties of the Graph Laplacian matrix to achieve feature smoothing and noise suppression, without explicitly computing the eigendecomposition. By integrating the Graph Laplacian smoothing prior into the GCN propagation rule, our method enhances the robustness and generalization ability of the learned representations, making it highly suitable for detecting DeepFake videos in challenging real-world scenarios with noisy and unreliable face sequences. The theoretical analysis of the convergence and generalization properties of our method further validates its effectiveness and provides a solid foundation for its application.

The robustness validation of the proposed method is the core of our investigation, particularly when applied to noisy face sequences containing many invalid faces. To achieve this, the use of representative benchmark datasets is essential. Therefore, we selected three well-established benchmark datasets for performance evaluation: FF++ [1], Celeb-DFv2 [3], and the large-scale DFDC dataset [2]. The FF++ dataset [1] comprises four distinct classes of manipulation methods: 1) DeepFakes (DF), 2) Face2Face (F2F), 3) FaceSwap (FS), and 4) NeuralTextures (NT). For each class, a set of 1,000 original videos was used to generate 1,000 manipulated versions, resulting in a total of 1,000 authentic and 4,000 doctored videos. The Celeb-DF dataset [3] contains 590 original videos and 5,639 manipulated counterparts, generated using improved generative adversarial networks at a resolution of $256\times 256$. To enhance the quality of the manipulated videos, a Kalman filter is employed in Celeb-DF [3] to mitigate temporal inconsistencies between successive frames. In addition to FF++ and Celeb-DF, we also utilize the DeepFake Detection Challenge (DFDC) dataset [2] to further validate the effectiveness of our proposed method. The DFDC dataset, created by Facebook in collaboration with other organizations, is a large-scale dataset designed to facilitate the development of DeepFake detection algorithms. It consists of over 100,000 videos, containing a mix of authentic and manipulated content generated using various state-of-the-art face swap** and facial reenactment techniques, ensuring a diverse and challenging set of DeepFakes for evaluation.

Furthermore, we assumed that each frame should contain at least one face to simulate adversarial attacks on face detectors in real-world scenarios. In cases where no face was detected in a frame, we replaced that frame with a black image, generating a noisy face sequence that allowed us to assess the robustness of GRACE under challenging conditions. Our experimental analysis employed three performance metrics: accuracy, Macro F1-Score, and Area Under the Receiver Operating Characteristic Curve (AUC). The Macro F1-Score accurately reflects the model’s performance under label imbalance situations. For simplicity, these metrics are referred to as Accuracy (Acc.), F1-Score, and AUC throughout the experimental sections.

The primary performance assessment comparing the handling of invalid facial images between our proposed model, GRACE, and various state-of-the-art schemes is provided in Table II. Under optimal conditions, where most facial images are valid, GRACE exhibits competitive results, holding its own against other cutting-edge DeepFake video detection methods such as Xception [6], MesoNet [5], $F^{3}$-Net [32], RECCE [27], CORE [26], TALL-Swin [53], and DFIL [28]. It is worth noting that TALL-Swin [53] is a video-based approach.

Specifically, the F1-Score of GRACE for DeepFake video detection, when evaluated on FF++ [1], Celeb-DF [3], and DFDC [2], slightly surpasses those of its contemporaries under clean cases (i.e., $m_{r}=0$). This outcome implies that the proposed GRACE with Graph Laplacian is effective and reliable for DeepFake video detection. However, in scenarios where partial face images are invalid due to purposeful attacks on face detectors, the performance of traditional frame-level methods, including Xception [6], MesoNet [5], $F^{3}$-Net [32], RECCE [27], CORE [26], UCF [30], and DFIL [28], may substantially deteriorate since they fail to consider noisy face sequences in real-world scenarios.

Similarly, the video-level DeepFake detection methods, TALL-Swin [53], which heavily rely on temporal cues, may suffer further performance degradation when the masking ratio increases. Invalid faces can cause landmark detection failures and incorrect temporal trajectories. Consequently, the F1-Score of TALL-Swin under a masking ratio of 0.8 in the testing phase is lower than 0.7, implying that all predictions would be categorized as either entirely fake or real. Likewise, the performance of another state-of-the-art video-based DeepFake detection method, TALL-Swin [53], is poor when $m_{r}$ is increased. In stark contrast, all quality indices of our proposed GRACE, evaluated on different datasets, display promising results, suggesting that GRACE is robust and reliable even under highly noisy face sequences (e.g., when $m_{r}=0.8$). Remarkably, since most DeepFake detection methods fail to discuss the impact of unreliable face sequences, the degraded performance is most likely predictable.

To further demonstrate the efficiency and practicality of the proposed GRACE method, we conduct a comprehensive complexity analysis and compare it with other state-of-the-art DeepFake detection methods. Table III presents the comparison results in terms of floating-point operations (FLOPs), multiply-accumulate operations (MACs), and the number of parameters for each method with $16\times 3\times 144\times 144$ tensor for the fair comparison. It is evident that GRACE achieves a remarkable balance between computational complexity and performance. With 70.751 trillion FLOPs, 35.246 trillion MACs, and 29.661 million parameters, GRACE exhibits a moderate computational overhead compared to other methods, such as TALL-Swin [29], UCF [30], and RECCE [27]. Notably, GRACE outperforms these methods in terms of FLOPs and MACs while maintaining a comparable number of parameters. Moreover, GRACE demonstrates superior performance in handling noisy face sequences, as shown in the experimental results, despite having a similar complexity to methods like CORE [26], Xception [6], and DFIL [28]. This highlights the effectiveness of the proposed feature entanglement, graph convolutional network, and graph Laplacian regularization techniques in learning discriminative and robust representations for DeepFake detection. The complexity analysis further substantiates GRACE as a practical and efficient solution for real-world DeepFake detection challenges, offering a compelling trade-off between computational resources and detection accuracy.

The detailed quantitative results, evaluated on the FF++ [1], Celeb-DF [3], and DFDC [2] datasets, are illustrated in Figures 3(a) and 3(c), respectively. In the clean case, i.e., when $m_{r}=0$, the performance of the proposed method is comparable to other state-of-the-art methods. It is observed that performance degradation becomes increasingly pronounced with a rise in the masking ratio during the testing phase, particularly when the masking ratio ($m_{r}$) exceeds 0.5. The performance of the previously established TALL-Swin [53] also declines when the masking ratio surpasses 0.2. A similar trend is discernible in Fig. 3(c), which evaluates the DFDC testing set. The performance of contemporary methods diminishes at higher masking ratios, whereas the proposed GRACE method maintains relatively high performance even at a masking ratio of 0.8. We also draw the AUC comparison between the proposed GRACE and other peer methods in Fig. 3. We show that the proposed GRACE significantly outperforms other state-of-the-art DeepFake detectors, especially under noisy face sequences.

More specifically, most existing DeepFake video/image detection algorithms do not address the impact of noisy face sequences. Although state-of-the-art face detectors perform exceptionally well under pristine conditions, their performance can be severely undermined when subjected to well-engineered post-processing techniques, particularly adversarial perturbations targeting the face detector. Our GRACE method successfully overcomes this shortcoming and introduces a novel and robust DeepFake video detection approach for real-world challenges.

To achieve optimal performance and robustness, we conducted a comprehensive ablation study to investigate the impact of various hyperparameters on the proposed GRACE method. This analysis provides valuable insights into the design choices and trade-offs involved in develo** an effective DeepFake video detection system for real-world scenarios with noisy face sequences. Table IV presents the performance comparison of GRACE under different hyperparameter settings, evaluated on the challenging FF++ dataset [1].

Table V presents an ablation study for the proposed modules in our GRACE, i.e., GCN, Graph Laplacian smooth prior, and Sparsity regularizer, where the performance is evaluated in noisy face sequences (say, $m_{r}=0.8$ and $m_{r}=0.7$). Note that when none of the proposed modules is adopted, we adopt the Transformer [46] as the classification head with four-head multi-head self-attention (MHSA) with the embedding size of 512 to meet a similar number of parameters with that of our GRACE, which could be treated as a variant of Convolutional Transformer. When we enable the GCN for the proposed feature entanglement and its affinity matrix, the performance of the DeepFake video detection under noisy face sequence, implying that the feature entanglement and its graph representation judiciously embeds the different spatiotemporal features into every node, thereby reducing the impact of invalid faces under noisy face sequences. Furthermore, the Graph Laplacian smooth prior could improve the robustness since it could filter noisy nodes that might contain many invalid faces without significantly increasing computational complexity. As shown in Fig.4, the convergence of the proposed GRACE with Graph Laplacian remains stable and shows outstanding performance on the FF++ [1] validation set with $m_{r}=0$. Finally, the sparsity regularizer significantly benefits the robustness of the DeepFake detection since we could enforce our GRACE learning essential and sparse features for the valid faces.

As previously discussed, DeepFake videos can be intentionally perturbed to evade detection by face detectors, rendering DeepFake detection ineffective. To emulate this real-world challenge, we employ an open-source adversarial attack on the MTCNN face detector [54], leveraging a PGD-like algorithm [39] with a maximum perturbation value of $\epsilon=0.04$, step size $\alpha_{\text{adv}}=0.01$, and the number of iterations $s=10$ to perturb the test sets of FF++ [1]. The step size $\alpha_{\text{adv}}$ determines the magnitude of each perturbation step applied to the input image during the iterative adversarial attack process. Assuming that each frame must contain at least one face detectable by MTCNN to assess the rate of missed detections, a black image will replace the frame when no face is detected. The results reveal that the average number of missed detected faces is 3.58 for the FF++ [1] test set, closely mirroring an $m_{r}=0.2$ scenario. However, the perturbed face sequences not only make it more difficult to recognize whether the video is fake due to the adversarial noise affecting the image quality, as exemplified in Fig. 5, but also introduce another challenge: the adversarial attack could cause the face detector to extract non-facial regions (e.g., background). This implies that the actual $m_{r}$ in this case could be even higher than the estimated $0.2$, as some of the detected faces might not be genuine facial regions. Despite these challenging conditions, our GRACE method maintains strong performance compared to other state-of-the-art methods, as illustrated in Table VI. Remarkably, the performance of all peer methods dropped significantly, partly due to the missed detections and partly because the adversarial noise introduces spatial distortions in the facial images. This real-world simulation further substantiates GRACE as a generalized, robust, and effective DeepFake detection model capable of handling noisy face sequences.

This study introduces a novel approach, GRACE, to address the challenge of DeepFake video detection in the presence of noisy face sequences. GRACE leverages feature entanglement with sparse constraints and a graph convolutional network with graph Laplacian regularization to effectively exploit the spatial-temporal correlations in face sequences while suppressing the impact of noise and distortions. The experimental results demonstrate the efficacy of GRACE in handling noisy face sequences and achieving state-of-the-art performance on several benchmark datasets.

However, it is essential to acknowledge the limitations of the current study and discuss potential future directions. One limitation is that while GRACE has shown strong performance on the evaluated datasets, its effectiveness on cross-dataset scenarios, where the training and testing data come from different sources, has not been extensively explored. The robustness of GRACE to domain shifts and variations in noise characteristics across different datasets requires further investigation. Nonetheless, it is worth emphasizing that GRACE represents the first dedicated effort to tackle the problem of noisy face sequences in DeepFake video detection, which has been largely overlooked in previous research. The proposed methodology and insights from this study lay a solid foundation for future work in this important direction.

Another aspect to consider is that GRACE currently does not incorporate masked learning strategies, which have shown promise in handling occlusions and missing data in various computer vision tasks. Integrating masked learning techniques into the GRACE framework could potentially further enhance its robustness to partial occlusions and incomplete face sequences. Moreover, the use of graph convolutional networks in GRACE allows for flexible processing of video frames, as the input frames are not required to be strictly sequential. This property could be leveraged to develop more efficient and adaptive sampling strategies for processing long video sequences.

It is also worth noting that while GRACE has demonstrated significant improvements over existing methods, there is still room for further enhancements. One direction could be to explore more advanced graph neural network architectures, such as graph attention networks or graph transformers, to better capture the complex dependencies and interactions among the spatial-temporal features. Additionally, incorporating prior knowledge or constraints specific to the DeepFake detection domain, such as the consistency of facial landmarks or the coherence of audio-visual signals, could potentially boost the performance and generalizability of the proposed approach.

In conclusion, GRACE represents a significant step forward in addressing the challenge of DeepFake video detection in the presence of noisy face sequences. While acknowledging the limitations and potential areas for improvement, we believe that the proposed methodology opens up new avenues for research in this critical domain. Future work could focus on extending GRACE to handle cross-dataset scenarios, integrating masked learning techniques, exploring more advanced graph neural network architectures, and incorporating domain-specific prior knowledge. As DeepFake techniques continue to evolve and become more sophisticated, develo** robust and reliable detection methods that can operate effectively in real-world scenarios with noisy and challenging data remains an ongoing research endeavor of paramount importance.