CFPL-FAS: Class Free Prompt Learning for Generalizable Face Anti-spoofing

Abstract

Domain generalization (DG) based Face Anti-Spoofing (FAS) aims to improve the model’s performance on unseen domains. Existing methods either rely on domain labels to align domain-invariant feature spaces, or disentangle generalizable features from the whole sample, which inevitably lead to the distortion of semantic feature structures and achieve limited generalization. In this work, we make use of large-scale VLMs like CLIP and leverage the textual feature to dynamically adjust the classifier’s weights for exploring generalizable visual features. Specifically, we propose a novel Class Free Prompt Learning (CFPL) paradigm for DG FAS, which utilizes two lightweight transformers, namely Content Q-Former (CQF) and Style Q-Former (SQF), to learn the different semantic prompts conditioned on content and style features by using a set of learnable query vectors, respectively. Thus, the generalizable prompt can be learned by two improvements: (1) A Prompt-Text Matched (PTM) supervision is introduced to ensure CQF learns visual representation that is most informative of the content description. (2) A Diversified Style Prompt (DSP) technology is proposed to diversify the learning of style prompts by mixing feature statistics between instance-specific styles. Finally, the learned text features modulate visual features to generalization through the designed Prompt Modulation (PM). Extensive experiments show that the CFPL is effective and outperforms the state-of-the-art methods on several cross-domain datasets.

1 Introduction

Face Anti-Spoofing (FAS) is an important step in protecting the security of face recognition systems from print-attack [58], replay-attack [4] and mask-attack [20, 6]. Despite the existing presentation attack detection (PAD) methods [30, 7, 55, 32, 52, 19, 21] obtain remarkable performance in intra-dataset experiments where training and testing data are from the same domain, their performance severely degraded in cross-dataset experiments due to large distribution discrepancies among different domains.

Figure 1: Comparison with existing DG FAS methods. (a) the previous methods either rely on a projector to align domain-invariant feature spaces with adversarial training, or disentangle generalizable features from the whole sample with a decoupler, which inevitably leads to the distortion of semantic structures and achieves limited generalization. (b) Our CFPL framework is built on CLIP to learn generalized visual features by using the text features as weights of the classifier.

Domain Generalization (DG) based FAS aims to mitigate the impact of distribution discrepancy by accessing multiple domains. As shown in Fig. 1 (a) (left), a typical strategy [40, 41, 12, 50] is rely on domain labels to learn a domain-invariant feature space by adversarial training, which is also generalized to unseen domains. While it is difficult to seek a compact and generalized feature space for all domains. Even, there is no guarantee that such a feature space exists among multiple domains due to large distribution discrepancies. Considering the manually labeled domain labels are arbitrary and subjective, which cannot truly reflect the diversity of samples in a domain, as shown in Fig. 1 (a) (right), another strategy [3, 27, 63] is based on the instance to disentangle or generate generalized features from liveness-irrelevant features by disentangled representation learning.Based on the above analysis, these methods either rely on domain labels to align domain-invariant feature spaces, or disentangle generalizable features from instance-specific and liveness-irrelevant features, which inevitably leads to the distortion of semantic feature structures and achieve limited generalization.

Rethinking the reason for the poor generalization of FAS can be attributed to the liveness-irrelevant features interfering with the classifier’s recognition of spoofing clues. If the weight of the classifier can be dynamically adjusted based on sample instances, such as, to weaken the interference factors and strengthen the generalized features, it will effectively improve its generalization. Inspired by large vision-language models like CLIP [39], which jointly trains an image encoder and a text encoder to predict the correct of pairings (image, text), As shown in Fig. 1 (b), we generate corrective text features by a high-capacity text encoder, which allows open-set visual concepts and broader semantic spaces compared to discrete domain labels. How to learn generalizable prompts for text encoder to adaptively adjust the classifier‘s weights? Unlike general DG task, which have domain information such as ‘This image belongs to Cartoon/Sketch/Art Painting/Photo‘, a FAS dataset is regarded as a domain, usually named with the publishing agency, such as MSU [51], CASIA [58], Idiap [4], or OULU [1], without providing any valuable domain semantic information to assist generalizable prompt learning.

In this work, without relying on domain semantics, the generalizable prompts can be based on image content and style learned by reducing their correlation [50, 63]. Based on this determination, inspired by BLIP-2 [14] and TGPT [44], we design two lightweight transformers, namely Content Q-Former (CQF) and Style Q-Former (SQF), to learn the expected prompts conditioned on content and style features by using a set of learnable query vectors, respectively. To further ensure CQF can learn to extract the visual representation that is most informative of the content description, we introduce a Prompt-Text Matched (PTM) supervision to optimize the learning of content prompts, where each sample’s content description is generated by template description. Due to the inability to accurately describe style information in text, instead, we propose a Diversified Style Prompt (DSP) technology to diversify style prompts by mixing feature statistics between instance-specific styles. Finally, the generalized visual features are learned through the designed Prompt Modulation (PM) function, which uses visual features as modulation factors. To sum up, the main contributions of this paper are summarized as follows:

• Instead of directly manipulating visual features, it is the first work to explore DG FAS via textual prompt learning, namely CFPL, which allows a broader semantic space to adjust the visual features to generalization.

• In order to release the requirement for categories in the text description, our CFPL first learns the prompts conditioned on content and style features with two lightweight transformers, namely Content Q-Former (CQF) and Style Q-Former (SQF). Then, the Prompts are further optimized through two improvements: (1) A Prompt-Text Matched (PTM) aims to ensure CQF learns semantic visual representation; (2) A Diversified Style Prompt (DSP) technology to diversify the learning of style prompts. Finally, the learned prompts modulate visual features to generalization through the designed Prompt Modulation (PM) function.

• Extensive cross-domain experiments show that the proposed CFPL is effective and outperforms the state-of-the-art (SOTA) methods by an undeniable margin.

2 Related Work

2.1 Face Anti-Spoofing

Methods on Intra-datasets.

The essence of FAS is a defensive measure for face recognition systems and has been studied for over a decade. Some CNN-based methods [31, 32, 45, 10, 29] design a unified framework of feature extraction and classification in an end-to-end manner. Intuitively, the live faces in any scene have consistent face-like geometry. Inspired by this, some works [30, 41, 49, 53] leverage the physical-based depth information instead of binary classification loss as supervision, which are more faithful attack clues in any domain. With the popularity of high-quality 2d attacks, i.e., OULU-NPU [1], SiW [30], CelebA-Spoof [57] and high-fidelity mask attacks, i.e., MARsV2 [26], WMCA [9, 37], HiFiMask [20, 18] and SuHiFiMask [6, 5] with more realistic in terms of color, texture, and geometry structure, it is very challenging to mine spoofing traces from the visible spectrum alone. Methods based on multimodal fusion [9, 56, 16, 17, 8] have proven to be effective in alleviating the above problems. The motivation for these methods is that indistinguishable fake faces may exhibit quite different properties under the other spectrum. In order to alleviate the limitation of consistency between testing and training modalities, flexible modality based methods [15, 22, 54] aims to improve the performance on any single modality by leveraging available multimodal data. However, above methods are not specially designed to solve the domain generalization.

Domain Generalization Methods.

Domain Adaptation (DA) [46, 33, 36] aims to minimize the distribution discrepancy between the source and target domain by leveraging the unlabeled target data. However, the target data is difficult to collect, or even unknown during training. Domain Generalization (DG) can conquer this by taking the advantage of multiple source domains without seeing any target data. MADDG [40], SSDG [12], DR-MD-Net [47] aim to learn a generalized feature space via adversarial training. SSAN [50] reduces the model’s overfitting of style by randomly assembling the content and style of the samples. RFM [41], MT-FAS [38], D${}^{2}$AM [3], and SDA [48] aim to find the generalized feature directions via meta-learning strategies. In addition to aligning a domain-invariant feature space, SA-FAS [43] encourages domain separability while aligning the live-to-spoof transition to be the same for all domains. Considering the domain information lies in the style features, ANRL [27] and SSAN [50] use IN strategy to separate complete representation into content and style features according to image statistics. DRDG [28] iteratively reweights the relative importance between samples to further improve the generalization. The latest emerging strategy is to improve the generalization with the help of domain-specific information. CIFAS [34] adopts causal intervention with backdoor adjustment to mitigate domain bias for learning generalizable features. AMEL [62] exploits the domain-specific feature as a complement to common domain-invariant features to further improve the generalization. IADG [63] learns generalizable visual features by weakening the features’ sensitivity to instance-specific styles. In addition to supervised training, Liu et al. [35] propose the first unsupervised DG framework for FAS, which could exploit large amounts unlabeled data to learn generalizable features.

2.2 Vision-Language Models (VLMs).

The vision-language models have undergone a leapfrog development since CLIP [39] was proposed. This approach has stimulated thinking and innovation in many fields, such as object detection, image generation [25], and image forgery detection [24, 23]. In terms of text models, GPT-3 already has strong language processing capabilities. By combining it with visual basic models to build, and adding some necessary link parameters, the trained vision-language model achieves a combined understanding of images and text. For example, BLIP [14] has made significant progress in multimodality by freezing the constructed image encoder and text encoder during training to train an additional small query transformer. Similarly, LLaVa and minigpt-4 reduce the cost of model training by simply linearly map** image features to the word embedding space. On the basis of deep exploration of VLM by researchers, we want to further extend VLM in FAS.

3 CFPL: Class Free Prompt Learning

Figure 2: Our CFPL is built on CLIP [39] consists of image encoder $\mathcal{V}$ and text encoder $\mathcal{T}$, and adaptes to FAS tasks via prompt learning with four contributions: (1) CQF and SQF. CFPL introduces two lightweight transformers, namely Content Q-Former (CQF) and Style Q-Former (SQF) to learn the different semantic prompts conditioned on content and style features from the image encoder by using a set of learnable query vectors, respectively; (2) Prompt-Text Matched (PTM) surpervision. The fixed template description of each sample is used as a supervise to ensure CQF learns semantic visual representation; (3) A Diversified Style Prompt (DSP). The style from each layer of the image encoder is diversified through mixing feature statistics; (4) Prompt Modulation (PM). The generalized visual feature is adjusted by the modulation factor, which is generated by the text feature through the designed modulation function.

3.1 Semanticized Prompts Generation

Visual Content and Style features.

Considering the challenge of DG FAS is the interference of liveness-irrelevant signals on spoofing cues, we need to model these two types of information with different prompts and alleviate this interference by reducing their correlation. Based on previous research [50, 63], the liveness-irrelevant signals lie more in the instance-specific styles while spoofing clues is an image attribute hidden in the content. Based on this determination, we design Content Q-Former (CQF) and Style Q-Former (SQF) to generate content and style prompts conditioned corresponding visual features, respectively. Inspired by Adaptive instance regularization (AdaIN) [11], given a sample, we first calculate the mean and standard deviation at $l$-th layer, i.e., $\mu(\boldsymbol{v}^{l})$ and $\sigma(\boldsymbol{v}^{l})$; Then, we concatenate them to obtain the style statistics $\boldsymbol{v}^{l}_{s}$ of this layer; Finally, the style feature $\boldsymbol{v}_{s}$ is obtained by averaging style statistics from all layers for this sample. We calculate the content feature $\boldsymbol{v}_{c}$ of the sample from the output of the image encoder by normalizing. The details are as follows:

$$\begin{split}&\boldsymbol{v}_{s}=\frac{\sum_{l=1}^{L}\boldsymbol{v}^{l}_{s}}{L},\boldsymbol{v}^{l}_{s}=\left[\mu(\boldsymbol{v}^{l})\left|\right|\sigma(\boldsymbol{v}^{l})\right],\boldsymbol{v}_{s}\in\mathbb{R}^{1\times 2d},\\ &\boldsymbol{v}_{c}=\frac{\boldsymbol{v}^{L}-\mu(\boldsymbol{v}^{L})}{\sigma(\boldsymbol{v}^{L})},\boldsymbol{v}_{c}\in\mathbb{R}^{n\times d}\end{split}$$ (1)

where $L$ is the total layers of the image encoder. $\left[\cdot\left|\right|\cdot\right]$ represents concatenating features along embedding dimension.

CQF and SQF.

As shown in Fig. 2, CQF and SQF share a similar backbone, which consists of alternating layers of multiheaded self-attention (MSA), multiheaded cross-attention (MCA) and MLP blocks. Firstly, we create $N$ learnable query embeddings $\boldsymbol{Q}=\left\{\boldsymbol{q}^{1},\boldsymbol{q}^{2},\cdots,\boldsymbol{q}^{N}\right\}\in\mathbb{R}^{N\times d}$ as input to the backbone, where each query has a dimension of $d=512$ (same dimension with multi-modal embedding space); Then, the queries interact with each other through MSA block, and interact with image features $\boldsymbol{v}\in\mathbb{R}^{d}$ through MCA block; Finally, we obtain the prompt $\boldsymbol{P}=\left\{\boldsymbol{p}^{1},\boldsymbol{p}^{2},\dots,\boldsymbol{p}^{N}\right\}\in\mathbb{R}^{N\times d}$ after the queries pass through MLP block. This process can be expressed as:

$$\begin{split}\boldsymbol{Q^{\prime}}&=\boldsymbol{Q}+\texttt{MSA}(\texttt{LN}(\boldsymbol{Q})),\boldsymbol{Q^{\prime}}\in\mathbb{R}^{N\times d}\\ \boldsymbol{Q^{\prime\prime}}&=\boldsymbol{Q^{\prime}}+\texttt{MCA}(\texttt{LN}(\boldsymbol{Q^{\prime}}),\texttt{LN}(\boldsymbol{v})),\boldsymbol{Q^{\prime\prime}}\in\mathbb{R}^{N\times d}\\ \boldsymbol{P}&=\boldsymbol{Q^{\prime\prime}}+\texttt{MLP}(\texttt{LN}(\boldsymbol{Q^{\prime\prime}})),\boldsymbol{P}\in\mathbb{R}^{N\times d}\end{split}$$ (2)

where Layernorm (LN) is applied before every block, and residual connections after every block. The MLP contains two layers with a GELU non-linearity. Based on this training mechanism, CQF and SQF can bridge the gap between visual and language modalities by interacting prompt queries with corresponding image features, and output the content prompt $\boldsymbol{P}_{c}$ and style prompt $\boldsymbol{P}_{s}$, respectively.

3.2 Generalized Prompt Optimization

Text Supervision in Content Prompt.

Due to the lack of semantics for CLIP in the FAS categories, it is not suitable to align queries and text representations with the concept of maximizing their mutual information. Instead, we guide the CQF to understand the FAS’s categories at a higher level with a binary classification task, where the model is asked to predict whether a prompt-text pair is matched (PTM).

Given a min-batch with $B$ samples, we first generate a text description containing category attribute for each sample, such as “a photo of a $\left\langle\mathrm{CLASS}\right\rangle$ face.", where “$\mathrm{CLASS}$" is “live" for real face and “fake" for spoof face, respectively; After that, the text descriptions $\boldsymbol{T}$ are transformed into text supervisions $\boldsymbol{S}\in\mathbb{R}^{B\times 77\times d}$ through Tokenizer and Embed layers, sequentially; Then, we construct positive and negative feature pairs of prompt-text for prediction by CQF. Specifically, we concatenate content prompt $\boldsymbol{P_{c}}\in\mathbb{R}^{B\times N\times d}$ and text supervision $\boldsymbol{S}$ according to the embedding dimension, and obtain the positive feature pairs $\boldsymbol{R}_{p}\in\mathbb{R}^{B\times N\times 2d}$. We adopt the hard negative mining strategy from ALBEF [13] to create informative negative pairs. Such as for each prompt, one negative text is selected with the contrastive similarity distribution, where texts that are more similar to the prompt have a higher chance of being sampled. A similar strategy for one hard negative prompt for each text. Therefore, we can obtain negative feature pairs $\boldsymbol{R}^{prompt}_{n}\in\mathbb{R}^{B\times N\times 2d}$ and $\boldsymbol{R}^{text}_{n}\in\mathbb{R}^{B\times N\times 2d}$ by mining prompt and text, respectively; After that, we concatenate all positive and negative feature pairs $\boldsymbol{R}_{p}$, $\boldsymbol{R}^{prompt}_{n}$, and $\boldsymbol{R}^{text}_{n}$ to obtain the joint features $\boldsymbol{R}\in\mathbb{R}^{3B\times N\times 2d}$ according to the batch dimension. This process can be expressed as:

$$\begin{split}\boldsymbol{S}&=\texttt{Embed}(\texttt{Tokenizer}(\boldsymbol{T})),\boldsymbol{S}\in\mathbb{R}^{{B\times 77\times d}},\\ \boldsymbol{S}&=\texttt{Mean\&Expand}(\boldsymbol{S}),\boldsymbol{S}\in\mathbb{R}^{{B\times N\times d}},\\ \boldsymbol{R}_{p}&=\left[\boldsymbol{P}\left|\right|\boldsymbol{S}\right]_{\texttt{2}},\boldsymbol{R}_{p}\in\mathbb{R}^{{B\times N\times 2d}},\\ \boldsymbol{R}&=\left[\boldsymbol{R}_{p}\left|\right|\boldsymbol{R}^{prompt}_{n}\left|\right|\boldsymbol{R}^{text}_{n}\right]_{\texttt{0}},\boldsymbol{R}\in\mathbb{R}^{{3B\times N\times 2d}}\\ \end{split}$$ (3)

where the number of word tokens in text supervisions is aligned with the number of queries in the content prompt by averaging (Mean) and expanding (Expand) $N$ times. $\left[\cdot\left|\right|\cdot\right]_{\texttt{dim}}$ represents concatenating features along dim dimension. Finally, the optimization of text supervision is achieved by predicting the matched and unmatched probabilities for the joint features $\boldsymbol{R}$:

$$\begin{split}\mathcal{L}_{ptm}=\sum_{i=1}^{3B}\mathcal{H}(\boldsymbol{y}^{ptm}_{i},\texttt{Mean}(\texttt{FC}^{ptm}(\boldsymbol{R}_{i})))\end{split}$$ (4)

where we feed each query embedding into a two-class linear classifier to obtain a logit, and average (Mean) the logits across all queries as the output matching score. $\mathcal{H}(.,.)$ is the cross-entropy loss, $\texttt{FC}^{ptm}$ is a fully-connected layer followed by softmax, and $\boldsymbol{y}^{ptm}\in\left\{0,1\right\}$ is a 2-dimensional one-hot vector representing the ground-truth label.

Method	OCI→M			OMI→C			OCM→I			ICM→O			avg.
	HTER$\downarrow$	AUC	TPR@ FPR=1%	HTER	AUC	TPR@ FPR=1%	HTER	AUC	TPR@ FPR=1%	HTER	AUC	TPR@ FPR=1%	HTER
MADDG [40]	17.69	88.06	-	24.50	84.51	-	22.19	84.99	-	27.98	80.02	-	23.09
DR-MD-Net [47]	17.02	90.10	-	19.68	87.43	-	20.87	86.72	-	25.02	81.47	-	20.64
RFMeta [41]	13.89	93.98	-	20.27	88.16	-	17.30	90.48	-	16.45	91.16	-	16.97
NAS-FAS [52]	19.53	88.63	-	16.54	90.18	-	14.51	93.84	-	13.80	93.43	-	16.09
D${}^{2}$AM [3]	12.70	95.66	-	20.98	85.58	-	15.43	91.22	-	15.27	90.87	-	16.09
SDA [48]	15.40	91.80	-	24.50	84.40	-	15.60	90.10	-	23.10	84.30	-	19.65
DRDG [28]	12.43	95.81	-	19.05	88.79	-	15.56	91.79	-	15.63	91.75	-	15.66
ANRL [27]	10.83	96.75	-	17.83	89.26	-	16.03	91.04	-	15.67	91.90	-	15.09
SSDG-R [12]	7.38	97.17	-	10.44	95.94	-	11.71	96.59	-	15.61	91.54	-	11.28
SSAN-R [50]	6.67	98.75	-	10.00	96.67	-	8.88	96.79	-	13.72	93.63	-	9.81
PatchNet [45]	7.10	98.46	-	11.33	94.58	-	13.40	95.67	-	11.82	95.07	-	10.91
SA-FAS [43]	5.95	96.55	-	8.78	95.37	-	6.58	97.54	-	10.00	96.23	-	7.82
IADG [63]	5.41	98.19	-	8.70	96.44	-	10.62	94.50	-	8.86	97.14	-	8.39
CFPL(Ours)	3.09	99.45	94.28	2.56	99.10	66.33	5.43	98.41	85.29	3.33	99.05	90.06	3.60
ViTAF*-5-shot [10]	2.92	99.62	91.66	1.40	99.92	98.57	1.64	99.64	91.53	5.39	98.67	76.05	2.83
FLIP-MCL* [42]	4.95	98.11	74.67	0.54	99.98	100.00	4.25	99.07	84.62	2.31	99.63	92.28	3.01
CFPL*(Ours)	1.43	99.28	98.57	2.56	99.10	66.33	5.43	98.41	85.29	2.50	99.42	94.72	2.98

Table 1: The results (%) of Protocol 1 on MSU-MFSD (M), CASIA-FASD (C), ReplayAttack (I), and OULU-NPU (O) datasets. Note that the $*$ indicates the corresponding method using CelebA-Spoof [57] as the supplementary source dataset and ‘5-shot’ represents 5 images from the target datasets participating in the training phase.

Diversified Style Prompt.

Due to the indescribability of the sample style, we are unable to complete this task using text supervision. Implicitly, we borrow a strategy from MixStyle [59] that mixes style feature statistics between instances to achieve diversification of style prompts.

Specifically, given the visual style stastics $\left[\mu(\boldsymbol{v}),\sigma(\boldsymbol{v})\right]$ of a min-batch, we first obtain reference statistics $\mu(\hat{\boldsymbol{v}})$ and $\sigma(\hat{\boldsymbol{v}})$ by shuffling the order of batch dimension for $\mu(\boldsymbol{v})$ and $\sigma(\boldsymbol{v})$, respectively; Then, we generate mixture of feature statistics $\gamma_{mix}$ and $\beta_{mix}$ through a weighted approach:

$$\begin{split}\gamma_{mix}=\lambda\sigma(\boldsymbol{v})+(1-\lambda)\sigma(\hat{\boldsymbol{v}}),\\ \beta_{mix}=\lambda\mu(\boldsymbol{v})+(1-\lambda)\mu(\hat{\boldsymbol{v}})\end{split}$$ (5)

where $\lambda$ is an instance-specific, random weight sampled from the beta distribution, $\lambda\sim Beta(\alpha,\alpha)$. $\alpha$ is set to 0.1 according to the suggestion in [59]. Finally, the mixture of style statistics $\left[\beta_{mix},\gamma_{mix}\right]$ are used to calculate style features according to Eq. 1.

3.3 Prompt Modulation on Visual Features

Class Free Prompt Modulation.

Due to the content and style prompts are generated based on sample instances, they are more suitable as a set of fine-tuning factors (class free) for adaptively recalibrating channel-wise visual feature responses, compared to using them as classifier’s weights (with class) to predict visual feature. In this work, we use prompts to distinguish between generalized features and liveness-irrelevant signals by explicitly modeling interdependencies between channels.

Concretely, we first input content prompt $\boldsymbol{P_{c}}$ and style prompt $\boldsymbol{P_{s}}$ into the text encoder to produce text features, $\boldsymbol{t_{c}}\in\mathbb{R}^{B\times d}$ and $\boldsymbol{t_{s}}\in\mathbb{R}^{B\times d}$, respectively; After that, we concatenate these two types of text features along the embedding dimension to obtain modulation features $\boldsymbol{t}\in\mathbb{R}^{B\times 2d}$ with rich visual concepts; Then, we employ a gating mechanism $\texttt{g}_{e}$ with a sigmoid activation $\delta$ to map modulation features to the weighting factors $\boldsymbol{w}\in\mathbb{R}^{B\times d}$. This process can be expressed as:

$$\begin{split}\boldsymbol{w}&=\delta(\texttt{g}_{e}(\boldsymbol{t},\mathbf{W}))=\delta(\mathbf{W}_{2}\sigma_{\textrm{relu}}(\mathbf{W}_{1}\boldsymbol{t})),\\ \boldsymbol{\tilde{v}}&=\left[\boldsymbol{\tilde{v}}^{1},\boldsymbol{\tilde{v}}^{2},\dots,\boldsymbol{\tilde{v}}^{d}\right],\boldsymbol{\tilde{v}}^{c}=\boldsymbol{w}^{c}\cdot\boldsymbol{v}^{c},\\ \mathcal{L}_{cls}&=\sum_{i=1}^{B}\mathcal{H}(\boldsymbol{y}^{cls}_{i},\texttt{FC}^{cls}(\boldsymbol{\tilde{v}}_{i})),\boldsymbol{\tilde{v}}\in\mathbb{R}^{B\times d}\end{split}$$ (6)

where $\sigma_{\textrm{relu}}$ is the ReLU function, $\mathbf{W}_{1}\in\mathbb{R}^{\frac{d}{r}\times 2d}$ and $\mathbf{W}_{2}\in\mathbb{R}^{d\times\frac{d}{r}}$ are trainable parameters for two fully connected (FC) layers in $\texttt{g}_{e}$ function. $r$ is a reduction ratio, with a value of 16 in this work. Finally, the adapted visual features $\boldsymbol{\tilde{v}}\in\mathbb{R}^{B\times d}$ are obtained by weighting the channel-wise feature $\boldsymbol{v}^{c}$ with the scalar $\boldsymbol{w}^{c}$, and append a fully-connected ($\texttt{FC}^{cls}$) layer followed by softmax to predict a two-class (i.e., live or fake) probability. $y^{cls}\in\left\{0,1\right\}$ is the label for live or spoof face.

Model Training and Inference.

In the training stage, parameters from two designed Q-Formers, i.e., CQF and SQF, two fully connected layers for classifiers, i.e., $\texttt{FC}^{ptm}$ and $\texttt{FC}^{cls}$, one gate function, i.e., $\texttt{g}_{e}$, and image encoder $\mathcal{V}$ are updated and the text encoder $\mathcal{T}$ is fixed. The full training objective of CFPL is:

$$\begin{split}\mathcal{L}_{total}=\mathcal{L}_{cls}+\mathcal{L}_{ptm}\end{split}$$ (7)

In the inference stage, our CQF and SQF will adaptively generate the semanticized prompt as input to the text encoder based on each sample instance. Finally, the text encoder generates continuous and widely adjustable modulation factors for weighting visual features to generalization.

Method	CS→W			SW→C			CW→S			avg.
	HTER$\downarrow$	AUC	TPR@ FPR=1%	HTER	AUC	TPR@ FPR=1%	HTER	AUC	TPR@ FPR=1%	HTER
ViT* [10]	7.98	97.97	73.61	11.13	95.46	47.59	13.35	94.13	49.97	10.82
ViTAF*-5-shot [10]	2.91	99.71	92.65	6.00	98.55	78.56	11.60	95.03	60.12	6.83
FLIP-MCL* [42]	4.46	99.16	83.86	9.66	96.69	59.00	11.71	95.21	57.98	8.61
CFPL*(Ours)	4.40	99.11	85.23	8.13	96.70	62.41	8.50	97.00	55.66	7.01
ViT [10]	21.04	89.12	30.09	17.12	89.05	22.71	17.16	90.25	30.23	18.44
CLIP-V [39]	20.00	87.72	16.44	17.67	89.67	20.70	8.32	97.23	57.28	15.33
CLIP [39]	17.05	89.37	8.17	15.22	91.99	17.08	9.34	96.62	60.75	13.87
CoOp [61]	9.52	90.49	10.68	18.30	87.47	11.50	11.37	95.46	40.40	13.06
CFPL (Ours)	9.04	96.48	25.84	14.83	90.36	8.33	8.77	96.83	53.34	10.88

Table 2: The results (%) of Protocol 2 on CASIA-SURF (S), CASIA-SURF CeFA (C), and WMCA (W) datasets. Note that the $*$ indicates the corresponding method using CelebA-Spoof [57] as the supplementary source dataset and ‘5-shot’ represents 5 images from the target datasets participating in the training phase.

4 Experimental Setup

Datasets, Protocols and Evaluation Metrics.

Following the prior work [10], two Protocols are used to evaluate the generalization in this work. For Protocol 1, we use four widely-used benchmark datasets, MSU-MFSD (M) [51], CASIA-FASD (C) [58], Idiap Replay-Attack (I) [4], and OULU-NPU (O) [1]. For Protocol 2, we use RGB samples in CASIA-SURF (S) [56], CASIA-SURF CeFA (C) [16], and WMCA (W) [9] datasets, which contain more subjects, diverse attack types, and rich collection environments. In each Protocol, we treat each dataset as a domain and apply the leave-one-out testing for generalization evaluation. We adopt three metrics to evaluate the performance of a model: (1) HTER. It computes the average of the FRR and the FAR. (2) AUC. It evaluates the theoretical performance of the model. (3) TPR at a fixed False Positive Rate (FPR=1%). It can be used to select a suitable trade-off threshold according to a given real application.

Implementation Details. We set the length of style and content queries to $16$, where each query has a dimension of $512$; The depth of the CQF and SQF is set to $1$. Style prompt diversification is activated in the training phase with a probability of 0.5 and does not participate in the test phase; All models are trained with a batch size of $12$, an Adam optimizer with a weight decay of 0.05. The minimum learning rate at the second stage is $1e-6$. We resize images to $224\times 224$, augmented with random resized crop** and horizontal flip**, and train all models with $500$ epochs.

4.1 Cross-domain Results

For Protocol 1, we report the results of recent SOTA methods on Tab. 1, such as SA-FAS [43] encourages domain separability while aligning the live-to-spoof transition; IADG [63] learns the generalizable feature by weakening the features’ sensitivity to instance-specific styles; FLIP-MCL* [42] improves the visual representations with the help of natural language, and ViTAF*-5-shot [10] tackles a real-world anti-spoofing when $5$ images are available from target datasets. From Tab. 1, without using CelebA-Spoof [57], it can be seen that our method achieves the best performance for all metrics on four test datasets. Specifically, on the HTER metric (similar conclusions on AUC), CFPL outperforms IADG [63] for all target domains with lower values: M (3.09% vs., 5.41%), C (2.56% vs., 8.70%), I (5.43% vs., 10.62%) and O (3.33% vs., 8.86%). Finally, an average HTER of 3.60% is achieved, significantly better than the previous best result of 7.82%. After introducing the CelebA-Spoof [57] dataset as the supplementary training data, CFPL* outperforms FLIP-MCL* [42] in terms of the average metric of HTER, such as 2.98% vs., 3.01%. It indicates that our algorithm aligns images with text representation by class-free prompt learning is superior to an ensemble of class descriptions. In addition, CFPL* is slightly inferior to the ViTAF*-5-shot [10] on the average of HTER, such as 2.98% vs., 2.83%, which uses 5 additional samples from target domain in training data, greatly alleviating the overfitting on the domain-specific distribution.

For Protocol 2, we list the results of different methods on Tab. 2. Without using CelebA-Spoof [57], our CFPL significantly surpasses several baselines in terms of the HTER metric when tested on the W and C domains. However, when tested on the S dataset, CFPL is slightly inferior to the CLIP-V [39] on the HTER (8.77% vs.8.32%), AUC (96.83% vs.97.23%) and TPR@FPR=1% (53.34% vs.57.28%) metrics. Due to the obvious spoofing traces on the CASIA-SURF dataset [56], relying solely on visual features is relatively generalizable. Similar to the conclusion of Protocol 1, CFPL* outperforms baseline ViT* [10] and FLIP-MCL* [42] when introducing the CelebA-Spoof [57] dataset into training data. For example, our CFPL* achieves the optimal mean on the HTER (7.01%) metric.

Baseline	PTM	DSP	PM	HTER(%)$\downarrow$	AUC(%)	TPR(%) @FPR=1%
CoOp [61]	-	-	-	8.78	94.77	43.71
$\checkmark$	-	-	-	8.11	96.09	51.59
$\checkmark$	$\checkmark$	-	-	7.50	96.39	54.78
$\checkmark$	$\checkmark$	$\checkmark$	-	7.08	96.79	57.61
$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	6.72	97.09	60.35

Table 3: Ablation of each component, where each result is the average on all sub-protocols.

4.2 Ablation Study

Effectiveness of each component. In order to evaluate the contribution of each component in our framework, such as Text Supervision (abbreviated as ‘PTM’), Diversification of Style Prompt (abbreviated as ‘DSP’), and Prompt Modulation (abbreviated as ‘PM’), we conduct ablation studies on Protocol 1 and 2 by gradually introducing one of them into the Baseline (abbreviated as ‘B’), where the Baseline is obtained by removing all contributions from the CFPL. And report the average results on all sub-protocols in Tab. 3.

Specifically, instead of modeling a prompt’s context words with free learnable vectors in CoOp [61], our ‘Baseline’ introduces two lightweight transformers CQF and SQF to learn the expected prompts conditioned on content and style features from the image encoder, achieving significant generalization benefits of -0.67% (HTER), +1.32% (AUC) and +7.88% (TPR@FPR=1%), respectively. After introducing the text supervision in the content prompt, the results of the three metrics are optimized to 7.50% (HTER), 96.39% (AUC), and 54.78% (TPR@FPR=1%), respectively. It indicates that promoting the content prompt carries sufficient category attributes that can be converted into generalization benefits. Further diversification of style statistics in style prompts can further expand performance benefits, i.e., -0.42% (HTER), +0.4% (AUC), and +2.83% (TPR@FPR=1%). Instead of using the designed gate function $\texttt{g}_{e}$, we calculate the modulation factor $\boldsymbol{w}$ by calculating the mean of content and style features, such as $\boldsymbol{w}=\left(\boldsymbol{t_{c}}+\boldsymbol{t_{s}}\right)/2,\boldsymbol{w}\in\mathbb{R}^{{B\times d}}$. From the Tab. 3, it can be seen that if removing the designed gate function, the generalization significantly decreased from 6.72% (HTER), 97.09% (AUC) and 60.35% (TPR@FPR=1%) to 7.08% (HTER), 96.79% (AUC) and 57.61% (TPR@FPR=1%).

Furthermore, in Fig. 3, we detailed the result of each method on three metrics across all sub-protocols, where the red line represents the Baseline, and the blue line represents our CFPL. From Fig. 3, it can be clearly seen that the blue line shrinks with the smallest area in the polar coordinate system of the HTER, while is distributed with the largest area for the AUC and TPR@FPR=1%. The opposite conclusion applies to the Baseline of red lines. Almost all methods have an undeniable advantage over Baseline.

Effect of the Structures of CQF and SQF. In Tab. 4, we list the results of CoCoOp [60], CFPL removing SQF (abbreviated as ‘CQF’), CFPL removing CQF (abbreviated as ‘SQF’), and CFPL on the ICM→O experiment. From Tab. 4, we can see that a simple two-layer bottleneck structure cannot effectively alleviate the FAS task with significant domain differences, such as achieving results of 6.80%, 97.27%, and 60.41% on metrics HTER, AUC, and TPR@FPR=1%. When replacing Meta-Net with CQF and SQF, we obtain performance benefits of -1.68% (HTER), +1.38% (AUC), and +13.26% (TPR@FPR=1%) and -1.96% (HTER), +1.48% (AUC) and +26.67% (TPR@FPR=1%), respectively. Finally, by collaborating with CQF and SQF, our CFPL achieves the best results, such as 3.33% (HTER), 99.05% (AUC), and 90.06% (TPR@FPR=1%). Furthermore, our CFPL further improves their performance through collaborative CQF and SQF, indicating that the CQF and SQF not only bring significant benefits but also have a positive collaborative effect.

Method	HTER(%)$\downarrow$	AUC(%)	TPR(%)@FPR=1%
CoCoOp [60]	6.80	97.27	60.41
CQF	5.12	98.65	73.67
SQF	4.84	98.75	87.08
CFPL	3.33	99.05	90.06

Table 4: Ablation of the structures for CQF and SQF on ICM→O.

HTER(%)$\downarrow$	Length
Depth	$\times$8	$\times$16	$\times$32	$\times$64
$\times$1	3.47	3.33	3.33	3.30
$\times$4	3.45	3.42	3.45	3.45
$\times$8	3.56	3.56	3.47	3.47
$\times$12	3.41	3.33	3.33	3.33

Table 5: Ablation of the length for Queries and the depth for Q-former on ICM→O. The optimal value for each row/column is represented in bold/italics.

(a)

(b)

(c)

Figure 3: The results of each method on three metrics across all sub-protocols, where the red line represents the Baseline, and the blue line represents our CFPL. For the HTER metric, the smaller area enclosed by lines, the better performance of the corresponding methods. The opposite conclusion applies to metrics AUC and TPR@FPR=1%.

The Length of Queries and the Depth of Q-Former. The number of learnable queries and the depth of CQF and SQF can also affect the performance. We search for the optimal value on the ICM→O experiment for the length of queries and the depth of Q-Former from two sets of changing values, such as $\left[8,16,32,64\right]$ for the former, and $\left[1,4,8,12\right]$ for the latter, respectively.

From Tab. 5, the following two conclusions can be drawn: (1) The length of the queries is set to around 16, achieving optimal performance. By observing at different depth settings, the number of learnable queries increases exponentially, the performance benefits of the model are subtle, and there is even a trend toward degradation. Specifically, at a length of 16, models at different depths achieve decent performance, with values of 3.33%, 3.42%, 3.56%, and 3.33% for HTER. (2) The depth of Q-Former has a negligible impact on the performance. In detail, under each length setting, the performance of models with different depths fluctuates around a certain value of HTER. For example, when the length is set to 8, the HTER is 3.45%, while when the length is 16, 32, and 64, the HTER is 3.33%. Based on the experimental results, we suggest setting the number of queries to 16 and the depth of the Q-Former to 1.

4.3 Visualization and Analysis

With attention-model explainability tool [2], we visually validate the superiority of the proposed CFPL from the visual attention maps, compared to the Baseline, which is obtained by removing all contributions from the CFPL. The results on all protocols are shown in Fig. 4, where the maps of the Baseline correspond to misclassified samples, while our CFPL correctly classifies these samples.

Specifically, for the OCM→I experiment on Protocol 1, the Baseline classifies live face errors due to focusing more on the background. Our CFPL correctly classifies it by correcting the focus area to the boundary between face and background. For the playback attack, the Baseline does not pay attention to spoofing clues, such as the reflection spot on the electronic screen, and the feature map area illuminated by our method. Similar conclusions can be drawn on other sub-protocols.

Figure 4: Using visualization tool [2], the attention maps on all sub-protocols from Protocol 1, where the Baseline caused classification errors due to its failure to detect spoofing regions, and our CFPL correctly classifies these samples by correcting the region of interest.

5 Conclusion

In this work, we target DG FAS via textual prompt learning for the first time, and present a cross-domain FAS framework CFPL, which utilizes two lightweight transformers, CQF and SQF, to learn the different semantic prompts conditioned on content and style features. Finally, we introduce text supervision, diverse style prompt, and prompt modulation to promote the generalization.

6 Acknowledgments

This work was supported by the National Key Research and Development Program of China under Grant 2021YFE0205700, Bei**g Natural Science Foundation JQ23016, the Science and Technology Development Fund of Macau Project 0123/2022/A3, 0096/2023/RIA2, and 0070/2020/AMJ, Guangdong Provincial Key R&D Programme: 2019B010148001, the Chinese National Natural Science Foundation Project 62276254, U23B2054, and the InnoHK program.