Magnitude-based Neuron Pruning for Backdoor Defense

Most of the recent works [19, 38] assume that backdoor and clean neurons hardly overlap in filter level, i.e., each filter is associated with either the backdoor task or the clean task. However, filters in the first quadrants in Fig 1 have positive BLC and positive CLC, which implies they contribute to both clean accuracy and backdoor behavior. Instead of simply defining a filter as backdoored or clean, we suggest a fine-grained approach with thresholds $\tau\geq 0,\epsilon\geq 0$ to categorize filters into backdoor, clean, hybrid and redundant filters. We follow the assumption mentioned in Section 3.1, that the backdoor and clean parameters are separated at the neuron level. A filter is composed of a large number of neurons, as it commonly contains multiple channels and kernel weights. Backdoor filters primarily consist of backdoor neurons that are crucial for the backdoor task but have minimal contribution to the clean task, while clean filters exhibit the opposite behavior. Hybrid filters are composed of both backdoor neurons and clean neurons, hence, pruning them results in an increase in both backdoor loss and clean loss. Redundant filters contain unimportant neurons and can be pruned without significantly affecting the overall performance of the model, as is widely observed in model compression studies [15, 11].

As weight decay is widely applied to reduce the complexity of DNNs, neurons of less importance tend to have smaller magnitudes. Based on the above assumption, magnitude-based model compression methods [15, 11] use the $l_{p}$-norm of filters as a statistical indicator of their contribution to the final prediction result of the model . Since CLC measures the importance of filters for model performance, we assume a positive correlation between neuron magnitude and the CLC in the clean parameter space $\theta_{c}$. Denoting the correlation by $C:\mathbb{R}\rightarrow\mathbb{R}$, every clean filter $\theta^{(i,j)}$ satisfies $\textrm{CLC}(F,i,j)=C(\|\theta^{(i,j)}\|_{p})$. Since the backdoor model is trained on both the backdoor task and the clean task, each filter $\theta^{(i,j)}$ can be decomposed into $\theta^{(i,j)}=\theta^{(i,j)}_{c}\cup\theta^{(i,j)}_{b}$. Its contribution to the clean task satisfies:

which indicates that the $l_{p}$-norm of the backdoor or hybrid filter does not correspond to its actual contribution to the clean task, for it contains additional parameters used to trigger the backdoor. This observation is consistent with the results in Fig 1, where the majority of backdoor filters have larger $l_{p}$-norms compared to clean filters with the same CLC value.

Our research is based on two fundamental assumptions about the neurons of the backdoored DNN: 1) A filter can be associated with both the backdoor task and the clean task, if so, it is composed of both clean and backdoor neurons. 2) The model is trained with weight decay or other parameter regularization techniques, thus the $l_{p}$ norm of a filter is positively correlated with its overall contribution to clean accuracy and backdoor behavior.

We adopt a typical defense setting where the defender has downloaded a backdoored model from an untrustworthy third party without knowledge of the attack or training data. We assume a small amount of clean data $\mathcal{D}_{d}$ is available for defense, which can be collected from the Internet or carefully selected from the training data. The defender aims to first determine if the model is backdoored, then remove the backdoor behavior from the infected model with minimum degradation to its clean accuracy.

As is discussed in Section 3.3, the $l_{p}$-norms of clean filters approximately follow a positive correlation with their contribution to clean accuracy, while backdoor or hybrid filters contain backdoor neurons and have larger $l_{p}$-norms that deviate from the positive correlation. The deviation inspires us to develop a strategy to prune filters with large $l_{p}$-norms while preserving (or recovering) the clean accuracy. For each filter, we apply a mask $m\in[0,1]$, which acts on the magnitude of each filter without changing the specific weight. For model $F$ with $L$ layers, the collection of all masks is denoted by $\mathcal{M}=\{\boldsymbol{m}_{i}\}_{i=1}^{L}$, where $\boldsymbol{m}_{i}\in[0,1]^{c_{i}}$ is a vector of masks for the $i$-th layer with $c_{i}$ filters. The masked network $F_{\mathcal{M}}$ has the same architecture as $F$ with weight matrices of all the convolutional layers set to $\boldsymbol{m}_{i}\odot\theta^{(i)}$. To expose the backdoor filters with large $l_{p}$-norms and minimal impact on the clean loss, we formulate our problem as follows:

where $\odot$ denotes the Hadamard product, $\boldsymbol{w}_{i}\in\mathbb{R}^{c_{i}}$ is the vector of $l_{2}$-norms of filters of the $i$-th convolutional layer, and $\lambda>0$ is a hyperparameter balancing the loss and the weight penalty term.

The clean suppression objective is designed to reduce the magnitude of most clean neurons and expose the backdoor behavior of the infected model:

where $\mu>0$ is the weight decay hyperparameter. Since the suppression of clean neurons both increases the classification loss and reduces the $l_{2}$ regularization term, during the optimization, magnitude of clean neurons decrease at a faster rate than backdoor and hybrid neurons.

In contrast to clean suppression, the clean preservation objective is designed to preserve an important subset of clean neurons. This includes most clean filters and part of hybrid filters that are critical for the clean accuracy. Similarly to the weight penalty process, we initialize $\boldsymbol{m}_{i}\in[1,2]^{c_{i}}$ for each filter and optimize masks to increase magnitude of filters as much as possible while retaining the clean loss:

MNP aims to prune filters that deviate from the magnitude-saliency correlation, as these filters may contain potential backdoor neurons. The weight penalty objective is designed to achieve this purpose. Conducting weight penalty process directly on backdoor models is enough to defend against classical attacks including BadNets [10] and WaNet [25]. However, we have empirically found that weight penalty alone is unable to thoroughly remove the backdoors injected by some advanced attacks like DFST [6] or LIRA [7], as backdoor filters produced by these attacks may have smaller magnitudes, making it difficult to distinguish them from redundant filters. Therefore, we adopt the clean suppression objective to reduce the magnitude of clean neurons in the backdoor model, thereby exposing the low-magnitude backdoor filters. The combination of clean suppression and weight penalty process is enough to purify most backdoor models. To further reduce the compromise in model performance, we conduct the clean preserving process on the original backdoor model to preserve critical clean neurons, which may be mistakenly pruned and cause degradation of clean accuracy in other backdoor mitigation methods. During the process, the masks of the backdoor and redundant filters keeps almost unchanged, while the mask of a subset of clean and hybrid filters that are critical for the clean accuracy significantly increase. MNP adds the masks obtained from the clean preserving process to the masks obtained from the weight penalty process, and prunes filters with lower mask values, thus better balancing clean accuracy and backdoor mitigation performance. More discussion about the mechanism of MNP can be found in Appendix A.2.

MNP detects backdoor models by measuring the magnitude-saliency correlation of neurons across the model. For each filter, the $l_{2}$-norm of its weights $\|\theta^{(i,j)}\|_{2}^{2}$ and $\textrm{CLC}(\theta,i,j)$ are chosen as the magnitude and the saliency metrics, respectively. The Spearsman’s rank correlation coefficient $\rho(F,\theta)$ between $\|\theta^{(i,j)}\|_{2}^{2}$ and $\textrm{CLC}(\theta,i,j)$ can be used to measure the strength of the magnitude-saliency correlation of model $F$. We compare the correlations of the original model $F(\cdot;\theta)$ and the suppressed model $F(\cdot;\theta_{s})$ obtained from the clean suppression process. In clean models, the unlearning process reduces the magnitude of clean neurons at approximately equal rates, so the magnitude-saliency correlation is maintained, while the correlation of backdoored models is significantly weakened. Given threshold $\delta\in[0,1]$, the set of backdoored models can be defined as follows:

Note that we have omitted the approximation of CLC and the normalization of the magnitude and saliency metrics. The detailed detection method can be found in Appendix A.3.

We evaluate MNP against 10 challenging attacks. These include 3 static attacks: BadNets [10], Trojan [20], Blend [5], 2 clean label attacks: CL [30] and SIG [1], 2 dynamic attacks: IAB [24] and WaNet [25], 2 feature space attacks: FC [27] and DSFT [6], and 1 adaptive attack LIRA [7]. Default settings from original papers and open-source codes are applied for most attacks, including backdoor trigger pattern and size. The backdoor label of all attacks is set to class 0, with a default poisoning rate of 10%. Attacks are performed on CIFAR-10 with ResNet18 and a 12-class subset of ImageNet with ResNet-34. For training setups, Stochastic Gradient Descent (SGD) is used with an initial learning rate 0.1, weight decay 5e-4, momentum 0.9, batch size 128 for 200 epochs on CIFAR-10, and batch size 64 for 300 epochs on ImageNet subset. A cosine scheduler is employed to adjust the learning rate.

We compare MNP with a total of 10 backdoor defense methods. These include 4 pruning-based backdoor mitigation methods: FP [19], ANP [34], CLP [37] and RNP [18], 3 non-pruning mitigation methods: NC [31], NAD [16] and I-BAU [35], as well as 4 detection methods: NC, AC [3], SC [29] and STRIP [8]. All the defenses share limited access to 1% benign training data except for CLP. Hyperparameters for these defenses are adjusted based on open-source codes to obtain best performance against different attacks. For MNP, we set hyperparameter $\mu$ as 0.001, $\lambda$ as 0.0005, suppression epochs $T$ as 20, learning rates $\eta_{1}$ as 0.01 and $\eta_{2}$ as 0.1. The detection threshold $\delta$ is set as $0.2$, and the pruning threshold $\epsilon$ is dynamically adjusted to prune 30 neurons for each backdoor model.

We adopt two metrics for evaluating backdoor mitigation performance: 1) Clean Accuracy (CA), which is the model’s accuracy on clean test data; 2) Attack Success Rate (ASR), which is the model’s accuracy on backdoored test data. We adopt 3) Detection Rate (DR) to evaluate detection performance, which is the accuracy of the defense in identifying backdoor models.

Table 1 presents the defense performance of 8 backdoor mitigation methods against 10 backdoor attacks on CIFAR-10. MNP outperforms other defense methods by cutting the average ASR down to 4.03% with a slight drop on CA (2.19% on average). In comparison, the state-of-the-art methods ANP, RNP, NAD and I-BAU reduce the average ASR to 14.31%, 7.60%, 14.03%, and 21.72%, respectively. An important observation is that each defense method has its limits. For instance, NAD achieves the lowest ASR (15.70%) on DFST but is outperformed by MNP on most other attacks. MNP shows weakness against SIG and FC, yet it has the best overall performance and highest clean accuracy in most settings. Nearly all defenses struggle against DFST and LIRA, suggesting the need for more sophisticated mechanisms to counter advanced attacks.

Table 2 presents the defense performance of 8 backdoor mitigation methods against 6 backdoor attacks on CIFAR-10. Note that some attacks are not conducted for ImageNet subset because of difficulties in reproduction on high-resolution dataset. Pruning larger models trained on high-resolution dataset poses a greater challenge due to the difficulty in locating backdoor neurons. However, MNP can effectively defend against existing attacks and still demonstrates an advantage in maintaining clean accuracy. For example, MNP can reduce the ASR of BadNet to 0.87%, even slightly improving the clean accuracy. MNP also generally outperforms most state-of-the-art defenses, except for a slightly lower clean accuracy (90.03%/90.31%) on Trojan compared to ANP and a slightly weaker ASR (10.63%/10.32%) on FC compared to RNP. Nevertheless, it is evident that MNP can better balance clean accuracy and ASR overall, as it cuts the average ASR down to 4.10% with only <1% decline in CA.

The clean suppression process of MNP can be use to detect backdoor models by the strength of magnitude-saliency correlation, and the experimental results can be found in 4 in Appendix B. MNP is able to detect backdoor models generated by six attacks and achieve a 98.00% DR, outperforming NC (51.33%), AC (89.75%), SC (83.08%), and STRIP (80.67%).

In this part, we evaluate the impact of defense data size on the performance of MNP with a backdoored ResNet18 attacked by BadNets. We use 0.1%(50), 0.5% (250), 1% (500) and 5% (2500) images from the CIFAR-10 training set for defense, respectively. Results in Fig 4 show that as defense data size increases, MNP demonstrates better backdoor mitigation performance. In some cases, MNP only needs 0.1% of clean samples to reduce the ASR of several attacks to < 5%, highlighting its potential to identify backdoor neurons in few-shot settings. The impact of defense data size is more pronounced on CA. As the number of clean samples grows, the distribution of the defense set gets closer to the original dataset, allowing MNP to more accurately preserve clean neurons. Overall, MNP can effectively eliminate backdoor neurons under few-shot settings, and 1% of clean samples is sufficient for MNP to limit the degradation of CA to < 3%.

We conducted experiments on MNP with different poisoning rates on the CIFAR-10. We have also tested the effectiveness of MNP against the all-to-all attack, where the target label of the backdoored sample is set to one plus the original label ($S(y)=y+1$). Experimental results are shown in Table 5. For attacks with poisoning rate $\geq$ 5%, MNP can effectively reduce the ASR to approximately 1% without causing significant CA decrease. For attacks with poisoning rates $\leq$ 1%, MNP can still maintain a certain level of defense performance. Additionally, MNP can effectively defend against all-to-all attacks with poisoning rates $\geq$ 5%, reducing the ASR to 1% with < 2% CA degradation.

The most critical hyperparameters in MNP are $\lambda$ and $\mu$, as higher values of $\lambda$ or $\mu$ cause the optimization process to be dominated by the magnitude of neurons rather than the classification loss of the model. We kept other hyperparameters the same as in 5.1 and adjusted $\mu$ to values of $0$, $1e^{-4}$, $1e^{-3}$, $1e^{-2}$, and $1e^{-1}$, and lambda to values of $0$, $5e^{-5}$, $5e^{-4}$, $5e^{-3}$, and $5e^{-2}$. The results in Fig 4 indicate that MNP achieves the best defense performance when $\mu$ and $\lambda$ are set to proper, smaller values. A larger $\mu$ simultaneously reduces the magnitude of both backdoor and clean neurons and make the clean suppression process ineffective. Similarly, a larger $\lambda$ makes MNP ignore the classification loss, which prevents it from effectively preserving clean neurons. Overall, setting $\lambda$ and $\mu$ to smaller values helps maintain stable performance, while appropriate larger $\lambda$ and $\mu$ helps achieve the best performance.