Gone but Not Forgotten: Improved Benchmarks for Machine Unlearning

Abstract

Machine learning models are vulnerable to adversarial attacks, including attacks that leak information about the model’s training data. There has recently been an increase in interest about how to best address privacy concerns, especially in the presence of data-removal requests. Machine unlearning algorithms aim to efficiently update trained models to comply with data deletion requests while maintaining performance and without having to resort to retraining the model from scratch, a costly endeavor. Several algorithms in the machine unlearning literature demonstrate some level of privacy gains, but they are often evaluated only on rudimentary membership inference attacks, which do not represent realistic threats. In this paper we describe and propose alternative evaluation methods for three key shortcomings in the current evaluation of unlearning algorithms. We show the utility of our alternative evaluations via a series of experiments of state-of-the-art unlearning algorithms on different computer vision datasets, presenting a more detailed picture of the state of the field.

1 Introduction

While incorporating new data into established machine learning models via fine-tuning is a well-studied problem, the inverse problem of removing data from those models has received less attention until recently. The field of machine unlearning [1] [2] [3] has emerged over the last several years in response to legal frameworks, such as the EU’s GDPR [4], that give citizens the “right to be forgotten.” To comply with such legal frameworks, companies may be required not only to remove user data from data structures, but also from machine learning models trained on that data. Machine unlearning can be applied to many data domains including images, videos, audio, and text – in this paper we focus on computer vision (CV) in response to recent research efforts to comply with legal requirements for user privacy.

The naïve approach to unlearning would be to erase the required data from a dataset and retrain a model from scratch. The removed data will have no influence on the model parameters or its output (ignoring indirect influences such as hyper-parameters optimized prior to removal), so an attacker would have no way of extracting information about the removed data. The major downside of this approach - and the main issue that the machine unlearning literature seeks to address - is the computational burden and hence the immense cost in terms of processes and technology. Retraining a model from scratch on unlearning requests that arrive in a stream may simply not be feasible at scale, especially for large datasets and models with hundreds of millions of parameters that require thousands of epochs on tens or hundreds of GPUs to train. Ideal unlearning algorithms would provide the privacy guarantees and performance characteristics of a retrained-from-scratch model and require far fewer compute resources. As such, machine unlearning involves balancing privacy, cost, and long-term performance.

To realistically represent conditions under practical attacks when benchmarking machine unlearning algorithms, we argue that researchers and practitioners should consider the following three essential characteristics:

1. Emphasis of worst-case metrics over average-case metrics and the use of strong adversarial attacks to provide a high-quality upper-bound on privacy.

2. Consideration of model update-based attacks (e.g. leakage), as in [5], which may cause unlearning to provide additional information to attackers.

3. Analysis of unlearning algorithm performance over repeated applications of unlearning (i.e. iterative unlearning), especially in regards to degradation of test accuracy performance of the unlearned models.

In response to these gaps, we propose a framework that incorporates a suite of improved benchmarks for the testing and evaluation of machine unlearning algorithms. To test our framework and demonstrate the utility of the improved benchmarks we conduct a benchmarking study of a variety of state-of-the-art (SoTA) unlearning algorithms with the goal of presenting a holistic assessment of unlearning algorithms. Our contributions are intended to demonstrate the importance of using more comprehensive evaluations.

1.1 Unlearning Definition and Notation

We are primarily interested in approximate unlearning where the influence of the forget set is practically removed, as opposed to exact unlearning where the influence must completely and provably removed. We define unlearning as follows. We have a dataset $\mathcal{D}$ partitioned into forget, retain, validation, and test sets $\mathcal{D}_{f}$, $\mathcal{D}_{r}$, $\mathcal{D}_{val}$, and $\mathcal{D}_{test}$, respectively. A model $M$ is trained on the training subset $\mathcal{D}_{train}=\mathcal{D}_{f}\cup\mathcal{D}_{r}$, where $\mathcal{D}_{f}\cap\mathcal{D}_{r}=\emptyset$. An unlearning algorithm $\mathcal{U}:M\times(\mathcal{D}_{r},\mathcal{D}_{f})\rightarrow M^{\prime}$ produces a new model $M^{\prime}$ which ideally has minimized the possibility of information leakage from $\mathcal{D}_{f}$, maintained model performance, and done so with minimal compute. Iterative unlearning requires an extended definition (omitted here for brevity), but intuitively, it is the repeated act of unlearning over time.

2 Evaluating Unlearning Algorithms

Prior works in unlearning [5, 6, 7, 8, 9, 10, 11, 12, 13] take inspiration from the Differential Privacy (DP) literature, using membership inference attacks (MIAs), which can determine whether a certain example was part of the training set, to demonstrate an empirical upper bound on privacy. Intuitively, such privacy auditing attacks demonstrate what is possible by an adversary, and thus high-confidence upper bounds are crucial for understanding vulnerabilities in a system.

2.1 Privacy as a Worst-Case Metric

We argue that worst-case measures of privacy are crucial for effective evaluations of unlearning algorithms. Users, in the absence of a strict guarantee of their own privacy, will care primarily about worst-case outcomes for themselves, rather than average- or even best-case metrics. When examining privacy through the lens of DP, as many in the unlearning field do, this becomes even more clear. Satisfying $(\epsilon,\delta)$-Differential Privacy requires a mathematical proof showing an algorithm satisfies a strict worst-case upper-bound on potential information leakage from $\mathcal{D}_{train}$ across all possible training examples. Therefore, strong unlearning evaluations will both use as effective MIAs as possible, and present the results of the attacks with worst-case metrics.

Through our literature review we have found that the most common MIA used to evaluate algorithms is a simple Logistic Regression classifier trained to predict whether or not a loss value comes from an example in $\mathcal{D}_{f}$ [6, 14, 15, 16, 17]. Furthermore, results are reported almost exclusively through either accuracy or recall - both of which are average case metrics. The study in [16] takes an important step forward in using stronger MIAs, adapting the online version of the SoTA Likelihood Ratio Attack (LiRA) from [18] to the unlearning setting. The ‘online’ version refers to the need to train new models for every membership query. Notably, they continue the precedent set in [18] by showing Receiver Operating Characteristic (ROC) curves with log-scales to show performance at very low false-positive rates - better demonstrating worst-case outcomes. The effectiveness of LiRA as an MIA makes it a much more realistic estimate of privacy.

One downside of [16]’s online-LiRA implementation is its computational complexity. The attack entails training 256 ‘shadow’ models (each on a random half of the training set) and then running an unlearning algorithm 10,000 times on random forget sets for each of the 256 shadow models. This amounts to over 2.5 million unlearning runs per algorithm. For any fine-tuning based algorithm, or any algorithm dealing with even moderately large models or data sets, this sort of evaluation is impractical.

To remedy this gap, we adapt the offline version of LiRA for unlearning in our benchmarking framework. ‘Offline’ means that no new shadow models need to be trained for new membership queries. This setup only requires a single unlearning run, making this attack much more feasible - especially in the iterative unlearning setting (Section 2.3). While it is not as strong an attack as online-LiRA, and therefore worse for privacy estimation, it is much stronger than the basic Logistic Regression MIA, so we recommend its use in cases where the online-LiRA attack is computationally infeasible.

While MIAs are often just used to rank unlearning algorithms, some approaches have been made to more directly estimate DP privacy parameters ($\epsilon$, $\delta$). The 2023 NeurIPS Machine Unlearning Competition [19] based their metric on group-level DP, a stricter formulation of DP which considers datasets differing by up to $k$ examples, which makes sense for unlearning as requests are likely to be batched. In fact, the competition metric used estimates of worst-case MIA performance to estimate $\epsilon$ for each example in $\mathcal{D}_{f}$, producing an entire distribution of privacy levels. While [19] averaged the per-example estimates of $\epsilon$ into a single score, we re-implement the per-example $\epsilon$ estimation to preserve the full distribution of privacy levels, which we find to be a valuable point of comparison.

2.2 Update Leakage

While intuitively one may think that unlearning can only result in positive outcomes with respect to the privacy of forgotten data, [5] demonstrate that, in regards to an attacker who already has the outputs of the base model, the act of unlearning may result in worse privacy than not unlearning at all. The idea of model update-leakage - where an adversary uses the difference in behaviors of two models as an attack vector - has been previously studied in iterative learning scenarios [20, 12], and [5] demonstrate that the issue is still relevant in the unlearning setting. [5] find that update-leakages are often stronger than standard attacks for both fully retrained and unlearned models, although the effect is weaker for the latter. In this regard unlearning may actually be preferential to retraining in terms of privacy, as it could find an optimal middle ground between the two types of attacks. We therefore evaluate unlearning algorithms on update-leakage attacks to:

1. Demonstrate an additional benefit of unlearning over retraining from scratch by showing less susceptibility to update-leakage attacks.

2. Ensure a ‘do no harm’ criteria is satisfied by showing an update-leakage attack does no worse than an attack on the base model.

To the best of our knowledge (through review of citations of [5]), the only unlearning algorithm benchmarked on an update-leakage attack, besides SISA in [5], is GraphEraser [21], an unlearning algorithm for graph data. We implement and run evaluations on the update-leak attack from [5], which works very similarly to online-LiRA, except uses outputs from both the base and the unlearned models.

2.3 Iterative Unlearning

The final piece of a comprehensive unlearning evaluation is studying how unlearning affects a model after repeated applications of an algorithm. If unlearning can truly be a replacement for full retraining, and deployers of ML models fully comply with data removal requests and destroy the base model trained on $\mathcal{D}_{f}$, then unlearning must be iteratively applied to unlearned models. Few papers consider this set-up, and those that do are usually special cases where unlearning can be performed in closed form or with convex optimization [10] [11] [3]. Surprisingly, we were able to locate only one unlearning paper that evaluates iterative test accuracy [22]. We have not encountered an iterative test accuracy evaluation in the vision domain.

In the iterative setting, it is required at each iteration to both ensure effective forgetting and, crucially, maintain model performance, as performance degradation tends to accumulate over time. Single forget set evaluations simply do not capture this important dimension of real-world performance. It should be noted that auditing privacy over many iterations may be prohibitively expensive, especially for attacks like online-LiRA that require hundreds of unlearning runs per iteration - so in this setting, our offline-LiRA implementation might be preferred. In either case, test-set performance can be computed directly after each unlearning iteration, so we advocate for its inclusion in any unlearning evaluation. We implement an iterative unlearning pipeline that handles the required dataset splitting and model management to allow for extending existing unlearning evaluations to the iterative setting – we focus primarily on test-set accuracy.

2.3.1 Preliminary Results

We conduct a preliminary evaluation of various unlearning algorithms on the CIFAR10 dataset [23] with a ResNet18 architecture [24], which was chosen due to its small computational footprint. We evaluate the following baselines and unlearning algorithms: Identity (baseline), Retrain (baseline), Finetune (baseline, finetune on $\mathcal{D}_{r}$), RandLabel [8], BadTeach [14], SCRUB+R [16], SSD [6], and SSD+FT (SSD followed by Finetune).

For each unlearning iteration, $\mathcal{D}_{f}$ is constructed by sampling 1% of $\mathcal{D}_{train}$, conditioned on samples that have not already been forgotten in prior iterations. The sequence of forget sets is identical across all experiments. The Identity algorithm is a control where no unlearning is applied. The base model used to test each unlearning algorithm is a ResNet18 model trained for 30 epochs. The hyperparameters for each unlearning algorithm were discovered by running 100 trials with the Optuna framework [25], optimized for minimizing $|0.5-\text{MIA Accuracy}|$ (aiming for the MIA to do no better than random) and maximizing validation accuracy.

In this iterative setting we find notable discrepancies in test accuracy amongst the various algorithms. As shown in Figure 1, BadTeach rapidly degrades model performance to random guessing, while other algorithms are able to maintain or in some cases increase accuracy over time. While these results only analyze test-set accuracy, and do not characterize either privacy or runtime, they demonstrate an important facet that we have observed to be considered in only one other unlearning evaluation [22].

Figure 1: Iterative unlearning results for ResNet18 on CIFAR10.

3 Discussion

Overall, our initial tests show mixed results across unlearning algorithms, emphasizing the need for a holistic evaluation across all three major requirements (test accuracy, runtime, and privacy). For our privacy evaluations, we find that success in defending the basic Logistic Regression MIA, utilized in most unlearning evaluations, does not necessarily translate to success under stronger MIAs. In terms of update-leakage attacks, we find that only some of the tested algorithms perform better than retraining, but none have yet been found to leak extra information. As we saw in Section 2.3.1, there are considerable discrepancies in test accuracy in the iterative setting. We also find that hyperparameter tuning is a key aspect of algorithm performance, and different hyperparameters trade off performance and privacy differently. If optimizing hyperparameters for privacy with an expensive attack like online-LiRA the tuning process can be extremely slow, potentially presenting a significant barrier to real-world adoption.

4 Future Work

We will run evaluations on a variety of model architectures and datasets to provide a rigorous presentation of the state of the field. Our code base will be open sourced to provide an easy-to-use toolkit for designing and evaluating machine unlearning algorithms.