A Survey of Privacy-Preserving Model Explanations:
Privacy Risks, Attacks, and Countermeasures

Many surveys have summarised different privacy issues on ML models (Biggio and Roli, 2018; Papernot et al., 2017; Machado et al., 2021; Liu et al., 2021), while others reviewed explanation methods for ML models (Gilpin et al., 2018; Bodria et al., 2023; Adadi and Berrada, 2018), but not both. For example, Rigaki et al. (Rigaki and Garcia, 2023) presented a thorough analysis of over 45 publications on privacy attacks in machine learning, spanning the last seven years. Hu et al. (Hu et al., 2022b) surveyed a special type of privacy attacks, called membership inference. On the other hand, others (Guidotti et al., 2018; Adadi and Berrada, 2018; Došilović et al., 2018) offered a comprehensive classification of model explanations to enhance interpretability and guided the selection of suitable methods for specific ML models and desired explanations.

Some existing surveys summarised adversarial attacks but presented partial coverage of privacy attacks on model explanations with basic introductions and limited discussions of the methods. Ferry et al. (Ferry et al., 2023b) examined the interplay between interpretability, fairness, and privacy, which are critical for responsible AI, particularly in high-stakes decision-making like college admissions and credit scoring. Baniecki et al. (Baniecki and Biecek, 2024) surveyed adversarial attacks on model explanations and fairness metrics, offered a unified taxonomy for clarity across related research areas, and discussed defensive strategies against such attacks. However, these papers are either too high-level or too specialised in non-privacy attacks.

Our survey presents an in-depth examination of privacy attacks on model explanations, diverging from previous work by its comprehensive nature. Rather than addressing the full spectrum of adversarial attacks, our study is specifically tailored to privacy attacks. This focus is due to the recent surge in these attacks and their significant potential to compromise the right to explanation (Goodman and Flaxman, 2017) and the right to privacy (Banisar, 2011). The threat posed by such privacy attacks could, in essence, challenge the very existence and usefulness of model explanations. Unlike the existing reviews that selected a very limited number of publications related to privacy attacks on model explanations (e.g. only two references are included in (Baniecki and Biecek, 2024)), we conduct a comprehensive search and include more than 50 related works in this survey. We delve into the underlying principles, theoretical frameworks, methodologies, and taxonomies, while also map** out potential trajectories for future research. Especially, our work encompasses the emerging field of privacy-preserving explanations (PrivEx), highlighting model explanations that inherently protect user privacy (Vo et al., 2023; Mochaourab et al., 2021; Harder et al., 2020).

Finding relevant research on this subject proved to be complex due to its incorporation of various topics such as data privacy, privacy attacks, explanations of models, explainable AI (XAI), and the development of privacy-preserving explanations. To navigate this breadth of concepts, we employed diverse keyword combinations about “privacy”, “explanation”, and specific attack types including “membership inference”, “data reconstruction”, “attribute inference”, “model extraction”, “model stealing”, “property inference”, and “model inversion”. Our initial search utilised platforms like Google Scholar, Semantic Scholar, and Scite.ai – an AI-enhanced search tool – to assemble a preliminary collection of studies. This selection was expanded through backward searches, analysing the references of initially chosen papers, and forward searches, identifying papers that cited the initial ones. Additionally, we manually verified the relevance and focus of these articles across various sources due to discrepancies, such as some studies addressing privacy in the context of safeguarding against manipulation attacks instead of privacy intrusions. Ultimately, this process culminated in nearly 50 pivotal research papers on the topic.

The main contributions of this article are:

• •

  Comprehensive Review: To the best of our knowledge, this study represents the inaugural effort to thoroughly examine privacy-preserving model explanations. We have collated and summarised a substantial body of literature, including papers published or in pre-print up to March 2024.

• •

  Connected Taxonomies: We have organised all existing literature on PrivEx according to various criteria, including the types of explanations targeted and the methodologies employed in attacks and defences. Fig. 2 showcases the taxonomy we have developed to structure these works.

• •

  Causal Analysis: Recent research has begun to investigate conditions that could lead to privacy leaks through model explanations, indicating that some explanation mechanisms inherently possess vulnerabilities. To this end, we dedicate a section to discuss the probable causes of these leaks.

• •

  Challenges and Future Directions: Designing privacy-preserving explanations for machine learning models is an emerging field of research. From the surveyed literature, we highlight unresolved issues and suggest several potential research directions into both the offensive and defensive aspects of privacy in model explanations.

• •

  Datasets and Metrics: In support of empirical research in PrivEx, we compile a comprehensive overview of datasets and evaluation metrics previously utilised in the field.

• •

  Online Updating Resource: To facilitate research in privacy-preserving model explanations, we have established an open-source repository¹¹1https://github.com/tamlhp/awesome-privex, which aggregates a collection of pertinent studies, including links to papers and available code.

The rest of the article is organised as follows. § 2 revisits model explanations, acting as foundations for privacy attacks. § 3 presents the taxonomy of privacy attacks on model explanations and provides in-depth descriptions, including threat model and attack scenarios. § 4 discusses the causes of privacy leaks in model explanations. § 5 explores countermeasures and a new class of privacy-preserving model explanations by design. § 6 provides the pinpoints to existing resources including source code, datasets, and evaluation metrics. Finally, § 7 contains a discussion on ongoing and upcoming research directions and § 8 concludes the survey.

The explanation function $\phi(D,f,x;\cdot)$ is predicated on identifying influential attributes (with the $\cdot$ symbol representing any potential additional inputs), and the explanation for the query $x$ is frequently referred to simply as $\phi(x)$ (Chang and Shokri, 2021). The value at the $i$-th index of a feature-based explanation, $\phi_{i}(x)$, quantifies the extent of influence the $i$-th feature exerts on the label ascribed to $x$. Ancona et al. (Ancona et al., 2018) have curated a comprehensive exposition of these attribution-focused explanation modalities, also termed attribution methods or numerical influential measures (Shokri et al., 2020).

• •

  (Vanilla) Gradients: Simonyan et al. (Simonyan et al., 2013) introduces gradient-based explanations, originally for image classification models, to emphasises important image pixels that affect the predictive outcomes. The explanation vector is defined as $\phi^{GRAD}(x)=\nabla_{x}f(x)$ or $\phi_{i}({x})=\frac{\partial f}{\partial x_{i}}({x})$ for each feature $i$. A high partial differential value indicates that a pixel significantly affects the prediction, and analysing the map these values (so-called gradient map) can explain a model’s decision-making (Miura et al., 2021). Shrikumar et al. (Shrikumar et al., 2017) suggest enhancing numerical explanations by using the input feature value multiplied by the gradient, $\phi_{i}({x})=x_{i}\times\frac{\partial f}{\partial x_{i}}({x})$.

• •

  Integrated Gradients: Sundararajan et al. (Sundararajan et al., 2017) advocate for an alternative to standard gradient computation by averaging gradients along a straight path from a baseline input $x^{BL}$ (often $x^{BL}=\vec{0}$) to the actual input. This method follows critical axioms like sensitivity and completeness. Sensitivity ensures that if there’s a prediction change due to $x_{i}$ not equaling $x_{BL,i}$, then $\phi_{i}({x})$ should not be zero. Completeness dictates that the sum of all attributions equals the change in prediction from the baseline to the input.

  (1)

  $$\phi^{INTG}({x}_{i})=(x_{i}-x_{BL,i})\cdot\int_{\alpha=0}^{1}\frac{\partial c({x}^{\alpha})}{\partial x^{\alpha}_{i}}\bigg{|}_{{x}^{\alpha}={x}+\alpha({x}-x^{BL})}.$$

• •

  Guided Backpropagation: Designed for networks with ReLU activations (others as well), Guided Backpropagation (Springenberg et al., 2014) modifies the gradient to only reflect paths with positive weights and activations, thereby considering only the positive evidence for a specific prediction.

• •

  Layer-wise Relevance Propagation (LRP): proposed by Klauschen et al. (Bach et al., 2015) to assign relevance from the output layer back to the input features. The relevance in each layer is proportionally distributed according to the contribution from neurons in the previous layer. The final attributions for the input are referred to as $\phi^{LRP}({x})$.

This method explains a black-box ML model or complex deep neural networks by computing a surrogate model that is interpretable by design (Shokri et al., 2021; Deng, 2019; Guidotti et al., 2018) that can emulate the overall predictive patterns of the original model (Naretto et al., 2022).

Example-based explanation (aka case-based interpretability or record-based explanation (Shokri et al., 2020)) uses comparable examples to create transparent explanations for machine learning decisions, offering an accessible way to understand model predictions by contrasting similar cases from the model’s database or generated data (Montenegro et al., 2022). Case-based interpretability techniques can create a range of explanatory examples, including:

• •

  Similar examples: are the closest matches from the training data with corresponding predictions to the case being analyzed, identified through a defined measure of similarity.

• •

  Typical examples: representing the epitome of a particular prediction, frequently utilized in models that focus on prototype learning.

• •

  Counterfactual examples: are similar examples but with differing predictions, highlighting the minimal changes needed for a different outcome. We dedicate a separate discussion on counterfactuals in the next subsection.

• •

  Semi-factual examples: are similar to the original case with the same prediction but positioned near the decision boundary, demonstrating the robustness of the prediction against variations typical of a different classification.

• •

  Influential examples: are key data points within a training set that have a significant impact on a model’s prediction for a given query instance (Koh and Liang, 2017). For explanatory purposes, we can provide the top $k$ influential points (Shokri et al., 2020).

These explanations can be sourced from existing datasets (i.e. $\phi(D,f,x;.)\in D$) (Koh and Liang, 2017) or crafted based on the original data (Kenny et al., 2021; Lipton, 2018).

Counterfactual explanations (aka algorithmic recourse) provide insights into how slight changes to input features could lead to different model outcomes, aiding in tasks like model debugging and ensuring regulatory compliance (Goethals et al., 2023; Kuppa and Le-Khac, 2021). The study in (Kuppa and Le-Khac, 2021) gives an illustration of counterfactual and other four sample categories (i. e., adversarial examples, local robustness (Zhang et al., 2024), invariant samples, and uncertainty samples) through the boundaries between human analyst and a learnt model (see Fig. 4). The application of counterfactual explanations varies with the model’s complexity and includes considerations such as model transparency, type compatibility, and adherence to constraints like feasibility and causality (Wachter et al., 2017; Dodge et al., 2019; Binns et al., 2018). The concept overlaps with other areas of research such as algorithmic recourse, inverse classification, and contrastive explanations (Karimi et al., 2021; Ustun et al., 2019; Laugel et al., 2017; Dhurandhar et al., 2018).

Kuppa et al. (Kuppa and Le-Khac, 2021) notes that methods for creating counterfactual explanations (CF) bear resemblance to those for generating adversarial examples (AE) in the way they both employ gradient-based optimization and surrogate models to find CF/AE for a given model. Some privacy attacks on adversarial examples can be used on counterfactual explanations (Kuppa and Le-Khac, 2021).

MIA aim to detect if data is part of a model’s training set (Shokri et al., 2019, 2021). Before model explanations, popular attacks are loss thresholding and likelihood ratio attack (LRT) (Pawelczyk et al., 2023). Loss thresholding identifies if a data point was in the training set by checking the model’s error rate against a threshold, requiring access to labels and model details (Yeom et al., 2018; Sablayrolles et al., 2019). LRT, in contrast, uses shadow models to compare confidence levels of data being in or out of the training set, calculating a likelihood ratio to predict membership without needing direct model access (Carlini et al., 2022). Pawelczyk et al. (Pawelczyk et al., 2023) designs a recourse-based attack (using counterfactual explanation) without access to the true labels and knowledge of the correct loss functions.

• •

  Threat model on gradient-based explanations: Most threat models are based on threshold-based attacks (Shokri et al., 2021). There are two key scenarios for this: the optimal threshold scenario, where the threshold is deduced from known data point memberships to gauge the maximum privacy risk; and the reference/shadow model scenario, which is more practical and assumes the attacker has some labeled data from the same distribution as the target model, as well as knowledge of the model’s architecture and hyperparameters in line with Kerckhoffs’s principle (Petitcolas, 2023). The attacker then trains a number of shadow models on this data to approximate the threshold, an approach that becomes more resource-intensive as the number of shadow models increases (Shokri et al., 2021).

• •

  Threat model on interpretable surrogates: Naretto et al. (Naretto et al., 2022) investigates how global explanation methods can potentially compromise the privacy. Specifically, the authors focus on TREPAN (Craven and Shavlik, 1994), an algorithm that explains neural network decisions by creating a surrogate Decision Tree (DT) model.

• •

  Threat model on counterfactuals: Pawelczyk et al. (Pawelczyk et al., 2023) formulates a membership inference game for attacking counterfactual explanations. The game features two participants: a model owner ($\mathcal{O}$) and an opponent ($\mathcal{A}$). Their actions are as follows. $\mathcal{O}$ selects a dataset for training from a population $D^{N}$, applying a training algorithm $T$ with a loss function $\ell$. Subsequently, $\mathcal{O}$ assigns a binary label $f_{\theta}(z)$ to each datapoint $z$ in $D_{t}$. Let $D_{t}^{0}$ be the segment of training data for which $f_{\theta}(x)=0$, and $D_{\theta,0}$ represent the conditional distribution $p(z)|f_{\theta}(z)=0$. $\mathcal{O}$ tosses a coin, and based on the outcome, selects a sample $x$ from either $D_{\theta,0}$ or $D_{t}^{+}$. Then, using the recourse algorithm $\phi$, $\mathcal{O}$ generates an alternate instance $x^{\prime}$ from $\phi(f_{\theta},x,D_{t})$ and sends the pair $(x^{\prime},x)$ to $\mathcal{A}$. In addition to the sample pair, $\mathcal{A}$ has the capability to make queries to $D$. It is presumed that $\mathcal{A}$ is fully aware of $\mathcal{O}$’s implementation specifics, including the training algorithm $T$ and the recourse algorithm $\phi$. $\mathcal{A}$ concludes the game by providing a binary guess $G$ signifying if $x$ belongs to $D_{t}$ (MEMBER) or does not ($x\notin D_{t}$, NON-MEMBER).

Liu et al. (Liu et al., 2024d) proposes a model-based attack that involves four main stages: training a shadow model, extracting attribution features, training an attack model, and inferring membership (see Fig. 6). The adversary starts by training a shadow model using an auxiliary dataset that is similar to the training data of the target model. Then, attribution maps are generated for a given sample, and perturbations are applied based on these maps to observe changes in predictions. Next, the adversary trains an attack model, typically a Multi-Layer Perceptron (MLP), using the attribution features combined with other data such as loss values and one-hot encoded class information to construct features indicative of membership.

• •

  Attacks on gradient-based explanations: Shokri et al. (Shokri et al., 2021) uses a threshold-based attack that infers membership based on the model’s confidence or its explanation output. A data point is classified as a member if the variance of the confidence scores $Var(f_{\theta}(x))$ or the variance of the explanation $Var(\phi(x))$ is below or equal to a certain threshold $\tau$. Attacks using explanation variance exploit the model’s certainty: when a model is sure about a prediction, explanation variance is low. However, near the decision boundary, even small changes can increase explanation variance. Models with certain activation functions like tanh, sigmoid, or softmax have steeper gradients, affecting how training data points are positioned relative to these boundaries (Shokri et al., 2021).

• •

  Attacks on interpretable surrogates: Naretto et al. (Naretto et al., 2022) develops an attacking procedure to assess the potential privacy risks of an interpretable surrogate (global explainer) that attempts to replicate the behavior of a black-box model. First, an MIA model, denoted as $A_{b}$, is trained to determine whether a specific data record, $x$, was included in the training dataset, $D_{train}^{b}$, of the black-box model $b$. This attack model leverages the black-box $b$ itself to classify the training data for the attack, making it specifically aimed at $b$. The attack training dataset $D_{train}^{a}$ is the same as $D_{Attack}^{B}$. Similarly, another MIA model, $A_{c}$, is developed to target the global explainer $c$, which serves as an interpretable stand-in for the black-box model $b$. This model is trained using $D_{train}^{a}$, but this time the labeling is done by $c$, not $b$.

• •

  Attacks on counterfactual explanations: The adversary has access to both the original instance $x$ and a counterfactual instance $x^{\prime}$. Models often overfit to training points, resulting in lower losses for these points compared to those on the test set (Shokri et al., 2021). Pawelczyk et al. (Pawelczyk et al., 2023) designs a distance-based attack where if the loss is below a certain threshold $\tau$, the point is considered a MEMBER of the training set. The counterfactual distance $c(x,x^{\prime})$ is effectively the distance to the model boundary, and even though algorithms that produce realistic recourses may not optimize for this distance, it can still be viewed as an approximation to the distance to the model boundary (Karimi et al., 2021; Pawelczyk et al., 2020a). The counterfactual distance-based attack is defined by $MI_{Distance}(x)$ as follows:

  (10)

  $$MI_{Distance}(x)=\begin{cases}\text{Member}&\text{if }c(x,x^{\prime})\geq\tau_{D}\\ \text{Non-member}&\text{if }c(x,x^{\prime})<\tau_{D}\end{cases}$$

  Another attack is using a Likelihood Ratio Test on top of the Counterfactual Distance (CFD) (Pawelczyk et al., 2023). The process involves calculating a baseline statistic $t_{0}$ using $c(x,x^{\prime})$ from the recourse output. If the initial statistic $t_{0}$ surpasses the critical threshold $z_{1-\alpha}$, which is the $1-\alpha$ quantile of the normal distribution $Z$, the algorithm designates the data point as a ‘Non-member’; and ‘Member’ otherwise. The key benefit is that it estimates the parameters $\mu_{out},\sigma_{out}$ only once for the non-membership scenario, reducing the computational load when assessing multiple data points $x^{\prime}$ (Sablayrolles et al., 2019).

  Huang et al. (Huang et al., 2023) proposes a CFD-based Likelihood Ratio Test (LRT) for linear classifiers built on the above Pawelczyk method (Pawelczyk et al., 2023). But the attack is simplified and one-sided as it only estimates parameters for data outside the training set, thus reducing computational complexity.

  Kuppa et al. (Kuppa and Le-Khac, 2021) develops an attack that leverages an auxiliary dataset $D_{aux}$ to train a shadow model $A_{MemInf}$. This is done by generating counterfactual examples $x_{cfi}$ for input samples $x_{i}$ and training a 1-nearest neighbor (1-NN) classifier to predict class membership based on proximity to these counterfactuals. If the prediction probability difference between the shadow model $A_{MemInf}$ and the target model $T$ is below a threshold $t$, the sample is deemed part of the training set. This inference is made under the assumption that if both models predict similarly for a sample, it implies the sample was significant in its prediction. The method is advantageous as it requires no direct access to the training set and iteratively uses counterfactuals to extract new data.

Based on model predictions and explanations, reconstruction attacks involve dataset reconstrcution attacks, model reconstruction attacks, and model inversion attacks (see Fig. 8).

• •

  Threat model: A machine learning model that is interpretable, like a decision tree, contains implicit information about its training dataset (Ferry et al., 2023a). This information can be formalized into a probabilistic dataset $\mathcal{D}$ consisting of $n$ examples, each with $d$ attributes. Every attribute $a_{k}$ has a domain $V_{k}$ covering all possible attribute values. The knowledge about an attribute $a_{k}$ for a given example $x_{i}$ is represented by a probability distribution across all possible values for that attribute, using the random variable $\mathcal{D}_{i,k}$. If a value $\mathcal{D}_{i,k}$ within $V_{k}$ has all the probability mass (i.e., $P(\mathcal{D}_{i,k}=v_{i,k})=1$), it’s deterministic. Conversely, a probabilistic dataset encompasses some uncertainty about attribute values.

• •

  Probabilistic Reconstruction Attacks: Earlier research (Gambs et al., 2012) proposes a method for constructing a probabilistic dataset $\mathcal{D}^{DT}$ from the structure of a trained decision tree $DT$. This probabilistic dataset reflects the decision tree’s implicit knowledge about its training dataset $\mathcal{D}^{Orig}$. The construction of this dataset is termed a probabilistic reconstruction attack, and by design, $\mathcal{D}^{DT}$ is compatible with $\mathcal{D}^{Orig}$, meaning the actual value $v_{i,k}^{Orig}$ of any attribute $a_{k}$ for any example $x_{i}$ is always among the set of possible values in the probabilistic reconstruction ($P(\mathcal{D}_{i,k}^{DT}=v_{i,k}^{Orig})>0$).

• •

  Attacks on Interpretable Models: Ferry et al. (Ferry et al., 2023a) discusses the possibility of a probabilistic reconstruction attack on interpretable models. In the general case, the success of the attack is calculated using the joint entropy of the dataset’s cells, which can be simplified if the variables of the model are statistically independent. For interpretable models like decision trees and rule lists, this assumption allows further decomposition of the computation (Ferry et al., 2023a).

• •

  Threat model: For a DNN with parameters $A\in\mathbb{R}^{h\times d}$ and $w\in\mathbb{R}^{h}$, where $A_{i}$ represents the ith row of A, three assumptions are posited: (1) Each row $A_{1},...,A_{h}$ is a unit vector; (2) No pair of rows $A_{i}$ and $A_{j}$ are collinear for $i\neq j$, satisfying $\langle A_{i},A_{j}\rangle\leq 1-c$ for some $c>0$; (3) The rows $A_{1},...,A_{h}$ are linearly independent. These assumptions are stated to be without loss of generality since they can be achieved by simple reparameterization of the network, such as scaling $w$ or $A$, or by reducing the hidden layer dimension.

• •

  General attacks: Under these assumptions, it is possible to learn the function with a sample complexity independent of the input dimension $d$ (Milli et al., 2019). Specifically, with a probability of $1-\delta$, an algorithm can find a function $\hat{f}$ such that $\hat{f}=f$. If the algorithm cannot find such a function, it will report the failure. Regardless of the outcome, the algorithm requires only $O\left(h\log\frac{h}{\delta}\right)$ queries to learn the function.

• •

  Attacks on gradient-based explanations: The algorithm involves recovering a matrix $Z$ and a sign vector $s$ (Milli et al., 2019). The matrix $Z$ is composed of either $w_{i}A_{i}$ or $-w_{i}A_{i}$, with the signs encapsulated in $s$. The function $f$ can then be reconstructed from $Z$ and $s$, utilizing the recovered structure to make predictions. The approach relies on exploiting the gradient structure of $f$ to identify the hyperplanes that partition the input space and uses binary search to recover the necessary components of $Z$ and $s$.

• •

  Threat model: We consider a machine learning model $f_{t}$ that processes confidential data $x$ from a set $X_{p}$ (for instance, facial images). It employs these private inputs to generate a prediction $\hat{y}_{t}$ (such as identifying emotions). An issue arises when an attacker gains access to the target prediction $\hat{y}_{t}$ and the explanation $\phi_{t}$ (due to reasons like a data breach, interception during transmission, or sharing on social media). One scenario is to assume that the attacker only has the compromised data, an independent dataset $x\in X_{a}$, and the ability to interact with the target model via black-box (Zhao et al., 2021b). The attacker does not require additional privileged information, such as blurred versions of the images. The objective of the attacker is to develop their own inversion model $f_{a}$ to reconstruct the original image $x$ from the model’s outputs $\hat{y}_{t},\phi_{t}$). Such a reconstruction would allow them to predict sensitive information from the reconstructed image $\hat{x}_{r}$, including the possibility of re-identifying the individual from the facial emotion recognition system (Hu et al., 2022a).

• •

  Attack on a single gradient-based explanation: To invert the target model $M_{t}$, a Transposed Convolutional Neural Network (TCNN) (Dumoulin and Visin, 2016) is devised to reconstruct a two-dimensional image $x_{r}$ from the one-dimensional prediction vector $y_{t}$ provided by $M_{t}$. The TCNN minimises the mean squared error (MSE) loss to approximate the original image. This TCNN incorporates various input forms, such as saliency maps and 2D explanations (Selvaraju et al., 2017; Simonyan et al., 2013), enhancing the reconstruction of $x_{r}$. Inputs can be processed by flattening the 2D explanations into a 1D vector and concatenating with the prediction vector, or by using a CNN to convert 2D patterns into a 1D feature embedding, following the approach used in CNN encoder-decoder networks and super-resolution techniques (ur Rehman et al., 2019; Zhang et al., 2020a). A U-Net architecture is employed to improve the reconstruction fidelity (Zhang et al., 2018). A hybrid model that combines flattened explanations with the U-Net structure is introduced in (Zhao et al., 2021b). The training objective for these models is defined by the image reconstruction loss function:

  (11)

  $$L_{r}=\sum_{x}(M^{a}_{i}(M_{t}(x))-x)^{2}$$

  where $x$ represents the original image, $M_{t}(x)=y_{t}$ denotes the prediction from the target model, and $M^{a}_{i}(M_{t}(x))=x_{r}$ is the reconstructed image output. Zhao et al. (Zhao et al., 2021b) conducts experiments on how different explanation methods, including gradients (Simonyan et al., 2013), CAM (Zhou et al., 2016), LRP (Bach et al., 2015), and blurred versions of the input images, affect the inversion model’s ability to capture information.

• •

  Attack on multiple gradient-based explanations: While many explanations clarify the reasons a model predicts a certain class within a set $C$, it is equally crucial to elucidate why it did not predict a different class $c^{\prime}\neq c$, offering contrastive insights (Miller, 2019). To facilitate this, certain techniques like Grad-CAM can generate explanations that are specific to a class based on the user’s query (Selvaraju et al., 2017). Nevertheless, this approach increases the risk to privacy as it provides additional information. Zhao et al. (Zhao et al., 2021b) makes use of these Alternative CAMs ($\Sigma$-CAM) by merging explanations across all classes in $|C|$ into a three-dimensional tensor, and they train their inversion models on this tensor rather than on a two-dimensional matrix representing a single explanation.

• •

  Attack on surrogate explanations: Interpretable surrogates could be harnessed for inversion attacks, even for models that do not provide target explanations. Zhao et al. (Zhao et al., 2021b) proposes an attack that predicts the target explanation and exploits that explanation to invert the original target data. Initially, an explainable surrogate target model $f_{a}$ is trained using the attacker’s dataset to generate a surrogate explanation $\widetilde{\phi}$. However, $\widetilde{\phi_{t}}$ is only accessible during the training phase and not during prediction. Consequently, an explanation inversion model $f_{e}$ is trained to reconstruct $\widetilde{\phi_{t}}$ as $\widehat{\phi_{r}}$ based on the target prediction $\widehat{y_{t}}$. The proposed loss function for minimising the surrogate explanation error is:

  (12)

  $$L_{\phi}=\sum_{x}\left(f_{e}(f_{t}(x))-\phi(f_{t}(x))\right)^{2}$$

  where $\phi(f)$ denotes the explanation of the model $f$, ${f_{t}(x)}={y_{t}}$ represents the surrogate target prediction, $\phi(f_{t}(x))=\widetilde{\phi_{t}}$ is the surrogate explanation, and $f_{e}(f_{t}(x))=\widehat{\phi_{r}}$ is the reconstructed surrogate explanation. This reconstructed explanation is available at prediction time. Finally, $\widehat{\phi_{r}}$ is fed into the image inversion model $\phi_{i}$ to finalize the model inversion attack. Given that $\widehat{\phi_{r}}$ is formatted similarly to $\widetilde{\phi_{t}}$, any explanation methods can be applied.

• •

  Attacks on confidence scores: Fredrikson et al. (Fredrikson et al., 2015) develops a model inversion attack by using a maximum a posteriori (MAP) estimator to compute $f(x_{1},\ldots,x_{d})$ for all possible values of the sensitive feature $x_{1}$, while exploiting confidence information from model predictions. Fredrikson et al. (Fredrikson et al., 2015) addresses the challenge of inverting high-dimensional features like facial recognition, where the inversion task becomes an optimization problem solved by gradient descent.

Attribute inference attacks, aka feature inference attacks, are designed to deduce specific attributes, such as gender, from individual data records by using accessible data like model predictions or explanations (Song and Shmatikov, 2020; Yeom et al., 2018) (see Fig. 9). These types of attacks are distinct from property inference attacks, which seek to ascertain broader dataset characteristics, like the training data’s gender ratio (Ganju et al., 2018; Melis et al., 2019; Zhang et al., 2021).

Duddu et al. (Duddu and Boutet, 2022) investigates a scenario where a machine learning model, $f_{target}$, is cloud-deployed within an MLaaS framework (e.g. Google Cloud, Microsoft Azure), capable of providing predictions and required explanations for any given input. Users can submit a private sample $x=\{x_{i}\}^{n}_{i=1}$ to the service provider and receive a prediction vector $\hat{y}=\{\hat{y}_{i}\}^{c}_{i=1}$, along with an explanation vector $\phi=\{\phi_{i}\}^{n}_{i=1}$ that pertains to a specific class. Although the service provider has the capacity to return multiple explanation vectors corresponding to different classes (Chen et al., 2018b), for practicality and without loss of generality, most works focuses on the use of one explanation vector for a specific class (Luo et al., 2022).