-
Detecting Brittle Decisions for Free: Leveraging Margin Consistency in Deep Robust Classifiers
Authors:
Jonas Ngnawé,
Sabyasachi Sahoo,
Yann Pequignot,
Frédéric Precioso,
Christian Gagné
Abstract:
Despite extensive research on adversarial training strategies to improve robustness, the decisions of even the most robust deep learning models can still be quite sensitive to imperceptible perturbations, creating serious risks when deploying them for high-stakes real-world applications. While detecting such cases may be critical, evaluating a model's vulnerability at a per-instance level using ad…
▽ More
Despite extensive research on adversarial training strategies to improve robustness, the decisions of even the most robust deep learning models can still be quite sensitive to imperceptible perturbations, creating serious risks when deploying them for high-stakes real-world applications. While detecting such cases may be critical, evaluating a model's vulnerability at a per-instance level using adversarial attacks is computationally too intensive and unsuitable for real-time deployment scenarios. The input space margin is the exact score to detect non-robust samples and is intractable for deep neural networks. This paper introduces the concept of margin consistency -- a property that links the input space margins and the logit margins in robust models -- for efficient detection of vulnerable samples. First, we establish that margin consistency is a necessary and sufficient condition to use a model's logit margin as a score for identifying non-robust samples. Next, through comprehensive empirical analysis of various robustly trained models on CIFAR10 and CIFAR100 datasets, we show that they indicate strong margin consistency with a strong correlation between their input space margins and the logit margins. Then, we show that we can effectively use the logit margin to confidently detect brittle decisions with such models and accurately estimate robust accuracy on an arbitrarily large test set by estimating the input margins only on a small subset. Finally, we address cases where the model is not sufficiently margin-consistent by learning a pseudo-margin from the feature representation. Our findings highlight the potential of leveraging deep representations to efficiently assess adversarial vulnerability in deployment scenarios.
△ Less
Submitted 26 June, 2024;
originally announced June 2024.
-
Layerwise Early Stop** for Test Time Adaptation
Authors:
Sabyasachi Sahoo,
Mostafa ElAraby,
Jonas Ngnawe,
Yann Pequignot,
Frederic Precioso,
Christian Gagne
Abstract:
Test Time Adaptation (TTA) addresses the problem of distribution shift by enabling pretrained models to learn new features on an unseen domain at test time. However, it poses a significant challenge to maintain a balance between learning new features and retaining useful pretrained features. In this paper, we propose Layerwise EArly STop** (LEAST) for TTA to address this problem. The key idea is…
▽ More
Test Time Adaptation (TTA) addresses the problem of distribution shift by enabling pretrained models to learn new features on an unseen domain at test time. However, it poses a significant challenge to maintain a balance between learning new features and retaining useful pretrained features. In this paper, we propose Layerwise EArly STop** (LEAST) for TTA to address this problem. The key idea is to stop adapting individual layers during TTA if the features being learned do not appear beneficial for the new domain. For that purpose, we propose using a novel gradient-based metric to measure the relevance of the current learnt features to the new domain without the need for supervised labels. More specifically, we propose to use this metric to determine dynamically when to stop updating each layer during TTA. This enables a more balanced adaptation, restricted to layers benefiting from it, and only for a certain number of steps. Such an approach also has the added effect of limiting the forgetting of pretrained features useful for dealing with new domains. Through extensive experiments, we demonstrate that Layerwise Early Stop** improves the performance of existing TTA approaches across multiple datasets, domain shifts, model architectures, and TTA losses.
△ Less
Submitted 4 April, 2024;
originally announced April 2024.
-
GROOD: GRadient-aware Out-Of-Distribution detection in interpolated manifolds
Authors:
Mostafa ElAraby,
Sabyasachi Sahoo,
Yann Pequignot,
Paul Novello,
Liam Paull
Abstract:
Deep neural networks (DNNs) often fail silently with over-confident predictions on out-of-distribution (OOD) samples, posing risks in real-world deployments. Existing techniques predominantly emphasize either the feature representation space or the gradient norms computed with respect to DNN parameters, yet they overlook the intricate gradient distribution and the topology of classification region…
▽ More
Deep neural networks (DNNs) often fail silently with over-confident predictions on out-of-distribution (OOD) samples, posing risks in real-world deployments. Existing techniques predominantly emphasize either the feature representation space or the gradient norms computed with respect to DNN parameters, yet they overlook the intricate gradient distribution and the topology of classification regions. To address this gap, we introduce GRadient-aware Out-Of-Distribution detection in interpolated manifolds (GROOD), a novel framework that relies on the discriminative power of gradient space to distinguish between in-distribution (ID) and OOD samples. To build this space, GROOD relies on class prototypes together with a prototype that specifically captures OOD characteristics. Uniquely, our approach incorporates a targeted mix-up operation at an early intermediate layer of the DNN to refine the separation of gradient spaces between ID and OOD samples. We quantify OOD detection efficacy using the distance to the nearest neighbor gradients derived from the training set, yielding a robust OOD score. Experimental evaluations substantiate that the introduction of targeted input mix-upamplifies the separation between ID and OOD in the gradient space, yielding impressive results across diverse datasets. Notably, when benchmarked against ImageNet-1k, GROOD surpasses the established robustness of state-of-the-art baselines. Through this work, we establish the utility of leveraging gradient spaces and class prototypes for enhanced OOD detection for DNN in image classification.
△ Less
Submitted 21 December, 2023;
originally announced December 2023.
-
Understanding Interventional TreeSHAP : How and Why it Works
Authors:
Gabriel Laberge,
Yann Pequignot
Abstract:
Shapley values are ubiquitous in interpretable Machine Learning due to their strong theoretical background and efficient implementation in the SHAP library. Computing these values previously induced an exponential cost with respect to the number of input features of an opaque model. Now, with efficient implementations such as Interventional TreeSHAP, this exponential burden is alleviated assuming…
▽ More
Shapley values are ubiquitous in interpretable Machine Learning due to their strong theoretical background and efficient implementation in the SHAP library. Computing these values previously induced an exponential cost with respect to the number of input features of an opaque model. Now, with efficient implementations such as Interventional TreeSHAP, this exponential burden is alleviated assuming one is explaining ensembles of decision trees. Although Interventional TreeSHAP has risen in popularity, it still lacks a formal proof of how/why it works. We provide such proof with the aim of not only increasing the transparency of the algorithm but also to encourage further development of these ideas. Notably, our proof for Interventional TreeSHAP is easily adapted to Shapley-Taylor indices and one-hot-encoded features.
△ Less
Submitted 5 December, 2022; v1 submitted 29 September, 2022;
originally announced September 2022.
-
Partial Order in Chaos: Consensus on Feature Attributions in the Rashomon Set
Authors:
Gabriel Laberge,
Yann Pequignot,
Alexandre Mathieu,
Foutse Khomh,
Mario Marchand
Abstract:
Post-hoc global/local feature attribution methods are progressively being employed to understand the decisions of complex machine learning models. Yet, because of limited amounts of data, it is possible to obtain a diversity of models with good empirical performance but that provide very different explanations for the same prediction, making it hard to derive insight from them. In this work, inste…
▽ More
Post-hoc global/local feature attribution methods are progressively being employed to understand the decisions of complex machine learning models. Yet, because of limited amounts of data, it is possible to obtain a diversity of models with good empirical performance but that provide very different explanations for the same prediction, making it hard to derive insight from them. In this work, instead of aiming at reducing the under-specification of model explanations, we fully embrace it and extract logical statements about feature attributions that are consistent across all models with good empirical performance (i.e. all models in the Rashomon Set). We show that partial orders of local/global feature importance arise from this methodology enabling more nuanced interpretations by allowing pairs of features to be incomparable when there is no consensus on their relative importance. We prove that every relation among features present in these partial orders also holds in the rankings provided by existing approaches. Finally, we present three use cases employing hypothesis spaces with tractable Rashomon Sets (Additive models, Kernel Ridge, and Random Forests) and show that partial orders allow one to extract consistent local and global interpretations of models despite their under-specification.
△ Less
Submitted 28 December, 2023; v1 submitted 25 October, 2021;
originally announced October 2021.
-
How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
Authors:
Florian Tambon,
Gabriel Laberge,
Le An,
Amin Nikanjam,
Paulina Stevia Nouwou Mindom,
Yann Pequignot,
Foutse Khomh,
Giulio Antoniol,
Ettore Merlo,
François Laviolette
Abstract:
Context: Machine Learning (ML) has been at the heart of many innovations over the past years. However, including it in so-called 'safety-critical' systems such as automotive or aeronautic has proven to be very challenging, since the shift in paradigm that ML brings completely changes traditional certification approaches.
Objective: This paper aims to elucidate challenges related to the certifica…
▽ More
Context: Machine Learning (ML) has been at the heart of many innovations over the past years. However, including it in so-called 'safety-critical' systems such as automotive or aeronautic has proven to be very challenging, since the shift in paradigm that ML brings completely changes traditional certification approaches.
Objective: This paper aims to elucidate challenges related to the certification of ML-based safety-critical systems, as well as the solutions that are proposed in the literature to tackle them, answering the question 'How to Certify Machine Learning Based Safety-critical Systems?'.
Method: We conduct a Systematic Literature Review (SLR) of research papers published between 2015 to 2020, covering topics related to the certification of ML systems. In total, we identified 217 papers covering topics considered to be the main pillars of ML certification: Robustness, Uncertainty, Explainability, Verification, Safe Reinforcement Learning, and Direct Certification. We analyzed the main trends and problems of each sub-field and provided summaries of the papers extracted.
Results: The SLR results highlighted the enthusiasm of the community for this subject, as well as the lack of diversity in terms of datasets and type of models. It also emphasized the need to further develop connections between academia and industries to deepen the domain study. Finally, it also illustrated the necessity to build connections between the above mention main pillars that are for now mainly studied separately.
Conclusion: We highlighted current efforts deployed to enable the certification of ML based software systems, and discuss some future research directions.
△ Less
Submitted 1 December, 2021; v1 submitted 26 July, 2021;
originally announced July 2021.
-
Out-of-distribution detection for regression tasks: parameter versus predictor entropy
Authors:
Yann Pequignot,
Mathieu Alain,
Patrick Dallaire,
Alireza Yeganehparast,
Pascal Germain,
Josée Desharnais,
François Laviolette
Abstract:
It is crucial to detect when an instance lies downright too far from the training samples for the machine learning model to be trusted, a challenge known as out-of-distribution (OOD) detection. For neural networks, one approach to this task consists of learning a diversity of predictors that all can explain the training data. This information can be used to estimate the epistemic uncertainty at a…
▽ More
It is crucial to detect when an instance lies downright too far from the training samples for the machine learning model to be trusted, a challenge known as out-of-distribution (OOD) detection. For neural networks, one approach to this task consists of learning a diversity of predictors that all can explain the training data. This information can be used to estimate the epistemic uncertainty at a given newly observed instance in terms of a measure of the disagreement of the predictions. Evaluation and certification of the ability of a method to detect OOD require specifying instances which are likely to occur in deployment yet on which no prediction is available. Focusing on regression tasks, we choose a simple yet insightful model for this OOD distribution and conduct an empirical evaluation of the ability of various methods to discriminate OOD samples from the data. Moreover, we exhibit evidence that a diversity of parameters may fail to translate to a diversity of predictors. Based on the choice of an OOD distribution, we propose a new way of estimating the entropy of a distribution on predictors based on nearest neighbors in function space. This leads to a variational objective which, combined with the family of distributions given by a generative neural network, systematically produces a diversity of predictors that provides a robust way to detect OOD samples.
△ Less
Submitted 11 September, 2023; v1 submitted 24 October, 2020;
originally announced October 2020.
-
Embeddability on functions: order and chaos
Authors:
Raphaël Carroy,
Yann Pequignot,
Zoltán Vidnyánszky
Abstract:
We study the quasi-order of topological embeddability on definable functions between Polish zero-dimensional spaces. We first study the descriptive complexity of this quasi-order restricted to the space of continuous functions. Our main result is the following dichotomy: the embeddability quasi-order restricted to continuous functions from a given compact space to another is either an analytic com…
▽ More
We study the quasi-order of topological embeddability on definable functions between Polish zero-dimensional spaces. We first study the descriptive complexity of this quasi-order restricted to the space of continuous functions. Our main result is the following dichotomy: the embeddability quasi-order restricted to continuous functions from a given compact space to another is either an analytic complete quasi-order or a well-quasi-order. We then turn to the existence of maximal elements with respect to embeddability in a given Baire class. It is proved that the class of continuous functions is the only Baire class to admit a maximal element. We prove that no Baire class admits a maximal element, except for the class of continuous functions which admits a maximum element.
△ Less
Submitted 22 February, 2018;
originally announced February 2018.
-
Finite versus infinite: an insufficient shift
Authors:
Yann Pequignot
Abstract:
The shift graph is defined on the space of infinite subsets of natural numbers by letting two sets be adjacent if one can be obtained from the other by removing its least element. We show that this graph is not a minimum among the graphs of the form $G_{f}$ defined on some Polish space $X$, where two distinct points are adjacent if one can be obtained from the other by a given Borel function…
▽ More
The shift graph is defined on the space of infinite subsets of natural numbers by letting two sets be adjacent if one can be obtained from the other by removing its least element. We show that this graph is not a minimum among the graphs of the form $G_{f}$ defined on some Polish space $X$, where two distinct points are adjacent if one can be obtained from the other by a given Borel function $f:X\to X$. This answers the primary outstanding question from \cite{Kechris19991}.
△ Less
Submitted 25 September, 2017; v1 submitted 5 December, 2016;
originally announced December 2016.
-
Towards Better: A motivated introduction to better-quasi-orders
Authors:
Yann Pequignot
Abstract:
The well-quasi-orders (WQO) play an important role in various fields such as Computer Science, Logic or Graph Theory. Since the class of WQOs lacks closure under some important operations, the proof that a certain quasi-order is WQO consists often of proving it enjoys a stronger and more complicated property, namely that of being a better-quasi-order (BQO).
Several articles contains valuable int…
▽ More
The well-quasi-orders (WQO) play an important role in various fields such as Computer Science, Logic or Graph Theory. Since the class of WQOs lacks closure under some important operations, the proof that a certain quasi-order is WQO consists often of proving it enjoys a stronger and more complicated property, namely that of being a better-quasi-order (BQO).
Several articles contains valuable introductory material to the theory of BQOs. However, a textbook entitled "Introduction to better-quasi-order theory" is yet to be written. Here is an attempt to give a motivated and self-contained introduction to the deep concept defined by Nash-Williams that we would expect to find in such a textbook.
△ Less
Submitted 22 November, 2017; v1 submitted 20 April, 2016;
originally announced April 2016.
