-
Transferability Metrics for Object Detection
Authors:
Louis Fouquet,
Simona Maggio,
Léo Dreyfus-Schmidt
Abstract:
Transfer learning aims to make the most of existing pre-trained models to achieve better performance on a new task in limited data scenarios. However, it is unclear which models will perform best on which task, and it is prohibitively expensive to try all possible combinations. If transferability estimation offers a computation-efficient approach to evaluate the generalisation ability of models, p…
▽ More
Transfer learning aims to make the most of existing pre-trained models to achieve better performance on a new task in limited data scenarios. However, it is unclear which models will perform best on which task, and it is prohibitively expensive to try all possible combinations. If transferability estimation offers a computation-efficient approach to evaluate the generalisation ability of models, prior works focused exclusively on classification settings. To overcome this limitation, we extend transferability metrics to object detection. We design a simple method to extract local features corresponding to each object within an image using ROI-Align. We also introduce TLogME, a transferability metric taking into account the coordinates regression task. In our experiments, we compare TLogME to state-of-the-art metrics in the estimation of transfer performance of the Faster-RCNN object detector. We evaluate all metrics on source and target selection tasks, for real and synthetic datasets, and with different backbone architectures. We show that, over different tasks, TLogME using the local extraction method provides a robust correlation with transfer performance and outperforms other transferability metrics on local and global level features.
△ Less
Submitted 27 June, 2023;
originally announced June 2023.
-
OpenAL: Evaluation and Interpretation of Active Learning Strategies
Authors:
W. Jonas,
A. Abraham,
L. Dreyfus-Schmidt
Abstract:
Despite the vast body of literature on Active Learning (AL), there is no comprehensive and open benchmark allowing for efficient and simple comparison of proposed samplers. Additionally, the variability in experimental settings across the literature makes it difficult to choose a sampling strategy, which is critical due to the one-off nature of AL experiments. To address those limitations, we intr…
▽ More
Despite the vast body of literature on Active Learning (AL), there is no comprehensive and open benchmark allowing for efficient and simple comparison of proposed samplers. Additionally, the variability in experimental settings across the literature makes it difficult to choose a sampling strategy, which is critical due to the one-off nature of AL experiments. To address those limitations, we introduce OpenAL, a flexible and open-source framework to easily run and compare sampling AL strategies on a collection of realistic tasks. The proposed benchmark is augmented with interpretability metrics and statistical analysis methods to understand when and why some samplers outperform others. Last but not least, practitioners can easily extend the benchmark by submitting their own AL samplers.
△ Less
Submitted 11 April, 2023;
originally announced April 2023.
-
Towards Clear Expectations for Uncertainty Estimation
Authors:
Victor Bouvier,
Simona Maggio,
Alexandre Abraham,
Léo Dreyfus-Schmidt
Abstract:
If Uncertainty Quantification (UQ) is crucial to achieve trustworthy Machine Learning (ML), most UQ methods suffer from disparate and inconsistent evaluation protocols. We claim this inconsistency results from the unclear requirements the community expects from UQ. This opinion paper offers a new perspective by specifying those requirements through five downstream tasks where we expect uncertainty…
▽ More
If Uncertainty Quantification (UQ) is crucial to achieve trustworthy Machine Learning (ML), most UQ methods suffer from disparate and inconsistent evaluation protocols. We claim this inconsistency results from the unclear requirements the community expects from UQ. This opinion paper offers a new perspective by specifying those requirements through five downstream tasks where we expect uncertainty scores to have substantial predictive power. We design these downstream tasks carefully to reflect real-life usage of ML models. On an example benchmark of 7 classification datasets, we did not observe statistical superiority of state-of-the-art intrinsic UQ methods against simple baselines. We believe that our findings question the very rationale of why we quantify uncertainty and call for a standardized protocol for UQ evaluation based on metrics proven to be relevant for the ML practitioner.
△ Less
Submitted 27 July, 2022;
originally announced July 2022.
-
Performance Prediction Under Dataset Shift
Authors:
Simona Maggio,
Victor Bouvier,
Léo Dreyfus-Schmidt
Abstract:
ML models deployed in production often have to face unknown domain changes, fundamentally different from their training settings. Performance prediction models carry out the crucial task of measuring the impact of these changes on model performance. We study the generalization capabilities of various performance prediction models to new domains by learning on generated synthetic perturbations. Emp…
▽ More
ML models deployed in production often have to face unknown domain changes, fundamentally different from their training settings. Performance prediction models carry out the crucial task of measuring the impact of these changes on model performance. We study the generalization capabilities of various performance prediction models to new domains by learning on generated synthetic perturbations. Empirical validation on a benchmark of ten tabular datasets shows that models based upon state-of-the-art shift detection metrics are not expressive enough to generalize to unseen domains, while Error Predictors bring a consistent improvement in performance prediction under shift. We additionally propose a natural and effortless uncertainty estimation of the predicted accuracy that ensures reliable use of performance predictors. Our implementation is available at https: //github.com/dataiku-research/performance_prediction_under_shift.
△ Less
Submitted 21 June, 2022;
originally announced June 2022.
-
Sample Noise Impact on Active Learning
Authors:
Alexandre Abraham,
Léo Dreyfus-Schmidt
Abstract:
This work explores the effect of noisy sample selection in active learning strategies. We show on both synthetic problems and real-life use-cases that knowledge of the sample noise can significantly improve the performance of active learning strategies. Building on prior work, we propose a robust sampler, Incremental Weighted K-Means that brings significant improvement on the synthetic tasks but o…
▽ More
This work explores the effect of noisy sample selection in active learning strategies. We show on both synthetic problems and real-life use-cases that knowledge of the sample noise can significantly improve the performance of active learning strategies. Building on prior work, we propose a robust sampler, Incremental Weighted K-Means that brings significant improvement on the synthetic tasks but only a marginal uplift on real-life ones. We hope that the questions raised in this paper are of interest to the community and could open new paths for active learning research.
△ Less
Submitted 21 October, 2022; v1 submitted 3 September, 2021;
originally announced September 2021.
-
Ensembling Shift Detectors: an Extensive Empirical Evaluation
Authors:
Simona Maggio,
Léo Dreyfus-Schmidt
Abstract:
The term dataset shift refers to the situation where the data used to train a machine learning model is different from where the model operates. While several types of shifts naturally occur, existing shift detectors are usually designed to address only a specific type of shift. We propose a simple yet powerful technique to ensemble complementary shift detectors, while tuning the significance leve…
▽ More
The term dataset shift refers to the situation where the data used to train a machine learning model is different from where the model operates. While several types of shifts naturally occur, existing shift detectors are usually designed to address only a specific type of shift. We propose a simple yet powerful technique to ensemble complementary shift detectors, while tuning the significance level of each detector's statistical test to the dataset. This enables a more robust shift detection, capable of addressing all different types of shift, which is essential in real-life settings where the precise shift type is often unknown. This approach is validated by a large-scale statistically sound benchmark study over various synthetic shifts applied to real-world structured datasets.
△ Less
Submitted 28 June, 2021;
originally announced June 2021.
-
Rebuilding Trust in Active Learning with Actionable Metrics
Authors:
Alexandre Abraham,
Léo Dreyfus-Schmidt
Abstract:
Active Learning (AL) is an active domain of research, but is seldom used in the industry despite the pressing needs. This is in part due to a misalignment of objectives, while research strives at getting the best results on selected datasets, the industry wants guarantees that Active Learning will perform consistently and at least better than random labeling. The very one-off nature of Active Lear…
▽ More
Active Learning (AL) is an active domain of research, but is seldom used in the industry despite the pressing needs. This is in part due to a misalignment of objectives, while research strives at getting the best results on selected datasets, the industry wants guarantees that Active Learning will perform consistently and at least better than random labeling. The very one-off nature of Active Learning makes it crucial to understand how strategy selection can be carried out and what drives poor performance (lack of exploration, selection of samples that are too hard to classify, ...).
To help rebuild trust of industrial practitioners in Active Learning, we present various actionable metrics. Through extensive experiments on reference datasets such as CIFAR100, Fashion-MNIST, and 20Newsgroups, we show that those metrics brings interpretability to AL strategies that can be leveraged by the practitioner.
△ Less
Submitted 19 February, 2021; v1 submitted 18 December, 2020;
originally announced December 2020.
-
Dataiku's Solution to SPHERE's Activity Recognition Challenge
Authors:
Maxime Voisin,
Leo Dreyfus-Schmidt,
Pierre Gutierrez,
Samuel Ronsin,
Marc Beillevaire
Abstract:
Our team won the second prize of the Safe Aging with SPHERE Challenge organized by SPHERE, in conjunction with ECML-PKDD and Driven Data. The goal of the competition was to recognize activities performed by humans, using sensor data. This paper presents our solution. It is based on a rich pre-processing and state of the art machine learning methods. From the raw train data, we generate a synthetic…
▽ More
Our team won the second prize of the Safe Aging with SPHERE Challenge organized by SPHERE, in conjunction with ECML-PKDD and Driven Data. The goal of the competition was to recognize activities performed by humans, using sensor data. This paper presents our solution. It is based on a rich pre-processing and state of the art machine learning methods. From the raw train data, we generate a synthetic train set with the same statistical characteristics as the test set. We then perform feature engineering. The machine learning modeling part is based on stacking weak learners through a grid searched XGBoost algorithm. Finally, we use post-processing to smooth our predictions over time.
△ Less
Submitted 9 October, 2016;
originally announced October 2016.
-
Stability conditions on Brauer tree algebras
Authors:
Léo Dreyfus-Schmidt
Abstract:
We study the space of stability conditions attached to the derived category of $A_{n}$-mod for $A_{n}$ the Brauer tree algebra of the line with $n$ edges. These algebras arise in the study of cyclic defect blocks of group algebras, and they are also related to the zig-zag algebras introduced by Huerfano and Khovanov. We show that for the Brauer tree algebra $A_{3}$, the connected component of the…
▽ More
We study the space of stability conditions attached to the derived category of $A_{n}$-mod for $A_{n}$ the Brauer tree algebra of the line with $n$ edges. These algebras arise in the study of cyclic defect blocks of group algebras, and they are also related to the zig-zag algebras introduced by Huerfano and Khovanov. We show that for the Brauer tree algebra $A_{3}$, the connected component of the natural heart of the space of stability conditions is simply connected. However, unlike certains examples arising in geometry, the Bridgeland homomorphism is not a covering map.
△ Less
Submitted 13 October, 2014; v1 submitted 9 June, 2014;
originally announced June 2014.
-
Splendid and perverse equivalences
Authors:
Léo Dreyfus-Schmidt
Abstract:
Inspired by the works of Rickard on splendid equivalences and of Chuang and Rouquier on perverse equivalences, we are here interested in the combination of both, a splendid perverse equivalence. This is naturally the right framework to understand the relations between global and local perverse equivalences between blocks of finite groups, as a splendid equivalence induces local derived equivalence…
▽ More
Inspired by the works of Rickard on splendid equivalences and of Chuang and Rouquier on perverse equivalences, we are here interested in the combination of both, a splendid perverse equivalence. This is naturally the right framework to understand the relations between global and local perverse equivalences between blocks of finite groups, as a splendid equivalence induces local derived equivalences via the Brauer functor. We prove that under certain conditions, we have an equivalence between a perverse equivalence between the homotopy category of p-permutation modules and local derived perverse equivalences, in the case of abelian defect group.
△ Less
Submitted 13 October, 2014; v1 submitted 9 June, 2014;
originally announced June 2014.
