-
Equity through Access: A Case for Small-scale Deep Learning
Authors:
Raghavendra Selvan,
Bob Pepin,
Christian Igel,
Gabrielle Samuel,
Erik B Dam
Abstract:
The recent advances in deep learning (DL) have been accelerated by access to large-scale data and compute. These large-scale resources have been used to train progressively larger models which are resource intensive in terms of compute, data, energy, and carbon emissions. These costs are becoming a new type of entry barrier to researchers and practitioners with limited access to resources at such…
▽ More
The recent advances in deep learning (DL) have been accelerated by access to large-scale data and compute. These large-scale resources have been used to train progressively larger models which are resource intensive in terms of compute, data, energy, and carbon emissions. These costs are becoming a new type of entry barrier to researchers and practitioners with limited access to resources at such scale, particularly in the Global South. In this work, we take a comprehensive look at the landscape of existing DL models for vision tasks and demonstrate their usefulness in settings where resources are limited. To account for the resource consumption of DL models, we introduce a novel measure to estimate the performance per resource unit, which we call the PePR score. Using a diverse family of 131 unique DL architectures (spanning 1M to 130M trainable parameters) and three medical image datasets, we capture trends about the performance-resource trade-offs. In applications like medical image analysis, we argue that small-scale, specialized models are better than striving for large-scale models. Furthermore, we show that using pretrained models can significantly reduce the computational resources and data required. We hope this work will encourage the community to focus on improving AI equity by develo** methods and models with smaller resource footprints.
△ Less
Submitted 19 March, 2024;
originally announced March 2024.
-
The Harmonic Edit Distance
Authors:
Bob Pepin
Abstract:
This short note introduces a new distance between strings, where the cost of an insertion or deletion is inversely proportional to the string length. It improves upon previous results by admitting a simple, explicit formula involving only the length of the longest common subsequence and satisfying the triangle inequality at the same time, while not requiring any parameter tuning.
This short note introduces a new distance between strings, where the cost of an insertion or deletion is inversely proportional to the string length. It improves upon previous results by admitting a simple, explicit formula involving only the length of the longest common subsequence and satisfying the triangle inequality at the same time, while not requiring any parameter tuning.
△ Less
Submitted 12 November, 2020; v1 submitted 8 November, 2020;
originally announced November 2020.
-
The documentational approach to didactics
Authors:
Luc Trouche,
Ghislaine Gueudet,
Birgit Pepin
Abstract:
This article is an updated version of an entry of the Encyclopedia of Mathematics Education (2018). In the same time, it is the seed of the HAL collection DAD-MULTILINGUAL, constituted by the translation of this entry in various languages.The documentational approach to didactics is a theory in mathematics education. Its first aim is to understand teachers' professional development by studying the…
▽ More
This article is an updated version of an entry of the Encyclopedia of Mathematics Education (2018). In the same time, it is the seed of the HAL collection DAD-MULTILINGUAL, constituted by the translation of this entry in various languages.The documentational approach to didactics is a theory in mathematics education. Its first aim is to understand teachers' professional development by studying their interactions with the resources they use and design in/for their teaching. In this text we briefly describe the emergence of the approach, its theoretical sources, its main concepts and the associated methodology. We illustrate these aspects with examples from different research projects. This synthetic presentation is written for researchers, but also for non-specialists (e.g. master students) interested in a first discovery of the documentational approach.
△ Less
Submitted 3 March, 2020;
originally announced March 2020.
-
Concentration Inequalities for Additive Functionals: a Martingale Approach
Authors:
Bob Pepin
Abstract:
This work shows how exponential concentration inequalities for additive functionals of stochastic processes over a finite time interval can be derived from concentration inequalities for martingales. The approach is entirely probabilistic and naturally includes time-inhomogeneous and non-stationary processes as well as initial laws concentrated on a single point. The class of processes studied inc…
▽ More
This work shows how exponential concentration inequalities for additive functionals of stochastic processes over a finite time interval can be derived from concentration inequalities for martingales. The approach is entirely probabilistic and naturally includes time-inhomogeneous and non-stationary processes as well as initial laws concentrated on a single point. The class of processes studied includes martingales, Markov processes and general square integrable processes. The general approach is complemented by a simple and direct method for martingales, diffusions and discrete-time Markov processes. The method is illustrated by deriving concentration inequalities for the Polyak-Ruppert algorithm, SDEs with time-dependent drift coefficients "contractive at infinity" with both Lipschitz and squared Lipschitz observables, some classical martingales and non-elliptic SDEs.
△ Less
Submitted 12 July, 2020; v1 submitted 25 October, 2018;
originally announced October 2018.
-
Time Averages of Markov Processes and Applications to Two-Timescale Problems
Authors:
Bob Pepin
Abstract:
We show a decomposition into the sum of a martingale and a deterministic quantity for time averages of the solutions to non-autonomous SDEs and for discrete-time Markov processes. In the SDE case the martingale has an explicit representation in terms of the gradient of the associated semigroup or transition operator. We show how the results can be used to obtain quenched Gaussian concentration ine…
▽ More
We show a decomposition into the sum of a martingale and a deterministic quantity for time averages of the solutions to non-autonomous SDEs and for discrete-time Markov processes. In the SDE case the martingale has an explicit representation in terms of the gradient of the associated semigroup or transition operator. We show how the results can be used to obtain quenched Gaussian concentration inequalities for time averages and to provide insights into the Averaging principle for two-timescale processes.
△ Less
Submitted 7 February, 2018; v1 submitted 20 October, 2017;
originally announced October 2017.
-
Towards a Quantitative Averaging Principle for Stochastic Differential Equations
Authors:
Bob Pepin
Abstract:
This work explores the use of a forward-backward martingale method together with a decoupling argument and entropic estimates between the conditional and averaged measures to prove a strong averaging principle for stochastic differential equations with order of convergence 1/2. We obtain explicit expressions for all the constants involved. At the price of some extra assumptions on the time margina…
▽ More
This work explores the use of a forward-backward martingale method together with a decoupling argument and entropic estimates between the conditional and averaged measures to prove a strong averaging principle for stochastic differential equations with order of convergence 1/2. We obtain explicit expressions for all the constants involved. At the price of some extra assumptions on the time marginals and an exponential bound in time, we loosen the usual boundedness and Lipschitz assumptions. We conclude with an application of our result to Temperature-Accelerated Molecular Dynamics.
△ Less
Submitted 15 September, 2017;
originally announced September 2017.
