-
Domain-Aware Augmentations for Unsupervised Online General Continual Learning
Authors:
Nicolas Michel,
Romain Negrel,
Giovanni Chierchia,
Jean-François Bercher
Abstract:
Continual Learning has been challenging, especially when dealing with unsupervised scenarios such as Unsupervised Online General Continual Learning (UOGCL), where the learning agent has no prior knowledge of class boundaries or task change information. While previous research has focused on reducing forgetting in supervised setups, recent studies have shown that self-supervised learners are more r…
▽ More
Continual Learning has been challenging, especially when dealing with unsupervised scenarios such as Unsupervised Online General Continual Learning (UOGCL), where the learning agent has no prior knowledge of class boundaries or task change information. While previous research has focused on reducing forgetting in supervised setups, recent studies have shown that self-supervised learners are more resilient to forgetting. This paper proposes a novel approach that enhances memory usage for contrastive learning in UOGCL by defining and using stream-dependent data augmentations together with some implementation tricks. Our proposed method is simple yet effective, achieves state-of-the-art results compared to other unsupervised approaches in all considered setups, and reduces the gap between supervised and unsupervised continual learning. Our domain-aware augmentation procedure can be adapted to other replay-based methods, making it a promising strategy for continual learning.
△ Less
Submitted 13 September, 2023;
originally announced September 2023.
-
New metrics for analyzing continual learners
Authors:
Nicolas Michel,
Giovanni Chierchia,
Romain Negrel,
Jean-François Bercher,
Toshihiko Yamasaki
Abstract:
Deep neural networks have shown remarkable performance when trained on independent and identically distributed data from a fixed set of classes. However, in real-world scenarios, it can be desirable to train models on a continuous stream of data where multiple classification tasks are presented sequentially. This scenario, known as Continual Learning (CL) poses challenges to standard learning algo…
▽ More
Deep neural networks have shown remarkable performance when trained on independent and identically distributed data from a fixed set of classes. However, in real-world scenarios, it can be desirable to train models on a continuous stream of data where multiple classification tasks are presented sequentially. This scenario, known as Continual Learning (CL) poses challenges to standard learning algorithms which struggle to maintain knowledge of old tasks while learning new ones. This stability-plasticity dilemma remains central to CL and multiple metrics have been proposed to adequately measure stability and plasticity separately. However, none considers the increasing difficulty of the classification task, which inherently results in performance loss for any model. In that sense, we analyze some limitations of current metrics and identify the presence of setup-induced forgetting. Therefore, we propose new metrics that account for the task's increasing difficulty. Through experiments on benchmark datasets, we demonstrate that our proposed metrics can provide new insights into the stability-plasticity trade-off achieved by models in the continual learning environment.
△ Less
Submitted 1 September, 2023;
originally announced September 2023.
-
Learning Representations on the Unit Sphere: Investigating Angular Gaussian and von Mises-Fisher Distributions for Online Continual Learning
Authors:
Nicolas Michel,
Giovanni Chierchia,
Romain Negrel,
Jean-François Bercher
Abstract:
We use the maximum a posteriori estimation principle for learning representations distributed on the unit sphere. We propose to use the angular Gaussian distribution, which corresponds to a Gaussian projected on the unit-sphere and derive the associated loss function. We also consider the von Mises-Fisher distribution, which is the conditional of a Gaussian in the unit-sphere. The learned represen…
▽ More
We use the maximum a posteriori estimation principle for learning representations distributed on the unit sphere. We propose to use the angular Gaussian distribution, which corresponds to a Gaussian projected on the unit-sphere and derive the associated loss function. We also consider the von Mises-Fisher distribution, which is the conditional of a Gaussian in the unit-sphere. The learned representations are pushed toward fixed directions, which are the prior means of the Gaussians; allowing for a learning strategy that is resilient to data drift. This makes it suitable for online continual learning, which is the problem of training neural networks on a continuous data stream, where multiple classification tasks are presented sequentially so that data from past tasks are no longer accessible, and data from the current task can be seen only once. To address this challenging scenario, we propose a memory-based representation learning technique equipped with our new loss functions. Our approach does not require negative data or knowledge of task boundaries and performs well with smaller batch sizes while being computationally efficient. We demonstrate with extensive experiments that the proposed method outperforms the current state-of-the-art methods on both standard evaluation scenarios and realistic scenarios with blurry task boundaries. For reproducibility, we use the same training pipeline for every compared method and share the code at https://github.com/Nicolas1203/ocl-fd.
△ Less
Submitted 16 February, 2024; v1 submitted 5 June, 2023;
originally announced June 2023.
-
Low Complexity Approaches for End-to-End Latency Prediction
Authors:
Pierre Larrenie,
Jean-François Bercher,
Olivier Venard,
Iyad Lahsen-Cherif
Abstract:
Software Defined Networks have opened the door to statistical and AI-based techniques to improve efficiency of networking. Especially to ensure a certain Quality of Service (QoS) for specific applications by routing packets with awareness on content nature (VoIP, video, files, etc.) and its needs (latency, bandwidth, etc.) to use efficiently resources of a network. Predicting various Key Performan…
▽ More
Software Defined Networks have opened the door to statistical and AI-based techniques to improve efficiency of networking. Especially to ensure a certain Quality of Service (QoS) for specific applications by routing packets with awareness on content nature (VoIP, video, files, etc.) and its needs (latency, bandwidth, etc.) to use efficiently resources of a network. Predicting various Key Performance Indicators (KPIs) at any level may handle such problems while preserving network bandwidth. The question addressed in this work is the design of efficient and low-cost algorithms for KPI prediction, implementable at the local level. We focus on end-to-end latency prediction, for which we illustrate our approaches and results on a public dataset from the recent international challenge on GNN [1]. We propose several low complexity, locally implementable approaches, achieving significantly lower wall time both for training and inference, with marginally worse prediction accuracy compared to state-of-the-art global GNN solutions.
△ Less
Submitted 31 January, 2023;
originally announced February 2023.
-
Low Complexity Adaptive Machine Learning Approaches for End-to-End Latency Prediction
Authors:
Pierre Larrenie,
Jean-François Bercher,
Olivier Venard,
Iyad Lahsen-Cherif
Abstract:
Software Defined Networks have opened the door to statistical and AI-based techniques to improve efficiency of networking. Especially to ensure a certain Quality of Service (QoS) for specific applications by routing packets with awareness on content nature (VoIP, video, files, etc.) and its needs (latency, bandwidth, etc.) to use efficiently resources of a network. Monitoring and predicting variou…
▽ More
Software Defined Networks have opened the door to statistical and AI-based techniques to improve efficiency of networking. Especially to ensure a certain Quality of Service (QoS) for specific applications by routing packets with awareness on content nature (VoIP, video, files, etc.) and its needs (latency, bandwidth, etc.) to use efficiently resources of a network. Monitoring and predicting various Key Performance Indicators (KPIs) at any level may handle such problems while preserving network bandwidth. The question addressed in this work is the design of efficient, low-cost adaptive algorithms for KPI estimation, monitoring and prediction. We focus on end-to-end latency prediction, for which we illustrate our approaches and results on data obtained from a public generator provided after the recent international challenge on GNN [12]. In this paper, we improve our previously proposed low-cost estimators [6] by adding the adaptive dimension, and show that the performances are minimally modified while gaining the ability to track varying networks.
△ Less
Submitted 31 January, 2023;
originally announced January 2023.
-
Contrastive Learning for Online Semi-Supervised General Continual Learning
Authors:
Nicolas Michel,
Romain Negrel,
Giovanni Chierchia,
Jean-François Bercher
Abstract:
We study Online Continual Learning with missing labels and propose SemiCon, a new contrastive loss designed for partly labeled data. We demonstrate its efficiency by devising a memory-based method trained on an unlabeled data stream, where every data added to memory is labeled using an oracle. Our approach outperforms existing semi-supervised methods when few labels are available, and obtain simil…
▽ More
We study Online Continual Learning with missing labels and propose SemiCon, a new contrastive loss designed for partly labeled data. We demonstrate its efficiency by devising a memory-based method trained on an unlabeled data stream, where every data added to memory is labeled using an oracle. Our approach outperforms existing semi-supervised methods when few labels are available, and obtain similar results to state-of-the-art supervised methods while using only 2.6% of labels on Split-CIFAR10 and 10% of labels on Split-CIFAR100.
△ Less
Submitted 22 November, 2022; v1 submitted 12 July, 2022;
originally announced July 2022.
-
On some interrelations of generalized $q$-entropies and a generalized Fisher information, including a Cramér-Rao inequality
Authors:
Jean-François Bercher
Abstract:
In this communication, we describe some interrelations between generalized $q$-entropies and a generalized version of Fisher information. In information theory, the de Bruijn identity links the Fisher information and the derivative of the entropy. We show that this identity can be extended to generalized versions of entropy and Fisher information. More precisely, a generalized Fisher information n…
▽ More
In this communication, we describe some interrelations between generalized $q$-entropies and a generalized version of Fisher information. In information theory, the de Bruijn identity links the Fisher information and the derivative of the entropy. We show that this identity can be extended to generalized versions of entropy and Fisher information. More precisely, a generalized Fisher information naturally pops up in the expression of the derivative of the Tsallis entropy. This generalized Fisher information also appears as a special case of a generalized Fisher information for estimation problems. Indeed, we derive here a new Cramér-Rao inequality for the estimation of a parameter, which involves a generalized form of Fisher information. This generalized Fisher information reduces to the standard Fisher information as a particular case. In the case of a translation parameter, the general Cramér-Rao inequality leads to an inequality for distributions which is saturated by generalized $q$-Gaussian distributions. These generalized $q$-Gaussians are important in several areas of physics and mathematics. They are known to maximize the $q$-entropies subject to a moment constraint. The Cramér-Rao inequality shows that the generalized $q$-Gaussians also minimize the generalized Fisher information among distributions with a fixed moment. Similarly, the generalized $q$-Gaussians also minimize the generalized Fisher information among distributions with a given $q$-entropy.
△ Less
Submitted 27 May, 2013;
originally announced May 2013.
-
Some results on a $χ$-divergence, an~extended~Fisher information and~generalized~Cramér-Rao inequalities
Authors:
Jean-François Bercher
Abstract:
We propose a modified $χ^β$-divergence, give some of its properties, and show that this leads to the definition of a generalized Fisher information. We give generalized Cramér-Rao inequalities, involving this Fisher information, an extension of the Fisher information matrix, and arbitrary norms and power of the estimation error. In the case of a location parameter, we obtain new characterizations…
▽ More
We propose a modified $χ^β$-divergence, give some of its properties, and show that this leads to the definition of a generalized Fisher information. We give generalized Cramér-Rao inequalities, involving this Fisher information, an extension of the Fisher information matrix, and arbitrary norms and power of the estimation error. In the case of a location parameter, we obtain new characterizations of the generalized $q$-Gaussians, for instance as the distribution with a given moment that minimizes the generalized Fisher information. Finally we indicate how the generalized Fisher information can lead to new uncertainty relations.
△ Less
Submitted 27 May, 2013;
originally announced May 2013.
-
Some properties of generalized Fisher information in the context of nonextensive thermostatistics
Authors:
Jean-François Bercher
Abstract:
We present two extended forms of Fisher information that fit well in the context of nonextensive thermostatistics. We show that there exists an interplay between these generalized Fisher information, the generalized $q$-Gaussian distributions and the $q$-entropies. The minimum of the generalized Fisher information among distributions with a fixed moment, or with a fixed $q$-entropy is attained, in…
▽ More
We present two extended forms of Fisher information that fit well in the context of nonextensive thermostatistics. We show that there exists an interplay between these generalized Fisher information, the generalized $q$-Gaussian distributions and the $q$-entropies. The minimum of the generalized Fisher information among distributions with a fixed moment, or with a fixed $q$-entropy is attained, in both cases, by a generalized $q$-Gaussian distribution. This complements the fact that the $q$-Gaussians maximize the $q$-entropies subject to a moment constraint, and yields new variational characterizations of the generalized $q$-Gaussians. We show that the generalized Fisher information naturally pop up in the expression of the time derivative of the $q$-entropies, for distributions satisfying a certain nonlinear heat equation. This result includes as a particular case the classical de Bruijn identity. Then we study further properties of the generalized Fisher information and of their minimization. We show that, though non additive, the generalized Fisher information of a combined system is upper bounded. In the case of mixing, we show that the generalized Fisher information is convex for $q\geq1.$ Finally, we show that the minimization of the generalized Fisher information subject to moment constraints satisfies a Legendre structure analog to the Legendre structure of thermodynamics.
△ Less
Submitted 4 June, 2013; v1 submitted 22 May, 2013;
originally announced May 2013.
-
On multidimensional generalized Cramér-Rao inequalities, uncertainty relations and characterizations of generalized $q$-Gaussian distributions
Authors:
J. -F. Bercher
Abstract:
In the present work, we show how the generalized Cramér-Rao inequality for the estimation of a parameter, presented in a recent paper, can be extended to the mutidimensional case with general norms on $\mathbb{R}^{n}$, and to a wider context. As a particular case, we obtain a new multidimensional Cramér-Rao inequality which is saturated by generalized $q$-Gaussian distributions. We also give anoth…
▽ More
In the present work, we show how the generalized Cramér-Rao inequality for the estimation of a parameter, presented in a recent paper, can be extended to the mutidimensional case with general norms on $\mathbb{R}^{n}$, and to a wider context. As a particular case, we obtain a new multidimensional Cramér-Rao inequality which is saturated by generalized $q$-Gaussian distributions. We also give another related Cramér-Rao inequality, for a general norm, which is saturated as well by these distributions. Finally, we derive uncertainty relations from these Cramér-Rao inequalities. These uncertainty relations involve moments computed with respect to escort distributions, and we show that some of these relations are saturated by generalized $q$-Gaussian distributions. These results introduce extended versions of Fisher information, new Cramér-Rao inequalities, and new characterizations of generalized $q$-Gaussian distributions which are important in several areas of physics and mathematics.
△ Less
Submitted 24 February, 2013; v1 submitted 8 November, 2012;
originally announced November 2012.
-
On generalized Cramér-Rao inequalities, generalized Fisher informations and characterizations of generalized q-Gaussian distributions
Authors:
J. -F. Bercher
Abstract:
This paper deals with Cramér-Rao inequalities in the context of nonextensive statistics and in estimation theory. It gives characterizations of generalized q-Gaussian distributions, and introduces generalized versions of Fisher information. The contributions of this paper are (i) the derivation of new extended Cramér-Rao inequalities for the estimation of a parameter, involving general q-moments o…
▽ More
This paper deals with Cramér-Rao inequalities in the context of nonextensive statistics and in estimation theory. It gives characterizations of generalized q-Gaussian distributions, and introduces generalized versions of Fisher information. The contributions of this paper are (i) the derivation of new extended Cramér-Rao inequalities for the estimation of a parameter, involving general q-moments of the estimation error, (ii) the derivation of Cramér-Rao inequalities saturated by generalized q-Gaussian distributions, (iii) the definition of generalized Fisher informations, (iv) the identification and interpretation of some prior results, and finally, (v) the suggestion of new estimation methods.
△ Less
Submitted 4 June, 2012;
originally announced June 2012.
-
A simple probabilistic construction yielding generalized entropies and divergences, escort distributions and q-Gaussians
Authors:
J. -F. Bercher
Abstract:
We give a simple probabilistic description of a transition between two states which leads to a generalized escort distribution. When the parameter of the distribution varies, it defines a parametric curve that we call an escort-path. The Rényi divergence appears as a natural by-product of the setting. We study the dynamics of the Fisher information on this path, and show in particular that the the…
▽ More
We give a simple probabilistic description of a transition between two states which leads to a generalized escort distribution. When the parameter of the distribution varies, it defines a parametric curve that we call an escort-path. The Rényi divergence appears as a natural by-product of the setting. We study the dynamics of the Fisher information on this path, and show in particular that the thermodynamic divergence is proportional to Jeffreys' divergence. Next, we consider the problem of inferring a distribution on the escort-path, subject to generalized moments constraints. We show that our setting naturally induces a rationale for the minimization of the Rényi information divergence. Then, we derive the optimum distribution as a generalized q-Gaussian distribution.
△ Less
Submitted 4 June, 2012;
originally announced June 2012.
-
On a (β,q)-generalized Fisher information and inequalities involving q-Gaussian distributions
Authors:
J. -F. Bercher
Abstract:
In the present paper, we would like to draw attention to a possible generalized Fisher information that fits well in the formalism of nonextensive thermostatistics. This generalized Fisher information is defined for densities on $\mathbb{R}^{n}.$ Just as the maximum Rényi or Tsallis entropy subject to an elliptic moment constraint is a generalized q-Gaussian, we show that the minimization of the g…
▽ More
In the present paper, we would like to draw attention to a possible generalized Fisher information that fits well in the formalism of nonextensive thermostatistics. This generalized Fisher information is defined for densities on $\mathbb{R}^{n}.$ Just as the maximum Rényi or Tsallis entropy subject to an elliptic moment constraint is a generalized q-Gaussian, we show that the minimization of the generalized Fisher information also leads a generalized q-Gaussian. This yields a generalized Cramér-Rao inequality. In addition, we show that the generalized Fisher information naturally pops up in a simple inequality that links the generalized entropies, the generalized Fisher information and an elliptic moment. Finally, we give an extended Stam inequality. In this series of results, the extremal functions are the generalized q-Gaussians. Thus, these results complement the classical characterization of the generalized q-Gaussian and introduce a generalized Fisher information as a new information measure in nonextensive thermostatistics.
△ Less
Submitted 17 January, 2013; v1 submitted 7 March, 2012;
originally announced March 2012.
-
Source coding with escort distributions and Renyi entropy bounds
Authors:
J. -F. Bercher
Abstract:
We discuss the interest of escort distributions and Rényi entropy in the context of source coding. We first recall a source coding theorem by Campbell relating a generalized measure of length to the Rényi-Tsallis entropy. We show that the associated optimal codes can be obtained using considerations on escort-distributions. We propose a new family of measure of length involving escort-distribution…
▽ More
We discuss the interest of escort distributions and Rényi entropy in the context of source coding. We first recall a source coding theorem by Campbell relating a generalized measure of length to the Rényi-Tsallis entropy. We show that the associated optimal codes can be obtained using considerations on escort-distributions. We propose a new family of measure of length involving escort-distributions and we show that these generalized lengths are also bounded below by the Rényi entropy. Furthermore, we obtain that the standard Shannon codes lengths are optimum for the new generalized lengths measures, whatever the entropic index. Finally, we show that there exists in this setting an interplay between standard and escort distributions.
△ Less
Submitted 15 September, 2011;
originally announced September 2011.
-
Escort entropies and divergences and related canonical distribution
Authors:
J. -F. Bercher
Abstract:
We discuss two families of two-parameter entropies and divergences, derived from the standard Rényi and Tsallis entropies and divergences. These divergences and entropies are found as divergences or entropies of escort distributions. Exploiting the nonnegativity of the divergences, we derive the expression of the canonical distribution associated to the new entropies and a observable given as an e…
▽ More
We discuss two families of two-parameter entropies and divergences, derived from the standard Rényi and Tsallis entropies and divergences. These divergences and entropies are found as divergences or entropies of escort distributions. Exploiting the nonnegativity of the divergences, we derive the expression of the canonical distribution associated to the new entropies and a observable given as an escort-mean value. We show that this canonical distribution extends, and smoothly connects, the results obtained in nonextensive thermodynamics for the standard and generalized mean value constraints.
△ Less
Submitted 15 September, 2011;
originally announced September 2011.
-
On some entropy functionals derived from Rényi information divergence
Authors:
Jean-François Bercher
Abstract:
We consider the maximum entropy problems associated with Rényi $Q$-entropy, subject to two kinds of constraints on expected values. The constraints considered are a constraint on the standard expectation, and a constraint on the generalized expectation as encountered in nonextensive statistics. The optimum maximum entropy probability distributions, which can exhibit a power-law behaviour, are de…
▽ More
We consider the maximum entropy problems associated with Rényi $Q$-entropy, subject to two kinds of constraints on expected values. The constraints considered are a constraint on the standard expectation, and a constraint on the generalized expectation as encountered in nonextensive statistics. The optimum maximum entropy probability distributions, which can exhibit a power-law behaviour, are derived and characterized. The Rényi entropy of the optimum distributions can be viewed as a function of the constraint. This defines two families of entropy functionals in the space of possible expected values. General properties of these functionals, including nonnegativity, minimum, convexity, are documented. Their relationships as well as numerical aspects are also discussed. Finally, we work out some specific cases for the reference measure $Q(x)$ and recover in a limit case some well-known entropies.
△ Less
Submitted 1 May, 2008;
originally announced May 2008.
-
An entropic view of Pickands' theorem
Authors:
J. -F. Bercher,
C. Vignat
Abstract:
It is shown that distributions arising in Renyi-Tsallis maximum entropy setting are related to the Generalized Pareto Distributions (GPD) that are widely used for modeling the tails of distributions. The relevance of such modelization, as well as the ubiquity of GPD in practical situations follows from Balkema-De Haan-Pickands theorem on the distribution of excesses (over a high threshold). We p…
▽ More
It is shown that distributions arising in Renyi-Tsallis maximum entropy setting are related to the Generalized Pareto Distributions (GPD) that are widely used for modeling the tails of distributions. The relevance of such modelization, as well as the ubiquity of GPD in practical situations follows from Balkema-De Haan-Pickands theorem on the distribution of excesses (over a high threshold). We provide an entropic view of this result, by showing that the distribution of a suitably normalized excess variable converges to the solution of a maximum Tsallis entropy, which is the GPD. This highlights the relevance of the so-called Tsallis distributions in many applications as well as some relevance to the use of the corresponding entropy.
△ Less
Submitted 6 May, 2008; v1 submitted 21 February, 2008;
originally announced February 2008.
