On Sequential Loss Approximation for Continual Learning

Continual learning, also known as incremental learning or lifelong learning, is learning from a sequence of datasets called tasks which are not necessarily identically distributed. When a neural network (including a generalized linear model, which is essentially a neural network with no hidden layers) is trained on a task and fine-tuned on a new task, it loses predictive performance on the old task. This phenomenon is known as catastrophic forgetting \parencitemccloskey_catastrophic_1989 and can be prevented by joint training on all tasks, but previous data may not be accessible due to computational or privacy constraints. Thus, the goal of continual learning is to prevent catastrophic forgetting without accessing previous data.

Continual-learning settings vary considerably, and three main types are commonly studied \parencitevan_de_ven_three_2022:

1. 1.

   Task-incremental learning, in which task IDs are provided and the classes change between tasks

2. 2.

   Domain-incremental learning, in which task IDs are not provided and the classes remain the same between tasks

3. 3.

   Class-incremental learning, in which task IDs are not provided and the classes change between tasks

Multi-headed models, in which there is one output head for each task, are commonly used for task-incremental learning. However, task-incremental learning has been criticized as task IDs make the problem of continual learning easier \parencitefarquhar_towards_2019. In fact, if there is only one class per task, then the task ID could be used to make a perfect prediction. In accordance with the desiderata proposed in [farquhar_towards_2019], we focus on class-incremental learning with single-headed models on more than two similar tasks with no access to previous data.

Sequential Bayesian inference provides an elegant approach to continual learning. We assume that the parameters of the model are random and use the previous posterior Probability Density Function (PDF) as the current prior PDF. For Maximum-A-Posteriori (MAP) prediction, this can be formulated as sequential loss approximation. In this work, we use Autodiff Quadratic Consolidation (AQC), which approximates the previous loss function with a quadratic function, and Neural Consolidation (NC), which approximates the previous loss function with a neural network. Since these methods are not scalable to large models with millions of parameters, we use a fixed pre-trained feature extractor for image datasets. We show empirically that for small datasets, neural-network approximation of the previous loss function leads to significant improvements in predictive performance over quadratic approximation of the previous loss function and that using tanh-output features improves the predictive performance of AQC.

Let $\theta$ be the parameters of the neural network, $x_{1:t}$ be the input data from time $1$ to $t$ and $y_{1:t}$ be the output data from time $1$ to $t$. The independence assumptions are described by the following Bayesian network. In particular, $x_{1:t}$ are assumed to be independent, and given $\theta$ and $x_{1:t}$, $y_{1:t}$ are assumed to be conditionally independent.

For $i=1,2,\ldots,t$, $(x_{i},y_{i})$ represents a task or dataset at time $i$, and the individual points in each task are assumed to be independent and identically distributed.

Tasks are assumed to be similar, so the likelihood PDF at time $t$, $p(y_{t}|\theta,x_{t})$, is assumed to be the same for all tasks. It is defined by the neural network, which maps $x_{t}$ to some parameters of $p(y_{t}|\theta,x_{t})$. In binary classification, $p(y_{t}|\theta,x_{t})$ is Bernoulli, and the neural network maps to the log-probability of the positive class, while in multi-class classification, $p(y_{t}|\theta,x_{t})$ is categorical, and the neural network maps to the un-normalized log-probabilities of the classes. The posterior PDF at time $t$, $p(\theta|x_{1:t},y_{1:t})$, can be obtained by recursive application of Bayes’ rule, where the posterior PDF at time $t-1$ is used as the prior PDF at time $t$:

where $z_{t}=\int_{-\infty}^{\infty}p(\theta|y_{1:t-1},x_{1:t-1})p(y_{t}|\theta,x_{t})d\theta$ is a normalization term that does not depend on $\theta$.

MAP prediction uses the maximum $\theta^{*}_{t}$ of the posterior PDF at time $t$ to make a point prediction. Since normalizing constants do not affect the maximum, $\theta^{*}_{t}$ is also equal to the maximum of the joint PDF $p(\theta,y_{1:t}|x_{1:t})$. Maximizing the PDF is equivalent to minimizing the negative log PDF. Thus, the total loss at time $t$ could be defined as $\mathfrak{L}_{t}(\theta)=-\ln p(\theta,y_{1:t}|x_{1:t})$ and the task loss at time $t$ as $\mathfrak{l}_{t}(\theta)=-\ln p(y_{t}|\theta,x_{t})$. This leads to a recursion of loss functions for $t=1,2,\ldots$:

Methods based on this recursion are also known as regularization-based methods as $\mathfrak{L}_{t-1}$ can be seen as regularizing $\mathfrak{l}_{t}$. Note that $\mathfrak{L}_{0}=\mathfrak{l}_{0}$, the negative log prior, may be in un-normalized form. In particular, a Gaussian prior PDF with zero mean vector and covariance matrix $\alpha I$ corresponds to an $L^{2}$ regularization term $\frac{1}{2}\alpha\lVert\theta\rVert_{2}^{2}$. In binary classification, $\mathfrak{l}_{t}$ is the sigmoid cross-entropy loss, while in multi-class classification, $\mathfrak{l}_{t}$ is the softmax cross-entropy loss.

There are several works using quadratic approximation of the previous loss function. Elastic Weight Consolidation (EWC) approximates the Hessian matrix by using a diagonal Fisher information matrix, which is computed by averaging the squared gradients over mini-batches \parencitekirkpatrick_overcoming_2017. However, it adds a penalty term for every new task, and [huszar_note_2018] points out that it is not well justified and that the penalty term can be updated by adding the Hessian matrices. Synpatic Intelligence (SI) also performs a diagonal approximation of the Hessian. However, it does so by using the change in loss during gradient descent, which is computed by summing the product of the gradient and the change in the parameter values during gradient descent \parencitezenke_continual_2017. Online Structured Laplace Approximation (OSLA) uses Kronecker factorization to perform a block-diagonal approximation of the Hessian, in which the diagonal blocks of the matrix correspond to a layer of the neural network \parenciteritter_online_2018.

The above methods are designed to be scalable to large models and have mainly been shown to perform well in Permuted MNIST \parencitegoodfellow_empirical_2015, with OSLA performing better than EWC and SI. In class-incremental Split MNIST, EWC and SI have been shown to perform as poorly as fine-tuning \parencitevan_de_ven_three_2022. It has also been observed that regularization-based methods generally perform worse than replay-based methods, which use stored or generated previous data \parencitemai_online_2022,van_de_ven_three_2022. In our work, we examine how much predictive performance we can achieve by using a full Hessian computation and whether the quadratic approximation can be improved by approximation with a neural network, which is a powerful function approximator.

Recently, two types of pre-training have been studied for continual learning: pre-training for initialization and pre-training for feature extraction. In the former, the parameters of the neural network are initialized with the pre-trained parameters rather than randomly, which has been reported to reduce forgetting \parencitelee_pre-trained_2023,mehta_empirical_2023. In the latter, a pre-trained model is used as a feature extractor, which has also been reported to reduce forgetting \parencitehu_continual_2021,li_continual_2022,yang_continual_2023. Intuitively, neural networks do not need to learn lower-level features continually as much as they need to learn higher-level features, just as humans do not need to learn continually to recognize edges, corners and shapes as much as they need to learn to recognize objects. It has also been empirically shown that forgetting is more pronounced in later layers of a neural network \parenciteliu_generative_2020. Thus, in our work, we use pre-training for feature extraction for image datasets.

In our experiments, we compare EWC, SI, AQC and NC in class-incremental-learning settings based on small and large datasets. For EWC, we use the variant with Huszár’s corrected penalty \parencitehuszar_note_2018 and the Fisher information matrix, which is computed based on the empirical distribution of $x$ and the conditional distribution of $y$ given $x$ defined by the model (rather than the empirical Fisher information matrix, which is computed based on the empirical distribution of $x$ and $y$) \parencitemartens_new_2020.

The experiments are run on an on-premise NVIDIA GeForce RTX 4090 GPU with 24 GB of memory. In every experiment, swish activation functions are used except in some pre-trained neural networks with tanh activation functions before the final layer, the regularization factor is $\alpha=\beta=0.1$ and optimization is done by using the Adam optimizer with learning rate 0.01. For Split Iris and Split Wine, the datasets are split randomly with 80% for training and 20% for testing, while for Split MNIST and Split CIFAR-10, the original dataset splits are used. Then, the each split dataset is split by class into multiple tasks. For NC, hyperparameters are chosen by manual tuning, while for EWC and SI, they are fixed at $\lambda=1$ and $\xi=1$, respectively. Evaluation is done by using the average accuracy, which is the average of testing accuracy scores of the tasks up to and including the current task.

We proposed Autodiff Quadratic Consolidation (AQC) and Neural Consolidation (NC) for continual learning based on sequential loss approximation. Although they are not scalable to very large neural networks, they can be used together with fixed pre-trained feature extractors. In class-incremental learning settings with small datasets, methods based on quadratic approximation of the loss function, such as EWC, SI and AQC, perform as poorly as fine-tuning, and NC performs better. With large datasets, when used with pre-training, AQC performs very well and its performance can be improved by using tanh-output features, probably because the output of the tanh function is between -1 and 1, leading to a loss function that can be better approximated with a quadratic function.

NC is conceptually appealing as neural networks are much more powerful function approximators than quadratic functions. However, in practice, it is difficult to tune the hyperparameters. One possible reason that it does not work well with large datasets is that larger datasets lead to steeper loss functions, which may affect gradient descent. Moreover, although we have access to the loss function to be approximated, we need to take a sample of points from it to train the consolidator neural network. It would be better if we could fit directly to the loss function without generating a sample of points.

We hope that our work provides some insights on sequential loss approximation for continual learning.