Efficient Expansion and Gradient Based Task Inference for Replay Free Incremental Learning

In incremental learning, tasks $\mathcal{T}_{1},\mathcal{T}_{2},\dots\mathcal{T}_{T}$ arrive sequentially. Each task $\mathcal{T}_{i}=\{(\mathbf{x}_{k},y_{k})\}_{k=1}^{N_{i}}$ where $\mathbf{x}_{k}$ has label $y_{k}\in\mathcal{Y}_{i}$ and $\mathcal{Y}_{i}$ is the set of all classes in task $\mathcal{T}_{i}$ ($|\mathcal{Y}_{i}|=K_{i}$). During training on the $i^{th}$ task, only $\mathcal{T}_{i}$ is available to train the model and $\mathcal{Y}_{i}\cap\mathcal{Y}_{j}=\phi$. Our proposed model can be applied to TIL as well as the CIL setting. During training and inference of the task $\mathcal{T}_{i}$, the task id $i$ is known in the TIL setting. However, in the CIL setting, task id $i$ is unknown during inference.

In this work, we propose an efficient, dynamically expandable model which grows with the number of tasks and accumulates all the previously learned knowledge. Let $\mathcal{M}_{\mathbf{\theta}_{i}}$ be the model for task $\mathcal{T}_{i}$ where $\mathbf{\theta}_{i}$ is the model parameter. The model parameter $\mathbf{\theta}_{i}=\{\Omega,\tau_{i},\tau_{2:i-1}\}$, i.e., it contains three types of parameters - global parameter ($\Omega$), task-specific parameter ($\tau_{i}$) and all the learned task-specific parameters before the current task ($\tau_{2:i-1}$). Therefore, the parameter $\mathbf{\theta}$ is growing with each novel task sequence and $|\mathbf{\theta}_{1}|<|\mathbf{\theta}_{2}|<\dots<|\mathbf{\theta}_{T}|$ where $|.|$ represents cardinality.

Now we will describe our expansion method for a particular task $\mathcal{T}_{i}$. For task $\mathcal{T}_{i-1}$, we can represent the $j^{th}$ convolutional layer for task model $\mathcal{M}_{\mathbf{\theta}_{i-1}}$ as $l_{i-1}^{j}\in\mathbb{R}^{c_{j}\times k\times k\times d_{j}}$ where $c_{j}$ is the number of filters, $k$ is the kernel size and $d_{j}$ is the number of feature maps of layer $(j-1)$. After training on the $(i-1)^{th}$ task, we freeze it’s parameter $\mathbf{\theta}_{i-1}$. Let us assume that $g_{j-1}$, $g_{j}$ and $g_{j+1}$ are the growth rates for the layers $(j-1)$, $j$ and $(j+1)$ respectively for the next task $\mathcal{T}_{i}$. Before training the model for task $\mathcal{T}_{i}$, we will grow the model by the defined growth rates and this growth is local to the $i^{th}$ task (and hence the name task-specific parameter). The $i^{th}$ task model adds $g_{j-1}$, $g_{j}$ and $g_{j+1}$ number of filters at the $(j-1)$, $j$ and $(j+1)$ layers respectively. Therefore, for the $i^{th}$ task at the $j^{th}$ layer, we have $c_{j}+g_{j}$ number of filters, but layer $(j-1)$ will also produce $c_{j-1}+g_{j-1}$ feature maps. To accommodate these feature maps, we have to increase the number of channels in each filter of layer $j$. So $l_{i}^{j}\in\mathbb{R}^{(c_{j}+g_{j})\times k\times k\times d\textquoteright_{j}}$ where $d\textquoteright_{j}=c_{j-1}+g_{j-1}$ is the number of feature maps after adding the task-specific filters to layer $(j-1)$. Similarly, the next layer $l_{i}^{j+1}\in\mathbb{R}^{(c_{j+1}+g_{j+1})\times k\times k\times d\textquoteright_{j+1}}$, where $d\textquoteright_{j+1}=c_{j}+g_{j}$. Figure 1 shows our expansion strategy at the particular layer $j$ during training of task $\mathcal{T}_{i}$. For each task, we train batch norm and final linear layer from scratch. So the global parameter ($\Omega$) is the parameter of the first task model without batch norm and final linear layer.

The proposed expansion-based model shows promising results in the TIL setting.

However, in the CIL setting, we don’t know the task id during inference; hence we have to predict it. In this work, we propose a robust method for task prediction that can predict the task id for a large number of task sequences.

Let $\mathcal{M}_{\mathbf{\theta}_{1}},\mathcal{M}_{\mathbf{\theta}_{2}}\dots\mathcal{M}_{\mathbf{\theta}_{T}}$ be the learned models for the tasks $\mathcal{T}_{1},\mathcal{T}_{2},\dots\mathcal{T}_{T}$ with parameters $\mathbf{\theta}_{1},\mathbf{\theta}_{2},\dots\mathbf{\theta}_{T}$. Once we have finished training for the task $\mathcal{T}_{i}$, we can predict the sample’s task id of any classes till the $i^{th}$ task. The gradient embedding of test sample $\mathbf{x}_{k}$, for the model with parameter $\theta_{i}$ over the loss function $\mathcal{L}(\mathbf{x}_{k},\mathbf{\theta}_{i})$, can be computed as:

$$\mathbf{\theta}^{\prime}_{ik}=\nabla_{\mathbf{\theta}_{i}}\mathcal{L}(\mathbf{x}_{k},\mathbf{\theta}_{i})$$

(1)

To predict the task id of sample $\mathbf{x}_{k}$, we measure the gradient norm using each model as follows:

$$\hat{i}=\operatorname*{arg\,min}_{i\in T}\frac{\|\mathbf{\theta}_{ik}^{{}^{\prime}}\|}{|\mathbf{\theta}_{ik}^{{}^{\prime}}|}$$

(2)

Here $|\mathbf{\theta}_{ik}^{\textquoteright}|$ is the number of parameters in the gradient vector. We need to normalize the embedding, as each task has a different gradient length and the norm of a longer embedding can be higher. Once we have task id $\hat{i}$ for the sample $\mathbf{x}_{k}$, we can choose the model $\mathcal{M}_{\mathbf{\theta}_{\hat{i}}}$ to evaluate the sample.

Minimizing the norm of loss gradient leads to flatter minima in weight space [59]. A flat minima is a large connected region in weight space where the loss remains approximately constant and it is optimal minima for generalization [10, 59]. In Eq. 2, we test the shape of the loss function $\mathcal{L}$ using $\mathbf{x}_{k}$ to estimate which model might perform better on that sample. We use $\ell_{1}$-norm as it gives marginally better results than $\ell_{2}$-norm. For a fair comparison with earlier baselines, we do not use the gradient norm loss during training time.

For a convolutional network with $L$ layers, we define the (flattened) gradient vector for layer $j$ as $\mathbf{\theta}_{ik}^{\textquoteright j}$ ($j=1\dots L$). An advantage of considering layer-wise gradient is that it allows us to use task and layer dependant gradient modification functions. Such functions can help us give more attention to certain parameters in a layer. Thus we can re-write the gradient embedding for task $\mathcal{T}_{i}$ and input $\mathbf{x}_{k}$ as the vector:

$$\mathbf{\theta}_{ik}^{\textquoteright}=\mathcal{CONCAT}(\mathcal{G}_{i}^{1}(\mathbf{\theta}_{ik}^{\textquoteright 1}),\mathcal{G}_{i}^{2}(\mathbf{\theta}_{ik}^{\textquoteright 2}),..,\mathcal{G}_{i}^{L}(\mathbf{\theta}_{ik}^{\textquoteright L}))$$

(3)

where $\mathcal{G}_{i}^{j}$ is the gradient modification function for task $i$ and layer $j$ and $\mathcal{CONCAT}$ concatenates multiple gradient vectors to create the final embedding vector. As our inference time grows linearly with the number of tasks, we can borrow the superposition idea of [47] to get sub-linear run times. However, in this paper, we do not focus on this direction. We present an overview of our entire method in Figure 2. Next we choose $\mathcal{L}(\mathbf{x}_{k},\mathbf{\theta}_{i})$ and $\mathcal{G}_{i}^{j}$.

Most expansion-based models [40, 44] consider the growth rate as a hyperparameter and irrespective of the task complexity, it remains constant across all tasks. Here, we propose an adaptive growth rate for the expansion parameter that depends on the task complexity. The calculation of task complexity, without availability of the previous task samples, is the key challenge. For the $i^{th}$ task, we leverage the current task samples $\mathcal{T}_{i}$, the $(i-1)^{th}$ task model and the mean gradient on the previous task samples $\mathcal{T}_{i-1}$ to construct a compatibility score. Subsequently, we use this score to decide the growth rate for the $i^{th}$ task.

Let’s assume that before training for task $\mathcal{T}_{i}$, we expand each layer $j$ of the $(i-1)^{th}$ task model by $g_{j}$ filters (i.e., growth rate is $g_{j}$). Using scaling factor $\alpha$ $\in$ [0, 1], we can write $g_{j}$ as:

$$g_{j}=\left[\alpha\times g_{j}^{min}+(1-\alpha)\times g_{j}^{max}\right]$$

(6)

where $g_{j}^{max}$ and $g_{j}^{min}$ are respectively the pre-defined maximum and minimum growth rates for layer $j$ and $\left[.\right]$ is the rounding function.

In case of static growth, the growth rate for a layer is constant for all tasks and $\alpha=0$. However, in adaptive growth, we expand our model based on the relative complexity of the novel task $\mathcal{T}_{i}$ and use $\alpha$ to encode how similar $\mathcal{T}_{i}$ is with $\mathcal{T}_{i-1}$. Since our models are optimized using gradient-based methods, we can measure the uncertainty of a sample using gradients [2]. Given the $(i-1)^{th}$ task model, we can compare the model confidences in two samples $\mathbf{x}_{1}$ and $\mathbf{x}_{2}$ using $\mathbf{\theta}_{(i-1)1}^{{}^{\prime}}$ and $\mathbf{\theta}_{(i-1)2}^{{}^{\prime}}$ from Eq. 1. For tasks $\mathcal{T}_{i-1}$ and $\mathcal{T}_{i}$, we can write the scaling factor $\alpha$ as:

$$\small\alpha=|\mathcal{MEAN}({\{\mathbf{\theta}_{(i-1)k}^{{}^{\prime}}\}}_{k\in\mathcal{T}_{i}})\cdot\mathcal{MEAN}({\{\mathbf{\theta}_{(i-1)k}^{{}^{\prime}}\}}_{k\in\mathcal{T}_{i-1}})|$$

(7)

where $\mathcal{MEAN}$ gives us the normalized mean vector from a set of input vectors. We can use this $\alpha$ to decide the growth rate for the $i^{th}$ task using Eq. 6. So for each task, we only need to remember the mean gradient of the last task. We can reduce the gradient cardinality to around $0.02\%$ of it’s original size using mean filters strategy. Based on the type of growth, we get two model variants - static parameter growth (SPG) model and adaptive parameter growth (APG) model. For a fair comparison between SPG and APG models, we set $g_{j}^{min}$ as 1 and $g_{j}^{max}$ as the growth rate used in SPG. The parameter growth rate is ablated in Sec. 5.2.

Our experiments are performed on three datasets - CIFAR-100 [16], ImageNet-100 [17] and TinyImageNet [19]. We divide the CIFAR-100 dataset into 5, 10, and 20 task sequences (let’s call them CIFAR100/5, CIFAR100/10 and CIFAR100/20 respectively). CIFAR100/5 has a smaller task sequence, but each task contains 20 classes. On the other hand, CIFAR100/20 contains only 5 classes per task but has a large number of tasks. We evaluate our method on this dataset using ResNet-18 [9] architecture for CIFAR datasets. For the ImageNet-100 dataset, we use the same class subset and class order as DER [50]. Similar to CIFAR-100, we divide the ImageNet-100 dataset into 5, 10, and 20 tasks. The ImageNet-100 dataset is challenging as it contains relatively bigger images, i.e., the image size is $224\times 224$. We use the standard ResNet-18 architecture for experimenting on this dataset. Tiny ImageNet is a smaller subset of the ImageNet [17] dataset, contains 200 classes with a $64\times 64$ resolution. To demonstrate the efficacy of our proposed model across different architectures, we use the VGG-16 [39] architecture with batch norm for the Tiny ImageNet dataset. We divide the dataset into 10 tasks and evaluate the model performance in the TIL setting. Furthermore, we explore our continual learning approach using the StackGAN-v2 [56] architecture for three diverse tasks - cats (ImageNet [5]), birds (CUB-200 [46]) and churches (LSUN [53]).

Our proposed method leverages network expansion to learn new tasks and does not use any pretrained model or sample storage for replay. Therefore, comparison with replay-based methods is not fair. So we compare our model with both regularization and expansion-based approaches. We use various regularization based methods like LwF [22], EWC [15], SI [55], MAS [1], SDC [54], DMC [57], L2T [51], IL2A [61], LwF+adBiC [41], SSRE [62] and FeTrIL [30] as our baselines. We also use expansion-based approaches like PNN [35], HAT [36], PB [24], PackNet [25], TFM [26], APD [51] and EFT [44] for comparison. For the continual GAN experiment, we consider EWC [15], MeRGAN-RA [48] and EFT [44] as baselines. As we share our hyperparameters with EFT [44], we borrow most baseline results from that paper; the rest are obtained by either using the paper’s publicly available code or by running the PyCIL [60] framework for that model.

We provide the model architecture, hyperparameter settings and other experimental details for CIFAR-100, ImageNet-100 and Tiny ImageNet datasets in the supplementary material. Moreover, in supplementary, we provide results for multiple seed values along with their standard deviations.

In the following section, we discuss the results in the TIL, CIL and generative continual learning settings. We use average incremental accuracy as the evaluation metric. For TIL, we compute it as the average accuracy over all tasks, including the initial one [44]. In case of CIL, we report the average accuracy for all seen classes [61, 44].

The CIFAR-100 [16] dataset contains $20$ superclasses where each superclass has $5$ similar classes. For this toy experiment, we consider $4$ superclasses - $aquatic\ mammals$, $fish$, $vehicles\ 1$ and $vehicles\ 2$. Using these superclasses, we construct two task sequences - $ordered$ and $mixed$. Each task sequence has two tasks and each task has 10 distinct classes. Task $1$ of the $ordered$ sequence contains $(\{aquatic\ mammals\}\cup\{fish\})$ classes, while task $2$ has $(\{vehicles\ 1\}\cup\{vehicles\ 2\})$ classes. On the other hand, the $mixed$ sequence picks $2-3$ classes from each superclass and constructs two tasks, each with 10 distinct classes. So the $ordered$ task sequence has two very different tasks, while the $mixed$ task sequence has two somewhat similar tasks. In this experiment, after training on the first task, we find the gradient similarity between the first and second tasks to decide the growth rate for the second task. For $ordered$ and $mixed$ task sequences, we compute the gradient similarities $\alpha_{ordered}$ and $\alpha_{mixed}$ using Eg. 7. We observe $\alpha_{mixed}$ $>$ $\alpha_{ordered}$ and ($\alpha_{mixed}$ - $\alpha_{ordered}$) = 0.33. This result confirms that our gradient measure can be used for finding similarities between tasks.

We define the parameter growth from task $\mathcal{T}_{i}$ to $\mathcal{T}_{i+1}$ as follows: $\frac{P_{i+1}-P_{i}+E_{i+1}}{P_{i}}$ where $P_{i}$ is the total number of parameters used during task $\mathcal{T}_{i}$ and $E_{i+1}$ is the number of parameters in task $\mathcal{T}_{i}$ which are trained from scratch during task $\mathcal{T}_{i+1}$ (let’s call them exclusive parameters). For example, batch norm and the final linear layer in the ResNet-18 architecture are trained from scratch for every new task in our proposed approach and are thus counted as exclusive parameters. We define average parameter growth for $T$ tasks as the mean of all $T$ parameter growths. Since methods like EFT [44] grow their networks from the first task, we define the parameter growth for the first task as the change of parameters from an equivalent standard network.

In Table 4, we compare both our variants with our best baseline EFT [44] on 5, 10 and 20 splits of CIFAR-100. We show that our variants achieve significantly better results without any significant increase in average parameter growth. In Table 5, we show that the variants have almost no degradation in accuracy even when we significantly reduce average parameter growth. This is due to the forward transfer from previous tasks. We also show the total number of parameters that were used during training of all 10 tasks.

We have also performed experiments to demonstrate the forward transfer ability of our proposed filter expansion model. In this experiment, we have explored the scenario where our model grows over just the global or shared parameter. This type of growth is used in masking [51, 47] and EFT [44]. We have also shown the result for our current approach where the parameter grows over the previous tasks to accumulate all the earlier knowledge. The result for the CIFAR100/10 setting is shown in Fig. 4. Our approach shows consistently better results across all tasks.

We have experimented with different variants of our proposed task prediction method on the same task models and present the results on the CIFAR100/5 split in Table 8. We start off with the ensemble class prediction approach of TCL [13] and show that it gives the lowest accuracy. Subsequently, we present the result for the vanilla entropy-based task prediction approach used in multiple previous works [47, 44]. Lastly, we explore different components of our gradient aggregation approach and show how the accuracy improves with every step. Our proposed approach leverages the gradients of cross-entropy, entropy weighted augmentations and mean filters leading to an absolute gain of $4.1\%$ over the vanilla entropy baseline. It should be noted that entropy weighted augmentations significantly outperform vanilla augmentations for initial tasks. For example, entropy weighted augmentations lead to respective relative gains of $1.1\%$ and $1.3\%$ over vanilla augmentations for the initial $\lceil\frac{T}{2}\rceil$ tasks (minus first task which has same accuracy for both baselines) on CIFAR100/5 ($T$=5) and CIFAR100/10 ($T$=10) splits.