Improved Baselines for Data-efficient
Perceptual Augmentation of LLMs

Here $n_{\text{fl}}=1$. The feature tokens are prepended to the sequence of textual token embeddings (including “BOS” token for OPT). They are propagated through the LLM, and removed from its final output. We use a causal attention mask, where each token can only attend to previous ones, including in-between inserted perceptual tokens.

Here we only require $n_{\text{fl}}\geqslant 1$. This method is additionally parametrized by a number of LLM layers $n_{\text{LLM}}$ where we inject tokens and a number of left-out layers at the end $n_{\text{left}}$. Then, if we note $L_{\text{LMM}}$ the total number of LLM layers, for each layer $i\in\{L_{\text{LMM}}-n_{\text{LLM}}-n_{\text{left}},\dots,L_{\text{LMM}}-n_{\text{left}}-1\}$, we inject feature tokens extracted from level $l_{i}=\lfloor\frac{(i-L_{\text{LMM}})*n_{\text{fl}}}{n_{\text{LLM}}}\rfloor$. For injection, we follow the same procedure as for first-layer token injection, where the feature tokens are prepended to the input sequence of the layer $i$. Additionally, we remove them from the output sequence of this layer.

This map** block is parametrized by the number of layers $L_{\textrm{QP}}$ and the number of query tokens $n_{\textrm{Q}}$. It takes as input a sequence of tokens of dimension $d_{\textrm{embed}}$. These tokens are concatenated to $n_{\textrm{Q}}$ query tokens, which are learnable parameters. The resulting sequence is then passed through a stack of $L_{\textrm{QP}}$ standard transformer encoder layers. We use a dropout of $0.1$, embedding dimension of $d_{\text{embed}}$, the GELU activation and 8 attention heads. Only the last $n_{\textrm{Q}}$ output tokens, corresponding to the query tokens, are considered as output.

Some of the resamplers use a common framework, based on local blocks of patches. This framework is parametrized by an embedding dimension $d_{\textrm{emb}}$, and a pooling function. The extracted feature tokens are first projected from dimension $d_{\textrm{feats}}$ to $d_{\textrm{embed}}$ using a linear layer. The patches tokens are then arranged on a 1D, 2D or 3D grid, depending on the modality, and grouped into blocks of dimension $4$ (1D case) or $2\times 2$ (2D case). Each block is pooled using the pooling function, resulting in a single token. This is similar to using a 1D or 2D pooling operation on the grid. The tokens are then rearranged as a sequence again, to which the CLS token is prepended, before being normalized using the RMSNorm, and finally projected with a linear layer from $d_{\textrm{embed}}$ to $d_{\textrm{LLM}}$.

is a parameter-efficient fine-tuning mechanism, parametrized by $n_{\textrm{pt}}$, the number of learned tokens. When used, $n_{\textrm{pt}}$ constant embedding vectors of dimension $d_{\textrm{LLM}}$ are learned and prepended before the textual tokens (and after the perceptual tokens) at the beginning of LLMs. We also use a causal padding attention mask with these tokens.

we use the feature extraction with $n_{\text{fl}}=1$ level, project all the feature tokens from dimension $d_{\text{feats}}$ to $d_{\text{LLM}}$ using a linear layer, and use first-layer token injection.

we use the same mechanism as for LiMBeR^(all), but only keep the single CLS token.

we use the feature extraction with $n_{\text{fl}}=1$ level, and a feature map** block consisting of a linear projection from dimension $d_{\text{feats}}$ to $d_{\text{embed}}=256$, followed by a QPMapper using $L_{\textrm{QP}}=4$ layers and $n_{\textrm{Q}}=32$ query tokens, and then a linear projection from $d_{\text{embed}}$ to the LLM inner dimension $d_{\text{LLM}}$. We then insert tokens with the first-layer injection mechanism.

we extract $n_{\text{fl}}=6$ levels of feature tokens, and only keep the CLS token from each level. We project each one from dimension $d_{\text{feats}}$ to $d_{\text{LLM}}$ with the same linear layer. Mapped tokens are inserted into $n_{\text{LLM}}=12$ inner layers, leaving out the last one ($n_{\text{left}}=1$).

we use a feature map** block consisting of a linear projection from dimension $d_{\textrm{feats}}$ to $d_{\text{embed}}=1024$, followed by a QPMapper using $L_{\textrm{QP}}=2$ layers and $n_{\textrm{Q}}=32$ query tokens, and a linear projection from $d_{\textrm{embed}}$ to the LLM inner dimension $d_{\textrm{LLM}}$.

we sample a subset of the tokens, using the following procedure. If the model outputs $n_{\text{tk}}$ patch tokens, we keep the CLS token, and $\lfloor fn_{\text{tk}}\rfloor$ uniformly sampled patch tokens, with $f$ the proportion of tokens we keep:

• •

  During training, with a probability of $\frac{1}{10}$, we set $f=f_{\text{max}}$. Otherwise, we sample $f^{\prime}\sim\mathcal{N}(f_{\text{mean}},f_{\text{std}})$, and set $f=\min(\max(f,f_{\text{min}}),f_{\text{max}})$ to restrict the values to the interval $[f_{\text{min}};f_{\text{max}}]$.

• •

  During inference, we set $f=f_{\text{max}}$.

• •

  We use $f_{\text{min}}=\frac{1}{16}$, $f_{\text{max}}=\frac{1}{2}$, $f_{\text{mean}}=\frac{1}{4}$ and $f_{\text{std}}=0.2$.

We project the resulting tokens from dimension $d_{\text{feats}}$ to $d_{\text{LLM}}$, normalize them using the RMSNorm, and project them again with a linear layer ($d_{\text{LLM}}$ to $d_{\text{LLM}}$) before injection inside the LLM.

we use a block-based resampler with $d_{\text{emb}}=d_{\text{LLM}}$. The pooling function is a linear projection from dimension $4\times d_{\text{emb}}$ to $d_{\text{emb}}$, taking as input the concatenation of all tokens from the same block.

we use a block-based resampler with $d_{\text{emb}}=768$. The pooling function is a QPMapper taking as input the 4 tokens of a single block, using $L_{\textrm{QP}}=4$ layers and a single query token ($n_{\textrm{Q}}=1$).

we use a block-based resampler with $d_{\text{emb}}=d_{\text{LLM}}$. The pooling function returns the mean of the tokens from the same block.

we extract $n_{\text{fl}}=4$ levels of feature tokens, and use the same feature map** as DePALM. In particular, the map** block is shared across all token levels, meaning that each sequence of tokens from each level goes through the same projection, with the same weights, but as separate batch elements. Mapped tokens are inserted into $n_{\text{LLM}}=12$ inner layers, with $n_{\text{left}}=3$. We also use prompt-tuning with $n_{\textrm{pt}}=16$.

we extract $n_{\text{fl}}=4$ levels of feature tokens, and use a single cross-attention block (detailed below) that will do both feature map** and injection. Injection takes place into the last $n_{\text{LLM}}=12$ inner layers, leaving out the $n_{\text{left}}=3$ last ones, similarly to the inner-layer injection mechanism, but using cross-attention instead of concatenation.

For the cross-attention block, see Figure S1, we first project the tokens from dimension $d_{\text{feats}}$ to $d_{\text{embed}}=1024$ using the linear projection $P_{\textrm{in}}$. They are then passed through a single transformer layer $\mathrm{Transf}$ (8 heads, dropout of 0.1), acting as a minimal resampler network. We also project the input textual tokens to the dimension $d_{\text{embed}}=1024$, using a two-layers MLP (the FFN block, with a hidden dimension of $d_{\text{embed}}$ and the GELU activation function), and normalize them using RMSNorm. We then use the perceptual tokens as keys and values, and the text ones as queries, in a cross-attention layer. We project the resulting tokens back to dimension $d_{\text{LLM}}$ using a linear projection, normalize them with RMSNorm, and add them to input textual tokens, using a tanh-gated residual connection. This residual connection takes the form $x^{(k-1)},x^{(k)}_{r},h^{(k)}\mapsto x^{(k-1)}+\tanh(h^{(k)})\times x^{(k)}_{r}$ where $x^{(k-1)}$ are the original textual tokens, $x^{(k)}_{r}$ the output of our cross-attention block, and $h^{(k)}$ a single learned float, initialized at $0$.

We train each model on a single node using eight V100-32G GPUs, using the following method:

• •

  Optimizer: we take AdamW, with a weight decay of 0.1 and a default learning rate of $\alpha_{\text{max}}=8\cdot 10^{-4}$, that we further adapt for each experiment.

• •

  Gradient clip**: we use a clip** value of $g_{\text{clip}}=0.8$, that we further adapt for each experiment.

• •

  Learning rate scheduler: we set a minimum learning rate of $\alpha_{\text{min}}=\alpha_{\text{max}}\cdot 10^{-4}$, with a cosine scheduler. During the first 20% of all iteration steps, we linearly warmup the effective learning rate from $\alpha_{\text{min}}$ to $\alpha_{\text{max}}$, then use the cosine scheduler to decrease it from $\alpha_{\text{max}}$ to $\alpha_{\text{min}}$.

• •

  Batch size: we use a batch size of 16 on each GPU, for an effective batch size of 128. On experiments where memory is an issue, we use gradient accumulation while to train on the same batch size.

• •

  Epochs: we use a base number of 8 epochs, and increase it on small datasets: we use 12 epochs on AudioCaps, 20 on TextCaps, OKVQA, AOKVQA and TextVQA, and 30 epochs on the two 1% settings.

• •

  Loss: we compute the loss only on the generate text. Additionally, we use a label smoothing value of $2\cdot 10^{-3}$ in the cross-entropy loss of the LLM.

• •

  Float precision: we load pre-trained models in float16, and train new weights in float32, using mixed-precision.

• •

  Duplicate inputs: we group together every training sample with the same perceptual and text input, but different outputs. During training, we select the target output for the loss randomly.

As each setting has a different training dynamic, which can be very sensitive to the learning rate, we use a grid search over a few values that we experimentally found to be working efficiently. We start by swee** over the learning rate values $\alpha_{\text{max}}\in\{1\cdot 10^{-3},8\cdot 10^{-4},4\cdot 10^{-4}\}$, and set the gradient clip** parameter as $g_{\text{clip}}=0.8$. On each experiment where the score significantly increase or decrease between the three runs, or where the results are noticeably lower than other methods or previous experiments, we further experiment with $g_{\text{clip}}=0.8$ for the same sweep over learning rates. We perform this sweep for each method-dataset pair, including baselines reproductions.

We also note that we used grid search on each parameter of our models (number of layers, etc.) on the COCO dataset, to confirm that they are at least a local optimum.

We use random augmentation during training of each model. Each time a training sample is used during the training loop, a random modification is applied, without increasing the dataset size.

• •

  Images: the normalized image is resized to $128\times 128$ with a random scale in $[0.5,1]$ and a random aspect ratio in $[\frac{3}{4},\frac{4}{3}]$, and horizontally flipped with probability $0.5$. Data is then augmented with the modified RandAugment procedure [45].

• •

  Audio: we augment the dataset using frequency masking with maximum length of 24, and time masking with maximum length of 96, after normalizing the audio.

• •

  Video: we normalize the videos, and use the same procedure as with images, with the same random scaling and flip** as with images, followed by the default RandAugment procedure implemented in pytorch.

due to memory limitations, when training LiMBeR⁽¹⁾ and DePALM^R-rand,L0 on MSRVTT, we average the patch tokens along the time dimension to reduce their number. This yields a representation with the dimension of the embedding of a single frame.