LoPT: Low-Rank Prompt Tuning for Parameter Efficient Language Models

Abstract

In prompt tuning, a prefix or suffix text is added to the prompt, and the embeddings (soft prompts) or token indices (hard prompts) of the prefix/suffix are optimized to gain more control over language models for specific tasks. This approach eliminates the need for hand-crafted prompt engineering or explicit model fine-tuning. Prompt tuning is significantly more parameter-efficient than model fine-tuning, as it involves optimizing partial inputs of language models to produce desired outputs.

In this work, we aim to further reduce the amount of trainable parameters required for a language model to perform well on specific tasks. We propose Low-rank Prompt Tuning (LoPT), a low-rank model for prompts that achieves efficient prompt optimization. The proposed method demonstrates similar outcomes to full parameter prompt tuning while reducing the number of trainable parameters by a factor of 5. It also provides promising results compared to the state-of-the-art methods that would require 10 to 20 times more parameters.

LoPT: Low-Rank Prompt Tuning for Parameter Efficient Language Models

Shouchang Guo, Sonam Damani, Keng-hao Chang Microsoft AI {shouchangguo,sodamani,kenchan}@microsoft.com

1 Introduction

With the success of large language models Touvron et al. (2023); Achiam et al. (2023); Jiang et al. (2023), it has become increasingly important for language models (LMs) to handle instructions effectively for customized agents and tasks. There are three essential categories of methods to adapt pre-trained language models to specific and customized needs: prompt engineering, model fine-tuning, and prompt tuning.

Prompt engineering Brown et al. (2020); Sanh et al. (2021); Chung et al. (2024) involves crafting handcrafted prompts and faces the challenge of getting LMs to consistently produce desired outputs with few-shot instructions. This effort may be difficult to generalize or extend to new tasks. Model fine-tuning Raffel et al. (2020) can perform very well for task-specific needs but requires explicit fine-tuning of a significant number of model parameters, even with parameter-efficient fine-tuning (PEFT) approaches Liu et al. (2022); Hu et al. (2021).

Prompt tuning (PT) Li and Liang (2021); Lester et al. (2021); Wen et al. (2024); Shi et al. (2022); Shin et al. (2020); Khashabi et al. (2021) is a promising method that lies between prompt engineering and model fine-tuning. Instead of handcrafting prompts, it optimizes a small number of prompt embeddings or indices with training data and has demonstrated capabilities comparable to those of model fine-tuning approaches Asai et al. (2022); Shi and Lipani (2023); Wang et al. (2023).

We focus on soft prompt tuning, which operates by adding a prefix or suffix to the existing inputs and optimizing the embeddings of this prefix or suffix. The embeddings, or the soft prompt matrix, has dimensions $n\times d$, where $n$ is the “tokens" length of soft prompts, and $d$ is the embedding size. The soft prompt length $n$ can be task specific to achieve desired outcomes. For example, more sophisticated tasks might benefit from longer soft prompts that allow for more parameters to be optimized.

In this work, we introduce a low-rank modeling approach for the soft prompt matrix, which effectively reduces the number of trainable parameters in prompt tuning without compromising performance. We find that soft prompt matrices are inherently low-rank due to their dimensionality, and we apply further dimensionality reduction through our proposed method. We demonstrate that the number of parameters required for tuning LMs to meet specific task requirements can be minimal. Additionally, the number of trainable parameters can be easily controlled by adjusting the rank of the soft prompt matrix.

Our approach distinguishes itself from existing methods by directly imposing low-rank constraints on the entire soft prompt to be trained. While recent work Shi and Lipani (2023) also explores low-rank matrices for prompt tuning, it restricts low-rankness to the differences or updates of a frozen baseline prompt, similar to the LoRA technique used in model fine-tuning Hu et al. (2021), and is only applied to a portion of the overall soft prompt.

Our primary contributions are:

• We introduce Low-rank Prompt Tuning (LoPT) that significantly reduces the number of trainable parameters required in prompt tuning.

• We achieve a 5-fold reduction in trainable parameters while maintaining performance comparable to the full-parameter prompt tuning.

• We demonstrate the efficacy of our method across 5 diverse datasets, showing substantial improvements in parameter efficiency compared to existing methods.

Our proposed parameter-efficient method would be particularly beneficial for computationally demanding prompt tuning needs in sophisticated tasks and large language models.

2 Method

2.1 Problem statement

In soft prompt tuning (Lester et al., 2021), we add a prefix or suffix to the original prompt and optimize the embeddings of this prefix or suffix as trainable parameters using supervised training data to achieve task-specific predictions.

Given a language model $\mathcal{M}$ with frozen network parameters $\bm{\theta}$ and embedding matrix $\bm{E}\in\mathbb{R}^{V\times d}$, where $V$ is the vocabulary size, $d$ is the embedding size, with each row of $\bm{E}$ representing a token in the vocabulary. We optimize trainable embeddings $\bm{X}\in\mathbb{R}^{n\times d}$ of the prefix, where $n$ is the number of soft tokens. The optimization problem can be formulated as:

$$\begin{array}[]{ll}\underset{\bm{X}}{\text{arg min\ }}\sum_{i}\mathcal{L}\left(\mathcal{M}\left(\left[\bm{X};\bm{I}_{i}\right];\bm{\theta}\right),\bm{y}_{i}\right),\end{array}$$ (1)

where $\mathcal{L}$ is the loss function for the task. For the $i$-th training sample, $\bm{I}_{i}\in\mathbb{R}^{t\times d}$ denotes tokenized embeddings of the original model input with sequence length $t$, and $\bm{y}_{i}$ is the label associated with this sample.

2.2 Our Low-Rank Prompt Tuning (LoPT)

Recent work (Lester et al., 2021; Shi and Lipani, 2023) demonstrates that prompt tuning could yield performance comparable to parameter-efficient model fine-tuning methods (Hu et al., 2021) with a significantly smaller amount of learnable parameters. In this work, we push the boundaries by exploring parameter-efficient prompt tuning to further reduce the number of trainable parameters without compromising accuracy.

Because the prefix or suffix length $n$ is often significantly smaller that the embedding dimension $d$ in prompt tuning, the rank of the soft prompt matrix $\bm{X}$ would inherently be constrained by $n$, making $\bm{X}$ low-rank. The potential similarity between neighboring embeddings in a prompt could also suggest that $\bm{X}$ is low-rank. Therefore, we explore this potential and impose constraints on $\bm{X}$ for dimensionality reduction and more efficient prompt tuning.

We propose two low-rank approximations for modeling $\bm{X}$. The proposed methods could drastically reduce the number of learnable parameters while maintaining performance comparable to full-parameter prompt tuning.

2.2.1 LoPT-1

For effective prompt tuning with a reduced and adjustable number of parameters, we propose to decomposite the low-rank prompt matrix $\bm{X}\in\mathbb{R}^{n\times d}$ as:

$$\bm{X}=\bm{U}\bm{V}.$$ (2)

In this formulation, $\bm{U}\in\mathbb{R}^{n\times r}$ and $\bm{V}\in\mathbb{R}^{r\times d}$ are the new trainable matrices. We train $\bm{U}$ and $\bm{V}$ simultaneously, transforming the prompt tuning optimization problem to the following:

$$\begin{array}[]{ll}\underset{\bm{U},\,\bm{V}}{\text{arg min\ }}\sum_{i}\mathcal{L}\left(\mathcal{M}\left(\left[\bm{U}\bm{V};\bm{I}_{i}\right];\bm{\theta}\right),\bm{y}_{i}\right).\end{array}$$ (3)

We initialize both $\bm{U}$ and $\bm{V}$ with uniform random values in the range of [-0.5, 0.5] at the beginning of training.

The number of trainable parameters is reduced to $r(n+d)$. As $n\ll d$, the total number of parameters can be significantly reduced compared to the original $nd$, especially with adjustable choices of $r<n$.

2.2.2 LoPT-2

we also introduce an empirical map** scheme for the low-rank approximation of $\bm{X}$, employing learnable linear projections and nonlinear thresholding operation to achieve effects analogous to singular value thresholding (Cai et al., 2010) and with reduced number of parameters for optimization. Specifically, we construct $\bm{X}$ as:

$$\bm{X}=\sigma(\bm{X}_{0}\bm{U})\bm{V},$$ (4)

where $\bm{X}_{0}\in\mathbb{R}^{n\times d}$ is a random initialization of $\bm{X}$, $\bm{U}\in\mathbb{R}^{d\times r}$ and $V\in\mathbb{R}^{r\times d}$ are linear projection matrices. $\sigma(\cdot)=\mathrm{max}(\cdot,0)$ represents the nonlinear thresholding operation that filters out negative values. Similar to LoPT-1, $\bm{U}$ and $\bm{V}$ are randomly initialized and optimized with function

$$\begin{array}[]{ll}\underset{\bm{U},\,\bm{V}}{\text{arg min\ }}\sum_{i}\mathcal{L}\left(\mathcal{M}\left(\left[\sigma(\bm{X}_{0}\bm{U})\bm{V};\bm{I}_{i}\right];\bm{\theta}\right),\bm{y}_{i}\right).\end{array}$$ (5)

The number of trainable parameters becomes $2rd$ rather than $nd$. By choosing a smaller projected dimension $r<n/2$, we can easily reduce redundancy in trainable parameters and improve time and memory efficiency. It is worth noting that for $n\ll d$, LoPT-1 is more parameter efficient than LoPT-2.

Implementation Simplification The proposed LoPT-2 map** for $\bm{X}$ improves parameter efficiency, and we propose a straightforward implementation. We use two linear layers for the linear projections $U$ and $V$, and apply an ELU (Clevert et al., 2015) function for the nonlinear thresholding operator $\sigma(\cdot)$. Empirically, we found that ELU performs better than ReLU (Nair and Hinton, 2010; Fukushima, 1969) and GELU (Hendrycks and Gimpel, 2016).

We demonstrate that the proposed low-rank modeling and formulations yield effective parameter reduction with promising outcomes.

Method	# Params	SST-2	AGNews
No LoPT	12.8k	92.8	91.8
LoPT-1 (ours)	2.58k	92.1	91.9
LoPT-2 (ours)	5.12k	90.9	90.0

Table 1: Accuracy (%) on the SST-2 and AGNews validation sets compares the proposed LoPT-1 and LoPT-2 to the baseline soft prompt tuning without low-rank factorization (No LoPT). The language model used is GPT-2 large with embedding dimension $d=1280$, and prompt length $n=10$. We set the rank $r=2$ for both LoPT-1 and LoPT-2, and calculate the # of parameters accordingly.

3 Experiments

3.1 Experiment Setup

Method	# Params	SST-2	BoolQ	RTE	WiC	CB
Fine-tuning1	220M	94.6	81.1	71.9	70.2	85.7
LoRA2	3.8M	94.3	81.3	75.5	68.3	92.9
PT3	76.8k	91.9	63.7	78.8	50.8	67.9
DePT3	76.8k	94.2	79.3	79.1	68.7	92.9
LoPT-1 (ours)	3.94k	92.9	76.5	73.8	55.1	90.4
LoPT-2 (ours)	7.68k	92.4	75.5	74.3	62.7	74.0

Table 2: Accuracy (%) on the SST-2 and SuperGLUE benchmarks for classification tasks. The language model is T5-Base with embedding dimension $d=768$. We set the rank $r=5$ and soft prompt length $n=20$ for both LoPT-1 and LoPT-2. Comparisons including Fine-tuning¹ from (Asai et al., 2022), LoRA² from (Sung et al., 2022), PT³ and DePT³ are from (Shi and Lipani, 2023).

Length	Rank	$\Delta$ # Params	SST-2
$n=10$	No LoPT	-	92.8
$n=10$	$r=1$	-89.92%	90.5
	$r=2$	-79.84%	92.1
	$r=5$	-49.61%	92.1
$n=20$	$r=1$	-89.84%	91.4
	$r=2$	-79.69%	92.8
	$r=5$	-49.22%	92.9
$n=30$	$r=1$	-89.77%	90.9
	$r=2$	-79.53%	92.2
	$r=5$	-48.83%	92.1

Table 3: Ablation study on LoPT-1: We evaluated various combinations of prompt length $n$ and rank $r$ using the SST-2 dataset and the GPT-2 large model. The numbers of trainable parameters are compared to the baseline prompt tuning, which has a fixed $n=10$ and no low-rank approximations. The parameter reduction rate is represented by $\Delta$ # Params. LoPT-1 with $n=20$ and $r=5$ achieves the highest accuracy (%).

Datasets We evaluate the proposed method on classification tasks using various datasets in English: the sentiment analysis task SST-2 (Socher et al., 2013), the 4-way topic classification task AGNews (Zhang et al., 2015), and datasets in the SuperGLUE benchmark (Wang et al., 2019). These include BoolQ (Clark et al., 2019), RTE (Giampiccolo et al., 2007), WiC (Pilehvar and Camacho-Collados, 2018), and CB (De Marneffe et al., 2019).

Training Details The proposed low-rank factorizations, LoPT-1 and LoPT-2, are optimized using GPT-2 large (774M parameters, $d=1280$) (Radford et al., 2019) and T5-base (220M parameters, $d=768$) (Raffel et al., 2020) models. We build upon the settings in (Ding et al., 2021; Wen et al., 2024), and optimize the prompts using the Adafactor optimizer (Shazeer and Stern, 2018) with a learning rate of 0.3. We apply soft prompt length $n$ of 10 or 20, and batch size of 8 for SuperGLUE datasets, and 16 for other data.

We set the rank parameter $r$ of LoPT-1 or LoPT-2 to $\lfloor\frac{n}{4}\rfloor$ for most experiments to achieve the desired level of trainable parameter reduction. In the case of prompt tuning without our proposed low-rank approximations, the number of trainable parameters is $nd$. For LoPT-1, the number of learnable parameters is $r(n+d)$. For LoPT-2, the trainable parameter amount is $2dr$.

3.2 Comparisons and Results

We compare the proposed parameter efficient approaches to vanilla soft prompt tuning using the GPT-2 large model, and evaluate their effectiveness with SST-2 and AGNews datasets. As presented in Table 1, LoPT-1 significantly reduces the number of trainable parameters from 12.8k to 2.58k, while maintaining accuracy levels comparable to full parameter prompt tuning. LoPT-2 achieves a 60% reduction in parameters and successfully preserves classification accuracy for both binary and multi-class classification tasks.

Our methods are compared against a variety of baselines including Fine-tuning, LoRA (Hu et al., 2021), PT (Lester et al., 2021), and DePT (Shi and Lipani, 2023) using the T5-base model. As shown in Table 2, LoPT-1 and LoPT-2 demonstrate promising performance, achieving reductions in trainable parameters by factors of 20 and 10, respectively. This marks a significant efficiency improvement over existing prompt tuning approaches, which are already noted for their high parameter efficiency.

It is noteworthy that LoPT-1 outperforms LoPT-2 on the CB dataset, while LoPT-2 excels over LoPT-1 on the WiC dataset. This suggests that both approaches could be strategically exploited to tailor the desired low-rank formation for optimal performance on specific tasks.

3.3 Ablation Study

Using the SST-2 task and the GPT-2 large model, Table 3 presents the accuracy of LoPT-1 with varying prompt lengths $n$ and ranks $r$ for the low-rank factorization. We observe that an increased prompt length does not necessarily lead to improved outcomes, and the combination of $n=20$ with $r=5$ or $r=2$ yield the highest accuracy. Given that $n$ is much smaller than $d$, the number of trainable parameters is primarily controlled by the rank parameter $r$ in LoPT, which can be easily adjusted to achieve parameter reduction.

3.4 Limitations

This work relies on the low-rank hypothesis and may not be effective when the prompt matrix is not low-rank. Regarding the performance of the proposed methods, further improvements could be achieved through hyper-parameter tuning.

4 Conclusion

In this work, we propose Low-rank Prompt Tuning (LoPT), a low-rank formulation of prompts that significantly reduces the number of trainable parameters for parameter-efficient prompt tuning of language models. We demonstrate that LoPT can decrease the number of trainable parameters by a factor of 10 or 20 while achieving promising performance across various datasets.

The proposed parameter-efficient method could be particularly beneficial for sophisticated tasks and large language models, where longer soft prompts are increasingly important for effective prompt tuning.