Selective Prompting Tuning for Personalized Conversations with LLMs

The CONVAI2 dataset, curated from the PersonaChat dataset Zhang et al. (2018), features a persona profile with four to five sentences for each interlocutor Dinan et al. (2019). This dataset has been established as a benchmark for personalized dialogue generation. Building upon this dataset, a variety of studies have explored different methods. For example, Wolf et al. (2019) extend the GPT2 model Radford et al. (2019) with fine-tuning techniques specific to persona-based conversations. Differently, Song et al. (2021) employed a tripartite BERT architecture Devlin et al. (2019), optimized through reinforcement learning, to craft responses. Similarly, Liu et al. (2020) introduces a transmitter-receiver model by applying reinforcement learning with custom rewards to refine the dialogue generation process. Cao et al. (2022) leverage model-agnostic data augmentation techniques to enrich the training dataset with pseudo-samples using models like GPT2 and BERT. Huang et al. (2023b) develop an adaptive attention mechanism that coherently integrates persona and context information. Huang et al. (2023a) propose a LAPDOG method to incorporate an external story corpus to enhance persona profiles for richer response generation. In contrast to those methods, the proposed SPT framework decomposes the task with multiple soft prompts without necessitating additional annotations or reliance on external corpora, which enables the generation of diverse and engaging responses while maintaining the integrity of the conversational context.

Language models (LMs) estimate text sequence probabilities, with recent models expanding from millions Radford et al. (2019); Zhang et al. (2022) to billions of parameters Brown et al. (2020); Zhang et al. (2022), and training corpora now including vast web texts and instructional data Ouyang et al. (2022); Touvron et al. (2023). Such advancements have notably improved the performance of LMs on various NLP tasks, especially in generating high-quality text for conversational applications. While those LMs are adept at providing user-centric responses, personalization remains a challenge. The prevalent strategy involves appending manually crafted hard prompts to LMs, which is overly simplistic and can result in the ‘lost in the middle’ problem Liu et al. (2023). This occurs when the output of the LM is generically correct but lacks personalized context, struggling to reconcile broad training data with specific user prompts. To counteract this, the proposed SPT method enables the LLM to adapt its responses to varying personalized contexts more effectively. As a result, SPT fosters the generation of dialogue responses that are not only consistent but also highly personalized, addressing the core challenge of maintaining context relevance in user interactions.

In persona-based dialogue sessions, a context is represented as $C=\{P,U\}$, where $P=\{p_{1},\ldots,p_{e}\}$ denotes the persona comprising $e$ sentences (e.g., $4\leq e\leq 5$) to provide background information for a machine interlocutor $m$ and $U=\{u_{h,1},u_{m,1},\ldots,u_{h,n}\}$ denotes the dialogue context initiated by the human $h$ to capture the exchange between human $h$ and machine $m$. The goal is to generate a machine’s response $r=u_{m,n}$ that aligns with its persona $P$ and the context $U$.

Figure 1 illustrates the SPT framework, consisting of a soft prompt group, a dense retriever, and a frozen LLM. Within this framework, the dense retriever selects an appropriate soft prompt from the soft prompt group by determining the closest match to the given context $C$. The chosen prompt is then merged with $C$ to guide the LLM to produce compelling responses. The SPT framework restricts the soft prompt group and dense retriever to be trainable, while maintaining the LLM in a frozen state, which could significantly reduce the memory footprint and optimize resource utilization during training.

To reduce computational overhead, the dense retriever $Ret$ utilizes two linear layers, i.e., $\mathrm{lin}_{C}$ and $\mathrm{lin}_{sp}$, for computing the similarity scores $\{s_{C,i}\}$. Those similarity scores are calculated using the context embedding $emb_{C}\in\mathbb{R}^{M\times D}$ obtained by the LLM’s word embedding layer $\mathrm{LLM}_{emb}$ and the soft prompt representation in $\mathbb{R}^{L\times D}$. The similarity score is computed as

where $\text{Avg}_{\text{dim}=0}(\cdot)$ denote the averaging operation along the length dimension to address the sequence length discrepancy between $emb_{C}$ and $sp_{i}$, $\text{Softplus}(\cdot)$ denotes the softplus activation function to ensure that $s_{C,i}$ remains in the range $[0,1]$ and enhance the numerical stability during training, and $s_{C,i}$ represents the normalized similarity score between the context $C$ and the soft prompt $sp_{i}$.

Navigating the lack of explicit annotations in complex dialogue scenarios poses a challenge in accurately guiding the retriever to assess the similarity between the context and each soft prompt. A naive method, which independently fine-tunes the entire soft prompt group and then selects candidates based on the similarity score during decoding, might lead to sub-optimal performance, akin to tuning a single soft prompt. To address this, we leverage context-driven losses from soft prompts, refining similarity score computations and enabling informed retriever decisions during training, as introduced in the next two subsections.

While the aforementioned losses aid in training, there is a risk that the retriever often retrieves a single prompt and stagnates in such sub-optimal states. To alleviate this and foster prompt diversity to retrieve more prompts, we propose a context-prompt contrastive loss. This loss refines prompt selection by adjusting similarity scores based on the textual similarity of distinct contexts, thereby preventing to always select a single soft prompt and promoting varied selections. Specifically, the context-prompt contrastive loss dynamically recalibrates the similarity scores between pairs of context contents, considering their textual resemblance. Mathematically, the context-prompt contrastive loss is formulated as

where $M(\cdot,\cdot)$ denotes a distance function such as BLEU Papineni et al. (2002), $\Gamma$ denotes a threshold, $s_{c_{i}}$ denotes a vector of cosine similarity scores between a context $c_{i}$ and soft prompts in the soft prompt group, and $\mathrm{cos}(\cdot,\cdot)$ denotes the cosine similarity.

The function $\mathcal{L}_{con}$ amplifies the cosine similarity for similar context pairs (i.e., $M(c_{i},c_{j})>\Gamma$) and dampens it for dissimilar pairs (i.e., $M(c_{i},c_{j})\leq\Gamma$). This contrastive strategy not only ensures the retriever’s alignment with the LLM’s evaluations but also fosters a rich diversity and distinctiveness among different dialogue contexts, significantly bolstering the framework’s overall adaptability.

To optimize the effectiveness of the soft prompts, we introduce a prompt fusion learning loss. This loss averages the predictive probabilities from all the soft prompts in the soft prompt group, aiming to aggregate a unified outcome that closely aligns with the desired output. The averaging operation in this loss smooths out variances and biases from individual prompts, thus improving the overall prediction accuracy and reliability. Formally, this loss is formulated as

By utilizing the collective strengths of diverse prompts, this loss enhances the model’s ability to generate context-appropriate responses.

The SPT framework hinges on the harmonious integration of the aforementioned loss functions, where each addresses a distinct aspect. The soft prompt loss (i.e., $\mathcal{L}^{LLM}$) ensures the LLM fidelity, the prompt selection loss (i.e., $\mathcal{L}_{selection}$) aligns the retriever’s similarity assessment with the LLM’s output, the context-prompt contrastive loss (i.e., $\mathcal{L}_{con}$) promotes diversity in prompt selection, and the prompt fusion learning loss (i.e., $\mathcal{L}_{fusion}$) enhance the overall performance for all the soft prompts. The overall objective of the SPT method is to minimize a composite loss function that encapsulates these individual components. Formally, the overall objective function $\mathcal{L}_{Total}$ for the SPT framework is formulated as

where $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are hyperparameters that control the relative contribution of each loss component. In our experiments, we simply set $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ to be $1$, which could achieve good performance.

By minimizing $\mathcal{L}_{Total}$ during training, the SPT framework effectively balances the fidelity to the LLM, the accuracy of the retriever, and the diversity in prompt selection, leading to an adaptive dialogue generation system.

During inference, the dense retriever selects the most appropriate soft prompt from the soft prompt group based on the given context. This selected prompt, along with the context, is then fed into the LLM to decode the final result. Formally, for a given context $C$, soft prompt group $SP$, and dense retriever $Ret$, the inference process proceeds as

where $sp_{i^{*}}$ denotes the selected soft prompt with index $i^{*}$ and $pred$ denotes the response generated by the LLM.

We conduct experiments on the ConvAI2 dataset Dinan et al. (2019), a benchmark for personalized dialogue generation. It comprises 8,939 training and 1,000 validation multi-turn conversations sourced from crowdworkers. Each dialogue includes persona profiles, each of which has four to five sentences to describe the background of each speaker, and the conversational history between the two interlocutors. By following Liu et al. (2020); Huang et al. (2023a), our experiments employ a self-persona setting where only the speaking interlocutor’s persona is revealed, maintaining the other’s persona as obscured.

All experiments are based on two LLMs, including OPT Zhang et al. (2022) and Llama2 Touvron et al. (2023) of different sizes, which serve as the foundation model for the proposed SPT method. We randomly initialize soft prompts using a standard Gaussian distribution. For OPT models, we set the soft prompt token length to $8$, and for the Llama2 model, we use a token length of $1$. The soft prompt group consists of $K=4$ candidates. Learning rates of different LLMs are recorded in Table 6 in the Appendix. The threshold $\Gamma$ in Eq. (4) is set to $20$.

We evaluate our model using a suite of established metrics for persona-based dialogue generation, including Unigram F1, BLEU, ROUGE, BERT Score, and textual unigram/bigram distinctness (denoted by DIST-1 and DIST-2). Unigram F1 measures the harmonic mean of precision and recall at the token level. BLEU Papineni et al. (2002) and ROUGE Lin (2004) evaluate the overlap of $n$-grams between the generated text and target reference. BERT score Zhang et al. (2019), using the deberta-xlarge-mnli model²²2https://github.com/Tiiiger/bert_score as recommended for its improved performance over roberta-large, captures the semantic similarity of text pairs. Unigram and bigram distinctness (denoted by DIST-1 and DIST-2) gauge the diversity of the generated text, where DIST_AVG denotes the average of DIST-1 and DIST-2.

Table 1 illustrates that the proposed SPT consistently outperforms the baseline models across various metrics. Notably, the OPT-2.7B-SPT and Llama2-7B-SPT models exhibit significant performance improvements (i.e., 33.04% and 26.26%, respectively). Those improvements affirm the effectiveness of the proposed SPT method in fostering more diverse and personalized responses.

For baseline models, we can see that there exists a common trade-off between linguistic quality and diversity. Specifically, the Llama2-7B model scores 17.12 in F1 and 1.99 in BLEU, but its diversity seems not so good (i.e., 2.80 in DIST-1 and 12.91 in DIST-2). This is in contrast to the OPT-125M model, which has lower linguistic scores (i.e., 10.79 in F1 and 1.61 in BLEU) but higher distinctness (i.e., 3.94 in DIST-1 and 13.67 in DIST-2). Different from those models, the proposed SPT method significantly enhances both diversity and linguistic quality, thereby avoiding the common compromise between linguistic enhancement and diversity.

Table 2 reveals the impact of different training losses on performance. Omitting the prompt fusion loss slightly increases the prediction diversity in terms of DIST_AVG but reduces the overall performance in terms of F1, BLEU, ROUGE, and BERT Score. One possible reason is that the prompt fusion loss contributes to the linguistic quality at the cost of the diversity. Excluding the context-prompt contrastive loss leads to a decline in all the evaluated metrics, which shows the effectiveness of the context-prompt contrastive loss. The absence of the prompt selection loss significantly affects the prediction diversity, causing the model to favor a single soft prompt, akin to utilizing a single prompt. The above results underscore the importance of each loss in enhancing the model performance and response diversity.

To see the prompt usage during the conversational process, we plot in Figure 2 the times each soft prompt is chosen during the entire conversation. According to Figure 2, we can see that in the OPT-1.3B-SPT model, prompt $sp_{3}$ is predominantly utilized for the initial stage in the conversation, $sp_{2}$ for the middle stage of the conversation, and $sp_{1}$ for the later stage of the conversation. For the Llama2-7B-SPT model, we have similar observations, indicating that soft prompts have functionalities in different stages of the conversation.

Moreover, Figure 3 explores the stylistic aspects of responses generated by different prompts, i.e., emojis in the generated responses. In the Llama2-7B-SPT model, $sp_{2}$, which is often used in the initial stage of the conversations, tends to generate emojis in the generated response. Differently, $sp_{3}$, often used in the late stage of the conversation, tends to generate few emoji in decoded responses. This phenomenon suggests a strategic use of emojis at different stages of the conversation.

Table 3 shows the effect of the number of soft prompts (i.e., $K$) to the model performance in terms of different metrics. Though the best performance occurs at different $K$’s for different performance metrics, the best performance for different metrics usually occurs when $K\leq 4$, which is likely due to the sizes of both the CONVAI2 dataset and the LLM used. Hence, in all the experiments, $K$ is set to be 4 by default.

As shown in Table 4, the SPT method with four single-token soft prompts outperforms the four-token prompt tuning method, highlighting effectiveness of the proposed SPT method. Moreover, SPT excels the eight-token prompt tuning method in terms of BLEU, ROUGE, and DIST_AVG, showing its effectiveness despite fewer trainable parameters.

As LoRA Hu et al. (2022) is another type of parameter-efficient finetuning method and has shown to be effective to utilize LLMs for different applications, we compare the proposed SPT method with it based on the Llama2-7B model under the condition that they have comparable numbers of trainable parameters. As shown in Table 4, LoRA exhibits improvements in the BLEU score and DIST_AVG but has lower ROUGE-L, BERT_F1, and F1 scores compared with the four-token prompt tuning method. Moreover, the proposed SPT method surpasses LoRA across all the evaluation metrics, highlighting its superior performance and affirming its effectiveness under the condition of comparable numbers of trainable parameters.

To compare the performance with In-Context Learning (ICL) on LLMs, we compare the SPT method with the zero-shot GPT-3.5 turbo with instructions. According to results shown in Table 4, we can see that ICL gains a higher diversity score (i.e., DIST_AVG) but lower scores in terms of other metrics. This implies that simply prompting a more powerful LLM without proper tuning is hard to gain comparable performance to tuning methods.

Table 5 presents BLEU scores between the model’s predictions and the system’s persona descriptions for different models. We can see that the prompt tuning method exhibit larger text overlap with the system’s persona, often leading to repetitive responses aligned with the persona. In contrast, the proposed SPT method has lower linguistic similarities to the persona, which results in more diverse and effective responses. This suggests that the proposed SPT method effectively balances the persona consistency and response diversity, avoiding the pitfalls of over-repetition.