Role-Play Zero-Shot Prompting with Large Language Models for Open-Domain Human-Machine Conversation

Quoting Reitz (2019), \saydialogue is more than just having a conversation. Genuine dialogue describes a way of interacting that is mutual, relational, attentive, and meaningful. Instruction-following models, even those optimized for conversation fall short from fulfilling some of these aspects of a genuine dialogue. Indeed, social aspects - mutuality, relationality and attention – especially are poorly displayed. Talking of mutuality, these systems often converse in a leader-follower structure where they are the follower and the user the leader – this is for instance evidenced by the words used to designate user entries in ChatGPT release blog: \sayqueries, \sayinstructions OpenAI (2022). Regarding relationality and attention, these systems lack of straightforward engagingness and personality consistency.

Commonly, these limitations have been tackled with finetuning. However, this is costly and data-dependant which data are scarce in languages other than English. As results, we formalize a general role-play prompt structure which is a more efficient and less expensive approach. Indeed, it avoids finetuning and rely on multilingual LLMs prompted in English, with external English data if needed (for instance persona from PersonaChat Dataset) and performs dialogue task in a desired target language here in French.

It is important to understand that Role-Play Prompting here is not restricted to playing a given character. To better understand that, let’s consider the simulacra and simulator framing in Shanahan et al. (2023). LLM is a simulator which swallowed myriads of simulacra during the pre-training and at each simulation it kind of randomly selects among any of them which oneS to display. The simulacrum is, as a matter of fact, not only about persona background Smith et al. (2020) but also thinking and writing styles, personal situation Smith et al. (2020), target language (for a multilingual simulator) and information processing (long-term memory, user personalization, response filtering etc.). All possible simulacra already exist in the simulator (LLM) but it doesn’t display all of them natively, in fact it can’t. Role-Prompting enters the chat to make the LLM favor simulacra that are suitable for a given dialogue task. In this paper, we derived two distinct dialogue tasks to assess the efficiency of this approach. The first task, referred as Persona task, uses Role-Play Prompting for enhancing humanness in conversation skills. The second one, referred as INT task, derives Role-Play to allow the LLM to talk about a simulacrum instead of interpreting it.

Open-domain dialogue belonging to the realm of complex tasks Santu and Feng (2023) makes the endeavour of role-play prompting more challenging. As a matter of fact, small variations in a prompt may hamper the model’s observed performance. For this reason, it is mandatory to define a general prompt structure that can be adapted later to different conversational tasks. Hence, with the concerns of dealing with the previously mentioned limitations we retain the following sections – each focusing on different aspects useful to a dialogue – as building blocks of a prompt engineering module:

1. 1.

   System Instructions $\mathcal{I}_{s}=\{i_{s,k}\}_{k=1}^{N_{i}}$: where $N_{i}$ is the number of instructions $i_{s,k}$ which sharply define the target task’s specifications and the global desired behaviour. This may include thinking (inferring) and writing (generation) styles.

2. 2.

   Situational context $\mathcal{C}^{t}=\{c_{k}\}_{k=1}^{N_{c}}$: each $c_{k}$ is a context information that may help the model better perform the desired task. As such, it evolves with time depending on the conversation flow. It can include personality information, image and scene description, summaries of old turns or information from external sub-modules.

3. 3.

   Response Instructions $\mathcal{I}_{a}=\{i_{a,k}\}_{k=1}^{N_{i}}$: These are final instructions to incite the LLM to respond to the users’ utterance with emphasis on the writing style, the target responding language and creativity with section 2) in mind.

4. 4.

   Conversation History $\mathcal{X}^{t}$: this part contains previous messages from the user ($x^{t}$) and the LLM ($y^{t}$). They can be truncated to the $k$ latest conversation turns to fit in the LLM token size limit or help the LLM focus on latest part of the conversation. In this case a summary of the $k$ removed turns $\tilde{\mathcal{X}^{k}}$ generated by an external module (also using PBL with an LLM) can be added in section 2). Conversation history, therefore becomes: $\mathcal{X}^{t-1}=\{(x^{t-k},y^{t-k}),...,(x^{t-1},y^{t-1})\}$

These sections can be further precised into subsections and their order is set to vary as it may be suitable to give more or less importance to one section than another for the final model’s response depending on the task at hand. This will be showcased in our two experimented tasks later on. Finally, the prompt builder returns: ${\mathcal{P}_{task}^{t}=\sigma_{task}(\mathcal{I}_{s},\mathcal{C}^{t},\mathcal{I}_{a},\mathcal{X}^{t-1})}$ where $\sigma_{task}$ is the most suitable permutation for the dialogue task at hand. Therefore at each turn, the model maximizes the following probability: $p(y^{t}|x^{t},\mathcal{P}_{task}^{t})$ to respond.

The system shown in Figure 1 is a pipeline of several modules. These include a web interface based on the Rasa X Rasa (2022) tool, modified to integrate voice functionality using the Google Chrome Speech-to-text and Text-to-speech APIs. This enables users to exchange easily with the agent, either by voice (recommended) or by text. Next, a module that constructs the prompt according to the general structure described in 3.2 from the user’s (textual) message and information both external and internal to the conversation. After generation, if the LLM responses are not valid, a filtering module is used to apply corrections before sending them to the user (cf. Annex C).

This task involves enhancing LLM’s conversational capabilities by using roles built from personality traits drawn from the PersonaChat dataset Zhang et al. (2018) embedded in $\mathcal{C}^{t}$ as external information (see conversation example in Appendix D.1).

If a model is capable of interpreting a role in order to embody a character, we can assume that it is also capable of talking about a role without interpreting it. To test the validity of this hypothesis, we propose a prompt designed to enable the LLM to converse about a specific topic. This topic is defined in the form of a role that the LLM will be encouraged to describe rather than interpret.

Collecting human-bot conversations is expensive. For this reason, we generated conversations between two instances, each model + prompt combination (self-chats). Their performance is evaluated in a Chatbot Arena style (Zheng et al., 2023). Evaluators compared two self-chats from different setups on each criterion and in general. The scores presented in Table 1 are Elo scores (Elo, 1978) calculated from the comparisons’ results. A total of 18 annotators evaluated 982 generated conversations of 10 rounds each (which correspond to around 70 dialogues per configuration and 5 to 14 battles per pair).

We can see that larger model sizes and instruction-tuning lead to better performance. On the one hand, Vicuna tops the chart, followed by Guanaco. On the other hand, LLaMA underperforms with the proposed prompt and the FSB prompt. As the Vicuna-33B + Advanced Prompt combination ranked first was too costly (latency, resources) for the collection of human-model conversations, the Vicuna-13B + Advanced Prompt combination was selected for collection and the next round of evaluations⁴⁴4Won 75 % of direct comparisons with Vicuna + Shallow despite being ranked behind overall..

Statistical data was computed for all conversations. Among these, the vocabulary size, which is the number of different lemmas in each message and in the conversation as a whole. All messages were lemmatized with the Spacy library’s morphosyntactic labeling tool (Honnibal et al., 2020) for French ⁵⁵5fr_core_news_sm available at https://spacy.io/models/fr?_x_tr_hist=true#fr_core_news_sm. The number of words per message for each speaker type is also reported to give another view of system performance.

Error occurrences in response generation have been evaluated on 100 self-chats for each setup. Each of these conversations contains 10 turns. Results for both tasks are reported in Table 5. All rates are computed over all turns. Details on errors are given in Appendix C.

Detection rates were calculated for the two prompts in the PersonaChat task. As several errors can occur for the same message in this task, the totals have not been calculated. Additionally, all detected errors are corrected.

On the other hand, errors in the INT task are corrected only if the proposed corrections comply with the response filtering rules (see Appendix C.2). For both types of error, we have calculated separate rates for detected and corrected errors. As these errors cannot occur in the same message, a total rate has been calculated.