Show, Don’t Tell: Aligning Language Models with Demonstrated Feedback

Consider a completion sampled from the expert policy, $y^{E}\sim\pi_{E}(\cdot|x)$. By virtue of being “expert”, $y^{E}$ is likely to have high reward, as $\pi_{E}$ is definitionally the reward maximizer in expectation. Consequently, we would expect samples from any other policy $\pi$ to have rewards less than or equal to those of $\pi_{E}$, i.e., $\forall\pi,\mathbb{E}_{\pi_{E}}[r(x,y)]\geq\mathbb{E}_{\pi}[r(x,y)]$. Using this observation, we can construct comparisons $(x,y^{E},y^{\pi})$ where $y^{E}\succeq y^{\pi}$ by simply sampling completions $y^{\pi}\sim\pi(\cdot|x)$ for every demonstration-prompt pair in $\mathcal{D}_{E}$. Though such comparisons are derived from policies instead of individual examples, they have proven effective in prior work [6]. A naïve approach for DITTO would then optimize Eq. 1 using this dataset and an off-the-shelf RLHF algorithm. Doing so would increase the probability of the expert responses while decreasing the probability of the current model samples, unlike standard finetuning which only does the former. Crucially, using samples from $\pi$ allows us to construct an unbounded preference dataset given only a few demonstrations. However, we can do better by considering the temporal aspect of the learning process.

Using comparisons only between the expert and single policy $\pi$ may be insufficient for obtaining good performance. Doing so decreases likelihoods only at that specific $\pi$, leading to the overfitting problems that plague SFT in low-data regimes. Analogous to replay in RL [29], we can consider data generated from all policies learned over time during RLHF.

At the first iteration, let the initial policy be $\pi_{0}$. We can sample from this policy to assemble a dataset $\mathcal{D}_{0}=\{(x,y^{\pi_{0}})\}$. Then, we can generate comparison data for RLHF as $y^{E}\succeq y^{\pi_{0}}$, which we denote as $\mathcal{D}_{E}\succeq\mathcal{D}_{0}$ for brevity. Using these induced comparisons, we update $\pi_{0}$ to obtain a new policy $\pi_{1}$. By definition, $\mathbb{E}_{\pi_{E}}[r(x,y)]\geq\mathbb{E}_{\pi_{1}}[r(x,y)]$ as well. It follows that we can also generate comparisons using $\pi_{1}$ as $\mathcal{D}_{E}\succeq\mathcal{D}_{1}$. Continuing this procedure, we generate a progressively more diverse comparison dataset using all prior policies. We refer to these as “replay” comparisons.

While this approach is theoretically consistent, it decreases the likelihood of the LM everywhere except at expert demonstrations. Though permissible in data rich scenarios, this may also lead to overfitting with a small $\mathcal{D}_{E}$. However, if we assume that the policy improves at each iteration, i.e. $\mathbb{E}_{\pi_{t+1}}[r(x,y)]\geq\mathbb{E}_{\pi_{t}}[r(x,y)]$, then we can also consider comparisons between policies during the course of learning. Unlike comparisons with the expert, we do not guarantee that this holds; in practice, however, we found that models tended to improve with each iteration, perhaps owing to the convexity of both reward modeling and Eq. 1. This lets us sample comparisons between the complete ranking of policies

The effect of adding these “intermodel” and “replay” comparisons is that the likelihoods of earlier samples (e.g., those in $\mathcal{D}_{1}$) are pushed down more than those of later samples (e.g., those in $\mathcal{D}_{t}$), smoothing the implicit reward landscape. Our practical implementation aggregates a handful of these intermodel comparisons in addition to comparisons with the expert.

In practice, the DITTO algorithm is an iterative procedure comprised of three simple components as outlined in Algorithm 1. First, we begin by running supervised fine-tuning on the set of expert demonstrations for a limited number of gradient steps. We set this to be the initial policy $\pi_{0}$. Second, we sample comparisons: at most $K$ times during the training process, we construct a new dataset $\mathcal{D}_{t}$ by sampling $M$ completions from $\pi_{t}$ for each of the $N$ demonstrations in $\mathcal{D}_{E}$ and add it to the ranking over policies Eq. 2. When sampling comparisons from Eq. 2 each batch $B$ is comprised of 70%“online” comparisons $\mathcal{D}_{E}\succeq\mathcal{D}_{t}$, 20% “replay” comparisons of the form $\mathcal{D}_{E}\succeq\mathcal{D}_{i<t}$, and 10% “intermodel comparisons” of the form $\mathcal{D}_{i\leq t}\succeq\mathcal{D}_{j<i}$. Finally, we update the policy using RLHF. Specifically, using batches sampled via the aforementioned procedure, we update the policy $\pi_{t}$ to obtain $\pi_{t+1}$ using the DPO [34] loss function

where $\sigma$ is the logistic function from the Bradley-Terry preference model. During each update, we do not update the reference model $\pi_{\textrm{ref}}$ from the SFT policy to avoid straying too far from initialization.

The first step in simplifying Eq. 3 is addressing the inner policy maximization. Fortunately, from Ziebart [56] we know that the policy objective $\mathcal{J}_{\text{\tiny{KL}}}$ has a closed form solution of the form $\pi^{\star}(y|x)=\pi_{\textrm{ref}}(y|x)e^{r(x,y)/\alpha}/Z(x)$ where $Z(x)$ is the partition function normalizing the distribution. Notably, this establishes a bijection between policies and reward functions which we can use to eliminate the inner optimization. By rearranging this solution, we can write the reward function $r$ as

Furthermore, prior work [35] shows that this reparameterization can express any reward function. Thus, we can perform a change of variables from $r$ to $\pi$ by substitution into Eq. 3, giving us the DITTO objective

Note that like DPO, we implicitly estimate the reward function. Unlike DPO, DITTO depends on an online dataset of preferences $\mathcal{D}^{\pi}$. At a minimum, the online preference dataset ought to contain comparisons $\pi_{E}\succeq\pi,\forall\pi$. However, any preferences consistent with the ground-truth reward function can additionally be used. We leave this exploration to future work.

One reason for DITTO’s relatively high performance is that it uses far more data than SFT by generating comparisons. Another is that online imitation learning methods can, in some circumstances, perform better than the demonstrator while SFT only mimics the demonstrations. While this is known in the IRL community, we show the following result in Appendix A to relate DITTO’s ability to extrapolate beyond the demonstrator to two divergence measures.

Measuring few-shot alignment with DITTO requires demonstrations from individuals instead of aggregated datasets. We therefore build on prior Author Attribution (AA) datasets. The AA task requires one to determine which author $a$ from a set of authors $A$ wrote a specific document. We can reframe prior AA classification tasks as effective alignment: aligning an LLM to a specific author should result in generations that are more likely to be attributed to the same author. We collect data from 20 distinct authors from two sources: (1) emails and blog posts from the CMCC dataset [17] that contain only one author and (2) news articles from the CCAT dataset [27]. For more dataset details, we refer the reader to Appendix 4.

Some of our benchmarks have more writing samples per author than others. While the original CCAT can have more than 50 samples per author, CMCC can have as few as 12. To control for sample count, we randomly select the smallest set of demonstrations available from each author across our training splits (12) for our experiments. We randomly select 10 authors from each dataset, use 7 samples to train, and split the remainder into test and validation. Table 4 in the Appendix describes the finalized train/val/test counts across each benchmark.

Alongside DITTO, we evaluate continued supervised fine-tuning (SFT), where we simply fine-tune on the expert demonstrations $\mathcal{D}_{E}.$ We also evaluate SPIN [10], an iterative self-play method designed to replace SFT. Finally, we test zero-shot and few-shot prompting, including demonstrations directly in the model’s context. For few-shot prompting, we add the train set of an author in-context. Our experiments require a base, instruction following LLM. We use Mistral Instruct v0.2 7B as a starting point [23] and train using LoRA [22]. Finally, we compare against zero/few-shot prompting with a more powerful LLM (GPT-4). Hyperparameter details are in Appendix C.

Given that our datasets contain a total of 20 authors, we must train and evaluate a large set of models (20 authors x 7 training paradigms = 140 models). To facilitate the evaluation process, we use GPT-4²²2We use the gpt-4-0613 version of GPT-4. We observed that Turbo versions of GPT-4 were more biased towards their own outputs. Queries were run between December 20th, 2023 to May 10th, 2024. to compare the outputs of models across various conditions. Prior work has used GPT to both annotate and evaluate text [53]. Performance lags behind human evaluation; however, GPT-4 eval generally outperforms other automatic metrics, allowing us to scale hyperparameter search and run evaluation in a more cost-effective manner.

In our setting, we use GPT-4 to determine if a text sounds more or less like a specific author. Given an author-written text $t$ and two pairs of generated text from different conditions $a$ and $b$, we prompt GPT-4 to select the text that most closely matches the validation or test text $t$, and compute averaged head-to-head win rates. Prompting details are outlined in Appendix D.

Our main results, evaluated with GPT-4 eval, are summarized in Table 1. Averaged across all authors, DITTO outperforms all baselines, with an average 77.09% win-rate across both CMCC (71.67%) and CCAT50 (82.50%). On CCAT50, DITTO outperforms all baselines across authors but one. On CMCC, DITTO outperforms all other baselines for 5/10 authors, followed by few-shot prompting for 3/10. While SFT serves as a strong baseline (56.78% on CMCC, 73.89% on CCAT), DITTO provides an average $\uparrow$11.7% pt. win rate improvement compared to SFT alone.

Prompted baselines also lag far behind DITTO, especially zero-shot (including closed-source) models (avg. $\downarrow$54.4% pt. decrease on Mistral, $\downarrow$51.5% pt. on GPT-4). While zero-shot GPT-4 is already finetuned using RLHF, we suspect that this training feedback differs significantly from that of authors in both CMCC and CCAT50. Adding few-shot examples to the prompt does help: win rates for few-shot prompting increase compared to zero-shot for both Mistral ($\uparrow$20.94% pt.) and GPT-4 ($\uparrow$22.95% pt.) based LLMs. However, including few-shot examples still falls behind applying DITTO (avg. $\downarrow$37.35% pt. decrease for Mistral; $\downarrow$26.99% pt. for GPT-4). We suspect the underlying RLHF priors for out-of-the-box LLMs are fairly strong. Qualitatively, few-shot generations still sound GPT-generated relative to DITTO (Table 6 in Appendix).

While we do test another self-improvement training method (SFT + SPIN), we find that performance is lower than DITTO (avg.$\downarrow$ 9.3% pt.)—we suspect that design decisions for SPIN (e.g., updating the reference policy, excluding interpolicy / replay comparisons) are targeted towards SFT-scale datasets. We ablate these decisions in §5.1 and propose reasons for performance degradation.

The user study consists of two parts. In the first part, we ask participants to specify four email-writing tasks (e.g., Write an email to your advisor asking for feedback). Participants are asked to provide two demonstrations for two of the tasks (4 training demonstrations in total). To help brainstorm tasks, we generate concrete task suggestions with GPT-4; participants could select from among these or provide their own custom tasks. We randomly split two task prompts into train, and saved two for testing; participants gave two demonstrations each for both the training prompts, to mimic a user willing to only put in minimal effort. Users were provided with default generations from GPT-4 to aid authoring demonstrations, which they could edit or ignore. In the second part, we use the two tasks from the test set and show participants generations across all methods. We sampled one output from each method (self-prompt, zero-shot, few-shot, SFT, and DITTO), and solicited 10 pairwise preferences for each test prompt (resulting in 20 preferences total for each user). In all, we collect a total of 320 pairwise preferences across 16 users. Additional user study details (e.g., interface, examples of demonstrated feedback, prompts for generating tasks, etc.) are in Appendix E.

Our user study results corroborate findings from static benchmarks. DITTO outperforms baseline methods in aligning to demonstrated preferences (Table 2), with DITTO (72.1% win-rate) > SFT (60.1%) > few-shot (48.1%) > self-prompt (44.2%) > zero-shot (25.0%). Additionally, users generally struggle with verbalizing preferences into prompts: self-prompting slightly underperforms providing demonstrations in a few-shot prompt, and substantially underperforms DITTO. We also qualitatively observe that users often edit nearly half of the default output from GPT-4 when authoring demonstrations (examples in Appendix E), with average normalized Levenshtein edit distance = 0.43. Large edits to the output alone highlight the effectiveness of demonstrated feedback as an interaction.