Triple Preference Optimization:
Achieving Better Alignment with Less Data in a Single Step Optimization

LLMs are trained across a wide array of tasks, demonstrating their remarkable versatility in solving diverse tasks Brown et al. (2020); Narayanan et al. (2021); Bubeck et al. (2023). However, their training on data of varying quality can lead to many issues, such as the generation of toxic or harmful text under certain contexts Perez et al. (2022); Ganguli et al. (2022), and in general, the generation of outputs that are not desired by humans. Hence, it is crucial to align LLMs with human expectations and preferences that prioritize their helpfulness, honesty, and harmlessness Bai et al. (2022).

Supervised Fine-Tuning (SFT) is a direct alignment method that involves fitting a model to human-written data Sanh et al. (2022). However, this approach fails to fully impart the human perspective to the model. During training, the model only receives a reference response for each input, thus lacking exposure to incorrect answers and preferences, which ultimately constrains its performance on downstream tasks Touvron et al. (2023).

A prominent method in AI alignment for LLMs is Reinforcement Learning with Human Feedback (RLHF) Ouyang et al. (2022). Despite its impressive performance relative to SFT, RLHF faces limitations such as instability and susceptibility to reward hacking Liu et al. (2024). Consequently, a recent approach called Direct Preference Optimization (DPO) Rafailov et al. (2023) has emerged. DPO is an RL-free method that directly optimizes human preferences by shifting from RL to simple binary cross-entropy. However, DPO encounters several limitations: 1) high dependency on the SFT part Tunstall et al. (2023), 2) tendency to overfit beyond a single epoch Azar et al. (2023), and 3) inefficient learning and memory utilization Xu et al. (2024).

To address these limitations, Various alignment methods have been proposed for dialogue systems Tunstall et al. (2023), harmful and helpfulness question answering Wu et al. (2023), summarization Zhao et al. (2023), and translation Xu et al. (2024) and all these studies include a separate SFT component. During SFT, models are fine-tuned to generate appropriate responses to the corresponding input prompts. Meanwhile, in DPO, models are fine-tuned to enhance the likelihood of generating preferred responses over less desirable ones and not to stray far away from the SFT model Rafailov et al. (2023).

In this paper, we introduce the Triple Preference Optimization (TPO), a new preference learning approach. In TPO, we combine the two separate optimization steps (supervised fine-tuning and preference learning) into a single step based on Pareto Front concept Lotov and Miettinen (2008), with the training data having both the gold standard response (as in SFT) and the preferences (as in PPO/DPO) in a consolidated format. Thus, our training data will be of the form (input prompt, gold standard response $(y_{ref})$, preferred response $(y_{w})$, less-preferred response $(y_{l})$). Specifically, we jointly optimize a policy model with $-\mathbb{E}_{\left(x,y_{ref}\right)\sim\mathcal{D}}\left[\log\pi_{\theta}\left(y_{ref}\mid x\right)\right]$ and $\begin{aligned} -\mathbb{E}_{\left(x,y_{w},y_{l}\right)\sim\mathcal{D}}\left[\operatorname{log}\sigma\left(\beta\log{\pi_{\theta}\left(y_{w}\mid x\right)}\right.\right.&\left.\left.-\beta\log{\pi_{\theta}\left(y_{l}\mid x\right)}\right)\right]\end{aligned}$ in one step (See Figure 1).

Our results show that TPO exhibits impressive performance compared to SFT across various benchmarks and outperforms other alignment methods such as DPO. Specifically, Mistral (7B), fine-tuned by TPO and trained with six times less data than other alignment techniques, outperforms SFT, DPO, KTO, IPO, CPO, and ORPO across nine benchmarks on the Open LLM Leaderboard. Notably, Mistral aligned with TPO achieved a +0.72 increase in the MT-Bench score over SFT.

Overall, TPO addresses two key shortcomings in alignment tasks. Firstly, by removing $\pi_{ref}$ justified in Section 3, TPO mitigates the inefficient learning and memory utilization issues observed in DPO, IPO, and KTO, allowing for more computational efficiency with less memory usage. Secondly, TPO enhances performance over SFT and other alignment methods by maximizing the likelihood of gold response, regularized by parameter $\boldsymbol{\alpha}$. and simultaneously optimizing between two preferences (preferred and less-preferred responses). Despite TPO’s need for three preferences and its higher cost relative to other methods, our findings reveal that it’s possible to considerably lessen the training data required and still achieve superior outcomes (See Table 1).

Our findings suggest that a separate SFT step is not necessary for TPO and, in certain scenarios, having one may even hinder TPO’s performance (See Tables 1 and 2).

We summarize our primary contributions as follows:

1. 1.

   We propose a new preference learning method called Triple Preferences Optimization (TPO) that simplifies the alignment process and reduces two stages to one stage.

2. 2.

   Theoretically, we derive the TPO objective and show that combining the human expectation data and preference dataset achieves better performance.

3. 3.

   Comprehensive experiments reveal that the TPO method, applied to two distinct baseline models—Mistral (7 B) and Phi-2 (2.7 B)—outperforms SFT, KTO, IPO, DPO, CPO, and ORPO in terms of performance across ten different benchmarks (refer to Tables 1, 2, and 3).

4. 4.

   Integrating the SFT step with the preference alignment step and moderating it with a regularization parameter ($\alpha$) enhances the model’s performance while reducing the data required for training (See Figure 3).

The performance of Large Language Models (LLMs) on a variety of tasks is remarkable Anil et al. (2023). Nonetheless, effectively aligning LLMs remains a significant challenge. Current studies have fine-tuned LLMs using datasets of human preferences, leading to improvements in translation Kreutzer et al. (2018), summarization Stiennon et al. (2022), story-telling Ziegler et al. (2019), instruction-following Ramamurthy et al. (2023), and dialogue systems.

RLHF Christiano et al. (2023), introduced in the literature, aims to optimize for maximum reward by interacting with a reward model trained using the Bradley-Terry (BT) model Bong and Rinaldo (2022), typically through reinforcement algorithms like Proximal Policy Optimization (PPO) Schulman et al. (2017). While RLHF enhances model performance, it faces challenges such as instability, reward hacking, and scalability inherent in reinforcement learning. Recent works have presented techniques to overcome these challenges by optimizing relative preferences without relying on reinforcement learning. Utilizing the Bradley-Terry (BT) model to optimize a model on preference datasets is instrumental in ensuring alignment with human preferences.

SLiC Zhao et al. (2023) introduced a novel method for ranking preferences generated by a supervised fine-tuned (SFT) model, incorporating calibration loss and regularization fine-tuning loss during training. Meanwhile, RRHF Yuan et al. (2023) trains the SFT model using a zero-margin likelihood contrastive loss, assuming multiple ranked responses for each input. While both SLiC and RRHF are effective, they lack theoretical foundations. In contrast, DPO offers a method to directly fit an SFT model to human preferences using the Bradley-Terry (BT) model, providing theoretical insights into the alignment process.

RSO Liu et al. (2024) merges the techniques of SLiC and DPO while introducing an improved approach for collecting preference pairs through statistical rejection sampling. IPO Azar et al. (2023) has mathematically revealed the limitations of the DPO approach concerning overfitting and generalization. It proposes a comprehensive objective for learning from human preferences. Zephyr Tunstall et al. (2023) has improved DPO by utilizing the distillation method.

KTO Ethayarajh et al. (2023), drawing inspiration from Kahneman and Tversky’s influential work on prospect theory Tversky and Kahneman (1992), seeks to maximize the utility of LLM outputs directly rather than optimizing the log-likelihood of preferences. By prioritizing the determination of whether a preference is desirable or undesirable, this method eliminates the requirement for two preferences for the same input.

Recently, CPO Xu et al. (2024) introduced an efficient method for learning preferences by combining maximum-likelihood loss with the DPO loss function, aiming to improve memory usage and learning efficiency. Additionally, ORPO Hong et al. (2024) proposed a novel approach by incorporating a penalty term to prevent the learning of unpreferred responses while enhancing the likelihood of learning preferred responses.

We observe two primary challenges in the alignment process addressed by the aforementioned studies. Firstly, alignment methods such as DPO require an SFT part or have better performance with an SFT part. Secondly, there are concerns regarding inefficient learning and memory usage. While the CPO has proven to be an effective learning approach, a conflict between its objectives may restrict the policy model’s performance. In this research, we investigate these limitations and seek to introduce a new algorithm to address them.

In this section, we introduce Triple Preference Optimization (TPO), a new approach to preference learning. This method optimizes a policy model ($\pi_{\theta}$) by maximizing the likelihood of the gold response and optimizing for the preferences simultaneously.

Typically, in NLP tasks, we utilize a dataset $D_{reference}=\{x^{i},y_{ref}^{i}\}^{N}_{i=1}$, where $x$ is the input and $y_{ref}$ is the gold standard response, crafted by humans or large models like GPT-4 and validated by humans. Additionally, for applying preference optimization methods, a dataset $D_{preference}=\{x^{i},y_{w}^{i},y_{l}^{i}\}^{N}_{i=1}$ is needed, where $y_{w}$ and $y_{l}$ are the preferred and unpreferred responses respectively, generated by smaller models such as LLaMA-3. The aim of TPO is to optimize three preferences concurrently. To achieve this, we merge the $reference$ and $preference$ datasets into one dataset $D_{TPO}=\{x^{i},y_{ref}^{i},y_{w}^{i},y_{l}^{i}\}^{N}_{i=1}$, establishing a response hierarchy of $y_{ref}\succ y_{w}\succ y_{l}$. Further details on the TPO objective will be discussed in the following subsection.

In this section, we present a comprehensive empirical analysis of TPO, yielding several key findings: 1) Phi-2+TPO and Mistral+TPO trained on 10K data outperform Phi-2+SFT and Mistral+SFT trained on 200K data by 12.7% and 7.2% on MT-Bench respectively. 2) Phi-2 fine-tuned with TPO surpasses the performance of models aligned with other methods on the MT-Bench. 3) Similarly, Mistral fine-tuned with TPO exceeds the performance of other alignment techniques across the majority of Open LLM Benchmarks. 4) Within the TPO method, the hyper-parameters $\alpha$ and $\beta$ play a critical role in influencing performance outcomes. 5) An ablation study focusing on batch size adjustments reveals that enlarging the batch size leads to improved performance for models optimized with TPO.

In this paper, we begin by addressing the limitations inherent in existing alignment methods. Typically, alignment techniques require an SFT component to achieve notable results. However, incorporating SFT introduces two primary challenges: firstly, fine-tuning a model using SFT demands a substantial dataset (for example, completing a chat task may require fine-tuning with 200K data points). Secondly, generating a preferences dataset by sampling from the SFT model poses additional difficulties, including determining the optimal configuration for producing preferred and less preferred responses. To mitigate these shortcomings, we introduce TPO, a new alignment approach aimed at concurrently optimizing for human preferences and gold responses. Our findings demonstrate the impressive performance of TPO compared to other alignment methods on ten benchmarks. Particularly, Mistral and Phi-2 fine-tuned by TPO achieve increases in the MT-Bench score of +0.72 and +1.27, respectively, compared to SFT, despite being trained on a dataset six times smaller. Another intriguing insight is the significant influence that the values of $\alpha$ and $\beta$ have on the model’s performance.

While TPO has demonstrated impressive performance compared to other alignment methods across various benchmarks, the requirement to prepare three preferences for each input in a dataset poses challenges. In this section, we outline potential directions for future work. Our evaluation of TPO focused on chat completion tasks, but we are particularly interested in examining its effectiveness in other areas, such as safety and reasoning. Another intriguing aspect for further study is investigating how the quality of reference and preferred responses affects TPO’s performance. Notably, our current findings suggest that the reference response is generally better than the preferred response. Investigating whether increasing the preferential difference between these responses enhances performance could yield valuable insights. Additionally, we are interested in exploring TPO’s effectiveness in larger models, such as those with 30 B or 70 B, which represents a promising avenue for future work. Drawing inspiration from the new method proposed in Chatterjee et al. (2024) for fine-tuning diffusion models, we are keen to investigate how these models perform when aligned using the TPO method.

We thank the anonymous reviewers for constructive suggestions and the Research Computing (RC) at Arizona State University (ASU) for providing computing resources for experiments. We acknowledge support by a 2023 Spring Amazon Research Award (ARA).