CoachLM: Automatic Instruction Revisions Improve the Data Quality in LLM Instruction Tuning

Before the primary revision, experts from group A conducted a preliminary filtering on the sampled 6k instruction pairs to exclude unsuitable pairs. As shown in Table III, a total of 1088 pairs were excluded, mainly due to missing or invalid key parts, excessive expertise or workload requirements, inclusion of unsupported multi-modal information, and overly toxic or sensitive content. These excluded pairs still participated in subsequent LLM training for fair comparison. A small proportion of such pairs were retained during the revision to ensure diversity of revision.

After excluding the 1088 filtered instruction pairs, the remaining 4.9k instruction pairs underwent the primary revision. To ensure an effective revision process, we adopted an expertise-based approach to assign instruction pairs to experts [28, 29]. Based on the categories proposed in [15], the instruction pairs were classified into three classes representing different levels of difficulty (i.e., expertise required) for revision. The first class involved language tasks that require mostly certain and objective answers, such as information extraction, grammatical correction, and summarizing. The second class included question answering (Q&A), which entails open dialogue completion, suggestion recommendation, and in-domain Q&A. Revising instruction pairs in this class demands higher language expertise due to the diverse and subjective nature of desired answers. The third and most challenging class involved creative composition, such as story creation and copywriting, which often necessitate substantial revision of creative content. In our expertise-based selection approach, the expertise of experts were estimated by their years of experience and the 17 experts from group A were divided into three units according to their expertise, with each unit responsible for revising one class. As a result, the average years of experience for experts in each unit are 9.4 years for language task performing, 11.2 years for Q&A, and 13.1 years for creative composition.

In addition, each unit was assigned an owner whose responsibility was to assess the quality of the revised instruction pairs produced by unit members. The revision process strictly adhered to the criteria outlined in Table II, following the principle of “making all necessary revisions,” regardless of the importance of the revised dimensions. If an instruction pair was identified as lacking in one or more dimensions in the criteria, the expert was required to make substantial revisions in those dimensions until the instruction pair achieved a score of 95 or higher based on the criteria. Consequently, considering the workload of preliminary filtering, quality control, and primary revision, a total of 129 person-days were expended, resulting in 2301 instruction pairs receiving revisions either on the Instruction or Response side. Among the 2.3k revised pairs, 1079 of them underwent revisions on Instruction.

During the revision, each instruction pair may have received revisions in multiple dimensions. The revised instruction pairs were categorized based on the primary type of revisions they underwent, and the distribution of each revision category is displayed in Table IV. For revisions on the Instruction side, approximately 68.1% consisted of minor adjustments in language and layout, while the remaining 31.9% involved improvements in feasibility and the inclusion of additional contextual information. As for Responses, the most common types of revisions comprised expanding the depth of the response or providing necessary supporting explanations, accounting for 43.7% of the revisions. Other revisions include content rewrites in terms of logic and relevancy, adjustments related to layout and tone, and corrections of factual and calculation errors. In order to ensure a diverse range of revisions, approximately 1.9% of the revisions were cases that should have fell into the categories listed in Table III. See more analysis details from the technical report in our repository.

CoachLM is trained by taking content revision as a type of instruction, which LLMs can follow via instruction tuning. Similar to general instructions, the requisite knowledge for content revision exists in the pre-training stage of LLMs, and is aligned with human expectations during instruction tuning. For instance, content-revision instructions found in the Alpaca52k dataset, such as “correct the grammatical errors in the sentence”, elicit the basic capacity of instruction-tuned LLMs like Alpaca to engage in content revision. Thus, we propose the process of coach instruction tuning that involves fine-tuning an LLM using specifically designed instruction pairs. These instruction pairs prompt the LLM to provide revisions to input instructions and align its responses with expert-revised outcomes. Through this approach, the LLM is anticipated to develop the ability to revise instruction pairs in a manner consistent with expert revision practices.

Specifically, given an instruction dataset $V$ of instruction pairs $x$ = (Instruction, Response) with $x\in V$, each instruction pair $x$ undergoes a revision through the expert revision process, resulting in a revised instruction pair $x_{r}$. The expert revision dataset $R$ is then formed, which comprises both the original and revised instruction pairs, denoted as $R=\{(x,x_{r})\ |\ x\in V\}$. During the coach instruction tuning process, each $(x,x_{r})\in R$ is leveraged to construct an instruction pair $x_{c}$, leading to an instruction dataset $C=\{x_{c}\ |\ x\in V\}$.

As shown in Fig. 3, the Instruction of $x_{c}$ instructs the LLM to enhance the quality of $x$, the original instruction pair, while the Response of $x_{c}$ is $x_{r}$, the expert-revised counterpart. When designing the Instruction component, we provide a succinct revision instruction that highlights the primary areas for revision based on the expert revision results. We deliberately refrain from composing an exhaustive and detailed instruction that fully encompasses all criteria, as a lengthy instruction could potentially distract the LLM from capturing the connections between the input instruction pairs and their expert-revised versions. Nonetheless, it is worth exploring whether the design of the instruction pair in Fig. 3 is optimal in future research.

Given an LLM with parameters $\theta$ as the initial model for coach instruction tuning, training the model on the constructed instruction dataset $C$ results in the adaption of the LLM’s parameters from $\theta$ to $\theta_{c}$, denoted as CoachLM. Specifically, $\theta_{c}$ is obtained by maximizing the probability of predicting the next tokens in the Response component of $x_{c}$, conditioned on the Instruction of $x_{c}\in C$, which is formulated as:

In the pre-LLM era, models were required to learn both task-specific knowledge and the alignment between task input and desired output. This is why training on negative samples was sometimes beneficial, as it provided the model with supplementary knowledge and boundaries for the task-specific information [19]. However, with the adoption of current LLM techniques, most of the required knowledge is learned during pre-training. Numerous pieces of evidence suggest that when fine-tuning an LLM through instruction tuning, the introduction of low-quality instruction pairs actually hinders the performance of the tuned LLM [20, 19, 18, 30]. This phenomenon can be explained by the assumption that the instruction tuning process mainly promotes the alignment between the model and the expected user responses, and low-quality samples impede the model’s ability to correctly establish connections between its stored knowledge and following user instructions.

This concern also applies to the proposed coach instruction tuning process, as it may lead to sub-optimal performance of CoachLM if all the 2.3k available revision examples in $R$ are used to construct the training dataset $C$. Although the expert revision process includes a quality control stage that ensures each revised instruction pair $x_{r}$ meets the criteria in Table II, the original instruction pair $x$ may still influence the overall quality of the constructed instruction pair $x_{c}$. If $x$ is already in good shape, only minor revisions are made to obtain $x_{r}$. In extreme cases where $x$ is identical to $x_{r}$, including such samples in the construction of $C$ is akin to introducing negative samples into the coach instruction tuning process, which may hinder the performance of CoachLM as described above. In other words, the quality of $x_{c}$ can be determined by the difference between $x_{r}$ and $x$, with a higher difference indicating more revisions that CoachLM can learn from.

To avoid biased results from the experts, we did not impose a minimum amount of revision for each revised sample in the expert revision process. Instead, we employ the edit distance metric to assess the quality of $(x,x_{r})\in R$ and define $\alpha$, the human input ratio, to determine the final subset of samples used in $C$. The edit distance, also known as the Levenshtein distance, quantifies the minimum number of single-character edits needed to transform one string into another [31]. The edit distance reflects the difference between $x$ and $x_{r}$, thereby measuring the quality of $x_{c}$. Then, by defining a ratio $\alpha$ between 0 and 1, we can ensure that $C_{\alpha}$ comprises human input samples from $R$ with the highest $\alpha$ proportion of edit distances. By replacing $C$ with $C_{\alpha}$ in Eq. (1), we obtain a CoachLM trained with a high-quality subset of the constructed instruction dataset $C$.

Through coach instruction tuning, CoachLM generates automatic revisions on input instruction pairs, creating a CoachLM-revised instruction dataset. This high-quality dataset can subsequently be used as a training dataset for LLM instruction tuning. Let $D$ represent an input instruction dataset (e.g., the Alpaca52k dataset), consisting of instruction pairs $x$. Each $x\in D$ is combined with the revision prompt shown in Fig. 3 to form an instruction pair $x^{\prime}_{c}\in D^{\prime}$, with an empty Response to be filled by CoachLM. The CoachLM-revised instruction dataset, denoted as $D_{c}$, is obtained by applying $\theta_{c}$, the CoachLM, on $D^{\prime}$:

In the experiment, a comprehensive evaluation of CoachLM is conducted using both automatic and human approaches, as shown in Table V.

As shown in Table VI, in addition to the CoachLM150 test set, we also utilize three popular public LLM test sets in our experiments, namely the Self-Instruct252 test set [14], the PandaLM170 test set [24], and the Vicuna80 test set [16]. The Self-Instruct252 test set was curated by Wang et al., who provided instructions under various application scenarios such as Gmail, Twitter, and Github, along with human responses. The PandaLM170 test set was created by sampling instructions from the Self-Instruct252 test set, with reference responses generated by ChatGPT. The Vicuna80 test set comprises instructions related to writing, role-play, math, and knowledge, for which the responses from Bard were used as reference responses due to the absence of human responses.

The experiments were conducted using 8 NVIDIA A100 GPUs. We explored different backbone models $\theta$ and different $\alpha$ values for CoachLM. In our main experiment, we used ChatGLM2 [32] as the backbone model, which has 6B parameters, and set $\alpha$ to 0.3. To efficiently adapt the backbone LLMs, we employed LoRA [33], a partial fine-tuning technique. See detailed parameter settings in our repository. CoachLM was trained for seven epochs with a learning rate of $2\times 10^{-4}$. For training the instruction-following models, we utilized the same settings as the official Alpaca repository⁷⁷7https://github.com/tatsu-lab/stanford_alpaca, with the exception of using different instruction datasets. During the inference stage, the beam size for decoding was set to one for all models.

By inputting every instruction pair from the Alpaca52k dataset into CoachLM for revisions as described in Eq. (2), a CoachLM-revised Alpaca52k dataset was obtained. We performed automatic post-processing on the outputs of CoachLM using regular expressions to remove invalid characters and repeated strings that were occasionally produced. Approximately 1.3% of the outputs were not valid instruction pairs and were replaced with the original instruction pairs. To avoid data leakage, instructions appeared in the training of CoachLM were kept from the inference and the original samples were directly adopted, which accounted for around 1.3% as well. Three examples revised by CoachLM are shown in Fig. 2.

Table VII presents the statistics of the Alpaca52k dataset before and after revision, including the average length and average edit distance at the word-level. The CoachLM-revised dataset showed significant revisions on Responses in most instruction pairs and resulted in longer responses on average compared with the original dataset, indicating the addition of substantial new content in the revised responses. In contrast, only around 8k instruction pairs exhibited revisions on Instructions. The relatively small number of revisions and nearly unchanged average length suggest that CoachLM primarily adjusted the logical and linguistic aspects of the Instructions without adding much new content.

As described in Section III-A1, ChatGPT is employed to rate the accuracy of each Response on a scale of 0-5 [20], which we utilized as an automatic quality metric for the entire dataset. Fig. 4 illustrates the significant improvement in the average rating of responses in the Alpaca52k dataset, rising from 3.95 to 4.31 after the revision by CoachLM. The original dataset had only 17.7% (around 9k as reported in [20]) of instruction pairs with a rating above 4.5. However, this ratio increased significantly to 78.9% in the CoachLM-revised dataset. This enhancement indicates that instead of refining the Alpaca52k dataset by discarding a majority of samples, the CoachLM-revised dataset predominantly consists of high-quality instruction pairs. As a result, it can positively impact the instruction tuning of LLMs, while preserving the integrity of the original dataset.

Since the evaluation approach of ChatGPT only covers Responses, we performed a human evaluation to assess the quality of both the Responses and Instructions, as described in Section III-A1. To achieve this, we randomly selected 150 instruction pairs from the revised dataset and obtained ratings from three independent reviewers who were unaware of the sample sources. Among these pairs, 18 had modifications in terms of Instructions made by CoachLM. The results, presented in Table VIII, indicate that after the revision by CoachLM, both the Instructions and Responses received higher average scores according to all three reviewers. Notably, the improvement in Responses was more pronounced for the 18 samples with modified Instructions compared with the entire subset, implying the importance of a feasible and accurate Instruction in enhancing the quality of Response.

In addition to automatic evaluation, human reviewers independently rated the responses generated by Alpaca-CoachLM and the original Alpaca model in the CoachLM150 test set. The reviewers were unaware of the sources of the responses. As shown in Table X, all reviewers consistently gave Alpaca-CoachLM a higher average score (ranging from 58.6 to 64.3) compared with the original Alpaca model. This improved performance of Alpaca-CoachLM further confirms the effectiveness of the revisions made by CoachLM, which successfully enhance the instruction-following ability of subsequently tuned LLMs by optimizing the quality of the underlying instruction dataset.