Beyond Performance:
Quantifying and Mitigating Label Bias in LLMs

When employing LLMs for classification tasks through prompting, the model is given a test example $x$, preceded by a context $C$. This context can contain a (potentially empty) set of examples of the task’s input-output map** $[(x^{1},y^{1}),\ldots,(x^{k},y^{k})]$, henceforth demonstrations, and may also include task instructions. To determine the model’s prediction from a set of answer choices $Y$, the likelihood it assigns to each continuation $y\in Y$ is computed, and the highest probability option is taken as the model prediction:

These output probabilities often exhibit label bias, where the model tends to assign higher probability to certain answers regardless of the input test example $x$ (Fig. 1). Multiple factors were posited to influence this bias, including the choice of verbalizers $Y$, the choice and order of in-context examples in $C$, and the overall textual features of task input $x$ Zhao et al. (2021); Fei et al. (2023).

The predominant approach to alleviate label bias is to calibrate the model’s output probabilities post-hoc, for a specific context prompt $C$. Such methods typically first estimate the model’s label bias using its output probabilities on a set of inputs, which can be content-free (e.g., “N/A” or random words from the task’s domain; Zhao et al. 2021; Fei et al. 2023) or ordinary task inputs Han et al. (2023). Next, calibration parameters are chosen based on this estimate, and used to adjust the original output probabilities during inference to generate the (hopefully unbiased) output.

Most LLM label bias analysis relies on indirect assessments. For instance, some work inspected improvements in overall performance gained after applying techniques to mitigate it Fei et al. (2023); Holtzman et al. (2021); Zhao et al. (2021). However, these do not indicate the extent of bias originally present, or that which remains after mitigation. We next examine approaches to measure this bias more directly, and define the metrics we use in this work. Importantly, we focus on label bias measures that could be used effectively both before and after applying mitigation techniques such as calibration.

Drawing from previous research on fairness and bias in machine learning, we observe that there are two distinct yet related aspects in which label bias can be measured in LLM predictions: through the probabilities assigned by the model to different answers, e.g., assigning the label “yes” with an average output probability of 0.55, while “no” with 0.45; and through the model’s predictions for different labels, e.g., achieving a recall of 0.50 for instances labeled “yes”, compared to 0.40 on “no” Mehrabi et al. (2021). Below we describe methods to measure each of these notions of bias.

We evaluate models on 279 diverse tasks from the Super-NaturalInstructions benchmark Wang et al. (2022). We select all available classification and multi-choice question answering tasks where the output space is a set of predefined labels, such as “yes/no” or “A/B/C”. We sample 1,000 evaluation examples for all tasks with larger data sizes, and additionally sample 32 held-out examples for computing the bias score metric (§2.3), and 64 more examples to use as a pool of instances for choosing in-context demonstrations and LoRA fine-tuning examples. We only include tasks with at least 300 evaluation examples in our experiments. For details on the selected tasks, see App. B.

We experiment with models of different sizes from three LLM families: Llama-2 7B and 13B Touvron et al. (2023), Mistral 7B Jiang et al. (2023a), and Falcon 7B and 40B Penedo et al. (2023). We use both the base and instruction fine-tuned versions of each model. We evaluate models using context prompts with $k\in\{0,2,4,8,16\}$ demonstrations, and average the results across $3$ different sets of demonstrations for each $k$. To control the evaluation budget, we run the more expensive Falcon 40B experiments with $k\in\{0,8,16\}$ averaged across $2$ sets of demonstrations. We use the task instructions and prompt template defined in Super-NaturalInstructions. For tasks where the answer choices $y\in Y$ have unequal token lengths, we use length-normalized log-likelihood to compute the output probabilities Holtzman et al. (2021). For additional implementation details, see App. A.

We evaluate the effects of three label bias mitigation methods: two calibration methods designed to correct a model’s label bias by adjusting its output scores; and few-shot LoRA fine-tuning Hu et al. (2022), which adapts the model to the task and its label distribution. We describe the methods below.

We begin by examining the performance and label bias of models with and without instruction-tuning. We report averaged results across all tasks for Llama-2 models in Fig. 2. Results for other models show similar trends (see App. C.1).

We first verify that, as expected, model performance (Fig. 2(a)) substantially improves with scale, with instruction tuning and with the number of demonstrations. We then consider the two bias metrics—RSD (Fig. 2(b)) and BiasScore (Fig. 2(c)). We observe that label bias is substantial across most evaluated settings: When prompted with two or no demonstrations, all models obtain high RSD values of 0.6 or more, with base models obtaining even higher values around 0.9. This implies a widespread disparity in model performance across classes in many of the evaluated tasks, and indicates that for most tasks, models primarily succeed on instances of certain classes, while consistently failing on others. Increasing the number of demonstrations to 8 helps reduce the bias, but RSD remains substantial at around 0.4, and adding further demonstrations results in little to no improvement.

Similarly, we find BiasScore improves considerably when using sufficient demonstrations, with models obtaining values as high as 0.25 when using no demonstrations, to around 0.05 for the best model and setting. High BiasScore values indicate the model is uncalibrated, and tends to make overly confident predictions on certain labels regardless of the input. Although BiasScore can be relatively small for some models—indicating their average output distribution is close to uniform—when observed together with high RSD, it implies that the model subtly but persistently assigns more probability mass to the preferred labels, resulting in substantial bias.

We further observe that, interestingly, both bias metrics show divergent trends. Although RSD values, much like model performance, sharply improve after instruction-tuning, the resulting models’ BiasScore is often higher than their base counterparts. Similarly, while RSD improves with scaling, the BiasScore of smaller models is lower.

We note that higher performance together with lower RSD means that the model’s performance has improved across most classes. In contrast, higher BiasScore indicates that its average predicted probabilities grew farther than uniform. Taken together, this implies that the scaled-up and instruction-tuned models are making more confident predictions on some classes, but not on others. This could mean more confident correct predictions on the preferred classes, or more confidently wrong predictions on others (or both). Altogether, this suggests more subtle forms of bias persist after instruction-tuning or scaling up Tal et al. (2022).

Overall, we find the two metrics to be complimentary due to their measurement of different aspects of label bias. We hence use both in further experiments to provide a more comprehensive understanding of label bias in model predictions.

We have seen that LLMs demonstrate extensive label bias across different models, scales and tasks (§4.1). We next examine techniques aimed at mitigating such bias, and assess the extent of label bias remaining after their application. We report our results for Llama-2 models in Fig. 3, and observe similar trends for other models (App. C.2).

We first consider the effect of bias mitigation on model performance (Fig. 3(a)) using the three methods described in §3.3: contextual calibration (CC), domain-context calibration (DC), and few-shot fine-tuning with LoRA. Compared to standard prompting (black lines), we find that applying CC (orange) provides little to no gains. Moreover, it can even undermine model performance, especially for instruction-tuned models, as previously observed by Fei et al. (2023). In contrast, DC (purple) can provide substantial performance gains, especially when using few or no in-context demonstrations, where baseline performance is relatively low. However, when calibrating instruction-tuned models prompted with a higher number of demonstrations, we find that DC mostly fails to improve performance. Finally, LoRA considerably improves performance in all cases (green in Fig. 3, upper row), vastly outperforming both CC and DC.

We next turn to measure label bias (Fig. 3(b) and 3(c)). Notably, here we observe that for the two calibration methods, changes in both RSD and BiasScore are correlated with changes in performance. We find that CC substantially worsens label bias in instruction-tuned models, and can also increase bias for base models. Conversely, while DC alleviates bias in many of the evaluated settings, it is largely unsuccessful in mitigating it when prompting instruction-tuned models with $8$ or more demonstrations. LoRA proves effective for improving RSD in all settings, but RSD values still remain relatively high. In contrast, BiasScore noticeably increases after LoRA fine-tuning, indicating that more subtle bias persists.

Overall, our results indicate that existing bias calibration approaches are insufficient for diminishing label bias in essential cases, particularly for instruction-tuned models. Further, while LoRA fine-tuning is effective in both improving performance and mitigating certain aspects of bias (though not others), it is also considerably more computationally expensive than calibration.

The output space for classification tasks often consists of labels with strong semantic meaning (e.g., “Positive” vs. “Negative”). Recent work has indicated that, when faced with such labels, models are affected by semantic priors from their pretraining or instruction-tuning Wei et al. (2023); Min et al. (2022b) that could affect label bias Fei et al. (2023).

We examine whether models exhibit lower label bias when the task’s labels are semantically equivalent and interchangeable. We extract all multi-choice QA tasks—with label spaces such as “A/B/C/D” or “1/2/3”—and all sentence completion tasks, where models choose a logical continuation for an input text between two options, usually labeled A and B. This results in $18$ tasks with semantically equivalent labels.

We compare label bias on this subset of tasks and the entire evaluation suite for Llama-2 models in Fig. 4, with results for other models largely following similar trends (App. C.3). We find that, in most cases, models demonstrate lower label bias on tasks with semantically equivalent labels. This is especially evident in settings with few or no demonstrations, where models are typically strongly biased (§4.1). Still, RSD levels for such tasks remain relatively high across all evaluated settings. Further, we observe that instruction-tuned models prompted with 8 or more demonstrations are often more biased on this subset of tasks. In summary, although using semantically equivalent labels may potentially mitigate bias in scenarios with limited demonstrations, LLMs still exhibit substantial label bias when faced with such labels.

Label imbalance in the in-context demonstration set was previously shown to amplify label bias Zhao et al. (2021) as well as decrease model performance Min et al. (2022a), but such results were derived on a restricted set of tasks. We use our evaluation suite to investigate the observed label bias and performance of models when varying the level of imbalance in the demonstrations. To establish a consistent definition of label imbalance across different tasks, we use the subset of binary classification tasks ($N=197$) with $k=8$ demonstrations. Given a task with labels $L=\{\ell_{A},\ell_{B}\}$ and a context $C$, we define $p_{\uparrow}$ as the proportion of the most frequent label in the demonstrations of $C$, such that $p_{\uparrow}$ attains values in $\{0.5,0.625,0.75,0.875,1.0\}$. Specifically, $p_{\uparrow}=0.5$ means the labels are perfectly balanced, and $p_{\uparrow}=1.0$ means the demonstrations only include examples for one of the labels.

For every task, we prompt Llama-2 (7B/13B) and Mistral (7B) models using 10 different sets of demonstrations, with 2 sets for each value of $p_{\uparrow}$: one where $\ell_{A}$ is the most frequent label in $C$, and another where $\ell_{B}$ is the most frequent, as well as two different balanced sets ($p_{\uparrow}=0.5$).⁷⁷7To build each set, we randomly select and permutate 8 demonstrations from a pool of 16 held-out examples, while controlling for the selected number of examples per label. We group measurements taken across different tasks and demonstration sets by their level of label imbalance $p_{\uparrow}$, and inspect the average results per level.

We report our results in Fig. 5. Examining the two bias metrics, RSD (Fig. 5(a)) and BiasScore (Fig. 5(b)), we observe that both pretrained and instruction-tuned models are resistant to label imbalance: Increased imbalance does not result in notable gains in bias, unless the imbalance is very extreme—specifically, when the demonstrations include only a single or no demonstrations for one of the labels ($p_{\uparrow}>0.75$). Interestingly, model performance follows the same trends (Fig. 5(c)). Overall, our results indicate that for most tasks, the impact of label imbalance in the demonstrations set is minimal, except for cases of severe imbalance.

The performance of LLMs in in-context learning was shown to be sensitive to the exact choice of demonstrations used to prompt the model Liu et al. (2022); Chang and Jia (2023). We examine whether such choices also impact the extent of label bias in model predictions. We assess the performance and bias of Llama-2 (7B/13B) and Mistral (7B) models across 5 different sets of $k=8$ demonstrations for each task in our evaluation suite. In addition to reporting the mean and standard deviation of each metric, we use several oracle methods to aggregate and choose a specific demonstration set per task when computing the overall cross-task performance and bias metrics. Specifically, we select the demonstration sets that attain the following, per task: best performance; worst performance; median performance; least bias; and most bias.

We report our results for Llama-2 7B base in Tab. 1, with other models showing similar trends (App. C.4). We find that label bias, similarly to model performance, is highly sensitive to the choice of demonstrations, as indicated by the high variance across sets. Interestingly, the set of demonstrations that attains the worst performance also leads to strong bias, and vice-versa. In fact, we find that performance and bias are anti-correlated, with strong Pearson correlation for RSD ($r=-0.74$) and moderate for BiasScore ($r=-0.30$), indicating that when LLMs underperform in classification, it is often due to prompts that exacerbate bias. We leave further research into demonstrations that lead to biased and unbiased predictions to future work.