Query Performance Prediction using Relevance Judgments Generated by Large Language Models

QPP has attracted lots of attention in the IR and NLP community and has been widely studied in ad-hoc search (Singh et al., 2023; Faggioli et al., 2023e, f; Datta et al., 2022c), conversational search (Faggioli et al., 2023c, d; Abbasiantaeb et al., 2023; Meng et al., 2023c, a; Sun et al., 2021; Meng, 2024; Meng et al., 2023b), question answering (Samadi and Rafiei, 2023; Hashemi et al., 2019), and image retrieval (Poesina et al., 2023). This paper focuses on QPP for ad-hoc search.

Typically, QPP methods are divided into two categories: pre- and post-retrieval methods (Carmel and Yom-Tov, 2010). The former predicts the difficulty of a given query by using features of the query and corpus, while the latter further uses features of a ranked list returned by a ranker for the query (Carmel and Yom-Tov, 2010). This paper focuses on post-retrieval QPP methods.

A large number of unsupervised and supervised post-retrieval QPP methods have been proposed (Carmel and Yom-Tov, 2010) for predicting the performance of lexical rankers, such as query likelihood (Lafferty and Zhai, 2001) and BM25 (Robertson et al., 2009). Unsupervised QPP methods can be classified into clarity-based (Cronen-Townsend et al., 2002), robustness-based (Zhou and Croft, 2007, 2006; Aslam and Pavlu, 2007), coherence-based (Arabzadeh et al., 2021a; Diaz, 2007), and score-based (Tao and Wu, 2014; Shtok et al., 2012; Cummins et al., 2011; Pérez-Iglesias and Araujo, 2010; Zhou and Croft, 2007). More recently, a set of supervised QPP methods have been proposed (Arabzadeh et al., 2021b; Hashemi et al., 2019; Zamani et al., 2018; Datta et al., 2022a, b; Chen et al., 2022; Khodabakhsh and Bagheri, 2023). NeuralQPP (Zamani et al., 2018) and Deep-QPP (Datta et al., 2022a) are optimized from scratch. NQA-QPP (Hashemi et al., 2019) and BERT-QPP (Arabzadeh et al., 2021b) fine-tune BERT (Devlin et al., 2019) to improve QPP effectiveness. Further, Datta et al. (2022c) propose qppBERT-PL, which considers list-wise-document information, while Chen et al. (2022) propose BERT-groupwise-QPP that considers both cross-query and cross-document information. Khodabakhsh and Bagheri (2023) propose a multi-task query performance prediction framework (M-QPPF), learning document ranking and QPP simultaneously.

Post-retrieval QPP methods designed for lexical rankers struggle to predict the retrieval quality of neural rankers (Faggioli et al., 2023f; Hashemi et al., 2019), motivating several new unsupervised post-retrieval QPP methods designed for neural rankers. Datta et al. (2022b) propose a weighted relative information gain-based model (WRIG), which assesses a neural ranker for a given query by considering the relative difference of predicted performance between the given query and its variants; Zendel et al. (2023) assess a neural re-ranker by measuring the entropy of scores returned by it; Faggioli et al. (2023e) propose neural-ranker-specific ways of calculating regularization terms used by unsupervised post-retrieval QPP methods; Vlachou and Macdonald (2023) propose an unsupervised coherence-based QPP method that employs neural embedding representations to assess dense retrievers; and Singh et al. (2023) propose pairwise rank preference-based QPP (QPP-PRP) for predicting the performance of a neural ranker by measuring the degree to which a pairwise neural re-ranker (e.g., DuoT5 (Pradeep et al., 2021)) agrees with the ranked list returned by the neural ranker.

We present a novel QPP perspective: we start by automatically generating relevance judgments for a ranked list for a query and then proceed to predict IR evaluation measures for the ranked list. To the best of our knowledge, no prior work addresses QPP from this perspective.

Unlike regression-based QPP models (Khodabakhsh and Bagheri, 2023; Chen et al., 2022; Datta et al., 2022a; Arabzadeh et al., 2021b; Hashemi et al., 2019), which require training separate models to predict different IR evaluation measures, the training of LLMs for judging relevance in the QPP-GenRE method that we propose is agnostic to a specific IR evaluation measure, and different measures can be derived from the same set of generated relevance judgments.

We also differ from qppBERT-PL (Datta et al., 2022c), which first predicts the number of relevant items for each chunk in a ranked list and then aggregates those numbers into a general QPP score. However, qppBERT-PL’s output is still presented as a single scalar, which is insufficient to accurately represent different evaluation measures; also, it is infeasible to predict arbitrary IR measures only using the number of relevant items in a ranked list.

The work closest to QPP-GenRE, which is still different, is QPP using effectiveness evaluation without relevance judgments (EEwRJ) (Mizzaro et al., 2018). The goal of EEwRJ methods is to predict search system effectiveness in a TREC-like environment. E.g., a method proposed by Soboroff et al. (2001) randomly samples items from a pool for a query and treats these items as relevant; the intuition is that if an item is ranked highly by many search systems, it is likely to be pooled and therefore considered relevant. Mizzaro et al. (2018) explore applying QPP EEwRJ (Mizzaro et al., 2018) methods to QPP. However, QPP using EEwRJ suffers from two limitations: (i) EEwRJrequires obtaining ranked lists returned by all search systems in a given TREC edition to predict the difficulty of a query, and (ii) EEwRJencounters normalization challenges when predicting the ranking quality for a ranked list returned by a specific search system (Mizzaro et al., 2018). QPP-GenRE does not face these limitations.

While fine-tuning pre-trained language models has given rise to many state-of-the-art results (Devlin et al., 2019), fully fine-tuning LLMs for a specific task on consumer-level hardware is typically infeasible (Zhu et al., 2023) because of the large number of parameters of LLMs. As a result, there are three prevailing ways to adapt LLMs to a specific task: zero-shot prompting, few-shot prompting, a.k.a.  in-context learning (ICL) (Dong et al., 2022; Brown et al., 2020), and parameter-efficient fine-tuning (PEFT) (Dettmers et al., 2023; Liu et al., 2022; Hu et al., 2021).

There is limited success in only prompting open-source LLMs for certain tasks (Qin et al., 2023). Zero-shot prompting instructs an LLM to perform a specific task by inputting a text instruction. To get a promising result, zero-shot prompting is usually based on instruction-tuned LLMs (Zhang et al., 2023b; Qin et al., 2023), such as Flan-T5 (Chung et al., 2022), Flan-UL2 (Tay et al., 2022). However, Sun et al. (2023a) show that the performance of zero-shot prompting degrades considerably if an LLM is fed an instruction that was not observable during its training. ICL inputs a few input-target pairs (a.k.a. demonstrations) to an LLM, which would make an LLM learn from analogy (Dong et al., 2022) without updating its parameters. However, ICL has a high computational cost because it needs to feed input-target pairs to an LLM for each prediction; also, ICL requires substantial manual prompt engineering because an LLM’s performance (Liu et al., 2022) is sensitive to the formatting of the prompt (e.g., the wording and the order of input-target pairs).

PEFT can solve the above limitations; it aims to adapt an LLM to a specific task by training only a small fraction of its parameters. \AcLoRA, a widely-used PEFT method (Gema et al., 2023; Liu and Low, 2023; Zhang et al., 2023a; Santilli and Rodolà, 2023), has been shown to achieve comparable performance to full-model fine-tuning (Dettmers et al., 2023; Lu et al., 2023); low-rank adaptation (LoRA) adds learnable low-rank adapters to each network layer of an LLM (Hu et al., 2021) while all original parameters of the LLM are frozen. QLoRA (Dettmers et al., 2023) further reduces the memory usage of LoRA without sacrificing performance; QLoRA first quantizes an LLM model to 4-bits before adding and optimizing low-rank adapters. Our work explores the possibility of training open-source LLMs with QLoRA to generate relevance judgments.

Automatically generating relevance judgments is a long-standing goal in IR that has been studied for multiple decades (Makary et al., 2017, 2016; Ravana et al., 2015; Nuray and Can, 2006, 2003; Soboroff et al., 2001). Recent studies have demonstrated promising results of using LLMs for the automatic generation of relevance judgments (Thomas et al., 2023; Faggioli et al., 2023a). In this paper we focus on studies into generating relevance judgments with discrete classes (e.g., “Relevant” or “Irrelevant”), instead of generating continuous relevance labels in real numbers (Yan et al., 2024). We discuss related studies into LLM-based automatic generation of relevance judgments from two dimensions: (i) how LLMs are used to generate relevance judgments, and (ii) their applications.

Recent studies have explored prompting commercial LLMs (e.g., GPT-3.5 and GPT 4) or open-source LLMs in zero- or few-shot manners. Specifically, Faggioli et al. (2023a) use zero- and few-shot prompting to instruct GPT-3.5 to predict the relevance of an item to a query. Thomas et al. (2023) instruct GPT-4 by zero-shot prompting, and add to the prompt a detailed query description and consider chain-of-thought (Wei et al., 2022). Similarly, Rahmani et al. (2024) follow Thomas et al. (2023)’s prompt to instruct GPT-4. Ma et al. (2024) instruct GPT-3.5 to generate relevance judgments for a domain-specific scenario, i.e., legal case retrieval (Ma et al., 2021); they use prompts specifically designed for this scenario. More recently, Upadhyay et al. (2024b) prompt GPT-4o in a zero-shot manner. Besides using commercial LLMs, only few studies (Zhang et al., 2024; Salemi and Zamani, 2024; Upadhyay et al., 2024a; Khramtsova et al., 2024) explore prompting open-source LLMs to generate relevance judgments. E.g., Khramtsova et al. (2024), Upadhyay et al. (2024a) and Salemi and Zamani (2024) prompt Flan-T5 (Chung et al., 2022), Vicuña-7B (Zheng et al., 2024) and Mistral (Jiang et al., 2023), respectively, in either zero-shot or few-shot manners. MacAvaney and Soldaini (2023) focus on a special scenario where a relevant item for a given query is already known and use Flan-T5 (Chung et al., 2022) to estimate the relevance of another item to the query given the known relevant item.

Recent studies have explored using LLM-generated relevance judgments to benefit (i) search system evaluation (Upadhyay et al., 2024a; Rahmani et al., 2024; Thomas et al., 2023; MacAvaney and Soldaini, 2023; Faggioli et al., 2023a), (ii) ranker selection (Khramtsova et al., 2024), (iii) item selection and retrieval quality evaluation in retrieval-augmented generation (RAG) (Zhang et al., 2024; Salemi and Zamani, 2024) and (iv) retriever fine-tuning (Ma et al., 2024). Concerning (i), recent studies (Rahmani et al., 2024; Upadhyay et al., 2024a; Abbasiantaeb et al., 2024; Faggioli et al., 2023a; Thomas et al., 2023; MacAvaney and Soldaini, 2023) explore evaluating search systems either entirely using LLM-generated relevance judgments or partially using LLM-generated relevance judgments (a.k.a. filling holes). They have demonstrated a high correlation between search system rankings based on LLM- and human-labelled relevance judgments. As to (ii), given a pool of dense retrievers, Khramtsova et al. (2024) select a suitable one for a target corpus by estimating their performance using LLM-generated queries and relevance judgments specific to the target corpus. For (iii), for item selection, Zhang et al. (2024) prompt LLMs to generate relevance judgments for retrieved candidate items in RAG; the items that are predicted as “relevant” are used for text generation. Zhang et al. (2024) observe that items selected via relevance prediction resulted in sub-optimal text generation quality. For retrieval quality evaluation, Salemi and Zamani (2024) generate relevance judgments for retrieved candidate items and aggregate those judgments into a score. However, Salemi and Zamani (2024) found that the aggregated score based on the LLM-generated relevance judgments achieves a low correlation with the text generation quality of RAG. Concerning (iv), Ma et al. (2024) fine-tune a legal case retriever on a training set augmented with LLM-generated relevance judgments. They show that fine-tuning a legal case retriever using the generated relevance judgments results in enhanced performance.

Our work differs from the studies mentioned above: (i) we explore the possibility of fine-tuning open-source LLMs for generating relevance judgments; unlike MacAvaney and Soldaini (2023), we focus on a more practical scenario wherein no relevant item is known in advance for each query; and (ii) we focus on QPP and predict the ranking quality of a ranked list for a query using LLM-generated relevance judgments, which previous studies have not explored.

Recent studies on using LLMs for re-ranking have witnessed remarkable progress (Meng et al., 2024; Askari et al., 2023; Zhuang et al., 2023b; Bommasani et al., 2023; Ma et al., 2023a, b; Zhuang et al., 2023a; Drozdov et al., 2023; Sachan et al., 2022; Zhang et al., 2023c; Pradeep et al., 2023b; Tang et al., 2023; Pradeep et al., 2023a; Ma et al., 2023b; Sun et al., 2023b; Zhuang et al., 2023c; Hou et al., 2023). There are four paradigms of LLM-based re-ranking: pointwise, pairwise, listwise, and setwise (Zhuang et al., 2023c). Given a query, pointwise re-rankers produce a relevance score for each item independently, and the final ranking is formed by sorting items by relevance score (Ma et al., 2023a; Drozdov et al., 2023; Zhuang et al., 2023b; Sachan et al., 2022). The pairwise paradigm (Qin et al., 2023) eliminates the need for computing relevance scores; given a query and a pair of items, a pairwise re-ranker estimates whether one item is more relevant than the other for the query. Listwise re-rankers (Zhang et al., 2023c; Pradeep et al., 2023b; Tang et al., 2023; Pradeep et al., 2023a; Ma et al., 2023b; Sun et al., 2023b) frame re-ranking as a pure generation task and directly output the reordered ranked list given a query and a ranked list return by first-stage retriever (Zhang et al., 2023c; Pradeep et al., 2023b; Tang et al., 2023; Pradeep et al., 2023a; Ma et al., 2023b; Sun et al., 2023b). Given the low efficiency of pairwise (multiple inference passes) and listwise (multiple decoding steps) re-rankers, the setwise paradigm (Zhuang et al., 2023c) is meant to improve the efficiency while retaining re-ranking effectiveness. Given a query and set of items, an LLM is asked which item is the most relevant one to the query; these items are reordered according to the LLM’s output logits of each item being chosen as the most relevant item to the query, which only requires one decoding step of an LLM.

Our work differs from this line of research because we generate explicit relevance judgments with discrete classes (e.g., “Relevant” or “Irrelevant”), whereas studies into LLMs for re-ranking aim to predict the relevance order of items. However, using LLMs for generating relevance judgments and for re-ranking are intrinsically the same task: relevance prediction. Thus, an LLM-based re-ranker has the potential to serve as a relevance judgment generator.

Our main contribution in this paper is the introduction of QPP-GenRE, a novel QPP framework, which, in theory, can be integrated with various relevance prediction approaches. To demonstrate the compatibility of QPP-GenRE with various relevance prediction approaches, we adapt a state-of-the-art pointwise LLM-based re-ranker, RankLLaMA (Ma et al., 2023a), into a relevance judgment generator by applying a threshold for its re-ranking scores; we then integrate QPP-GenRE with this adapted RankLLaMA. Note that adapting other types of LLM-based re-rankers, e.g., pairwise and listwise LLM-based re-rankers into relevance judgment generators is challenging because they do not explicitly produce explicit relevance scores. Exploring their use as relevance judgment generators is beyond the scope of this paper.

We propose QPP-GenRE, which consists of two steps: (i) generating relevance judgments using LLMs, and (ii) predicting IR evaluation measures. In (i), we employ an LLM to generate relevance judgments for the top-$n$ retrieved items in the ranked list for a given query; to improve LLM’s effectiveness in generating relevance judgments, we fine-tune an LLM with PEFT using human-labeled relevance judgments. In (ii), we regard the generated relevance judgments as pseudo labels to calculate different IR evaluation measures.

In this section, we study the following research questions:

1. RQ1

   To what extent does QPP-GenRE improve QPP quality for lexical and neural rankers in terms of RR@10, a precision-oriented IR metric, compared to state-of-the-art baselines?

2. RQ2

   To what extent does QPP-GenRE improve QPP quality for lexical and neural rankers in terms of nDCG@10, an IR metric considering recall, compared to state-of-the-art baselines?

3. RQ3

   How does judging depth in a ranked list affect the prediction of nDCG@10, an IR metric that considers recall?

4. RQ4

   To what extent do fine-tuning and the choice of LLMs affect the quality of generated relevance judgments and QPP?

We experiment with 4 widely-used IR datasets from the TREC 2019–2022 deep learning (TREC-DL) tracks (Craswell et al., 2022, 2021, 2020, 2019). These datasets provide relevance judgments in multi-graded relevance scales per query. TREC-DL 19, 20, 21 and 22 have 43, 54, 53 and 76 queries, respectively. TREC-DL 19/20 and TREC-DL 21/22 are based on the MS MARCO V1 and MS MARCO V2 passage ranking collections respectively. In the V1 edition, the corpus comprises 8.8 million passages while the V2 edition has over 138 million passages.

We consider BM25 (Robertson et al., 2009) as a lexical ranker, and ANCE (Xiong et al., 2021) as a neural-based dense retriever. To increase the comparability and reproducibility of our paper, we get the retrieval results of both rankers using the publicly available resource from Pyserini (Lin et al., 2021). We get BM25’s retrieval result with top-1000 retrieved items per query on the TREC-DL 19–22 datasets using the default parameters ($k1=0.9$, $b=0.4$). BM25’s actual nDCG@10 values are 0.506, 0.480, 0.446 and 0.269 on TREC-DL 19, 20, 21 and 22, respectively. We get ANCE’s retrieval result with top-1000 retrieved items per query on TREC-DL 19–20, using the publicly available dense vector index of ANCE on MS MARCO V1. ANCE’s actual nDCG@10 values are 0.645 and 0.646 on TREC-DL 19 and 20, respectively. We rely on the publicly available dense vector index of ANCE; at the time of writing, there is no dense vector index of ANCE publicly available on MS MARCO V2 for TREC-DL 21 and 22.³³3Building the dense vector index on MS MARCO V2 with over 138 million passages is resource-intensive and out of the scope of our work.

We consider three groups of baselines: unsupervised post-retrieval QPP methods, supervised post-retrieval QPP methods, and the LLM-based QPP methods. Specifically, we consider the following unsupervised QPP approaches that already showed high correlation with actual retrieval performance in previous work:

• •

  Clarity (Cronen-Townsend et al., 2002) computes the KL divergence between language models (Lavrenko and Croft, 2001) induced from the top-$k$ items in a ranked list and the corpus.

• •

  Weighted information gain (WIG) (Zhou and Croft, 2007) calculates the difference between retrieval scores of the top-$k$ items in a ranked list and the retrieval score of the entire corpus.

• •

  Normalized query commitment (NQC) (Shtok et al., 2012) calculates the standard deviation of retrieval scores of the top-$k$ items in a ranked list to a query; the standard deviation is normalized by the retrieval score of the entire corpus to the query.

• •

  $\sigma_{max}$ (Pérez-Iglesias and Araujo, 2010) computes the standard deviation of retrieval scores from the first item to each point in a ranked list and outputs the maximum standard deviation.

• •

  n($\sigma_{x\%}$) (Cummins et al., 2011) calculates the standard deviation for each query by considering the items whose retrieval scores are at least $x\%$ of the top retrieval score in a ranked list.

• •

  Score magnitude and variance (SMV) (Tao and Wu, 2014) considers both the magnitude of retrieval scores (WIG) and their variance (NQC).

• •

  UEF(NQC) (Shtok et al., 2010) uses a pseudo-effective reference list to improve QPP quality; we follow (Arabzadeh et al., 2023; Datta et al., 2022c; Arabzadeh et al., 2021b) to use NQC as a base predictor.

• •

  RLS(NQC) (Roitman, 2017) generates and selects both pseudo-effective and pseudo-ineffective reference lists; we use NQC as a base predictor because Roitman (2017) show that RLS works better with NQC.

• •

  QPP-PRP (Singh et al., 2023) measures the degree to which a pairwise neural re-ranker (DuoT5 (Pradeep et al., 2021)) agrees with the ranked list for the query.

• •

  Dense-QPP (Arabzadeh et al., 2023) is robustness-based and designed for dense retrievers only: it injects noise neural representation of the given query, and then measures the similarity between ranked lists for the original query and perturbed query representations. Note that Dense-QPP (Arabzadeh et al., 2023) is designed for predicting the ranking quality of neural-based retrievers; it cannot predict the ranking quality of BM25.

We follow established best practices (Datta et al., 2022c; Zamani et al., 2018; Carmel and Yom-Tov, 2010; Hauff et al., 2008; Cronen-Townsend et al., 2002) to evaluate QPP by measuring linear correlation by Pearson’s $\rho$ as well as ranked-based correlation through Kendall’s $\tau$ correlation coefficients between the actual and predicted performance of a query set.⁴⁴4We also consider scaled Mean Absolute Ranking Error (sMARE) (Faggioli et al., 2022, 2021b) and draw the same conclusion as Pearson’s $\rho$ and Kendall’s $\tau$; we include the results in our repository at https://github.com/ChuanMeng/QPP-GenRE. As for target IR metrics, we consider the two primary official IR metrics used in TREC DL 19–22 (Craswell et al., 2022, 2021, 2020, 2019), RR@10 (precision-oriented) and nDCG@10 (considering recall); recent QPP studies (Arabzadeh et al., 2023; Faggioli et al., 2023e; Khodabakhsh and Bagheri, 2023) consider either or both of these metrics as their target metrics. Following (Datta et al., 2022c), we use relevance scale $\geq$ 2 as positive to compute actual binary IR measures (e.g., RR). When calculating correlation for nDCG@10, the actual values of nDCG@10 are calculated by human-labeled and multi-graded relevance judgments, while the nDCG@10 values predicted by QPP-GenRE are based on its generated binary judgments.

For all unsupervised QPP baselines, we tune the hyper-parameters for predicting the ranking quality of a ranker (either BM25 or ANCE) on TREC-DL 19 (TREC-DL 21) based on Pearson’s $\rho$ correlation for predicting the ranking quality of the same ranker on TREC-DL 20 (TREC-DL 22), and vice versa. We select the cut-off value $k$ for Clarity, NQC, WIG, SMV and so on from $\{5,10,15,20,25,50,100,300,500,1000\}$. n($\sigma_{x\%}$) has a a hyper-parameter $x$, which we choose from the set $\{0.25,0.4,0.5,0.6,0.75,0.9\}$.

To predict the performance of a certain ranker (either BM25 or ANCE), we train all supervised QPP baselines based on the ranked list returned by the target ranker. To predict a certain IR evaluation measure, regression-based methods (Khodabakhsh and Bagheri, 2023; Arabzadeh et al., 2021b; Hashemi et al., 2019) are trained to learn to output the target evaluation measure during training. However, our preliminary result shows that training supervised QPP baselines, especially for regression-based supervised methods (Khodabakhsh and Bagheri, 2023; Arabzadeh et al., 2021b; Hashemi et al., 2019), on the training set of MS MARCO V1 leads to inferior QPP quality for predicting the performance of ANCE. We hypothesize that this is because ANCE was originally trained on the training set of MS MARCO V1 (Xiong et al., 2021), and so the ranked list returned by ANCE on the training set of MS MARCO V1 would have higher quality than the ranked list returned by ANCE on the evaluation sets; therefore, supervised QPP methods that share the same training set as ANCE, tend to predict inflated performance on the evaluation sets, leading to degraded QPP quality. To solve the issue and ensure the consistency of the paper, we train all supervised QPP methods (including QPP-GenRE) on the development set of MS MARCO V1 (6980 queries) for predicting the performance of either BM25 or ANCE. We train all supervised QPP methods for 5 epochs and pick the best checkpoint for predicting the performance of a ranker on TREC-DL 19 (TREC-DL 21) based on Pearson’s $\rho$ correlation for predicting the performance of the same ranker on TREC-DL 20 (TREC-DL 22) and vice versa. All supervised QPP baselines use bert-base-uncased,⁵⁵5https://github.com/huggingface/transformers a constant learning rate (0.00002), and the Adam optimizer (Kingma and Ba, 2015).

For QPP-LLM, we prompt LLaMA-7B with the top-$k$ retrieved items, where $k$ is set to 10. For QPP-LLM (few-shot), we randomly sample demonstration examples from the development set of MS MARCO V1; our preliminary experiments show that sampling 2 demonstrations works best. For QPP-LLM (fine-tuned), we fine-tune LLaMA-7B using PEFT as QPP-GenRE fine-tunes LLMs.

We equip QPP-GenRE with LLaMA-7B for judging relevance. We use a recent PEFT method, 4-bit QLoRA (Dettmers et al., 2023), to fine-tune LLaMA-7B. The training of judging relevance needs positive and negative items per query. For positive items, we use the items annotated as relevant in qrels per query; we randomly sample one negative item from the ranked list (1,000 items) returned by BM25 per query. We fine-tune LLaMA-7B for 5 epochs on the development set of MS MARCO V1, taking about an hour and a half on an NVIDIA A100 GPU (40GB).

To answer RQ1, we compare QPP-GenRE and all baselines in predicting the performance of BM25 and ANCE w.r.t. a widely-used precision-oriented metric, RR@10; see Table 2. We have three main observations.

First, our proposed method, QPP-GenRE, outperforms all baselines in terms of both correlation coefficients on all datasets when predicting the performance of both rankers. In particular, we observe that QPP-GenRE outperforms QPP-PRP (Singh et al., 2023), which is a recently proposed baseline by 84% (0.292 vs. 0.538) in terms of Pearson’s $\rho$ when predicting RR@10 for BM25 on TREC-DL 19.

Second, QPP-LLM (few-shot) gets the worst result compared to other approaches. While QPP-LLM (fine-tuning) performs slightly better than QPP-LLM (few-shot), its performance is still limited in most cases. This indicates that it is ineffective for an LLM to model QPP in a straightforward way of directly predicting a score.

Third, there is no clear winner among the baselines, and the performance of baselines shows a bigger variance than QPP-GenRE across different datasets and rankers. E.g., the unsupervised method WIG achieves a good result among baselines for assessing BM25 on TREC-DL 20, while it gets nearly zero correlation coefficients on TREC-DL 22 when assessing BM25. Conversely, QPP-GenRE consistently achieves the best performance across datasets and rankers, thus showing robust performance.

To answer RQ2, Table 4 lists the performance of QPP-GenRE along with all the baselines on assessing BM25 and ANCE in terms of nDCG@10. For QPP-GenRE, we universally set the judging depth $n$ to 200 for all evaluation sets. The result reveals that by judging only 200 items per query, we can achieve state-of-the-art QPP quality in terms of nDCG@10 for all rankers on all evaluation sets; we will investigate the impact of judging depth on QPP-GenRE’s performance in the next section. Also, QPP-LLM (few-shot) and QPP-LLM (fine-tuning) are among the worst-performing baselines, showing that the LLMs struggle to generate numerical scores. Different from the results for RQ1, most QPP methods tend to perform better when predicting nDCG@10 than RR@10; this observation indicates that predicting RR@10 is a more challenging task.

To answer RQ3, as detailed in Section 4.3.2, for predicting IDCG, we devise an approximation strategy and use the items in the top $n$ ranks of the ranked list $L$ that are predicted as relevant by QPP-GenRE to approximate all the relevant items for a query in the corpus. To investigate the impact of the value of $n$ on the quality of the prediction, we investigate the relationship between the QPP quality of predicting nDCG@10 and the judgment depth to answer the following question: What depth of relevance judgment $n$ do we need to consider to get a satisfactory performance for predicting nDCG@10? In Figure 3, we plot the correlation coefficients between actual nDCG@10 values and nDCG@10 values predicted by QPP-GenRE for different judging depths in {10, 25, 50, 75, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1,000} on TREC-DL 19 and 20. We also show exact QPP results with depths at 10, 100 and 1000 in Table 4.

Table 4 reveals that, by judging only 10 items in the ranked list of BM25 and ANCE, we can already outperform all the baselines and achieve state-of-the-art QPP quality on half of the evaluation sets we used, e.g., assessing BM25 on TREC-DL 19 and 21, and assessing ANCE on TREC-DL 19. While judging deeper in the ranked list is essential for predicting recall-oriented measures, satisfactory QPP quality is still attainable with a relatively shallow depth. Moreover, Figure 3 illustrates that judging the top 200 items in a ranked list already reaches the saturation point for assessing BM25, i.e., there is no significant improvement by judging a higher number of items, while judging less than 100 top items reaches the saturation point for ANCE. We speculate that this is because ANCE has better retrieval quality than BM25, and more relevant items would appear earlier in the ranked list of ANCE than BM25; therefore, a shallower judging depth suffices to approximate all relevant items in the corpus. This emphasizes the need to consider retrieval quality when determining the optimal judgment depth for various rankers.

To answer RQ4, we analyze the impact of fine-tuning and the choice of LLMs on the quality of generated relevance judgments and QPP. We consider LLaMA-7B (Touvron et al., 2023a), Llama-3-8B (AI@Meta, 2024), and Llama-3-8B-Instruct (AI@Meta, 2024) under two settings: (i) trained with PEFT on human relevance labels (following the same fine-tuning setup as in Section 5.6), and (ii) few-shot prompted (in-context learning) .⁶⁶6We randomly sample human-labeled demonstration examples from the same set used for fine-tuning LLMs; each example is a triplet (query, passage, relevant/irrelevant); our experiments show that two examples work best, one with relevant passages and one with irrelevant passages. We do not report the results for a zero-shot setting because our preliminary experiments show that prompting these LLMs in a zero-shot way yields pretty poor performance.

To evaluate the performance of judging relevance, we compute Cohen’s $\kappa$ metric to measure the agreement between relevance judgments made by the TREC assessors (i.e., relevance judgments in the qrels) and relevance judgments automatically generated by a fine-tuned or few-shot LLM, on TREC-DL 19–22. Faggioli et al. (2023a) reported the relevance judgment agreement in terms of Cohen’s $\kappa$ between TREC assessors and GPT-3.5 (text-davinci-003) on TREC-DL 21; we also consider their Cohen’s $\kappa$ value for comparison. To evaluate QPP quality, we compute the Pearson’s $\rho$ correlation coefficients between BM25’s actual nDCG@10 values and those predicted by QPP-GenRE using relevance judgments generated by an LLM, on TREC-DL 19–22.⁷⁷7We do not report the Pearson’s $\rho$ correlation for GPT-3.5 (text-davinci-003) because the relevance judgments generated by Faggioli et al. (2023a) are not available to us. The judging depth is set to 1000 in a ranked list. We report the results in Table 5.

We have two observations. First, fine-tuning improves the quality of relevance judgment generation and QPP for all LLMs on all datasets. Specifically, all fine-tuned LLMs exhibit improved relevance judgment agreement with the TREC assessors on TREC-DL 19–22. After fine-tuning, LLaMA-7B and Llama-3-8B achieve “fair” agreement with the TREC assessors on TREC-DL 19, 20 and 21,⁸⁸8Note that unlike the qrels files for TREC-DL 19, 20, and 21 which are fully manually annotated, the qrels file for TREC-DL 22 is constructed by first detecting near-duplicate items and manually judging only one representative item from each near-duplicate cluster for a given query (Craswell et al., 2021); this difference may result in variation in Cohen’s $\kappa$ values of LLMs across TREC-DL 19, 20, 21, and TREC-DL 22. Llama-3-8B-Instruct (fine-tuned) even achieves “moderate” agreement on TREC-DL 21 (a Cohen’s $\kappa$ value of 0.418). All fine-tuned LLMs exhibit a higher Cohen’s $\kappa$ value than the commercial LLM, GPT-3.5 (text-davinci-003). All fine-tuned LLMs surpass their corresponding few-shot counterpart on all datasets in terms of Pearson’s $\rho$. This reveals that fine-tuning is an effective way to improve the quality of LLMs in generating relevance judgments, which finally translates to better QPP quality.

Second, newly released and instruction-tuned LLMs generally perform better. Llama-3-8B shows improved relevance judgment agreement and QPP quality compared to LLaMA-7B in most cases. Llama-3-8B-Instruct further enhances relevance judgment generation and QPP quality over both Llama-3-8B and LLaMA-7B across most cases. Notably, Llama-3-8B-Instruct (few-shot) even performs better than or equally as well as LLaMA-7B (fine-tuned) on TREC-DL 19 and 22. This finding implies that with a more effective LLM QPP-GenRE has the potential to achieve imrpoved QPP performance.

To show QPP-GenRE’s compatibility with other types of relevance prediction methods instead of directly asking an LLM to explicitly generate explicit relevance judgments, we adapt a state-of-the-art pointwise LLM-based re-ranker, RankLLaMA (Ma et al., 2023a), into a relevance judgment generator, and then integrate QPP-GenRE with the adapted RankLLaMA. Specifically, we translate a re-ranking score into a relevance judgment by applying a threshold: an item is deemed as “relevant” if its re-ranking score meets or exceeds a given threshold value. We analyze Pearson’s $\rho$ and Kendall’s $\tau$ correlation coefficients between BM25’s actual nDCG@10 values and those predicted by QPP-GenRE integrated with RankLLaMA w.r.t. different threshold values on TREC-DL 19 and 20. We employ RepLLaMA (7B) from Tevatron.⁹⁹9https://github.com/texttron/tevatron/tree/main/examples/rankllama RankLLaMA’s re-ranking scores for BM25 range from -12.93 to 89.90 for TREC-DL 19 and from -14.38 to 8.82 for TREC-DL 20. Thresholds are set at intervals of 0.5. The judging depth is set to 1,000 in a ranked list.

We report the results in Figure 4. We find that RepLLaMA achieves the highest QPP quality on both datasets when the threshold is 1. At this particular threshold, RepLLaMA achieves high Pearson’s $\rho$ values of 0.789 and 0.788 on TREC-DL 19 and 20, respectively. These values exceed those of fine-tuned LLaMA-7B, which achieves Pearson’s $\rho$ values of 0.715 and 0.627 on TREC-DL 19 and 20, respectively, as well as Llama-3-8B-Instruct, which achieves Pearson’s $\rho$ values of 0.647 and 0.743 on TREC-DL 19 and 20, respectively (see Figure 5).¹⁰¹⁰10Note that the comparison is not fair because (i) LLaMA-7B, Llama-3-8B-Instruct and all other supervised QPP methods used in this paper are trained on the development set of MS MARCO V1, while RankLLaMA (Ma et al., 2023a) was trained on the training set of MS MARCO V1, which is much larger. (ii) We employ the official version of MS MARCO V1, while RankLLaMA (Ma et al., 2023a) uses the Tevatron version of MS MARCO V1, where passages are enriched with document titles; Lassance and Clinchant (2023) reveal that incorporating titles leads to enhanced ranking performance. This means that a state-of-the-art pointwise LLM-based re-ranker can be adapted into an effective relevance judgment generator. The high QPP quality achieved by RankLLaMA demonstrates QPP-GenRE’s compatibility with other types of relevance prediction methods besides directly using LLMs as relevance judgment generators (i.e., asking an LLM to explicitly generate explicit relevance judgments).

However, compared to directly regarding an LLM as a relevance judgment generator, adapting an LLM-based re-ranker into a relevance judgment generator requires tuning an appropriate threshold. As demonstrated, re-ranking scores are not normalized and their ranges vary across datasets. Directly using a re-ranker as a relevance judgment generator can cause issues in real-world scenarios. Extra calibration work for re-ranking scores might be necessary.

As QPP-GenRE computes QPP based on generated relevance judgments, we analyze QPP errors from the perspective of relevance judgment generation. Figure 5 shows the QPP errors of QPP-GenRE integrated with LLaMA-7B in predicting the performance of BM25 and ANCE in terms of RR@10 on TREC-DL 19 and 20; the error is defined as the distance between the RR@10 values predicted by QPP-GenRE and actual RR@10 values, namely “predicted RR@10 minus actual RR@10.” We find that most RR@10 values predicted by QPP-GenRE tend to be smaller than the actual RR@10 values, indicating that QPP-GenRE performs less effectively in identifying relevant items than irrelevant ones in the top of the ranked list. Table 6 shows the confusion matrices that compare relevance judgments made by TREC assessors (i.e., relevance judgments in qrels) and QPP-GenRE integrated with LLaMA-7B on TREC-DL 19 and 20. We find that QPP-GenRE tends to wrongly predict some relevant items as irrelevant (false negatives), which provides a further interpretation of the QPP errors we found above. Therefore, reducing false negatives in generating relevance judgments is a potential way to improve the QPP quality of QPP-GenRE. We leave this exploration for future work.

Table 7 shows the online QPP latency of QPP-GenRE integrated with LLaMA-7B and other BERT-based supervised QPP baselines, on TREC-DL 19, on a single NVIDIA A100 GPU. We compute the inference latency when queries are processed individually. For QPP-GenRE, we consider judging depths at 10, 100, and 200; QPP-GenRE can use batch acceleration for judging items for the same query because each item in a ranked list for a query is independent of each other.¹¹¹¹11qppBERT-PL first splits a ranked list with 100 items into 25 chunks and then predicts the number of relevant items in each chunk. For a fair comparison, we put 25 chunks into one batch for acceleration. Although QPP-GenRE is more expensive than all baselines when predicting one measure due to the much larger parameter size of LLaMA-7B compared to BERT, QPP-GenRE has lower latency compared to some baselines when predicting multiple IR evaluation measures because multiple measures can be derived from the same set of relevance judgments at no additional cost. E.g., while QPP-GenRE is 56% more expensive than M-QPPF for predicting one measure, it becomes more efficient when predicting 2 or more metrics than M-QPPF. Nevertheless, we acknowledge that QPP-GenRE has higher computational costs than supervised QPP methods when predicting a single measure. Conversely, regression-based QPP baselines (NQA-QPP, BERTQPP and M-QPPF) need to train separate models for different IR evaluation metrics. Although qppBERT-PL is not optimized to learn to output one specific IR evaluation measure, qppBERT-PL does not achieve a promising QPP quality (see Table 2 and 4).

We argue that QPP-GenRE’s latency is still much smaller than some high-performing LLM-based re-rankers. E.g., Sun et al. (2023b) show that a GPT-4-based listwise re-ranker needs 10 API calls (one call takes 3,200ms) to re-rank 100 items for a query, resulting in 32,000ms in total, which is around 20 times worse than QPP-GenRE’s latency with a judging depth of 100. QPP-GenRE can well fit some knowledge-intensive professional search scenarios where QPP quality is prioritized or users may have a higher tolerance level for latency, such as patent search (Lupu et al., 2013), legal search (Tomlinson et al., 2007). Besides using QPP online, QPP can also be used to analyze a search system’s performance in an offline setting (Faggioli et al., 2023d).