DUQGen: Effective Unsupervised Domain Adaptation of Neural Rankers by Diversifying Synthetic Query Generation

Abstract

State-of-the-art neural rankers pre-trained on large task-specific training data such as MS-MARCO, have been shown to exhibit strong performance on various ranking tasks without domain adaptation, also called zero-shot. However, zero-shot neural ranking may be sub-optimal, as it does not take advantage of the target domain information. Unfortunately, acquiring sufficiently large and high quality target training data to improve a modern neural ranker can be costly and time-consuming. To address this problem, we propose a new approach to unsupervised domain adaptation for ranking, DUQGen, which addresses a critical gap in prior literature, namely how to automatically generate both effective and diverse synthetic training data to fine tune a modern neural ranker for a new domain. Specifically, DUQGen produces a more effective representation of the target domain by identifying clusters of similar documents; and generates a more diverse training dataset by probabilistic sampling over the resulting document clusters. Our extensive experiments, over the standard BEIR collection, demonstrate that DUQGen consistently outperforms all zero-shot baselines and substantially outperforms the SOTA baselines on 16 out of 18 datasets, for an average of 4% relative improvement across all datasets. We complement our results with a thorough analysis for more in-depth understanding of the proposed method’s performance and to identify promising areas for further improvements.

DUQGen: Effective Unsupervised Domain Adaptation of Neural Rankers by Diversifying Synthetic Query Generation

Ramraj Chandradevan, Kaustubh D. Dhole, Eugene Agichtein Department of Computer Science Emory University Atlanta, USA-30307 {rchan31,kdhole,yagicht}@emory.edu

1 Introduction

Large Language Models (LLMs) have enabled new state-of-the-art performance in neural ranking Yan et al. (2019); Kamps et al. (2020); Hu et al. (2022); Nogueira et al. (2020a). An effective approach has been to train the LLMs on a large-scale general ranking task such as MS-MARCO passage or document ranking Bajaj et al. (2016) or Wikipedia retrieval Sun and Duh (2020), to learn task-specific features, which are often shared across other domains and datasets. The resulting rankers can then be used, without any adaptation (or in a zero-shot way) for a wide range of ranking tasks. For example, the BEIR Thakur et al. (2021) benchmark had demonstrated SOTA or near-SOTA performance of several zero-shot neural rankers on a diverse set of retrieval tasks.

Figure 1: DUQGen: an unsupervised domain-adaptation framework for neural ranking.

However, when switching to specialized domains such as finance or scientific documents, zero-shot ranking performance should benefit from additional information for the target domain. For training modern neural rankers, acquiring sufficiently large and high-quality target training data of query-document pairs, to improve a modern neural ranker, can be costly and time-consuming. Hence, there has been significant interest in various approaches to domain adaptation for neural rankers, with varying degrees of supervision, including unsupervised approaches using synthetically generated queries, documents, or both query-document pairs Sachan et al. (2022); Bonifacio et al. (2022); Jeronymo et al. (2023); Dai et al. (2022); Askari et al. (2023).

Unfortunately, most of the previously reported results did not exceed the ranking performance compared to the current SOTA zero-shot models, as evaluated on the BEIR benchmark Thakur et al. (2021). In other words, to the best of our knowledge, no previously reported unsupervised ranking adaptation method demonstrated consistent improvements over large neural SOTA zero-shot rankers.

In this work, we investigate whether it possible to improve the ranking performance of a pre-trained SOTA neural ranker for a given target domain, through unsupervised domain adaptation (UDA). To address this question, we first identify a critical requirement for the synthetic training data to be effective for ranking adaptation: that the generated training data should be both representative of the target domain, and sufficiently diverse to force changes to the ranking model at the appropriate representation level, without causing over-fitting or catastrophic forgetting (i.e., degrading performance on the original ranking tasks).

Specifically, we propose a new method DUQGen, which stands for Diversified Unsupervised Query Generation.  DUQGenintroduces a general approach for ranking domain adaptation, which focuses on selecting representative and diverse set of documents and query pairs for training a neural ranker. DUQGen only requires access to a (part of) target document collection to be searched, and can improve any pre-trained neural ranker. DUQGen is illustrated in Figure 1, and introduces the following innovations compared to previous unsupervised ranking adaptation approaches: (1) representing the target document collection as document clusters; (2) diversifying the synthetic query generation by probabilistic sampling over the resulting document clusters; and (3) prompting a large LLM for query generation with in-context examples to generate queries from the selected documents. As we show experimentally, these innovations are responsible for consistent improvements over the previous SOTA baselines for ranking adaptation on almost all BEIR benchmarks, as well as consistent improvements over zero-shot performance of SOTA neural rankers. In summary, our contributions include:

1. DUQGen, a general and effective unsupervised approach for domain adaption of neural rankers via synthetic query generation for training.

2. A novel and general method for creating representative and diverse synthetic query data for a given collection via clustering and probabilistic sampling.

3. Comprehensive experiments demonstrating that DUQGen consistently outperforms all SOTA baselines on 16 out of 18 BEIR datasets, and thorough analysis of the components of DUQGen responsible for the improvements. We release all our code and models publicly.

Next, we describe prior work on domain adaptation of neural rankers in more detail, to place our contributions in context.

2 Related Work

In this section, we discuss the prior works that help to establish the problem and navigate the solution.

2.1 Neural Rankers

Recently, transformer-based pre-trained language models have demonstrated impressive effectiveness in neural rankers Lin et al. (2021b). A neural ranker returns an order list of documents given a query, where a relevancy score between query and document dense embedding representations is used for sorting. Extensive studies have been conducted on both dense retrievers and re-rankers Mitra and Craswell (2018). In comparison to encoder-based rankers (MonoBERT and DuoBERT) Nogueira et al. (2019), encoder-decoder Nogueira et al. (2020b) and decoder-based Ma et al. (2023) rankers exhibit notably superior performance with a larger margin. While ColBERT Khattab and Zaharia (2020), Contriever Izacard et al. (2021), and GTR Ni et al. (2022) perform competitively as dense retrievers, MonoT5-3B Nogueira et al. (2020b) is widely adapted for re-ranking purposes.

2.2 Unsupervised Domain Adaptation for Neural Rankers

Despite the remarkable ranking performances demonstrated by recent pre-trained language models in zero-shot settings, they often encounter catastrophic failures in real-world deployment scenarios. The main factor contributing to these failures is Domain-Shift Zhu and Hauff (2022) or Domain Divergence Ramesh Kashyap et al. (2021). Domain-Shift has been a subject of exploration for decades, including recent investigations in domain adaptation Lupart et al. (2023). Traditionally, it is assumed that the source and target domains share samples drawn from the same distribution. Previous studies have addressed this issue by quantifying domain divergence through various measures, such as geometric measures, information-theoretic measures, and higher-order measures Ramesh Kashyap et al. (2021). Ultimately, these measures contribute to the development of novel solutions for domain adaptation in neural rankers.

Solutions addressing domain divergence typically fall into two categories: (1) representation learning; and (2) data selection. Representation learning approaches primarily address UDA, with a focus on learning domain-invariant representations Bousmalis et al. (2016); Cohen et al. (2018) or pre-training a zero-shot ranker. On the other hand, data selection assumes that not all samples contribute equally to domain representation Axelrod et al. (2011), highlighting the importance of identifying effective target domain samples. Improper selection of target data during fine-tuning has the potential to undermine the impact of source pre-training. Our research centers on the issue of improper representation of the target domain leading to diminished performances in neural rankers. Therefore, DUQGen aims to identify representative and diverse target samples that can be effective during the fine-tuning process.

2.3 Synthetic IR Data Generation

The increasing power of Large Language models have prompted numerous studies to focus on utilizing LLMs for the creation of high-quality training data. Several previous works have explored unsupervised synthetic data generation for fine-tuning ranking models, including GPL Wang et al. (2022), InPars Bonifacio et al. (2022), InPars-v2 Jeronymo et al. (2023), DocGen-RL Askari et al. (2023), GenQ Thakur et al. (2021), and Promptagator Dai et al. (2022). These frameworks use random document sampling or random seed queries to start their pipelines, which leaves room for improvement.

Each of the previously mentioned works utilizes distinct strategies employed alongside their data synthesis processes. GPLWang et al. (2022), for instance, combines a T5-based Raffel et al. (2020) query generator with a pseudo-labeling cross-encoder to enhance robust learning. InPars and InPars-v2 methods utilize GPT-3 and GPT-J query generators along with different filtering strategies to eliminate low-quality synthetic queries. DocGen-RL introduces an RL-driven guided approach combined with document synthesis using BLOOM Scao et al. (2022). GenQ, on the other hand, fine-tunes TAS-B Hofstätter et al. (2021) with queries generated from an MS-MARCO fine-tuned T5-base generator. Promptagator employs a pipeline similar to InPars, but with improved components, such as a random million document samples, a 137B FLAN query generator, and a strong consistency filter to prune 8 million synthetic queries through a relatively complicated and expensive process. Notably, none of the methods mentioned above take into consideration the significance of identifying domain-representative documents or diversifying the resulting queries. Consequently, the fine-tuned performances appears to fall short of zero-shot performances in many cases.

The quality of the generated training queries significantly affects the end retrieval performances. Despite the utilization of strong query generators (BLOOM and GPT-3), the domain query representation can still be improved. For instance, InPars employed a prompt containing in-context examples from MS-MARCO training data, yet it still maintains a domain representation gap during in-context generation. Furthermore, their query generation did not address the need for diversity among the generated training samples. Additionally, they incorporated a complex filtering step to prune the generated queries, which we show can be avoided. These methods fine-tuned rankers using large-scale synthetic data, ranging from 100k to 1M examples. In contrast, we argue that judicious selection of training samples can obviate the necessity for such large-scale generation, reducing the required amount of synthetic training data by a factor of x1000.

3 Methodology

DUQGen, shown in Figure 1, consists of four components – domain document selection, domain query generation, negative pairs mining, and fine-tuning. We now cover each component in detail.

3.1 Domain Document Selection

We propose to represent a target domain with clusters and each clusters with its sampled documents. Therefore, in this section we describe them in three stages, namely collection document clustering, probabilistic document sampling, and diversified document selection.

3.1.1 Collection Document Clustering

Representing a large-scale target collection of documents with limited training data is challenging. Therefore, we propose to divide the collection into portions, and then sample documents within each portion. We use a clustering approach for the collection representation. Moreover, we can achieve diverse topical documents to represent the domain. We start with the full collection of documents and apply a preprocessing step, where we discard short span documents, filtering out noisy documents. Then we use a SOTA text encoder, viz. Contriever Izacard et al. (2021), to encode each of the documents. Using the document embeddings $v_{D_{i}}$, we apply clustering (e.g., $K$-Means) technique, where $K$ is a hyper-parameter to tune.

3.1.2 Probabilistic Document Sampling

Representing each cluster within large data collections is challenging since the resultant clusters can often be of imbalanced sizes. Let’s take $k^{th}$ cluster size as $c_{k}$ and collection size as $C$, where ($1\leq c_{k}\leq C$). We ideally want to sample more number of documents from larger size clusters in proportion to the cluster size. If $cluster_{k}$ and $D_{i}$ represent $k^{th}$ cluster and its $i^{th}$ document, the probability of selecting $D_{i}$ from $cluster_{k}$ is $Pr(D_{i}|cluster_{k})\propto c_{k}\forall D_{i}\in cluster_{k}$.

We intend to sample $N$ number of synthetic training examples from $K$ number of clusters, where $N\geq K$. Therefore, we design a stratified expression to determine the document sample size $N_{k}$ for $k^{th}$ cluster, given by

$$N^{0}_{k}=1+\left\lfloor\frac{c_{k}}{C}(N-K)\right\rfloor$$

$$P=N-\sum_{k=1}^{K}N^{0}_{k}$$

$$N_{k}=\begin{cases}N^{0}_{k}+1&\text{if $k\in\operatorname*{argsort}_{top-P}(c_{k})$}\\ N^{0}_{k}&\text{if $k\not\in\operatorname*{argsort}_{top-P}(c_{k})$}\end{cases}$$

where $N^{0}_{k}$ and $P$ are intermediate sample size and integer number. $\left\lfloor*\right\rfloor$ operation finds the floor integer value.

Now that we determined the sample size for each clusters, we define our sampling approach. Let’s take $d_{i}$ as the similarity (e.g. cosine similarity) between document $D_{i}$ and its corresponding cluster centroid. We define an exponential value $e^{d_{i}}$ as the representative of how close $D_{i}$ is to its cluster centroid. Therefore, $Pr(D_{i}|cluster_{k})$ becomes the normalized softmax given by:

$$Pr(D_{i}|cluster_{k})=\frac{e^{d_{i}/T}}{\sum_{j=1}^{c_{k}}e^{d_{j}/T}}$$ (1)

$$d_{i}=cosine(v_{D_{i}},\frac{1}{c_{k}}\sum_{j=1}^{c_{k}}v_{D_{j}})\vspace{-0.2cm}$$ (2)

where $T$ is the softmax temperature and $v_{D_{i}}$ is the $i^{th}$ document embedding. Intuitively, a document likelihood to be selected to generate an associated query is proportional to the document similarity to its cluster centroid.

3.1.3 Diversified Document Selection

Now we sample $N_{k}$ number of documents from each cluster $cluster_{k}$ and pool them to obtain the required training size documents $N$. Different sample sets can be drawn from the aforementioned sampling approach with different choices of random seed values. Therefore, to improve selection robustness in the sampling process, we apply a diversity measure, namely Maximal Marginal Relevance (MMR) Carbonell and Goldstein (1998). We first iterate the sampling process $m$ times ($m=5$) to obtain different sample sets. Then we apply MMR on the pooled documents from $m$ sets to select top-$N_{k}$ documents for $cluster_{k}$ as shown:

$$\begin{split}\operatorname*{argmax}_{D_{i}\in R\setminus S}\Bigl{[}\lambda Sim_{1}(D_{i},D_{k})\\ -(1-\lambda)\max_{D_{j}\in S}Sim_{2}(D_{i},D_{j})\Bigr{]}\end{split}$$ (3)

where $D_{k}$ is the document closest to the cluster centroid, $\lambda$ is a trade-off weight (to be tuned) between similarity to cluster centroid and diversity, $R$ is the pooled documents, $S$ is a subset of documents already selected from $R$, and $Sim_{1}$ and $Sim_{2}$ that can be same or different, but we used the $cosine$ similarity for both instances.

3.2 Synthetic Query Generation

Figure 2: Prompt template with in-context examples for synthetic query generation for the SCIDOCS dataset.

Query generation is an essential component in an unsupervised data generation pipeline for ranking models. Queries represent a target domain w.r.t. the user’s information need and the domain-task by taking different types, such as questions, headlines, keywords, or claims. Therefore, we use a LLM to generate a synthetic in-domain query for each sampled document. We few-shot prompt the LLM to generate such training queries similar to the existing work of Bonifacio et al. (2022). However, our contribution lies in showing that the in-domain few-shot examples (query-document pairs) help to achieve high-quality of queries compared to out-of-domain generic MS-MARCO examples. On each domain, we create a handful (e.g., 3) human generated queries for the few-shot example documents with minimal human effort, and an example prompt is shown in Figure 2.

3.3 Negative Pairs Mining

After obtaining the domain specific documents and queries, we should generate both positive and negative query-document pairs. First, the positive query-document pairs can be easily generated by map** the synthetic queries with their corresponding original (seed) documents. Second, the negative query-document pairs can be generated from hard negative mining, described in the standard practices Izacard et al. (2021); Xiong et al. (2020); Karpukhin et al. (2020). We parse the synthetic queries to any first-stage retrievers, such as BM25 Robertson and Zaragoza (2009), ColBERT Khattab and Zaharia (2020), or Contriever Izacard et al. (2021), to get top-$x$ documents. Then we pick the bottom-$num_{neg}$ documents from the top-$x$ to map against the synthetic queries, where 1:$num_{neg}$ is the positive:negative document pair ratio.

3.4 Fine-tuning with our Synthetic Data

Our domain adaptation framework can be applied on any ranking models with any weights initialization. To establish a strong competitor, we leverage the task pre-trained model (on MS-MARCO), and sequentially fully fine-tune with our own generated synthetic data. We also adapt the same hyper-parameter settings used in the MS-MARCO pre-training stage for fair deliverable.

4 Experiments

In this section, we provide details of our experimental setup to demonstrate the effectiveness of DUQGen.

4.1 Datasets and Metrics

We employed all 18 datasets from BEIR collection, ranging on diverse retrieval tasks, to assess the effectiveness of our domain adaptation framework on standard out-of-distribution datasets. Utilizing the multi-field index from Pyserini Lin et al. (2021a) for all datasets, we retrieved the top-100 and top-200 documents from lexical and dense first-stage retrievers respectively. Subsequently, we restricted re-ranking to top-100 BM25 documents and top-200 dense retriever documents. Since we evaluate our approach on both first-stage retrieval and re-ranking, we measured both nDCG@10 and R@100.

4.2 Ranking Models

We fine-tuned ColBERT Khattab and Zaharia (2020) and MonoT5-3B Nogueira et al. (2020b), namely DUQGen-retriever and DUQGen-reranker, to show the effectiveness in both dense retrieval and re-ranking. During evaluation, we tested two multi-stage ranking pipelines: (1) DUQGen-reranker: a fine-tuned MonoT5-3B re-ranking BM25 top-100 and (2) DUQGen-retriever + DUQGen-reranker: a fine-tuned MonoT5-3B re-ranking a fine-tuned ColBERT top-200 documents.

4.3 Baselines

We chose strong competitive rankers as baselines to highlight the effectiveness of our proposed domain adapted ranker.

BM25: Traditional lexical sparse retrieval. We replicated the BM25 scores from scratch.

Zero-shot (ZS) Models: A fine-tuned ranker on MS-MARCO dataset, includes MonoT5-3B and ColBERT.

InPars Bonifacio et al. (2022): An unsupervised training data generation framework for ranking. Synthetic queries are generated from randomly selected documents using few-shot prompting GPT-3 Curie model. Language model likelihood is used as a filtering step to pick top-10k high-quality synthetic queries before fine-tuning any ranker. Based on the reasons provided by Askari et al. (2023), we do not compare against InPars-v2.

DocGen-RL Askari et al. (2023): An RL-driven framework to generate documents from queries. Also an iterative approach, based on expand, highlight, and generate stages, generates documents from queries to prepare training data.

Promptagator++ Dai et al. (2022): As the SOTA methods closest to our work, we evaluate against Promptagator++. This methods operates by randomly selecting 1 million documents from the target collection. It utilizes 8-shot prompting with a 137 billion-parameter FLAN model Wei et al. (2022) to create 8 queries per document. Following consistency filtering, 1 million queries are selected to train a GTR-Base dual-encoder and cross-encoder Ni et al. (2022).

We directly utilized the scores reported by authors for DocGen-RL and Promptagator++. For the remaining baselines, we employed their corresponding HuggingFace Wolf et al. (2020) models to re-run the inference.

4.4 Tools and Implementation

Various tools were employed for distinct stages in our pipeline, utilizing Contriever Izacard et al. (2021) for text encoding, Faiss Johnson et al. (2019) for $k$-Means clustering, and Llama2-7B-Chat Touvron et al. (2023) for query generation, Pyserini for BM25 baseline and hard negative mining, and PyTorch for standard fine-tuning. Throughout our experiments, documents were represented using their title along with the text. Initially, collection documents were filtered for noise by excluding those with a character length less than 300 (can vary across datasets). Greedy decoding with a temperature of 0.0 was employed for the LLM to generate queries.

4.5 Hyper-Parameter Tuning

In our methodology section, we introduced several hyper-parameters, all of which underwent tuning to determine the optimal values. These include the temperature $T=1$ (Equation 1), MMR weight $\lambda=1.0$, number of clusters $K=1000$, and training sample size $N=1000$ for ColBERT and $N=1000$ and $5000$ for MonoT5-3B fine-tuning. We tuned the varying number of in-context examples and found the optimal performance with 3-shot prompting (also used in InPars). Additionally, through tuning different prompt templates, we discovered that a simple InPars-style template, displayed in Figure 2, consistently yields superior retrieval performance across datasets. For the process of hard negative mining, we set the first stage retriever hits $x=100$ and the number of negatives per positive pair $num_{neg}=4$.

We fine-tuned MonoT5-3B using a batch size of 8, gradient accumulation steps of 16, learning rate of $2e^{-5}$, AdamW optimizer with weight decay of 0.01 and warm-up ratio of 0.1, and epochs of 1. To fine-tune ColBERT, we adapted its official pre-training hyper-parameters, including a batch size of 32, a learning rate of $3e^{-6}$, and a maximum sequence length of 300.

The scale and quality of synthetic data depend on the training examples, $N$, and number of clusters, $K$, which we optimize in the subsequent subsections.

4.5.1 Clustering Optimization

To represent target domain, we employed $K$-Means algorithm, where $K$ denotes the number of clusters. We identified the optimal $K$ for each dataset through an unsupervised method, known as the Elbow method Thorndike (1953). The elbow method computes the Sum of Squared Error (SSE) for each value of $K$, where SSE is calculated as the sum of cosine distances between every collection document and its closest cluster centroid. The optimal $K$ consistently aligns at a fixed point of 1000 across all evaluation datasets, irrespective of variations in corpus size, domain properties, or domain-divergence from MS-MARCO.

Datasets (→)	covid	nfc	bio	nq	hotpot	fiqa	signal	news	robust	arg	touché	stack	quora	dbp	scidocs	fever	climate	scifact	avg
Models ($\downarrow$)	Retriever
BM25	.656	.325	.465	.329	.603	.236	.330	.398	.407	.315	.367	.299	.789	.313	.158	.753	.213	.665	.423
Zero-shot ColBERT	.706	.305	.480	.523	.590	.318	.270	.390	.392	.404	.209	.350	.853	.392	.144	.771	.184	.672	.442
DUQGen-retriever	.751	.325†	.497†	.530†	.614†	.336†	.271	.399	.411†	.425†	.234†	.363†	.857†	.401	.155†	.805†	.196†	.688†	.459
	BM25 Top-100 + Re-ranker
Zero-shot MonoT5-3B	.830	.373	.559	.579	.718	.462	.321	.473	.566	.316	.311	.421	.848	.408	.193	.849	.278	.760	.515
InPars	.803	-	-	.313	-	.352	-	-	.510	-	-	-	-	.351	-	-	-	-	-
DocGen-RL	-	-	-	.517	.663	-	-	-	-	-	-	-	-	-	-	.720	-	-	-
DUQGen-reranker(1k)	.862†∗	.382†	.588†	.593†∗	.748†∗	.458∗	.333	.483	.591†∗	.393†	.320	.439†	.895†	.422†∗	.200†	.890†∗	.310†	.757	.537
DUQGen-reranker(5k)	.836†∗	.376	.590†	.588†∗	.740†∗	.465∗	.300	.449	.571∗	.427†	.269	.439†	.894†	.421†∗	.202†	.891†∗	.288†	.761	.528
	Dense Retriever Top-200 + Re-ranker
GTR (base) retriever	.539	.308	.271	.495	.535	.349	.261	.337	.437	.511	.205	.357	.881	.347	.149	.660	.241	.600	.416
Promptagator++	.762	.370	-	-	.736	.494	-	-	-	.630	.381	-	-	.434	.201	.866	.203	.731	.528$
DUQGen-retriever	.751	.325	.497	.530	.613	.336	.270	.399	.411	.425	.234	.363	.857	.403	.154	.805	.196	.688	.459
+ DUQGen-reranker(1k)	.851	.402	.594	.671	.769	.511	.275	.477	.636	.511	.331	.462	.898	.484	.203	.901	.309	.758	.558
+ DUQGen-reranker(5k)	.817	.398	.602	.661	.768	.517	.253	.422	.609	.575	.262	.463	.896	.482	.202	.903	.284	.762	.549

Table 1: Main results comparing nDCG@10 scores between DUQGen and baselines on BEIR datasets. The best scores across each ranking setting are highlighted in bold. Avg score marked by ^$ calculated across only 11 datasets. DUQGen-reranker(1k) and (5k) represent the MonoT5-3B fine-tuned with 1k and 5k training examples correspondingly. Statistical significance reported using two-tailed paired t-test with Bonferroni correction (p < 0.05), against Zero-shot counterparts ($\dagger$) and best of InPars or DocGen-RL ($*$). Promptagator++ was fine-tuned on GTR base, thus we reported GTR scores for comparison.

4.5.2 Optimal Training Sample Size Discovery

By fixing the optimum number of clusters $K$ at 1000, we determined an optimal training sample size $N$, that proved effective across all datasets. To tune for $N$, we utilized FiQA and NQ as dev datasets, referencing prior work (InPars-v2) which demonstrated improved performance on FiQA and a declined performance on NQ compared to the zero-shot scores. Table 2 displays nDCG@10 values for various instances of $N$, with $K$ fixed at 1000. Our analysis led us to select optimum $N=1000$ for ColBERT and both $N=1000$ and $5000$ for MonoT5-3B fine-tuning across the datasets.

	$N$	FiQA	NQ
MonoT5-3B	(ZS) 0	.4617$\dagger$	.5792
	1k	.4581	.5934
	5k	.4646	.5880$\dagger$
	10k	.4553	.5777
ColBERT	(ZS) 0	.3183	.5228
	1k	.3356$\dagger$	.5301
	5k	.3388	.5233$\dagger$
	10k	.3306	.5171

Table 2: Ranking performances evaluated on nDCG@10 across the scale of training sample size $N$ on dev datasets. Bold and $\dagger$ indicate the best and second-best scores across benchmarks for each ranker.

5 Results and Discussion

In this section, we present our main experimental results and delve into the key observations. We first describe our primary findings, reported using nDCG@10 in Table 1 comparing between baselines and our approach within each ranking setting. Second, we report the first-stage retrieval performances, measured using R@100 in Table 3.

5.1 Re-ranking Results

Datasets (→) Models ($\downarrow$)	covid	nfc	bio	nq	hotpot	fiqa	signal	news	robust	arg	touché	stack	quora	dbp	scidocs	fever	climate	scifact	avg
BM25	.498	.250	.714	.760	.740	.540	.370	.422	.375	.942	.538	.606	.973	.398	.356	.931	.436	.908	.598
Zero-shot ColBERT	.473	.255	.664	.911	.747	.598	.278	.369	.311	.885	.436	.625	.989	.458	.345	.934	.447	.878	.589
DUQGen-retriever	.544†	.272†	.691†	.915	.769†	.615†	.291	.380†	.321†	.906†	.474†	.645†	.990	.493†	.356†	.948†	.465†	.899†	.610

Table 3: Comparison of R@100 scores across baselines and DUQGen. The best scores for each dataset are highlighted in bold. Statistical significance reported using two-tailed paired t-test with Bonferroni correction (p < 0.05), against Zero-shot counterpart ($\dagger$).

In Table 1, it is evident that DUQGen consistently surpasses the SOTA baselines in most cases, exhibiting notable improvements in performance. Specifically, DUQGen consistently and substantially outperforms both InPars and DocGen-RL re-rankers, showcasing average relative enhancements of 26% and 17% respectively across the evaluation datasets they share. When compared to Promptagator++, DUQGen demonstrates an average relative improvement of 4% across the shared evaluation datasets. Remarkably, DUQGen surpasses Promptagator++ in performance, utilizing merely 1000 LLM calls and fine-tuning with only 1000 training pairs, in contrast to Promptagator++’s requirement of generating 8 million queries using a 137B LLM and fine-tuning with 1 million training pairs. This highlights the effectiveness of our efficient and robust approach compared to the complex, resource-intensive, and exhaustive training methods based on reinforcement learning.

In many instances, the performance of the SOTA baselines degraded, compared to zero-shot counterparts. For instance, both InPars and DocGen-RL consistently demonstrate performance decreases relative to the zero-shot MonoT5-3B, with Avg. decrements of 18% and 11% respectively across the evaluation datasets they share (DocGen-RL also underperforms compared to zero-shot MonoT5-base, as shown in Table  6). On the other hand, DUQGen consistently surpasses all zero-shot models across all BEIR datasets, whether trained with 1,000 or 5,000 synthetic training examples.

Interestingly, training DUQGen-reranker with only 1,000 synthetic examples exhibited a slight performance improvement compared to training with 5,000 synthetic examples on 13 out of 18 datasets, indicating the sample efficiency of our approach. In the future, it may be feasible to automatically determine the minimum training size ($N$) for each dataset or task.

5.2 First-Stage Retrieval Results

In Table 3, similar to nDCG@10 scores, R@100 also demonstrates more substantial improvements for larger domain-shifts (7.1% on TREC-COVID and 3.8%${}^{\ref{abs_imp}}$ on Touché-2020) and limited improvements for smaller domain-shifts (.4%${}^{\ref{abs_imp}}$ on NQ). On average, DUQGen enhances zero-shot ColBERT by 2.1%${}^{\ref{abs_imp}}$ on BEIR datasets.

LLM	Prompt	Size	FiQA	NQ
(Zero-shot) No LLM		-	.3702	.5404
LLAMA-2 7B Chat	ms-marco	7B	.3736	.5371
	in-domain	7B	.3811	.5444
LLAMA-2 13B Chat	in-domain	13B	.3912	.5370
BLOOM-3B		3B	.3380	.5193
BLOOM-7B1		7.1B	.3634	.5172
gpt-3.5-turbo		20B	.3742	.5466

Table 4: nDCG@10 performances across different LLMs for query generation ($K=1$, $N=5000$) with MonoELECTRA re-ranker fine-tuned on the generations

	10k	.391	.391	.399	.396	.398	.389
		(+2%)	(+2%)	(+3%)	(+3%)	(+3%)	(+2%)
	5k	.381	.390	.395	.392	.387	-
		(+1%)	(+2%)	(+3%)	(+2%)	(+2%)
	1k	.382	.385	.390	.384	-	-
		(+1%)	(+2%)	(+2%)	(+1%)
	500	.381	.381	.381	-	-	-
		(+1%)	(+1%)	(+1%)
Number of Training Examples (N)	100	.359	.371	-	-	-	-
		(-1%)	(+0%)
		1	100	500	1k	5k	10k
		Number of Clusters (K) Zero-shot MonoELECTRA score on FiQA is 0.370

	10k	.532	.547	.542	.546	.540	.546
		(-1%)	(+1%)	(+0%)	(+1%)	(+0%)	(+1%)
	5k	.544	.542	.544	.551	.543	-
		(+0%)	(+0%)	(+0%)	(+1%)	(+0%)
	1k	.546	.547	.550	.548	-	-
		(+1%)	(+1%)	(+1%)	(+1%)
	500	.537	.548	.548	-	-	-
		(-0%)	(+1%)	(+1%)
Number of Training Examples (N)	100	.539	.546	-	-	-	-
		(-0%)	(+1%)
		1	100	500	1k	5k	10k
		Number of Clusters (K) Zero-shot MonoELECTRA score on NQ is 0.540

Table 5: Fine-tuned MonoELECTRA re-ranking performances in nDCG@10 for different values of $K$ and $N$ on FiQA (left) and NQ (right). (+%) and (-%) indicate integer rounded superior or inferior performance percentage against zero-shot scores.

6 Analysis

In this section, we report our analysis of DUQGen’s performance, which includes examining the need for clustering, confirming the choice of the query generator, and validating the quality of the generated queries.

6.1 Effect of Clustering for Domain Adaptation

We employ clustering to represent the target domain and number of training samples to force diversity during fine-tuning. However, we question whether clustering genuinely contributes to the process and, if so, how it influences the overall performance. Additionally, we take the training sample size $N$, into account. In Table 5, we illustrate the combined effect of both the $K$ and $N$ on MonoELECTRA top-100 BM25 re-ranking performances, measured in nDCG@10. MonoELECTRA is used in the analysis Sections 6.1 and 6.2 in order to measure the amplified performance improvements in a smaller model, as described in the previous section.

Table 5 confirms our decision to select $N=5000$ for MonoELECTRA. Notably, this figure highlights that the most substantial and consistent improvements occur around the values of {K=1000, N=5000} across both datasets. Performances without clustering ($K=1$) often fall below zero-shot in both datasets, especially NQ exhibiting the poorest performances.

6.2 Effect of Query Generators

We conducted an ablation study on query generation to assess how the quality of generated queries impacts overall retrieval performance. Table 4 displays the performances of MonoELECTRA fine-tuned with queries generated by various LLMs, including LLAMA2-Chat (7B and 13B), BLOOM (3B and 7B), and GPT-3.5-turbo Brown et al. (2020).

In comparison to the zero-shot re-ranking scores, LLAMA-2 7B was deemed the optimal choice for our query generator. LLAMA-2 7B with 3-shot in-domain prompts exhibits higher improvements on both dev datasets, surpassing gpt-3.5-turbo. While LLAMA-2 13B demonstrates superior performance to 7B on FiQA, it falls below the zero-shot performance in NQ, attributed to its large model capacity and sensitivity to prompts Zhao et al. (2021). BLOOM generates short queries lacking context, despite having sufficient contextual query examples from 3-shot examples. GPT-3.5-turbo generates high-quality queries, resulting in improved performance over zero-shot, but tends to be unstable with few-shot prompts, suggesting potential for further prompt engineering to enhance performance on each dataset. Our second main contribution involves using in-domain 3-shot prompts to generate queries over the ms-marco prompt, showcasing notable improvements on LLAMA-2 7B model.

6.3 Examples of DUQGen Queries

(a)

(b)

Figure 3: Example queries generated by DUQGen on (a) Quora and (b) TREC-Covid datasets. Pr denotes the $Pr(D_{i}|cluster_{k})$ where $D_{i}$ and $cluster_{k}$ refer to $i^{th}$ document and $k^{th}$ cluster.

So far, we have evaluated the effectiveness of DUQGen using quantitative measures and are now shifting our focus to examining the actual queries produced by our method. Figure 3 presents ten example queries generated from the Quora and TREC-Covid datasets, each representing distinct tasks and domains. In Figure 3, the synthetic queries are sampled across different clusters with different probability scores $Pr(D_{i}|cluster_{k})$. For instance, in Figure 3(a), we observe that in the Quora duplicate question retrieval task, each cluster corresponds to sub-topics of the target domain representation, such as monetary bank transfers, religion, exams in India, energy, and programming languages. Within each cluster, diverse queries are sampled using different probabilistic scores to aid in learning the domain representation. Additionally, the generated queries contain sufficient context or entities to retrieve pertinent information from its respective collection. This analysis of the generated queries further validates the effectiveness of our approach in generating a diverse and representative set of high-quality queries.

7 Conclusions

We proposed a general unsupervised domain adaptation method  DUQGen, which can be used to fine-tune any ranking model for a given target domains. DUQGen introduced significant innovations over the previously reported unsupervised domain adaptation methods. Specifically, DUQGen proposes representing the target domain collection with document clustering; an effective method to diversify the synthetically generated queries, and an effective prompting strategy for using an LLM to generate more effective and representative synthetic training data. We experimentally demonstrated that DUQGen is both scalable and effective, as it uses only a few thousands of synthetic training examples, while consistently improves over the SOTA zero-shot rankers, and significantly outperforms the SOTA methods for unsupervised domain adaptation methods in most cases. We complemented the strong empirical performance of DUQGen with an in-depth analysis of the components to quantify their contributions. Together, the presented techniques and experimental results significantly advance neural ranking adaptation, establish a new state-of-the-art in neural ranking, and suggest promising directions for future improvements.

8 Limitations

Our proposed methodology involves two pivotal steps: (1) clustering; and (2) query generation. First, we employ Contriever as our text encoder to produce embeddings for clustering. While we anticipate that it will produce high-quality document representation and prove to be useful in our work, we did not assess other document embeddings. Future work could directly address the question of choosing the appropriate embedding for clustering.

Secondly, we employed the Faiss library to implement $K$-Means clustering. However, as the collection size scales up over the millions, clustering becomes impractical. Consequently, Faiss resorts to sampling the collection and then training their algorithm. This loss of information during sampling could propagate as errors in the final retrieval scores. However, given that large collections typically contain dense clusters, the process of sampling for clustering in such cases may pose less problem.

Akin to many previous studies Zhao et al. (2021), we often encountered a lack of robustness of LLMs and their sensitivity to minor changes in the prompt affecting subsequent retrieval performance. Future work could explore strategies to mitigate this robustness through techniques like calibration Zhao et al. (2021) and perform corresponding studies to see the impact on reranking.

9 Ethical Considerations

Retrieval systems may give rise to a variety of ethical issues, such as the potential for bias, which can result in the preferential treatment of specific perspectives, a lack of transparency due to the opaque nature of deep learning models, obscuring the reasons behind the ranking of documents, and, in extreme cases, the facilitation of echo chambers. Therefore, it is essential to conduct thorough testing of these systems both prior to and during their deployment.

As shown by our work, the performance of downstream retrievers can be rightly influenced by the LLMs employed for generating synthetic queries. Given that LLMs can produce content that is inaccurate or entirely fabricated, there’s a risk that they might generate problematic queries, especially if applied to sensitive datasets. Although this issue may appear less critical in a scenario such as ours where the generated content is intended solely as training material for a following retriever, there is still a potential for generating harmful and toxic queries. Such queries could lead the retriever towards biased outcomes. Therefore, it is imperative to assess these systems to mitigate these risks against biases of the data generator.

Appendix A Appendix

		Dense Retriever		Re-ranker using BM25 Top-100
Size (→)		110M		110M		220M		3B
Ranker (→)	BM25	ColBERT		MonoELECTRA		MonoT5-base		MonoT5-3B
Datasets ($\downarrow$)		Zero-shot	DUQGen- retriever	Zero-shot	DUQGen- reranker(5k)	Zero-shot	DUQGen- reranker(5k)	Zero-shot	DUQGen- reranker(5k)
covid	.656	.706	.751(+6%)	.730	.761(+4%)	.814	.853(+5%)	.830	.836(+1%)
nfc	.325	.305	.325(+6%)	.280	.356(+27%)	.357	.368(+3%)	.373	.376(+1%)
bio	.465	.480	.497(+4%)	.502	.523(+4%)	.531	.566(+7%)	.559	.590(+6%)
nq	.329	.523	.530(+1%)	.540	.551(+2%)	.540	.550(+2%)	.579	.588(+2%)
hotpot	.603	.590	.614(+4%)	.691	.709(+3%)	.698	.721(+3%)	.718	.740(+3%)
fiqa	.236	.318	.336(+5%)	.370	.392(+6%)	.391	.400(+2%)	.462	.465(+1%)
signal	.330	.270	.271(+0%)	.297	.284(–4%)	.316	.309(–2%)	.321	.300(–6%)
news	.398	.390	.399(+2%)	.445	.411(–8%)	.459	.470(+2%)	.473	.449(-5%)
robust	.407	.392	.411(+5%)	.440	.479(+9%)	.518	.538(+4%)	.566	.571(+1%)
arg	.315	.404	.425(+5%)	.233	.327(+40%)	.188	.383(+103%)	.316	.427(+35%)
touché	.367	.209	.234(+12%)	.278	.261(–6%)	.305	.347(+14%)	.311	.269(–13%)
stack	.299	.350	.363(+4%)	.339	.387(+14%)	.389	.405(+4%)	.421	.439(+4%)
quora	.789	.853	.857(+0%)	.730	.873(+20%)	.845	.888(+5%)	.848	.894(+5%)
dbp	.313	.392	.401(+2%)	.278	.389(+40%)	.395	.406(+3%)	.408	.421(+3%)
scidocs	.158	.146	.155(+6%)	.162	.182(+12%)	.171	.186(+9%)	.193	.202(+5%)
fever	.753	.771	.805(+4%)	.816	.867(+6%)	.826	.878(+6%)	.849	.891(+5%)
climate	.213	.184	.196(+7%)	.246	.296(+21%)	.251	.268(+7%)	.278	.288(+3%)
scifact	.665	.672	.688(+2%)	.684	.727(+6%)	.730	.746(+2%)	.760	.761(+0%)
avg	.423	.442	.459(+4%)	.448	.487(+9%)	.485	.516(+6%)	.515	.528(+3%)

Table 6: Comparison of nDCG@10 scores across different model sizes and different ranking families. The best scores are highlighted in bold. Blue and red colored percentage values indicate the relative improvements in performance compared to the corresponding zero-shot baseline. Suffix (5k) refers to the training size used to fine-tune corresponding models.

A.1 Effect on Model Sizes

Different model sizes and different ranker families have been shown to exhibit different performances. Therefore, we fine-tuned MonoELECTRA Clark et al. (2020), MonoT5-base, MonoT5-3B, and ColBERT to show the effectiveness of our approach against the model sizes. We pre-trained the MonoELECTRA re-ranker on MS-MARCO using a batch size of 32, learning rate of $2e^{-5}$, AdamW optimizer with weight decay of 0.01 and warm-up ratio of 0.1, regression loss, and a maximum sequence length of 512. We continued using the same hyper-parameters for fine-tuning too. For MonoT5-base fine-tuning we only changed the batch size to 8 and kept the remaining hyper-parameters same as of MonoELECTRA.

The complete performances of DUQGen against each zero-shot baseline measured in nDCG@10 is reported in Table 6. DUQGen shows consistent improvements over corresponding zero-shot baselines across almost all datasets and across all model sizes. The larger the model size, the higher the performance improvements are, for example, the average score across all datasets increases from .487 to .528 between MonoELECTRA and MonoT5-3B. It is important to note that the performance improvements achieved by DUQGen are larger (4% Avg.) in smaller models and smaller (1% Avg.) in larger models. Fine-tuning on ColBERT shows consistent improvements across all datasets with no drop in performance. Thus showcasing the robustness of our approach to deploying dense retrievers in practical systems.

A.2 Running Time of DUQGen

DUQGen is a cost-effective and easily scalable to a large corpus size.  DUQGen needs to apply the pipeline only once per domain to generate training data to fine-tune a ranker. Then, at inference time, no additional complexity overheard is added. The training dataset generation process is a sequence of three steps, each fully controllable by a small set of parameters, and can be scaled to a large document collection as our experiments show. In fact, the evaluation datasets of NQ, FEVER, BioASQ are 2.5 Million, 5.5 Million, and 14.9 Million documents respectively. To give an idea of the time complexity, single-pass clustering of the full NQ collection required 8 minutes on a single machine; query generation using LLama2-7B to create $N=5,000$ synthetic training examples was the slowest step which required 1.2 hours, and can be increased or reduced easily by modifying N. The fine-tuning of the largest ranker model MonoT5-3B required 38 minutes${}^{\ref{machine2}}$.

A.3 Query Generation Prompts

(a)

(b)

Figure 4: Example prompts used for (a) NQ and (b) FiQA dataset.

Using additional prompts can be helpful in showcasing the diversity of tasks in BEIR and highlighting the minimal effort required for any domain using our approach. DUQGen can be utilized across a wide range of retrieval tasks with significantly reduced human effort, requiring only a few labeled examples as demonstrations. Thus, in Figures 4(a) and 4(b), we present the additional prompts utilized for dev datasets (NQ and FiQA) to exemplify the diverse tasks and the minimal effort involved.