-
Navigable Graphs for High-Dimensional Nearest Neighbor Search: Constructions and Limits
Authors:
Haya Diwan,
**rui Gou,
Cameron Musco,
Christopher Musco,
Torsten Suel
Abstract:
There has been significant recent interest in graph-based nearest neighbor search methods, many of which are centered on the construction of navigable graphs over high-dimensional point sets. A graph is navigable if we can successfully move from any starting node to any target node using a greedy routing strategy where we always move to the neighbor that is closest to the destination according to…
▽ More
There has been significant recent interest in graph-based nearest neighbor search methods, many of which are centered on the construction of navigable graphs over high-dimensional point sets. A graph is navigable if we can successfully move from any starting node to any target node using a greedy routing strategy where we always move to the neighbor that is closest to the destination according to a given distance function. The complete graph is navigable for any point set, but the important question for applications is if sparser graphs can be constructed. While this question is fairly well understood in low-dimensions, we establish some of the first upper and lower bounds for high-dimensional point sets. First, we give a simple and efficient way to construct a navigable graph with average degree $O(\sqrt{n \log n })$ for any set of $n$ points, in any dimension, for any distance function. We compliment this result with a nearly matching lower bound: even under the Euclidean metric in $O(\log n)$ dimensions, a random point set has no navigable graph with average degree $O(n^α)$ for any $α< 1/2$. Our lower bound relies on sharp anti-concentration bounds for binomial random variables, which we use to show that the near-neighborhoods of a set of random points do not overlap significantly, forcing any navigable graph to have many edges.
△ Less
Submitted 28 May, 2024;
originally announced May 2024.
-
DeeperImpact: Optimizing Sparse Learned Index Structures
Authors:
Soyuj Basnet,
Jerry Gou,
Antonio Mallia,
Torsten Suel
Abstract:
A lot of recent work has focused on sparse learned indexes that use deep neural architectures to significantly improve retrieval quality while kee** the efficiency benefits of the inverted index. While such sparse learned structures achieve effectiveness far beyond those of traditional inverted index-based rankers, there is still a gap in effectiveness to the best dense retrievers, or even to sp…
▽ More
A lot of recent work has focused on sparse learned indexes that use deep neural architectures to significantly improve retrieval quality while kee** the efficiency benefits of the inverted index. While such sparse learned structures achieve effectiveness far beyond those of traditional inverted index-based rankers, there is still a gap in effectiveness to the best dense retrievers, or even to sparse methods that leverage more expensive optimizations such as query expansion and query term weighting.
We focus on narrowing this gap by revisiting and optimizing DeepImpact, a sparse retrieval approach that uses DocT5Query for document expansion followed by a BERT language model to learn impact scores for document terms. We first reinvestigate the expansion process and find that the recently proposed Doc2Query -- query filtration does not enhance retrieval quality when used with DeepImpact. Instead, substituting T5 with a fine-tuned Llama 2 model for query prediction results in a considerable improvement. Subsequently, we study training strategies that have proven effective for other models, in particular the use of hard negatives, distillation, and pre-trained CoCondenser model initialization. Our results substantially narrow the effectiveness gap with the most effective versions of SPLADE.
△ Less
Submitted 6 July, 2024; v1 submitted 27 May, 2024;
originally announced May 2024.
-
Faster Learned Sparse Retrieval with Block-Max Pruning
Authors:
Antonio Mallia,
Torten Suel,
Nicola Tonellotto
Abstract:
Learned sparse retrieval systems aim to combine the effectiveness of contextualized language models with the scalability of conventional data structures such as inverted indexes. Nevertheless, the indexes generated by these systems exhibit significant deviations from the ones that use traditional retrieval models, leading to a discrepancy in the performance of existing query optimizations that wer…
▽ More
Learned sparse retrieval systems aim to combine the effectiveness of contextualized language models with the scalability of conventional data structures such as inverted indexes. Nevertheless, the indexes generated by these systems exhibit significant deviations from the ones that use traditional retrieval models, leading to a discrepancy in the performance of existing query optimizations that were specifically developed for traditional structures. These disparities arise from structural variations in query and document statistics, including sub-word tokenization, leading to longer queries, smaller vocabularies, and different score distributions within posting lists. This paper introduces Block-Max Pruning (BMP), an innovative dynamic pruning strategy tailored for indexes arising in learned sparse retrieval environments. BMP employs a block filtering mechanism to divide the document space into small, consecutive document ranges, which are then aggregated and sorted on the fly, and fully processed only as necessary, guided by a defined safe early termination criterion or based on approximate retrieval requirements. Through rigorous experimentation, we show that BMP substantially outperforms existing dynamic pruning strategies, offering unparalleled efficiency in safe retrieval contexts and improved tradeoffs between precision and efficiency in approximate retrieval tasks.
△ Less
Submitted 2 May, 2024;
originally announced May 2024.
-
Faster Learned Sparse Retrieval with Guided Traversal
Authors:
Antonio Mallia,
Joel Mackenzie,
Torsten Suel,
Nicola Tonellotto
Abstract:
Neural information retrieval architectures based on transformers such as BERT are able to significantly improve system effectiveness over traditional sparse models such as BM25. Though highly effective, these neural approaches are very expensive to run, making them difficult to deploy under strict latency constraints. To address this limitation, recent studies have proposed new families of learned…
▽ More
Neural information retrieval architectures based on transformers such as BERT are able to significantly improve system effectiveness over traditional sparse models such as BM25. Though highly effective, these neural approaches are very expensive to run, making them difficult to deploy under strict latency constraints. To address this limitation, recent studies have proposed new families of learned sparse models that try to match the effectiveness of learned dense models, while leveraging the traditional inverted index data structure for efficiency. Current learned sparse models learn the weights of terms in documents and, sometimes, queries; however, they exploit different vocabulary structures, document expansion techniques, and query expansion strategies, which can make them slower than traditional sparse models such as BM25. In this work, we propose a novel indexing and query processing technique that exploits a traditional sparse model's "guidance" to efficiently traverse the index, allowing the more effective learned model to execute fewer scoring operations. Our experiments show that our guided processing heuristic is able to boost the efficiency of the underlying learned sparse model by a factor of four without any measurable loss of effectiveness.
△ Less
Submitted 24 April, 2022;
originally announced April 2022.
-
Learning Passage Impacts for Inverted Indexes
Authors:
Antonio Mallia,
Omar Khattab,
Nicola Tonellotto,
Torsten Suel
Abstract:
Neural information retrieval systems typically use a cascading pipeline, in which a first-stage model retrieves a candidate set of documents and one or more subsequent stages re-rank this set using contextualized language models such as BERT. In this paper, we propose DeepImpact, a new document term-weighting scheme suitable for efficient retrieval using a standard inverted index. Compared to exis…
▽ More
Neural information retrieval systems typically use a cascading pipeline, in which a first-stage model retrieves a candidate set of documents and one or more subsequent stages re-rank this set using contextualized language models such as BERT. In this paper, we propose DeepImpact, a new document term-weighting scheme suitable for efficient retrieval using a standard inverted index. Compared to existing methods, DeepImpact improves impact-score modeling and tackles the vocabulary-mismatch problem. In particular, DeepImpact leverages DocT5Query to enrich the document collection and, using a contextualized language model, directly estimates the semantic importance of tokens in a document, producing a single-value representation for each token in each document. Our experiments show that DeepImpact significantly outperforms prior first-stage retrieval approaches by up to 17% on effectiveness metrics w.r.t. DocT5Query, and, when deployed in a re-ranking scenario, can reach the same effectiveness of state-of-the-art approaches with up to 5.1x speedup in efficiency.
△ Less
Submitted 24 April, 2021;
originally announced April 2021.
-
KeyAuth: Bringing Public-key Authentication to the Masses
Authors:
Travis Z. Suel
Abstract:
Passwords are a fragile, inadequate, and insecure tool for authenticating users, and are especially fraught with problems when used to secure access to network resources and services. In many cases, passwords provide a false sense of security. Creating passwords which are both secure (i.e., hard for attackers to guess) and easy for humans to remember is, at best, a paradoxical task because these t…
▽ More
Passwords are a fragile, inadequate, and insecure tool for authenticating users, and are especially fraught with problems when used to secure access to network resources and services. In many cases, passwords provide a false sense of security. Creating passwords which are both secure (i.e., hard for attackers to guess) and easy for humans to remember is, at best, a paradoxical task because these two criteria are diametrically opposed. Fortunately, a far more secure and user-friendly alternative is available. Public-key cryptography provides a means of both identifying and authenticating users without the need for passwords. KeyAuth is a generic and universal implementation of public-key authentication aimed at supplanting password-based authentication and significantly improving the security of network accessible resources by enhancing the usability of frequently used authentication mechanisms. KeyAuth is an application-, language-, operating system-, and protocol-independent public-key authentication service.
△ Less
Submitted 5 September, 2012;
originally announced September 2012.
