-
MS MARCO Web Search: a Large-scale Information-rich Web Dataset with Millions of Real Click Labels
Authors:
Qi Chen,
Xiubo Geng,
Corby Rosset,
Carolyn Buractaon,
**gwen Lu,
Tao Shen,
Kun Zhou,
Chenyan Xiong,
Yeyun Gong,
Paul Bennett,
Nick Craswell,
Xing Xie,
Fan Yang,
Bryan Tower,
Nikhil Rao,
Anlei Dong,
Wenqi Jiang,
Zheng Liu,
Mingqin Li,
Chuanjie Liu,
Zengzhong Li,
Rangan Majumder,
Jennifer Neville,
Andy Oakley,
Knut Magne Risvik
, et al. (6 additional authors not shown)
Abstract:
Recent breakthroughs in large models have highlighted the critical significance of data scale, labels and modals. In this paper, we introduce MS MARCO Web Search, the first large-scale information-rich web dataset, featuring millions of real clicked query-document labels. This dataset closely mimics real-world web document and query distribution, provides rich information for various kinds of down…
▽ More
Recent breakthroughs in large models have highlighted the critical significance of data scale, labels and modals. In this paper, we introduce MS MARCO Web Search, the first large-scale information-rich web dataset, featuring millions of real clicked query-document labels. This dataset closely mimics real-world web document and query distribution, provides rich information for various kinds of downstream tasks and encourages research in various areas, such as generic end-to-end neural indexer models, generic embedding models, and next generation information access system with large language models. MS MARCO Web Search offers a retrieval benchmark with three web retrieval challenge tasks that demand innovations in both machine learning and information retrieval system research domains. As the first dataset that meets large, real and rich data requirements, MS MARCO Web Search paves the way for future advancements in AI and system research. MS MARCO Web Search dataset is available at: https://github.com/microsoft/MS-MARCO-Web-Search.
△ Less
Submitted 13 May, 2024;
originally announced May 2024.
-
Learning without gradient descent encoded by the dynamics of a neurobiological model
Authors:
Vivek Kurien George,
Vikash Morar,
Weiwei Yang,
Jonathan Larson,
Bryan Tower,
Shweti Mahajan,
Arkin Gupta,
Christopher White,
Gabriel A. Silva
Abstract:
The success of state-of-the-art machine learning is essentially all based on different variations of gradient descent algorithms that minimize some version of a cost or loss function. A fundamental limitation, however, is the need to train these systems in either supervised or unsupervised ways by exposing them to typically large numbers of training examples. Here, we introduce a fundamentally nov…
▽ More
The success of state-of-the-art machine learning is essentially all based on different variations of gradient descent algorithms that minimize some version of a cost or loss function. A fundamental limitation, however, is the need to train these systems in either supervised or unsupervised ways by exposing them to typically large numbers of training examples. Here, we introduce a fundamentally novel conceptual approach to machine learning that takes advantage of a neurobiologically derived model of dynamic signaling, constrained by the geometric structure of a network. We show that MNIST images can be uniquely encoded and classified by the dynamics of geometric networks with nearly state-of-the-art accuracy in an unsupervised way, and without the need for any training.
△ Less
Submitted 23 March, 2021; v1 submitted 16 March, 2021;
originally announced March 2021.
-
A Simple Lifelong Learning Approach
Authors:
Joshua T. Vogelstein,
Jayanta Dey,
Hayden S. Helm,
Will LeVine,
Ronak D. Mehta,
Tyler M. Tomita,
Haoyin Xu,
Ali Geisa,
Qingyang Wang,
Gido M. van de Ven,
Chenyu Gao,
Weiwei Yang,
Bryan Tower,
Jonathan Larson,
Christopher M. White,
Carey E. Priebe
Abstract:
In lifelong learning, data are used to improve performance not only on the present task, but also on past and future (unencountered) tasks. While typical transfer learning algorithms can improve performance on future tasks, their performance on prior tasks degrades upon learning new tasks (called forgetting). Many recent approaches for continual or lifelong learning have attempted to maintain perf…
▽ More
In lifelong learning, data are used to improve performance not only on the present task, but also on past and future (unencountered) tasks. While typical transfer learning algorithms can improve performance on future tasks, their performance on prior tasks degrades upon learning new tasks (called forgetting). Many recent approaches for continual or lifelong learning have attempted to maintain performance on old tasks given new tasks. But striving to avoid forgetting sets the goal unnecessarily low. The goal of lifelong learning should be to use data to improve performance on both future tasks (forward transfer) and past tasks (backward transfer). In this paper, we show that a simple approach -- representation ensembling -- demonstrates both forward and backward transfer in a variety of simulated and benchmark data scenarios, including tabular, vision (CIFAR-100, 5-dataset, Split Mini-Imagenet, and Food1k), and speech (spoken digit), in contrast to various reference algorithms, which typically failed to transfer either forward or backward, or both. Moreover, our proposed approach can flexibly operate with or without a computational budget.
△ Less
Submitted 11 June, 2024; v1 submitted 27 April, 2020;
originally announced April 2020.