-
Nomic Embed Vision: Expanding the Latent Space
Authors:
Zach Nussbaum,
Brandon Duderstadt,
Andriy Mulyar
Abstract:
This technical report describes the training of nomic-embed-vision, a highly performant, open-code, open-weights image embedding model that shares the same latent space as nomic-embed-text. Together, nomic-embed-vision and nomic-embed-text form the first unified latent space to achieve high performance across vision, language, and multimodal tasks.
This technical report describes the training of nomic-embed-vision, a highly performant, open-code, open-weights image embedding model that shares the same latent space as nomic-embed-text. Together, nomic-embed-vision and nomic-embed-text form the first unified latent space to achieve high performance across vision, language, and multimodal tasks.
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
-
Tracking the perspectives of interacting language models
Authors:
Hayden Helm,
Brandon Duderstadt,
Youngser Park,
Carey E. Priebe
Abstract:
Large language models (LLMs) are capable of producing high quality information at unprecedented rates. As these models continue to entrench themselves in society, the content they produce will become increasingly pervasive in databases that are, in turn, incorporated into the pre-training data, fine-tuning data, retrieval data, etc. of other language models. In this paper we formalize the idea of…
▽ More
Large language models (LLMs) are capable of producing high quality information at unprecedented rates. As these models continue to entrench themselves in society, the content they produce will become increasingly pervasive in databases that are, in turn, incorporated into the pre-training data, fine-tuning data, retrieval data, etc. of other language models. In this paper we formalize the idea of a communication network of LLMs and introduce a method for representing the perspective of individual models within a collection of LLMs. Given these tools we systematically study information diffusion in the communication network of LLMs in various simulated settings.
△ Less
Submitted 17 June, 2024;
originally announced June 2024.
-
Nomic Embed: Training a Reproducible Long Context Text Embedder
Authors:
Zach Nussbaum,
John X. Morris,
Brandon Duderstadt,
Andriy Mulyar
Abstract:
This technical report describes the training of nomic-embed-text-v1, the first fully reproducible, open-source, open-weights, open-data, 8192 context length English text embedding model that outperforms both OpenAI Ada-002 and OpenAI text-embedding-3-small on short and long-context tasks. We release the training code and model weights under an Apache 2 license. In contrast with other open-source m…
▽ More
This technical report describes the training of nomic-embed-text-v1, the first fully reproducible, open-source, open-weights, open-data, 8192 context length English text embedding model that outperforms both OpenAI Ada-002 and OpenAI text-embedding-3-small on short and long-context tasks. We release the training code and model weights under an Apache 2 license. In contrast with other open-source models, we release a training data loader with 235 million curated text pairs that allows for the full replication of nomic-embed-text-v1. You can find code and data to replicate the model at https://github.com/nomic-ai/contrastors
△ Less
Submitted 2 February, 2024;
originally announced February 2024.
-
GPT4All: An Ecosystem of Open Source Compressed Language Models
Authors:
Yuvanesh Anand,
Zach Nussbaum,
Adam Treat,
Aaron Miller,
Richard Guo,
Ben Schmidt,
GPT4All Community,
Brandon Duderstadt,
Andriy Mulyar
Abstract:
Large language models (LLMs) have recently achieved human-level performance on a range of professional and academic benchmarks. The accessibility of these models has lagged behind their performance. State-of-the-art LLMs require costly infrastructure; are only accessible via rate-limited, geo-locked, and censored web interfaces; and lack publicly available code and technical reports. In this paper…
▽ More
Large language models (LLMs) have recently achieved human-level performance on a range of professional and academic benchmarks. The accessibility of these models has lagged behind their performance. State-of-the-art LLMs require costly infrastructure; are only accessible via rate-limited, geo-locked, and censored web interfaces; and lack publicly available code and technical reports. In this paper, we tell the story of GPT4All, a popular open source repository that aims to democratize access to LLMs. We outline the technical details of the original GPT4All model family, as well as the evolution of the GPT4All project from a single model into a fully fledged open source ecosystem. It is our hope that this paper acts as both a technical overview of the original GPT4All models as well as a case study on the subsequent growth of the GPT4All open source ecosystem.
△ Less
Submitted 6 November, 2023;
originally announced November 2023.
-
Comparing Foundation Models using Data Kernels
Authors:
Brandon Duderstadt,
Hayden S. Helm,
Carey E. Priebe
Abstract:
Recent advances in self-supervised learning and neural network scaling have enabled the creation of large models, known as foundation models, which can be easily adapted to a wide range of downstream tasks. The current paradigm for comparing foundation models involves evaluating them with aggregate metrics on various benchmark datasets. This method of model comparison is heavily dependent on the c…
▽ More
Recent advances in self-supervised learning and neural network scaling have enabled the creation of large models, known as foundation models, which can be easily adapted to a wide range of downstream tasks. The current paradigm for comparing foundation models involves evaluating them with aggregate metrics on various benchmark datasets. This method of model comparison is heavily dependent on the chosen evaluation metric, which makes it unsuitable for situations where the ideal metric is either not obvious or unavailable. In this work, we present a methodology for directly comparing the embedding space geometry of foundation models, which facilitates model comparison without the need for an explicit evaluation metric. Our methodology is grounded in random graph theory and enables valid hypothesis testing of embedding similarity on a per-datum basis. Further, we demonstrate how our methodology can be extended to facilitate population level model comparison. In particular, we show how our framework can induce a manifold of models equipped with a distance function that correlates strongly with several downstream metrics. We remark on the utility of this population level model comparison as a first step towards a taxonomic science of foundation models.
△ Less
Submitted 7 January, 2024; v1 submitted 8 May, 2023;
originally announced May 2023.
-
A partition-based similarity for classification distributions
Authors:
Hayden S. Helm,
Ronak D. Mehta,
Brandon Duderstadt,
Weiwei Yang,
Christoper M. White,
Ali Geisa,
Joshua T. Vogelstein,
Carey E. Priebe
Abstract:
Herein we define a measure of similarity between classification distributions that is both principled from the perspective of statistical pattern recognition and useful from the perspective of machine learning practitioners. In particular, we propose a novel similarity on classification distributions, dubbed task similarity, that quantifies how an optimally-transformed optimal representation for a…
▽ More
Herein we define a measure of similarity between classification distributions that is both principled from the perspective of statistical pattern recognition and useful from the perspective of machine learning practitioners. In particular, we propose a novel similarity on classification distributions, dubbed task similarity, that quantifies how an optimally-transformed optimal representation for a source distribution performs when applied to inference related to a target distribution. The definition of task similarity allows for natural definitions of adversarial and orthogonal distributions. We highlight limiting properties of representations induced by (universally) consistent decision rules and demonstrate in simulation that an empirical estimate of task similarity is a function of the decision rule deployed for inference. We demonstrate that for a given target distribution, both transfer efficiency and semantic similarity of candidate source distributions correlate with empirical task similarity.
△ Less
Submitted 12 November, 2020;
originally announced November 2020.
-
NeuroStorm: Accelerating Brain Science Discovery in the Cloud
Authors:
Gregory Kiar,
Robert J. Anderson,
Alex Baden,
Alexandra Badea,
Eric W. Bridgeford,
Andrew Champion,
Vikram Chandrashekhar,
Forrest Collman,
Brandon Duderstadt,
Alan C. Evans,
Florian Engert,
Benjamin Falk,
Tristan Glatard,
William R. Gray Roncal,
David N. Kennedy,
Jeremy Maitin-Shepard,
Ryan A. Marren,
Onyeka Nnaemeka,
Eric Perlman,
Sharmishtaas Seshamani,
Eric T. Trautman,
Daniel J. Tward,
Pedro Antonio Valdés-Sosa,
Qing Wang,
Michael I. Miller
, et al. (2 additional authors not shown)
Abstract:
Neuroscientists are now able to acquire data at staggering rates across spatiotemporal scales. However, our ability to capitalize on existing datasets, tools, and intellectual capacities is hampered by technical challenges. The key barriers to accelerating scientific discovery correspond to the FAIR data principles: findability, global access to data, software interoperability, and reproducibility…
▽ More
Neuroscientists are now able to acquire data at staggering rates across spatiotemporal scales. However, our ability to capitalize on existing datasets, tools, and intellectual capacities is hampered by technical challenges. The key barriers to accelerating scientific discovery correspond to the FAIR data principles: findability, global access to data, software interoperability, and reproducibility/re-usability. We conducted a hackathon dedicated to making strides in those steps. This manuscript is a technical report summarizing these achievements, and we hope serves as an example of the effectiveness of focused, deliberate hackathons towards the advancement of our quickly-evolving field.
△ Less
Submitted 20 March, 2018; v1 submitted 8 March, 2018;
originally announced March 2018.