-
Paloma: A Benchmark for Evaluating Language Model Fit
Authors:
Ian Magnusson,
Akshita Bhagia,
Valentin Hofmann,
Luca Soldaini,
Ananya Harsh Jha,
Oyvind Tafjord,
Dustin Schwenk,
Evan Pete Walsh,
Yanai Elazar,
Kyle Lo,
Dirk Groeneveld,
Iz Beltagy,
Hannaneh Hajishirzi,
Noah A. Smith,
Kyle Richardson,
Jesse Dodge
Abstract:
Language models (LMs) commonly report perplexity on monolithic data held out from training. Implicitly or explicitly, this data is composed of domains$\unicode{x2013}$varying distributions of language. Rather than assuming perplexity on one distribution extrapolates to others, Perplexity Analysis for Language Model Assessment (Paloma), measures LM fit to 585 text domains, ranging from nytimes.com…
▽ More
Language models (LMs) commonly report perplexity on monolithic data held out from training. Implicitly or explicitly, this data is composed of domains$\unicode{x2013}$varying distributions of language. Rather than assuming perplexity on one distribution extrapolates to others, Perplexity Analysis for Language Model Assessment (Paloma), measures LM fit to 585 text domains, ranging from nytimes.com to r/depression on Reddit. We invite submissions to our benchmark and organize results by comparability based on compliance with guidelines such as removal of benchmark contamination from pretraining. Submissions can also record parameter and training token count to make comparisons of Pareto efficiency for performance as a function of these measures of cost. We populate our benchmark with results from 6 baselines pretrained on popular corpora. In case studies, we demonstrate analyses that are possible with Paloma, such as finding that pretraining without data beyond Common Crawl leads to inconsistent fit to many domains.
△ Less
Submitted 16 December, 2023;
originally announced December 2023.
-
Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation
Authors:
Hao Peng,
Qingqing Cao,
Jesse Dodge,
Matthew E. Peters,
Jared Fernandez,
Tom Sherborne,
Kyle Lo,
Sam Skjonsberg,
Emma Strubell,
Darrell Plessas,
Iz Beltagy,
Evan Pete Walsh,
Noah A. Smith,
Hannaneh Hajishirzi
Abstract:
Rising computational demands of modern natural language processing (NLP) systems have increased the barrier to entry for cutting-edge research while posing serious environmental concerns. Yet, progress on model efficiency has been impeded by practical challenges in model evaluation and comparison. For example, hardware is challenging to control due to disparate levels of accessibility across diffe…
▽ More
Rising computational demands of modern natural language processing (NLP) systems have increased the barrier to entry for cutting-edge research while posing serious environmental concerns. Yet, progress on model efficiency has been impeded by practical challenges in model evaluation and comparison. For example, hardware is challenging to control due to disparate levels of accessibility across different institutions. Moreover, improvements in metrics such as FLOPs often fail to translate to progress in real-world applications. In response, we introduce Pentathlon, a benchmark for holistic and realistic evaluation of model efficiency. Pentathlon focuses on inference, which accounts for a majority of the compute in a model's lifecycle. It offers a strictly-controlled hardware platform, and is designed to mirror real-world applications scenarios. It incorporates a suite of metrics that target different aspects of efficiency, including latency, throughput, memory overhead, and energy consumption. Pentathlon also comes with a software library that can be seamlessly integrated into any codebase and enable evaluation. As a standardized and centralized evaluation platform, Pentathlon can drastically reduce the workload to make fair and reproducible efficiency comparisons. While initially focused on natural language processing (NLP) models, Pentathlon is designed to allow flexible extension to other fields. We envision Pentathlon will stimulate algorithmic innovations in building efficient models, and foster an increased awareness of the social and environmental implications in the development of future-generation NLP models.
△ Less
Submitted 18 July, 2023;
originally announced July 2023.
-
How To Train Your (Compressed) Large Language Model
Authors:
Ananya Harsh Jha,
Tom Sherborne,
Evan Pete Walsh,
Dirk Groeneveld,
Emma Strubell,
Iz Beltagy
Abstract:
With the increase in the size of large language models (LLMs), we need compression methods that can reduce the model size while preserving the generality and zero-shot promptability of the model. This goal is more ambitious than the typical compression setup, which reduces the model's size at the expense of specializing it to a specific end-task. To study this, we develop a task-agnostic compressi…
▽ More
With the increase in the size of large language models (LLMs), we need compression methods that can reduce the model size while preserving the generality and zero-shot promptability of the model. This goal is more ambitious than the typical compression setup, which reduces the model's size at the expense of specializing it to a specific end-task. To study this, we develop a task-agnostic compression pipeline with a large-scale evaluation comprising language modeling perplexity and 12 zero-shot end-tasks. Our results show that a simple layer-wise pruning followed by continued language model pretraining matches or outperforms three existing state-of-the-art baselines while being 1.5x more computationally efficient. However, unlike typical task-specialized compression, our best-compressed model significantly underperforms a similar-sized model trained from scratch. We posit the half-sized pretrained model as an upper bound for task-agnostic compression and call for future work to bridge this gap under a reasonable token budget. Our findings highlight the inadequacy of existing compression methods for LLMs and establish a requirement for new methods that preserve a model's generality and zero-shot promptability under compression. We release our code and evaluation setup to facilitate reproducibility and help iterate on method design.
△ Less
Submitted 18 November, 2023; v1 submitted 24 May, 2023;
originally announced May 2023.
-
Method for in-solution, high-throughput T1 relaxometry using fluorescent nanodiamonds
Authors:
Erin. S. Grant,
Mina Barzegar Amiri Olia,
Ella. P. Walsh,
Liam T. Hall,
Gawain McColl,
David A. Simpson
Abstract:
Fluorescent nanodiamonds (FNDs) have been exploited as sensitive quantum probes for nanoscale chemical and biological sensing applications, with the majority of demonstrations to date relying on the detection of single FNDs. This places significant limits on the measurement time, throughput and statistical significance of a measured result as there is usually marked inhomogeneity within FND sample…
▽ More
Fluorescent nanodiamonds (FNDs) have been exploited as sensitive quantum probes for nanoscale chemical and biological sensing applications, with the majority of demonstrations to date relying on the detection of single FNDs. This places significant limits on the measurement time, throughput and statistical significance of a measured result as there is usually marked inhomogeneity within FND samples. Here we have developed a measurement platform that can report the T1 spin relaxation time from a large ensemble of FNDs in solution. We first describe a refined sensing protocol for this modality and then use it to identify the optimal FND size for the detection of paramagnetic targets. Our approach is simple to set up, robust and can be used for rapid material characterisation or a variety of in-situ quantum sensing applications.
△ Less
Submitted 27 November, 2022;
originally announced November 2022.
