Large Language Model Routing with Benchmark Datasets
Authors:
Tal Shnitzer,
Anthony Ou,
MÃrian Silva,
Kate Soule,
Yuekai Sun,
Justin Solomon,
Neil Thompson,
Mikhail Yurochkin
Abstract:
There is a rapidly growing number of open-source Large Language Models (LLMs) and benchmark datasets to compare them. While some models dominate these benchmarks, no single model typically achieves the best accuracy in all tasks and use cases. In this work, we address the challenge of selecting the best LLM out of a collection of models for new tasks. We propose a new formulation for the problem,…
▽ More
There is a rapidly growing number of open-source Large Language Models (LLMs) and benchmark datasets to compare them. While some models dominate these benchmarks, no single model typically achieves the best accuracy in all tasks and use cases. In this work, we address the challenge of selecting the best LLM out of a collection of models for new tasks. We propose a new formulation for the problem, in which benchmark datasets are repurposed to learn a "router" model for this LLM selection, and we show that this problem can be reduced to a collection of binary classification tasks. We demonstrate the utility and limitations of learning model routers from various benchmark datasets, where we consistently improve performance upon using any single model for all tasks.
△ Less
Submitted 27 September, 2023;
originally announced September 2023.
Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via Full-Stack Integration
Authors:
Hasan Genc,
Seah Kim,
Alon Amid,
Ameer Haj-Ali,
Vighnesh Iyer,
Pranav Prakash,
Jerry Zhao,
Daniel Grubb,
Harrison Liew,
Howard Mao,
Albert Ou,
Colin Schmidt,
Samuel Steffl,
John Wright,
Ion Stoica,
Jonathan Ragan-Kelley,
Krste Asanovic,
Borivoje Nikolic,
Yakun Sophia Shao
Abstract:
DNN accelerators are often developed and evaluated in isolation without considering the cross-stack, system-level effects in real-world environments. This makes it difficult to appreciate the impact of System-on-Chip (SoC) resource contention, OS overheads, and programming-stack inefficiencies on overall performance/energy-efficiency. To address this challenge, we present Gemmini, an open-source*,…
▽ More
DNN accelerators are often developed and evaluated in isolation without considering the cross-stack, system-level effects in real-world environments. This makes it difficult to appreciate the impact of System-on-Chip (SoC) resource contention, OS overheads, and programming-stack inefficiencies on overall performance/energy-efficiency. To address this challenge, we present Gemmini, an open-source*, full-stack DNN accelerator generator. Gemmini generates a wide design-space of efficient ASIC accelerators from a flexible architectural template, together with flexible programming stacks and full SoCs with shared resources that capture system-level effects. Gemmini-generated accelerators have also been fabricated, delivering up to three orders-of-magnitude speedups over high-performance CPUs on various DNN benchmarks.
* https://github.com/ucb-bar/gemmini
△ Less
Submitted 9 July, 2021; v1 submitted 22 November, 2019;
originally announced November 2019.
