-
RouterBench: A Benchmark for Multi-LLM Routing System
Authors:
Qitian Jason Hu,
Jacob Bieker,
Xiuyu Li,
Nan Jiang,
Benjamin Keigwin,
Gaurav Ranganath,
Kurt Keutzer,
Shriyash Kaustubh Upadhyay
Abstract:
As the range of applications for Large Language Models (LLMs) continues to grow, the demand for effective serving solutions becomes increasingly critical. Despite the versatility of LLMs, no single model can optimally address all tasks and applications, particularly when balancing performance with cost. This limitation has led to the development of LLM routing systems, which combine the strengths…
▽ More
As the range of applications for Large Language Models (LLMs) continues to grow, the demand for effective serving solutions becomes increasingly critical. Despite the versatility of LLMs, no single model can optimally address all tasks and applications, particularly when balancing performance with cost. This limitation has led to the development of LLM routing systems, which combine the strengths of various models to overcome the constraints of individual LLMs. Yet, the absence of a standardized benchmark for evaluating the performance of LLM routers hinders progress in this area. To bridge this gap, we present RouterBench, a novel evaluation framework designed to systematically assess the efficacy of LLM routing systems, along with a comprehensive dataset comprising over 405k inference outcomes from representative LLMs to support the development of routing strategies. We further propose a theoretical framework for LLM routing, and deliver a comparative analysis of various routing approaches through RouterBench, highlighting their potentials and limitations within our evaluation framework. This work not only formalizes and advances the development of LLM routing systems but also sets a standard for their assessment, paving the way for more accessible and economically viable LLM deployments. The code and data are available at https://github.com/withmartian/routerbench.
△ Less
Submitted 28 March, 2024; v1 submitted 18 March, 2024;
originally announced March 2024.
-
Data Assimilation using ERA5, ASOS, and the U-STN model for Weather Forecasting over the UK
Authors:
Wenqi Wang,
Jacob Bieker,
Rossella Arcucci,
César Quilodrán-Casas
Abstract:
In recent years, the convergence of data-driven machine learning models with Data Assimilation (DA) offers a promising avenue for enhancing weather forecasting. This study delves into this emerging trend, presenting our methodologies and outcomes. We harnessed the UK's local ERA5 850 hPa temperature data and refined the U-STN12 global weather forecasting model, tailoring its predictions to the UK'…
▽ More
In recent years, the convergence of data-driven machine learning models with Data Assimilation (DA) offers a promising avenue for enhancing weather forecasting. This study delves into this emerging trend, presenting our methodologies and outcomes. We harnessed the UK's local ERA5 850 hPa temperature data and refined the U-STN12 global weather forecasting model, tailoring its predictions to the UK's climate nuances. From the ASOS network, we sourced T2m data, representing ground observations across the UK. We employed the advanced kriging method with a polynomial drift term for consistent spatial resolution. Furthermore, Gaussian noise was superimposed on the ERA5 T850 data, setting the stage for ensuing multi-time step synthetic observations. Probing into the assimilation impacts, the ASOS T2m data was integrated with the ERA5 T850 dataset. Our insights reveal that while global forecast models can adapt to specific regions, incorporating atmospheric data in DA significantly bolsters model accuracy. Conversely, the direct assimilation of surface temperature data tends to mitigate this enhancement, tempering the model's predictive prowess.
△ Less
Submitted 15 January, 2024;
originally announced January 2024.
-
Discovering Effective Policies for Land-Use Planning with Neuroevolution
Authors:
Risto Miikkulainen,
Olivier Francon,
Daniel Young,
Elliot Meyerson,
Clemens Schwingshackl,
Jacob Bieker,
Hugo Cunha,
Babak Hodjat
Abstract:
How areas of land are allocated for different uses, such as forests, urban areas, and agriculture, has a large effect on the terrestrial carbon balance, and therefore climate change. Based on available historical data on land-use changes and a simulation of the associated carbon emissions and removals, a surrogate model can be learned that makes it possible to evaluate the different options availa…
▽ More
How areas of land are allocated for different uses, such as forests, urban areas, and agriculture, has a large effect on the terrestrial carbon balance, and therefore climate change. Based on available historical data on land-use changes and a simulation of the associated carbon emissions and removals, a surrogate model can be learned that makes it possible to evaluate the different options available to decision-makers efficiently. An evolutionary search process can then be used to discover effective land-use policies for specific locations. Such a system was built on the Project Resilience platform and evaluated with the Land-Use Harmonization dataset LUH2 and the bookkee** model BLUE. It generates Pareto fronts that trade off carbon impact and amount of land-use change customized to different locations, thus providing a potentially useful tool for land-use planning.
△ Less
Submitted 1 April, 2024; v1 submitted 20 November, 2023;
originally announced November 2023.
-
Comparing the carbon costs and benefits of low-resource solar nowcasting
Authors:
Ben Dixon,
María Pérez-Ortiz,
Jacob Bieker
Abstract:
Solar PV yield nowcasting is used to help anticipate peaks and troughs in demand to support grid integration. This paper compares multiple low-resource approaches to nowcasting solar PV yield, using a dataset of UK satellite imagery and solar PV energy readings over a 1 to 4-hour time range. The paper also estimates the carbon emissions generated and averted by deploying models, and finds that eve…
▽ More
Solar PV yield nowcasting is used to help anticipate peaks and troughs in demand to support grid integration. This paper compares multiple low-resource approaches to nowcasting solar PV yield, using a dataset of UK satellite imagery and solar PV energy readings over a 1 to 4-hour time range. The paper also estimates the carbon emissions generated and averted by deploying models, and finds that even small models that could be deployable in low-resource settings may have a benefit several orders of magnitude greater than its carbon cost. The paper also examines prediction errors and the activations in a CNN.
△ Less
Submitted 10 October, 2022;
originally announced October 2022.
-
The Evolution of Bulge-Dominated Field Galaxies from z~1 to the Present
Authors:
Charity Woodrum,
Inger Jørgensen,
R. Scott Fisher,
Lindsey Oberhelman,
Ricardo Demarco,
Taylor Contreras,
Jacob Bieker
Abstract:
We analyze the stellar populations and evolutionary history of bulge-dominated field galaxies at redshifts 0.3<z<1.2 as part of the Gemini/HST Galaxy Cluster Project (GCP). High signal-to-noise optical spectroscopy from the Gemini Observatory and imaging from the Hubble Space Telescope are used to analyze a total of 43 galaxies, focusing on the 30 passive galaxies in the sample. Using the size-mas…
▽ More
We analyze the stellar populations and evolutionary history of bulge-dominated field galaxies at redshifts 0.3<z<1.2 as part of the Gemini/HST Galaxy Cluster Project (GCP). High signal-to-noise optical spectroscopy from the Gemini Observatory and imaging from the Hubble Space Telescope are used to analyze a total of 43 galaxies, focusing on the 30 passive galaxies in the sample. Using the size-mass and velocity dispersion-mass relations for the passive field galaxies we find no significant evolution of sizes or velocity dispersions at a given dynamical mass between z~1 and the present. We establish the Fundamental Plane and study mass-to-light (M/L) ratios. The M/L vs. dynamical mass relation shows that the passive field galaxies follow a relation with a steeper slope than the local comparison sample, consistent with cluster galaxies in the GCP at z=0.86. This steeper slope indicates that the formation redshift is mass dependent, in agreement with "downsizing," meaning that the low mass galaxies formed their stars more recently while the high mass galaxies formed their stars at higher redshift. The zero point differences of the scaling relations for the M/L ratios imply a formation redshift of z_form=1.35(+0.10)(-0.07) for the passive field galaxies. This is consistent with the (Hdelta_A + Hgamma_A)' line index which implies a formation redshift of z_form=1.40(+0.60)(-0.18).
△ Less
Submitted 22 August, 2017;
originally announced August 2017.
