-
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
Authors:
Saleh Ashkboos,
Amirkeivan Mohtashami,
Maximilian L. Croci,
Bo Li,
Martin Jaggi,
Dan Alistarh,
Torsten Hoefler,
James Hensman
Abstract:
We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, as well as to th…
▽ More
We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism and to the KV cache. The result is a quantized model where all matrix multiplications are performed in 4-bits, without any channels identified for retention in higher precision. Our quantized LLaMa2-70B model has losses of at most 0.29 WikiText-2 perplexity and retains 99% of the zero-shot performance. Code is available at: https://github.com/spcl/QuaRot.
△ Less
Submitted 30 March, 2024;
originally announced April 2024.
-
SliceGPT: Compress Large Language Models by Deleting Rows and Columns
Authors:
Saleh Ashkboos,
Maximilian L. Croci,
Marcelo Gennari do Nascimento,
Torsten Hoefler,
James Hensman
Abstract:
Large language models have become the cornerstone of natural language processing, but their use comes with substantial costs in terms of compute and memory resources. Sparsification provides a solution to alleviate these resource constraints, and recent works have shown that trained models can be sparsified post-hoc. Existing sparsification techniques face challenges as they need additional data s…
▽ More
Large language models have become the cornerstone of natural language processing, but their use comes with substantial costs in terms of compute and memory resources. Sparsification provides a solution to alleviate these resource constraints, and recent works have shown that trained models can be sparsified post-hoc. Existing sparsification techniques face challenges as they need additional data structures and offer constrained speedup with current hardware. In this paper we present SliceGPT, a new post-training sparsification scheme which replaces each weight matrix with a smaller (dense) matrix, reducing the embedding dimension of the network. Through extensive experimentation, we show that SliceGPT can remove up to 25% of the model parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models while maintaining 99%, 99% and 90% zero-shot task performance of the dense model respectively. Our sliced models run on fewer GPUs and run faster without any additional code optimization: on 24GB consumer GPUs we reduce the total compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance in transformer networks, which enables SliceGPT and we hope it will inspire and enable future avenues to reduce memory and computation demands for pre-trained models. Code is available at: https://github.com/microsoft/TransformerCompression
△ Less
Submitted 9 February, 2024; v1 submitted 26 January, 2024;
originally announced January 2024.
-
Exposure Inequality in People Recommender Systems: The Long-Term Effects
Authors:
Francesco Fabbri,
Maria Luisa Croci,
Francesco Bonchi,
Carlos Castillo
Abstract:
People recommender systems may affect the exposure that users receive in social networking platforms, influencing attention dynamics and potentially strengthening pre-existing inequalities that disproportionately affect certain groups.
In this paper we introduce a model to simulate the feedback loop created by multiple rounds of interactions between users and a link recommender in a social netwo…
▽ More
People recommender systems may affect the exposure that users receive in social networking platforms, influencing attention dynamics and potentially strengthening pre-existing inequalities that disproportionately affect certain groups.
In this paper we introduce a model to simulate the feedback loop created by multiple rounds of interactions between users and a link recommender in a social network. This allows us to study the long-term consequences of those particular recommendation algorithms. Our model is equipped with several parameters to control (i) the level of homophily in the network, (ii) the relative size of the groups, (iii) the choice among several state-of-the-art link recommenders, and (iv) the choice among three different user behavior models, that decide which recommendations are accepted or rejected.
Our extensive experimentation with the proposed model shows that a minority group, if homophilic enough, can get a disproportionate advantage in exposure from all link recommenders. Instead, when it is heterophilic, it gets under-exposed. Moreover, while the homophily level of the minority affects the speed of the growth of the disparate exposure, the relative size of the minority affects the magnitude of the effect. Finally, link recommenders strengthen exposure inequalities at the individual level, exacerbating the "rich-get-richer" effect: this happens for both the minority and the majority class and independently of their level of homophily.
△ Less
Submitted 15 December, 2021;
originally announced December 2021.
-
Online parameter inference for the simulation of a Bunsen flame using heteroscedastic Bayesian neural network ensembles
Authors:
Maximilian L. Croci,
Ushnish Sengupta,
Matthew P. Juniper
Abstract:
This paper proposes a Bayesian data-driven machine learning method for the online inference of the parameters of a G-equation model of a ducted, premixed flame. Heteroscedastic Bayesian neural network ensembles are trained on a library of 1.7 million flame fronts simulated in LSGEN2D, a G-equation solver, to learn the Bayesian posterior distribution of the model parameters given observations. The…
▽ More
This paper proposes a Bayesian data-driven machine learning method for the online inference of the parameters of a G-equation model of a ducted, premixed flame. Heteroscedastic Bayesian neural network ensembles are trained on a library of 1.7 million flame fronts simulated in LSGEN2D, a G-equation solver, to learn the Bayesian posterior distribution of the model parameters given observations. The ensembles are then used to infer the parameters of Bunsen flame experiments so that the dynamics of these can be simulated in LSGEN2D. This allows the surface area variation of the flame edge, a proxy for the heat release rate, to be calculated. The proposed method provides cheap and online parameter and uncertainty estimates matching results obtained with the ensemble Kalman filter, at less computational cost. This enables fast and reliable simulation of the combustion process.
△ Less
Submitted 26 April, 2021;
originally announced April 2021.
-
Real-time parameter inference in reduced-order flame models with heteroscedastic Bayesian neural network ensembles
Authors:
Ushnish Sengupta,
Maximilian L. Croci,
Matthew P. Juniper
Abstract:
The estimation of model parameters with uncertainties from observed data is a ubiquitous inverse problem in science and engineering. In this paper, we suggest an inexpensive and easy to implement parameter estimation technique that uses a heteroscedastic Bayesian Neural Network trained using anchored ensembling. The heteroscedastic aleatoric error of the network models the irreducible uncertainty…
▽ More
The estimation of model parameters with uncertainties from observed data is a ubiquitous inverse problem in science and engineering. In this paper, we suggest an inexpensive and easy to implement parameter estimation technique that uses a heteroscedastic Bayesian Neural Network trained using anchored ensembling. The heteroscedastic aleatoric error of the network models the irreducible uncertainty due to parameter degeneracies in our inverse problem, while the epistemic uncertainty of the Bayesian model captures uncertainties which may arise from an input observation's out-of-distribution nature. We use this tool to perform real-time parameter inference in a 6 parameter G-equation model of a ducted, premixed flame from observations of acoustically excited flames. We train our networks on a library of 2.1 million simulated flame videos. Results on the test dataset of simulated flames show that the network recovers flame model parameters, with the correlation coefficient between predicted and true parameters ranging from 0.97 to 0.99, and well-calibrated uncertainty estimates. The trained neural networks are then used to infer model parameters from real videos of a premixed Bunsen flame captured using a high-speed camera in our lab. Re-simulation using inferred parameters shows excellent agreement between the real and simulated flames. Compared to Ensemble Kalman Filter-based tools that have been proposed for this problem in the combustion literature, our neural network ensemble achieves better data-efficiency and our sub-millisecond inference times represent a savings on computational costs by several orders of magnitude. This allows us to calibrate our reduced-order flame model in real-time and predict the thermoacoustic instability behaviour of the flame more accurately.
△ Less
Submitted 11 October, 2020;
originally announced November 2020.
