-
CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks
Authors:
Andrei Tomut,
Saeed S. Jahromi,
Abhijoy Sarkar,
Uygar Kurt,
Sukhbinder Singh,
Faysal Ishtiaq,
Cesar Muñoz,
Prabdeep Singh Bajaj,
Ali Elborady,
Gianni del Bimbo,
Mehrazin Alizadeh,
David Montero,
Pablo Martin-Ramiro,
Muhammad Ibrahim,
Oussama Tahiri Alaoui,
John Malcolm,
Samuel Mugel,
Roman Orus
Abstract:
Large Language Models (LLMs) such as ChatGPT and LlaMA are advancing rapidly in generative Artificial Intelligence (AI), but their immense size poses significant challenges, such as huge training and inference costs, substantial energy demands, and limitations for on-site deployment. Traditional compression methods such as pruning, distillation, and low-rank approximation focus on reducing the eff…
▽ More
Large Language Models (LLMs) such as ChatGPT and LlaMA are advancing rapidly in generative Artificial Intelligence (AI), but their immense size poses significant challenges, such as huge training and inference costs, substantial energy demands, and limitations for on-site deployment. Traditional compression methods such as pruning, distillation, and low-rank approximation focus on reducing the effective number of neurons in the network, while quantization focuses on reducing the numerical precision of individual weights to reduce the model size while kee** the number of neurons fixed. While these compression methods have been relatively successful in practice, there is no compelling reason to believe that truncating the number of neurons is an optimal strategy. In this context, this paper introduces CompactifAI, an innovative LLM compression approach using quantum-inspired Tensor Networks that focuses on the model's correlation space instead, allowing for a more controlled, refined and interpretable model compression. Our method is versatile and can be implemented with - or on top of - other compression techniques. As a benchmark, we demonstrate that a combination of CompactifAI with quantization allows to reduce a 93% the memory size of LlaMA 7B, reducing also 70% the number of parameters, accelerating 50% the training and 25% the inference times of the model, and just with a small accuracy drop of 2% - 3%, going much beyond of what is achievable today by other compression techniques. Our methods also allow to perform a refined layer sensitivity profiling, showing that deeper layers tend to be more suitable for tensor network compression, which is compatible with recent observations on the ineffectiveness of those layers for LLM performance. Our results imply that standard LLMs are, in fact, heavily overparametrized, and do not need to be large at all.
△ Less
Submitted 13 May, 2024; v1 submitted 25 January, 2024;
originally announced January 2024.
-
Financial Risk Management on a Neutral Atom Quantum Processor
Authors:
Lucas Leclerc,
Luis Ortiz-Guitierrez,
Sebastian Grijalva,
Boris Albrecht,
Julia R. K. Cline,
Vincent E. Elfving,
Adrien Signoles,
Loïc Henriet,
Gianni Del Bimbo,
Usman Ayub Sheikh,
Maitree Shah,
Luc Andrea,
Faysal Ishtiaq,
Andoni Duarte,
Samuel Mugel,
Irene Caceres,
Michel Kurek,
Roman Orus,
Achraf Seddik,
Oumaima Hammammi,
Hacene Isselnane,
Didier M'tamon
Abstract:
Machine Learning models capable of handling the large datasets collected in the financial world can often become black boxes expensive to run. The quantum computing paradigm suggests new optimization techniques, that combined with classical algorithms, may deliver competitive, faster and more interpretable models. In this work we propose a quantum-enhanced machine learning solution for the predict…
▽ More
Machine Learning models capable of handling the large datasets collected in the financial world can often become black boxes expensive to run. The quantum computing paradigm suggests new optimization techniques, that combined with classical algorithms, may deliver competitive, faster and more interpretable models. In this work we propose a quantum-enhanced machine learning solution for the prediction of credit rating downgrades, also known as fallen-angels forecasting in the financial risk management field. We implement this solution on a neutral atom Quantum Processing Unit with up to 60 qubits on a real-life dataset. We report competitive performances against the state-of-the-art Random Forest benchmark whilst our model achieves better interpretability and comparable training times. We examine how to improve performance in the near-term validating our ideas with Tensor Networks-based numerical simulations.
△ Less
Submitted 3 April, 2024; v1 submitted 6 December, 2022;
originally announced December 2022.
-
Quantum artificial vision for defect detection in manufacturing
Authors:
Daniel Guijo,
Victor Onofre,
Gianni Del Bimbo,
Samuel Mugel,
Daniel Estepa,
Xabier De Carlos,
Ana Adell,
Aizea Lojo,
Josu Bilbao,
Roman Orus
Abstract:
In this paper we consider several algorithms for quantum computer vision using Noisy Intermediate-Scale Quantum (NISQ) devices, and benchmark them for a real problem against their classical counterparts. Specifically, we consider two approaches: a quantum Support Vector Machine (QSVM) on a universal gate-based quantum computer, and QBoost on a quantum annealer. The quantum vision systems are bench…
▽ More
In this paper we consider several algorithms for quantum computer vision using Noisy Intermediate-Scale Quantum (NISQ) devices, and benchmark them for a real problem against their classical counterparts. Specifically, we consider two approaches: a quantum Support Vector Machine (QSVM) on a universal gate-based quantum computer, and QBoost on a quantum annealer. The quantum vision systems are benchmarked for an unbalanced dataset of images where the aim is to detect defects in manufactured car pieces. We see that the quantum algorithms outperform their classical counterparts in several ways, with QBoost allowing for larger problems to be analyzed with present-day quantum annealers. Data preprocessing, including dimensionality reduction and contrast enhancement, is also discussed, as well as hyperparameter tuning in QBoost. To the best of our knowledge, this is the first implementation of quantum computer vision systems for a problem of industrial relevance in a manufacturing production line.
△ Less
Submitted 5 January, 2024; v1 submitted 9 August, 2022;
originally announced August 2022.
