-
CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks
Authors:
Andrei Tomut,
Saeed S. Jahromi,
Abhijoy Sarkar,
Uygar Kurt,
Sukhbinder Singh,
Faysal Ishtiaq,
Cesar Muñoz,
Prabdeep Singh Bajaj,
Ali Elborady,
Gianni del Bimbo,
Mehrazin Alizadeh,
David Montero,
Pablo Martin-Ramiro,
Muhammad Ibrahim,
Oussama Tahiri Alaoui,
John Malcolm,
Samuel Mugel,
Roman Orus
Abstract:
Large Language Models (LLMs) such as ChatGPT and LlaMA are advancing rapidly in generative Artificial Intelligence (AI), but their immense size poses significant challenges, such as huge training and inference costs, substantial energy demands, and limitations for on-site deployment. Traditional compression methods such as pruning, distillation, and low-rank approximation focus on reducing the eff…
▽ More
Large Language Models (LLMs) such as ChatGPT and LlaMA are advancing rapidly in generative Artificial Intelligence (AI), but their immense size poses significant challenges, such as huge training and inference costs, substantial energy demands, and limitations for on-site deployment. Traditional compression methods such as pruning, distillation, and low-rank approximation focus on reducing the effective number of neurons in the network, while quantization focuses on reducing the numerical precision of individual weights to reduce the model size while kee** the number of neurons fixed. While these compression methods have been relatively successful in practice, there is no compelling reason to believe that truncating the number of neurons is an optimal strategy. In this context, this paper introduces CompactifAI, an innovative LLM compression approach using quantum-inspired Tensor Networks that focuses on the model's correlation space instead, allowing for a more controlled, refined and interpretable model compression. Our method is versatile and can be implemented with - or on top of - other compression techniques. As a benchmark, we demonstrate that a combination of CompactifAI with quantization allows to reduce a 93% the memory size of LlaMA 7B, reducing also 70% the number of parameters, accelerating 50% the training and 25% the inference times of the model, and just with a small accuracy drop of 2% - 3%, going much beyond of what is achievable today by other compression techniques. Our methods also allow to perform a refined layer sensitivity profiling, showing that deeper layers tend to be more suitable for tensor network compression, which is compatible with recent observations on the ineffectiveness of those layers for LLM performance. Our results imply that standard LLMs are, in fact, heavily overparametrized, and do not need to be large at all.
△ Less
Submitted 13 May, 2024; v1 submitted 25 January, 2024;
originally announced January 2024.
-
Boosting Defect Detection in Manufacturing using Tensor Convolutional Neural Networks
Authors:
Pablo Martin-Ramiro,
Unai Sainz de la Maza,
Sukhbinder Singh,
Roman Orus,
Samuel Mugel
Abstract:
Defect detection is one of the most important yet challenging tasks in the quality control stage in the manufacturing sector. In this work, we introduce a Tensor Convolutional Neural Network (T-CNN) and examine its performance on a real defect detection application in one of the components of the ultrasonic sensors produced at Robert Bosch's manufacturing plants. Our quantum-inspired T-CNN operate…
▽ More
Defect detection is one of the most important yet challenging tasks in the quality control stage in the manufacturing sector. In this work, we introduce a Tensor Convolutional Neural Network (T-CNN) and examine its performance on a real defect detection application in one of the components of the ultrasonic sensors produced at Robert Bosch's manufacturing plants. Our quantum-inspired T-CNN operates on a reduced model parameter space to substantially improve the training speed and performance of an equivalent CNN model without sacrificing accuracy. More specifically, we demonstrate how T-CNNs are able to reach the same performance as classical CNNs as measured by quality metrics, with up to fifteen times fewer parameters and 4% to 19% faster training times. Our results demonstrate that the T-CNN greatly outperforms the results of traditional human visual inspection, providing value in a current real application in manufacturing.
△ Less
Submitted 26 April, 2024; v1 submitted 29 December, 2023;
originally announced January 2024.
-
Application of Tensor Neural Networks to Pricing Bermudan Swaptions
Authors:
Raj G. Patel,
Tomas Dominguez,
Mohammad Dib,
Samuel Palmer,
Andrea Cadarso,
Fernando De Lope Contreras,
Abdelkader Ratnani,
Francisco Gomez Casanova,
Senaida Hernández-Santana,
Álvaro Díaz-Fernández,
Eva Andrés,
Jorge Luis-Hita,
Escolástico Sánchez-Martínez,
Samuel Mugel,
Roman Orus
Abstract:
The Cheyette model is a quasi-Gaussian volatility interest rate model widely used to price interest rate derivatives such as European and Bermudan Swaptions for which Monte Carlo simulation has become the industry standard. In low dimensions, these approaches provide accurate and robust prices for European Swaptions but, even in this computationally simple setting, they are known to underestimate…
▽ More
The Cheyette model is a quasi-Gaussian volatility interest rate model widely used to price interest rate derivatives such as European and Bermudan Swaptions for which Monte Carlo simulation has become the industry standard. In low dimensions, these approaches provide accurate and robust prices for European Swaptions but, even in this computationally simple setting, they are known to underestimate the value of Bermudan Swaptions when using the state variables as regressors. This is mainly due to the use of a finite number of predetermined basis functions in the regression. Moreover, in high-dimensional settings, these approaches succumb to the Curse of Dimensionality. To address these issues, Deep-learning techniques have been used to solve the backward Stochastic Differential Equation associated with the value process for European and Bermudan Swaptions; however, these methods are constrained by training time and memory. To overcome these limitations, we propose leveraging Tensor Neural Networks as they can provide significant parameter savings while attaining the same accuracy as classical Dense Neural Networks. In this paper we rigorously benchmark the performance of Tensor Neural Networks and Dense Neural Networks for pricing European and Bermudan Swaptions, and we show that Tensor Neural Networks can be trained faster than Dense Neural Networks and provide more accurate and robust prices than their Dense counterparts.
△ Less
Submitted 10 March, 2024; v1 submitted 18 April, 2023;
originally announced April 2023.
-
Quantum-Inspired Tensor Neural Networks for Option Pricing
Authors:
Raj G. Patel,
Chia-Wei Hsing,
Serkan Sahin,
Samuel Palmer,
Saeed S. Jahromi,
Shivam Sharma,
Tomas Dominguez,
Kris Tziritas,
Christophe Michel,
Vincent Porte,
Mustafa Abid,
Stephane Aubert,
Pierre Castellani,
Samuel Mugel,
Roman Orus
Abstract:
Recent advances in deep learning have enabled us to address the curse of dimensionality (COD) by solving problems in higher dimensions. A subset of such approaches of addressing the COD has led us to solving high-dimensional PDEs. This has resulted in opening doors to solving a variety of real-world problems ranging from mathematical finance to stochastic control for industrial applications. Altho…
▽ More
Recent advances in deep learning have enabled us to address the curse of dimensionality (COD) by solving problems in higher dimensions. A subset of such approaches of addressing the COD has led us to solving high-dimensional PDEs. This has resulted in opening doors to solving a variety of real-world problems ranging from mathematical finance to stochastic control for industrial applications. Although feasible, these deep learning methods are still constrained by training time and memory. Tackling these shortcomings, Tensor Neural Networks (TNN) demonstrate that they can provide significant parameter savings while attaining the same accuracy as compared to the classical Dense Neural Network (DNN). In addition, we also show how TNN can be trained faster than DNN for the same accuracy. Besides TNN, we also introduce Tensor Network Initializer (TNN Init), a weight initialization scheme that leads to faster convergence with smaller variance for an equivalent parameter count as compared to a DNN. We benchmark TNN and TNN Init by applying them to solve the parabolic PDE associated with the Heston model, which is widely used in financial pricing theory.
△ Less
Submitted 10 March, 2024; v1 submitted 28 December, 2022;
originally announced December 2022.
-
Financial Risk Management on a Neutral Atom Quantum Processor
Authors:
Lucas Leclerc,
Luis Ortiz-Guitierrez,
Sebastian Grijalva,
Boris Albrecht,
Julia R. K. Cline,
Vincent E. Elfving,
Adrien Signoles,
Loïc Henriet,
Gianni Del Bimbo,
Usman Ayub Sheikh,
Maitree Shah,
Luc Andrea,
Faysal Ishtiaq,
Andoni Duarte,
Samuel Mugel,
Irene Caceres,
Michel Kurek,
Roman Orus,
Achraf Seddik,
Oumaima Hammammi,
Hacene Isselnane,
Didier M'tamon
Abstract:
Machine Learning models capable of handling the large datasets collected in the financial world can often become black boxes expensive to run. The quantum computing paradigm suggests new optimization techniques, that combined with classical algorithms, may deliver competitive, faster and more interpretable models. In this work we propose a quantum-enhanced machine learning solution for the predict…
▽ More
Machine Learning models capable of handling the large datasets collected in the financial world can often become black boxes expensive to run. The quantum computing paradigm suggests new optimization techniques, that combined with classical algorithms, may deliver competitive, faster and more interpretable models. In this work we propose a quantum-enhanced machine learning solution for the prediction of credit rating downgrades, also known as fallen-angels forecasting in the financial risk management field. We implement this solution on a neutral atom Quantum Processing Unit with up to 60 qubits on a real-life dataset. We report competitive performances against the state-of-the-art Random Forest benchmark whilst our model achieves better interpretability and comparable training times. We examine how to improve performance in the near-term validating our ideas with Tensor Networks-based numerical simulations.
△ Less
Submitted 3 April, 2024; v1 submitted 6 December, 2022;
originally announced December 2022.
-
Financial Index Tracking via Quantum Computing with Cardinality Constraints
Authors:
Samuel Palmer,
Konstantinos Karagiannis,
Adam Florence,
Asier Rodriguez,
Roman Orus,
Harish Naik,
Samuel Mugel
Abstract:
In this work, we demonstrate how to apply non-linear cardinality constraints, important for real-world asset management, to quantum portfolio optimization. This enables us to tackle non-convex portfolio optimization problems using quantum annealing that would otherwise be challenging for classical algorithms. Being able to use cardinality constraints for portfolio optimization opens the doors to n…
▽ More
In this work, we demonstrate how to apply non-linear cardinality constraints, important for real-world asset management, to quantum portfolio optimization. This enables us to tackle non-convex portfolio optimization problems using quantum annealing that would otherwise be challenging for classical algorithms. Being able to use cardinality constraints for portfolio optimization opens the doors to new applications for creating innovative portfolios and exchange-traded-funds (ETFs). We apply the methodology to the practical problem of enhanced index tracking and are able to construct smaller portfolios that significantly outperform the risk profile of the target index whilst retaining high degrees of tracking.
△ Less
Submitted 24 August, 2022;
originally announced August 2022.
-
Quantum artificial vision for defect detection in manufacturing
Authors:
Daniel Guijo,
Victor Onofre,
Gianni Del Bimbo,
Samuel Mugel,
Daniel Estepa,
Xabier De Carlos,
Ana Adell,
Aizea Lojo,
Josu Bilbao,
Roman Orus
Abstract:
In this paper we consider several algorithms for quantum computer vision using Noisy Intermediate-Scale Quantum (NISQ) devices, and benchmark them for a real problem against their classical counterparts. Specifically, we consider two approaches: a quantum Support Vector Machine (QSVM) on a universal gate-based quantum computer, and QBoost on a quantum annealer. The quantum vision systems are bench…
▽ More
In this paper we consider several algorithms for quantum computer vision using Noisy Intermediate-Scale Quantum (NISQ) devices, and benchmark them for a real problem against their classical counterparts. Specifically, we consider two approaches: a quantum Support Vector Machine (QSVM) on a universal gate-based quantum computer, and QBoost on a quantum annealer. The quantum vision systems are benchmarked for an unbalanced dataset of images where the aim is to detect defects in manufactured car pieces. We see that the quantum algorithms outperform their classical counterparts in several ways, with QBoost allowing for larger problems to be analyzed with present-day quantum annealers. Data preprocessing, including dimensionality reduction and contrast enhancement, is also discussed, as well as hyperparameter tuning in QBoost. To the best of our knowledge, this is the first implementation of quantum computer vision systems for a problem of industrial relevance in a manufacturing production line.
△ Less
Submitted 5 January, 2024; v1 submitted 9 August, 2022;
originally announced August 2022.
-
Quantum-Inspired Tensor Neural Networks for Partial Differential Equations
Authors:
Raj Patel,
Chia-Wei Hsing,
Serkan Sahin,
Saeed S. Jahromi,
Samuel Palmer,
Shivam Sharma,
Christophe Michel,
Vincent Porte,
Mustafa Abid,
Stephane Aubert,
Pierre Castellani,
Chi-Guhn Lee,
Samuel Mugel,
Roman Orus
Abstract:
Partial Differential Equations (PDEs) are used to model a variety of dynamical systems in science and engineering. Recent advances in deep learning have enabled us to solve them in a higher dimension by addressing the curse of dimensionality in new ways. However, deep learning methods are constrained by training time and memory. To tackle these shortcomings, we implement Tensor Neural Networks (TN…
▽ More
Partial Differential Equations (PDEs) are used to model a variety of dynamical systems in science and engineering. Recent advances in deep learning have enabled us to solve them in a higher dimension by addressing the curse of dimensionality in new ways. However, deep learning methods are constrained by training time and memory. To tackle these shortcomings, we implement Tensor Neural Networks (TNN), a quantum-inspired neural network architecture that leverages Tensor Network ideas to improve upon deep learning approaches. We demonstrate that TNN provide significant parameter savings while attaining the same accuracy as compared to the classical Dense Neural Network (DNN). In addition, we also show how TNN can be trained faster than DNN for the same accuracy. We benchmark TNN by applying them to solve parabolic PDEs, specifically the Black-Scholes-Barenblatt equation, widely used in financial pricing theory, empirically showing the advantages of TNN over DNN. Further examples, such as the Hamilton-Jacobi-Bellman equation, are also discussed.
△ Less
Submitted 10 August, 2022; v1 submitted 3 August, 2022;
originally announced August 2022.
-
Dynamic Portfolio Optimization with Real Datasets Using Quantum Processors and Quantum-Inspired Tensor Networks
Authors:
Samuel Mugel,
Carlos Kuchkovsky,
Escolastico Sanchez,
Samuel Fernandez-Lorenzo,
Jorge Luis-Hita,
Enrique Lizaso,
Roman Orus
Abstract:
In this paper we tackle the problem of dynamic portfolio optimization, i.e., determining the optimal trading trajectory for an investment portfolio of assets over a period of time, taking into account transaction costs and other possible constraints. This problem is central to quantitative finance. After a detailed introduction to the problem, we implement a number of quantum and quantum-inspired…
▽ More
In this paper we tackle the problem of dynamic portfolio optimization, i.e., determining the optimal trading trajectory for an investment portfolio of assets over a period of time, taking into account transaction costs and other possible constraints. This problem is central to quantitative finance. After a detailed introduction to the problem, we implement a number of quantum and quantum-inspired algorithms on different hardware platforms to solve its discrete formulation using real data from daily prices over 8 years of 52 assets, and do a detailed comparison of the obtained Sharpe ratios, profits and computing times. In particular, we implement classical solvers (Gekko, exhaustive), D-Wave Hybrid quantum annealing, two different approaches based on Variational Quantum Eigensolvers on IBM-Q (one of them brand-new and tailored to the problem), and for the first time in this context also a quantum-inspired optimizer based on Tensor Networks. In order to fit the data into each specific hardware platform, we also consider doing a preprocessing based on clustering of assets. From our comparison, we conclude that D-Wave Hybrid and Tensor Networks are able to handle the largest systems, where we do calculations up to 1272 fully-connected qubits for demonstrative purposes. Finally, we also discuss how to mathematically implement other possible real-life constraints, as well as several ideas to further improve the performance of the studied methods.
△ Less
Submitted 6 December, 2021; v1 submitted 30 June, 2020;
originally announced July 2020.
