-
TQCompressor: improving tensor decomposition methods in neural networks via permutations
Authors:
V. Abronin,
A. Naumov,
D. Mazur,
D. Bystrov,
K. Tsarova,
Ar. Melnikov,
I. Oseledets,
S. Dolgov,
R. Brasher,
M. Perelshtein
Abstract:
We introduce TQCompressor, a novel method for neural network model compression with improved tensor decompositions. We explore the challenges posed by the computational and storage demands of pre-trained language models in NLP tasks and propose a permutation-based enhancement to Kronecker decomposition. This enhancement makes it possible to reduce loss in model expressivity which is usually associ…
▽ More
We introduce TQCompressor, a novel method for neural network model compression with improved tensor decompositions. We explore the challenges posed by the computational and storage demands of pre-trained language models in NLP tasks and propose a permutation-based enhancement to Kronecker decomposition. This enhancement makes it possible to reduce loss in model expressivity which is usually associated with factorization. We demonstrate this method applied to the GPT-2$_{small}$. The result of the compression is TQCompressedGPT-2 model, featuring 81 mln. parameters compared to 124 mln. in the GPT-2$_{small}$. We make TQCompressedGPT-2 publicly available. We further enhance the performance of the TQCompressedGPT-2 through a training strategy involving multi-step knowledge distillation, using only a 3.1% of the OpenWebText. TQCompressedGPT-2 surpasses DistilGPT-2 and KnGPT-2 in comparative evaluations, marking an advancement in the efficient and effective deployment of models in resource-constrained environments.
△ Less
Submitted 29 January, 2024;
originally announced January 2024.
-
Tetra-AML: Automatic Machine Learning via Tensor Networks
Authors:
A. Naumov,
Ar. Melnikov,
V. Abronin,
F. Oxanichenko,
K. Izmailov,
M. Pflitsch,
A. Melnikov,
M. Perelshtein
Abstract:
Neural networks have revolutionized many aspects of society but in the era of huge models with billions of parameters, optimizing and deploying them for commercial applications can require significant computational and financial resources. To address these challenges, we introduce the Tetra-AML toolbox, which automates neural architecture search and hyperparameter optimization via a custom-develop…
▽ More
Neural networks have revolutionized many aspects of society but in the era of huge models with billions of parameters, optimizing and deploying them for commercial applications can require significant computational and financial resources. To address these challenges, we introduce the Tetra-AML toolbox, which automates neural architecture search and hyperparameter optimization via a custom-developed black-box Tensor train Optimization algorithm, TetraOpt. The toolbox also provides model compression through quantization and pruning, augmented by compression using tensor networks. Here, we analyze a unified benchmark for optimizing neural networks in computer vision tasks and show the superior performance of our approach compared to Bayesian optimization on the CIFAR-10 dataset. We also demonstrate the compression of ResNet-18 neural networks, where we use 14.5 times less memory while losing just 3.2% of accuracy. The presented framework is generic, not limited by computer vision problems, supports hardware acceleration (such as with GPUs and TPUs) and can be further extended to quantum hardware and to hybrid quantum machine learning models.
△ Less
Submitted 28 March, 2023;
originally announced March 2023.
-
Do we live in a [quantum] simulation? Constraints, observations, and experiments on the simulation hypothesis
Authors:
Florian Neukart,
Anders Indset,
Markus Pflitsch,
Michael Perelshtein
Abstract:
The question "What is real?" can be traced back to the shadows in Plato's cave. Two thousand years later, Rene Descartes lacked knowledge about arguing against an evil deceiver feeding us the illusion of sensation. Descartes' epistemological concept later led to various theories of sensory experiences. The concept of "illusionism", proposing that even the very conscious experience we have is an il…
▽ More
The question "What is real?" can be traced back to the shadows in Plato's cave. Two thousand years later, Rene Descartes lacked knowledge about arguing against an evil deceiver feeding us the illusion of sensation. Descartes' epistemological concept later led to various theories of sensory experiences. The concept of "illusionism", proposing that even the very conscious experience we have is an illusion, is not only a red-pill scenario found in the 1999 science fiction movie "The Matrix" but is also a philosophical concept promoted by modern tinkers, most prominently by Daniel Dennett. Reflection upon a possible simulation and our perceived reality was beautifully visualized in "The Matrix", bringing the old ideas of Descartes to coffee houses around the world. Irish philosopher Bishop Berkeley was the father of what was later coined as "subjective idealism", basically stating that "what you perceive is real". With the advent of quantum technologies based on the control of individual fundamental particles, the question of whether our universe is a simulation isn't just intriguing. Our ever-advancing understanding of fundamental physical processes will likely lead us to build quantum computers utilizing quantum effects for simulating nature quantum-mechanically in all complexity, as famously envisioned by Richard Feynman. In this article, we outline constraints on the limits of computability and predictability in/of the universe, which we then use to design experiments allowing for first conclusions as to whether we participate in a simulation chain. Eventually, in a simulation in which the computer simulating a universe is governed by the same physical laws as the simulation, the exhaustion of computational resources will halt all simulations down the simulation chain unless an external programmer intervenes, which we may be able to observe.
△ Less
Submitted 12 December, 2022; v1 submitted 7 December, 2022;
originally announced December 2022.
-
Hybrid quantum ResNet for car classification and its hyperparameter optimization
Authors:
Asel Sagingalieva,
Mo Kordzanganeh,
Andrii Kurkin,
Artem Melnikov,
Daniil Kuhmistrov,
Michael Perelshtein,
Alexey Melnikov,
Andrea Skolik,
David Von Dollen
Abstract:
Image recognition is one of the primary applications of machine learning algorithms. Nevertheless, machine learning models used in modern image recognition systems consist of millions of parameters that usually require significant computational time to be adjusted. Moreover, adjustment of model hyperparameters leads to additional overhead. Because of this, new developments in machine learning mode…
▽ More
Image recognition is one of the primary applications of machine learning algorithms. Nevertheless, machine learning models used in modern image recognition systems consist of millions of parameters that usually require significant computational time to be adjusted. Moreover, adjustment of model hyperparameters leads to additional overhead. Because of this, new developments in machine learning models and hyperparameter optimization techniques are required. This paper presents a quantum-inspired hyperparameter optimization technique and a hybrid quantum-classical machine learning model for supervised learning. We benchmark our hyperparameter optimization method over standard black-box objective functions and observe performance improvements in the form of reduced expected run times and fitness in response to the growth in the size of the search space. We test our approaches in a car image classification task and demonstrate a full-scale implementation of the hybrid quantum ResNet model with the tensor train hyperparameter optimization. Our tests show a qualitative and quantitative advantage over the corresponding standard classical tabular grid search approach used with a deep neural network ResNet34. A classification accuracy of 0.97 was obtained by the hybrid model after 18 iterations, whereas the classical model achieved an accuracy of 0.92 after 75 iterations.
△ Less
Submitted 29 September, 2023; v1 submitted 10 May, 2022;
originally announced May 2022.
-
Practical application-specific advantage through hybrid quantum computing
Authors:
Michael Perelshtein,
Asel Sagingalieva,
Karan Pinto,
Vishal Shete,
Alexey Pakhomchik,
Artem Melnikov,
Florian Neukart,
Georg Gesek,
Alexey Melnikov,
Valerii Vinokur
Abstract:
Quantum computing promises to tackle technological and industrial problems insurmountable for classical computers. However, today's quantum computers still have limited demonstrable functionality, and it is expected that scaling up to millions of qubits is required for them to live up to this touted promise. The feasible route in achieving practical quantum advantage goals is to implement a hybrid…
▽ More
Quantum computing promises to tackle technological and industrial problems insurmountable for classical computers. However, today's quantum computers still have limited demonstrable functionality, and it is expected that scaling up to millions of qubits is required for them to live up to this touted promise. The feasible route in achieving practical quantum advantage goals is to implement a hybrid operational mode that realizes the cohesion of quantum and classical computers. Here we present a hybrid quantum cloud based on a memory-centric and heterogeneous multiprocessing architecture, integrated into a high-performance computing data center grade environment. We demonstrate that utilizing the quantum cloud, our hybrid quantum algorithms including Quantum Encoding (QuEnc), Hybrid Quantum Neural Networks and Tensor Networks enable advantages in optimization, machine learning, and simulation fields. We show the advantage of hybrid algorithms compared to standard classical algorithms in both the computational speed and quality of the solution. The achieved advance in hybrid quantum hardware and software makes quantum computing useful in practice today.
△ Less
Submitted 10 May, 2022;
originally announced May 2022.
