-
CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks
Authors:
Andrei Tomut,
Saeed S. Jahromi,
Abhijoy Sarkar,
Uygar Kurt,
Sukhbinder Singh,
Faysal Ishtiaq,
Cesar Muñoz,
Prabdeep Singh Bajaj,
Ali Elborady,
Gianni del Bimbo,
Mehrazin Alizadeh,
David Montero,
Pablo Martin-Ramiro,
Muhammad Ibrahim,
Oussama Tahiri Alaoui,
John Malcolm,
Samuel Mugel,
Roman Orus
Abstract:
Large Language Models (LLMs) such as ChatGPT and LlaMA are advancing rapidly in generative Artificial Intelligence (AI), but their immense size poses significant challenges, such as huge training and inference costs, substantial energy demands, and limitations for on-site deployment. Traditional compression methods such as pruning, distillation, and low-rank approximation focus on reducing the eff…
▽ More
Large Language Models (LLMs) such as ChatGPT and LlaMA are advancing rapidly in generative Artificial Intelligence (AI), but their immense size poses significant challenges, such as huge training and inference costs, substantial energy demands, and limitations for on-site deployment. Traditional compression methods such as pruning, distillation, and low-rank approximation focus on reducing the effective number of neurons in the network, while quantization focuses on reducing the numerical precision of individual weights to reduce the model size while kee** the number of neurons fixed. While these compression methods have been relatively successful in practice, there is no compelling reason to believe that truncating the number of neurons is an optimal strategy. In this context, this paper introduces CompactifAI, an innovative LLM compression approach using quantum-inspired Tensor Networks that focuses on the model's correlation space instead, allowing for a more controlled, refined and interpretable model compression. Our method is versatile and can be implemented with - or on top of - other compression techniques. As a benchmark, we demonstrate that a combination of CompactifAI with quantization allows to reduce a 93% the memory size of LlaMA 7B, reducing also 70% the number of parameters, accelerating 50% the training and 25% the inference times of the model, and just with a small accuracy drop of 2% - 3%, going much beyond of what is achievable today by other compression techniques. Our methods also allow to perform a refined layer sensitivity profiling, showing that deeper layers tend to be more suitable for tensor network compression, which is compatible with recent observations on the ineffectiveness of those layers for LLM performance. Our results imply that standard LLMs are, in fact, heavily overparametrized, and do not need to be large at all.
△ Less
Submitted 13 May, 2024; v1 submitted 25 January, 2024;
originally announced January 2024.
-
Multi-disk clutch optimization using quantum annealing
Authors:
John D. Malcolm,
Alexander Roth,
Mladjan Radic,
Pablo Martin-Ramiro,
Jon Oillarburu,
Borja Aizpurua,
Roman Orus,
Samuel Mugel
Abstract:
In this work, we develop a new quantum algorithm to solve a combinatorial problem with significant practical relevance occurring in clutch manufacturing. It is demonstrated how quantum optimization can play a role in real industrial applications in the manufacturing sector. Using the quantum annealer provided by D-Wave Systems, we analyze the performance of the quantum and quantum-classical hybrid…
▽ More
In this work, we develop a new quantum algorithm to solve a combinatorial problem with significant practical relevance occurring in clutch manufacturing. It is demonstrated how quantum optimization can play a role in real industrial applications in the manufacturing sector. Using the quantum annealer provided by D-Wave Systems, we analyze the performance of the quantum and quantum-classical hybrid solvers and compare them to deterministic- and random-algorithm classical benchmark solvers. The continued evolution of the quantum technology, indicating an expectation for even greater relevance in the future is discussed and the revolutionary potential it could have in the manufacturing sector is highlighted.
△ Less
Submitted 5 April, 2024; v1 submitted 11 August, 2022;
originally announced August 2022.
-
Multi Antenna Radar System for American Sign Language (ASL) Recognition Using Deep Learning
Authors:
Gavin MacLaughlin,
Jack Malcolm,
Syed Ali Hamza
Abstract:
This paper investigates RF-based system for automatic American Sign Language (ASL) recognition. We consider radar for ASL by joint spatio-temporal preprocessing of radar returns using time frequency (TF) analysis and high-resolution receive beamforming. The additional degrees of freedom offered by joint temporal and spatial processing using a multiple antenna sensor can help to recognize ASL conve…
▽ More
This paper investigates RF-based system for automatic American Sign Language (ASL) recognition. We consider radar for ASL by joint spatio-temporal preprocessing of radar returns using time frequency (TF) analysis and high-resolution receive beamforming. The additional degrees of freedom offered by joint temporal and spatial processing using a multiple antenna sensor can help to recognize ASL conversation between two or more individuals. This is performed by applying beamforming to collect spatial images in an attempt to resolve individuals communicating at the same time through hand and arm movements. The spatio-temporal images are fused and classified by a convolutional neural network (CNN) which is capable of discerning signs performed by different individuals even when the beamformer is unable to separate the respective signs completely. The focus group comprises individuals with varying expertise with sign language, and real time measurements at 77 GHz frequency are performed using Texas Instruments (TI) cascade radar.
△ Less
Submitted 30 March, 2022;
originally announced March 2022.
-
On the p-width of finite simple groups
Authors:
Alexander J. Malcolm
Abstract:
In this paper we measure how efficiently a finite simple group $G$ is generated by its elements of order $p$, where $p$ is a fixed prime. This measure, known as the $p$-width of $G$, is the minimal $k\in \mathbb{N}$ such that any $g\in G$ can be written as a product of at most $k$ elements of order $p$. Using primarily character theoretic methods, we sharply bound the $p$-width of some low rank fa…
▽ More
In this paper we measure how efficiently a finite simple group $G$ is generated by its elements of order $p$, where $p$ is a fixed prime. This measure, known as the $p$-width of $G$, is the minimal $k\in \mathbb{N}$ such that any $g\in G$ can be written as a product of at most $k$ elements of order $p$. Using primarily character theoretic methods, we sharply bound the $p$-width of some low rank families of Lie type groups, as well as the simple alternating and sporadic groups.
△ Less
Submitted 17 February, 2021; v1 submitted 2 March, 2020;
originally announced March 2020.
-
The p-width of the alternating groups
Authors:
Alexander J. Malcolm
Abstract:
Let $p$ be a fixed prime. For a finite group generated by elements of order $p$, the $p$-width is defined to be the minimal $k\in\mathbb{N}$ such that any group element can be written as a product of at most $k$ elements of order $p$. Let $A_{n}$ denote the alternating group of even permutations on $n$ letters. We show that the $p$-width of $A_{n}$ $(n\geq p)$ is at most $3$. This result is sharp,…
▽ More
Let $p$ be a fixed prime. For a finite group generated by elements of order $p$, the $p$-width is defined to be the minimal $k\in\mathbb{N}$ such that any group element can be written as a product of at most $k$ elements of order $p$. Let $A_{n}$ denote the alternating group of even permutations on $n$ letters. We show that the $p$-width of $A_{n}$ $(n\geq p)$ is at most $3$. This result is sharp, as there are families of alternating groups with $p$-width precisely 3, for each prime $p$.
△ Less
Submitted 8 December, 2017; v1 submitted 13 October, 2017;
originally announced October 2017.
-
An analytic evaluation of Kane fermion magneto-optics in two and three dimensions
Authors:
John D. Malcolm,
Elisabeth J. Nicol
Abstract:
We calculate and present an analytic form of the magneto-optical conductivity for the gapped low-energy Kane model in two and three dimensions separately. The two-dimensional case maps onto the $α$-$\mathcal{T}_3$ model at a particular value of $α=1/\sqrt{3}$. In two dimensions, two chiral sectors exist, between which there are no optically activated transitions. In three dimensions, the extra dim…
▽ More
We calculate and present an analytic form of the magneto-optical conductivity for the gapped low-energy Kane model in two and three dimensions separately. The two-dimensional case maps onto the $α$-$\mathcal{T}_3$ model at a particular value of $α=1/\sqrt{3}$. In two dimensions, two chiral sectors exist, between which there are no optically activated transitions. In three dimensions, the extra dimension of dispersion mixes the two sectors so that intra- and inter-sector transitions can occur. The latter type of transition can be separated out via circular polarizations of light and shows a distinct signature in the transverse conductivity.
△ Less
Submitted 30 November, 2016;
originally announced November 2016.
-
The involution width of finite simple groups
Authors:
Alexander J. Malcolm
Abstract:
For a finite group generated by involutions, the involution width is defined to be the minimal $k\in\mathbb{N}$ such that any group element can be written as a product of at most $k$ involutions. We show that the involution width of every non-abelian finite simple group is at most $4$. This result is sharp, as there are families with involution width precisely 4.
For a finite group generated by involutions, the involution width is defined to be the minimal $k\in\mathbb{N}$ such that any group element can be written as a product of at most $k$ involutions. We show that the involution width of every non-abelian finite simple group is at most $4$. This result is sharp, as there are families with involution width precisely 4.
△ Less
Submitted 21 November, 2016;
originally announced November 2016.
-
Frequency-dependent polarizability, plasmons, and screening in the 2D pseudospin-1 dice lattice
Authors:
John D. Malcolm,
Elisabeth J. Nicol
Abstract:
We calculate the dynamic polarizability under the random phase approximation for the dice lattice. This two-dimensional system gives rise to massless Dirac fermions with pseudospin-1 in the low-energy quantum excitation spectrum, providing a Dirac-cone plus flat-band dispersion. Due to the presence of the flat band, the polarizability shows key differences to that of graphene (the pseudospin-1/2 D…
▽ More
We calculate the dynamic polarizability under the random phase approximation for the dice lattice. This two-dimensional system gives rise to massless Dirac fermions with pseudospin-1 in the low-energy quantum excitation spectrum, providing a Dirac-cone plus flat-band dispersion. Due to the presence of the flat band, the polarizability shows key differences to that of graphene (the pseudospin-1/2 Dirac material). We find that the plasmon branch is pinched in to a single point, $ω_p=q=μ$, independent of the background dielectric constant. Finally, screening effects are discussed with regard to impurities.
△ Less
Submitted 11 March, 2016; v1 submitted 25 January, 2016;
originally announced January 2016.
-
Magneto-optics of massless Kane fermions: Role of the flat band and unusual Berry phase
Authors:
John D. Malcolm,
Elisabeth J. Nicol
Abstract:
HgCdTe at a critical cadmium do** has a bulk dispersion which includes two linear cones meeting at a single point at zero energy, intersecting a nearly flat band, similar to the pseudospin-1 Dirac-Weyl system. In the presence of a finite magnetic field, these bands condense into highly degenerate Landau levels. We have numerically calculated the frequency-dependent magneto-optical and zero-field…
▽ More
HgCdTe at a critical cadmium do** has a bulk dispersion which includes two linear cones meeting at a single point at zero energy, intersecting a nearly flat band, similar to the pseudospin-1 Dirac-Weyl system. In the presence of a finite magnetic field, these bands condense into highly degenerate Landau levels. We have numerically calculated the frequency-dependent magneto-optical and zero-field conductivity of this material using the Kane model. The calculations show good agreement with recent experimental measurements. We discuss the signature of the flat band and the split peaks of the magneto-optics in terms of general pseudospin-s models and propose that the system exhibits a non-π-quantized Berry phase, found in recent theoretical work.
△ Less
Submitted 9 July, 2015; v1 submitted 19 March, 2015;
originally announced March 2015.
-
Gas bubble dynamics in soft materials
Authors:
J. M. Solano-Altamirano,
John D. Malcolm,
Saul Goldman
Abstract:
Epstein and Plesset's seminal work on the rate of gas bubble dissolution and growth in a simple liquid is generalized to render it applicable to a gas bubble embedded in a soft elastic medium. Both the underlying diffusion equation and the expression for the gas bubble pressure were modified to allow for the non-zero shear modulus of the elastic medium. The extension of the diffusion equation resu…
▽ More
Epstein and Plesset's seminal work on the rate of gas bubble dissolution and growth in a simple liquid is generalized to render it applicable to a gas bubble embedded in a soft elastic medium. Both the underlying diffusion equation and the expression for the gas bubble pressure were modified to allow for the non-zero shear modulus of the elastic medium. The extension of the diffusion equation results in a trivial shift (by an additive constant) in the value of the diffusion coefficient, and does not change the form of the rate equations. But the use of a Generalized Young-Laplace equation for the bubble pressure resulted in significant differences on the dynamics of bubble dissolution and growth, relative to a simple liquid medium. Depending on whether the salient parameters (solute concentration, initial bubble radius, surface tension, and shear modulus) lead to bubble growth or dissolution, the effect of allowing for a non-zero shear modulus in the Generalized Young-Laplace equation is to speed up the rate of bubble growth, or to reduce the rate of bubble dissolution, respectively. The relation to previous work on visco-elastic materials is discussed, as is the connection of this work to the problem of Decompression Sickness (specifically, "the bends"). Examples of tissues to which our expressions can be applied are provided. Also, a new phenomenon is predicted whereby, for some parameter values, a bubble can be metastable and persist for long times, or it may grow, when embedded in a homogeneous under-saturated soft elastic medium.
△ Less
Submitted 13 October, 2014; v1 submitted 10 September, 2014;
originally announced September 2014.
-
Magneto-optics of general pseudospin-s two-dimensional Dirac-Weyl fermions
Authors:
John D. Malcolm,
Elisabeth J. Nicol
Abstract:
The popularity of graphene--a pseudospin-1/2 two-dimensional Dirac-Weyl material--has prompted the search for related materials and the characterization of their properties. In this work, the magneto-optical conductivity is calculated for systems that obey the general pseudospin-s two-dimensional Dirac-Weyl Hamiltonian, with particular focus on s = {1/2, 1, 3/2, 2}. This generalizes calculations t…
▽ More
The popularity of graphene--a pseudospin-1/2 two-dimensional Dirac-Weyl material--has prompted the search for related materials and the characterization of their properties. In this work, the magneto-optical conductivity is calculated for systems that obey the general pseudospin-s two-dimensional Dirac-Weyl Hamiltonian, with particular focus on s = {1/2, 1, 3/2, 2}. This generalizes calculations that have been made for s = 1/2 and follows previous work on the optical response of these systems in zero field. In the presence of a magnetic field, Landau levels condense out of the 2s+1 energy bands. As the chemical potential in a system is shifted, patterns arise in the appearance and disappearance of certain peaks within the optical spectra. These patterns are markedly different for each case considered, creating unique signatures in the magneto-optics. The general structure of each spectrum and how they compare is discussed.
△ Less
Submitted 8 July, 2014; v1 submitted 6 June, 2014;
originally announced June 2014.
-
Interferometry with Bose-Einstein Condensates in Microgravity
Authors:
H. Müntinga,
H. Ahlers,
M. Krutzik,
A. Wenzlawski,
S. Arnold,
D. Becker,
K. Bongs,
H. Dittus,
H. Duncker,
N. Gaaloul,
C. Gherasim,
E. Giese,
C. Grzeschik,
T. W. Hänsch,
O. Hellmig,
W. Herr,
S. Herrmann,
E. Kajari,
S. Kleinert,
C. Lämmerzahl,
W. Lewoczko-Adamczyk,
J. Malcolm,
N. Meyer,
R. Nolte,
A. Peters
, et al. (19 additional authors not shown)
Abstract:
Atom interferometers covering macroscopic domains of space-time are a spectacular manifestation of the wave nature of matter. Due to their unique coherence properties, Bose-Einstein condensates are ideal sources for an atom interferometer in extended free fall. In this paper we report on the realization of an asymmetric Mach-Zehnder interferometer operated with a Bose-Einstein condensate in microg…
▽ More
Atom interferometers covering macroscopic domains of space-time are a spectacular manifestation of the wave nature of matter. Due to their unique coherence properties, Bose-Einstein condensates are ideal sources for an atom interferometer in extended free fall. In this paper we report on the realization of an asymmetric Mach-Zehnder interferometer operated with a Bose-Einstein condensate in microgravity. The resulting interference pattern is similar to the one in the far-field of a double-slit and shows a linear scaling with the time the wave packets expand. We employ delta-kick cooling in order to enhance the signal and extend our atom interferometer. Our experiments demonstrate the high potential of interferometers operated with quantum gases for probing the fundamental concepts of quantum mechanics and general relativity.
△ Less
Submitted 24 January, 2013;
originally announced January 2013.
