-
Training of Physical Neural Networks
Authors:
Ali Momeni,
Babak Rahmani,
Benjamin Scellier,
Logan G. Wright,
Peter L. McMahon,
Clara C. Wanjura,
Yuhang Li,
Anas Skalli,
Natalia G. Berloff,
Tatsuhiro Onodera,
Ilker Oguz,
Francesco Morichetti,
Philipp del Hougne,
Manuel Le Gallo,
Abu Sebastian,
Azalia Mirhoseini,
Cheng Zhang,
Danijela Marković,
Daniel Brunner,
Christophe Moser,
Sylvain Gigan,
Florian Marquardt,
Aydogan Ozcan,
Julie Grollier,
Andrea J. Liu
, et al. (3 additional authors not shown)
Abstract:
Physical neural networks (PNNs) are a class of neural-like networks that leverage the properties of physical systems to perform computation. While PNNs are so far a niche research area with small-scale laboratory demonstrations, they are arguably one of the most underappreciated important opportunities in modern AI. Could we train AI models 1000x larger than current ones? Could we do this and also…
▽ More
Physical neural networks (PNNs) are a class of neural-like networks that leverage the properties of physical systems to perform computation. While PNNs are so far a niche research area with small-scale laboratory demonstrations, they are arguably one of the most underappreciated important opportunities in modern AI. Could we train AI models 1000x larger than current ones? Could we do this and also have them perform inference locally and privately on edge devices, such as smartphones or sensors? Research over the past few years has shown that the answer to all these questions is likely "yes, with enough research": PNNs could one day radically change what is possible and practical for AI systems. To do this will however require rethinking both how AI models work, and how they are trained - primarily by considering the problems through the constraints of the underlying hardware physics. To train PNNs at large scale, many methods including backpropagation-based and backpropagation-free approaches are now being explored. These methods have various trade-offs, and so far no method has been shown to scale to the same scale and performance as the backpropagation algorithm widely used in deep learning today. However, this is rapidly changing, and a diverse ecosystem of training techniques provides clues for how PNNs may one day be utilized to create both more efficient realizations of current-scale AI models, and to enable unprecedented-scale models.
△ Less
Submitted 5 June, 2024;
originally announced June 2024.
-
Efficient Computation Using Spatial-Photonic Ising Machines: Utilizing Low-Rank and Circulant Matrix Constraints
Authors:
Richard Zhipeng Wang,
James S. Cummins,
Marvin Syed,
Nikita Stroev,
George Pastras,
Jason Sakellariou,
Symeon Tsintzos,
Alexis Askitopoulos,
Daniele Veraldi,
Marcello Calvanese Strinati,
Silvia Gentilini,
Davide Pierangeli,
Claudio Conti,
Natalia G. Berloff
Abstract:
We explore the potential of spatial-photonic Ising machines (SPIMs) to address computationally intensive Ising problems that employ low-rank and circulant coupling matrices. Our results indicate that the performance of SPIMs is critically affected by the rank and precision of the coupling matrices. By develo** and assessing advanced decomposition techniques, we expand the range of problems SPIMs…
▽ More
We explore the potential of spatial-photonic Ising machines (SPIMs) to address computationally intensive Ising problems that employ low-rank and circulant coupling matrices. Our results indicate that the performance of SPIMs is critically affected by the rank and precision of the coupling matrices. By develo** and assessing advanced decomposition techniques, we expand the range of problems SPIMs can solve, overcoming the limitations of traditional Mattis-type matrices. Our approach accommodates a diverse array of coupling matrices, including those with inherently low ranks, applicable to complex NP-complete problems. We explore the practical benefits of low-rank approximation in optimization tasks, particularly in financial optimization, to demonstrate the real-world applications of SPIMs. Finally, we evaluate the computational limitations imposed by SPIM hardware precision and suggest strategies to optimize the performance of these systems within these constraints.
△ Less
Submitted 3 June, 2024;
originally announced June 2024.
-
Coupling Light with Matter for Identifying Dominant Subnetworks
Authors:
Airat Kamaletdinov,
Natalia G. Berloff
Abstract:
We present a novel light-matter platform that uses complex-valued oscillator networks, a form of physical neural networks, to identify dominant subnetworks and uncover indirect correlations within larger networks. This approach offers significant advantages, including low energy consumption, high processing speed, and the immediate identification of co- and counter-regulated nodes without post-pro…
▽ More
We present a novel light-matter platform that uses complex-valued oscillator networks, a form of physical neural networks, to identify dominant subnetworks and uncover indirect correlations within larger networks. This approach offers significant advantages, including low energy consumption, high processing speed, and the immediate identification of co- and counter-regulated nodes without post-processing. The effectiveness of this approach is demonstrated through its application to biological networks, and we also propose its applicability to a wide range of other network types.
△ Less
Submitted 27 May, 2024;
originally announced May 2024.
-
Analog Iterative Machine (AIM): using light to solve quadratic optimization problems with mixed variables
Authors:
Kirill P. Kalinin,
George Mourgias-Alexandris,
Hitesh Ballani,
Natalia G. Berloff,
James H. Clegg,
Daniel Cletheroe,
Christos Gkantsidis,
Istvan Haller,
Vassily Lyutsarev,
Francesca Parmigiani,
Lucinda Pickup,
Antony Rowstron
Abstract:
Solving optimization problems is challenging for existing digital computers and even for future quantum hardware. The practical importance of diverse problems, from healthcare to financial optimization, has driven the emergence of specialised hardware over the past decade. However, their support for problems with only binary variables severely restricts the scope of practical problems that can be…
▽ More
Solving optimization problems is challenging for existing digital computers and even for future quantum hardware. The practical importance of diverse problems, from healthcare to financial optimization, has driven the emergence of specialised hardware over the past decade. However, their support for problems with only binary variables severely restricts the scope of practical problems that can be efficiently embedded. We build analog iterative machine (AIM), the first instance of an opto-electronic solver that natively implements a wider class of quadratic unconstrained mixed optimization (QUMO) problems and supports all-to-all connectivity of both continuous and binary variables.Beyond synthetic 7-bit problems at small-scale, AIM solves the financial transaction settlement problem entirely in analog domain with higher accuracy than quantum hardware and at room temperature. With compute-in-memory operation and spatial-division multiplexed representation of variables, the design of AIM paves the path to chip-scale architecture with 100 times speed-up per unit-power over the latest GPUs for solving problems with 10,000 variables. The robustness of the AIM algorithm at such scale is further demonstrated by comparing it with commercial production solvers across multiple benchmarks, where for several problems we report new best solutions. By combining the superior QUMO abstraction, sophisticated gradient descent methods inspired by machine learning, and commodity hardware, AIM introduces a novel platform with a step change in expressiveness, performance, and scalability, for optimization in the post-Moores law era.
△ Less
Submitted 20 June, 2023; v1 submitted 25 April, 2023;
originally announced April 2023.
-
Analog Photonics Computing for Information Processing, Inference and Optimisation
Authors:
Nikita Stroev,
Natalia G. Berloff
Abstract:
This review presents an overview of the current state-of-the-art in photonics computing, which leverages photons, photons coupled with matter, and optics-related technologies for effective and efficient computational purposes. It covers the history and development of photonics computing and modern analogue computing platforms and architectures, focusing on optimization tasks and neural network imp…
▽ More
This review presents an overview of the current state-of-the-art in photonics computing, which leverages photons, photons coupled with matter, and optics-related technologies for effective and efficient computational purposes. It covers the history and development of photonics computing and modern analogue computing platforms and architectures, focusing on optimization tasks and neural network implementations. The authors examine special-purpose optimizers, mathematical descriptions of photonics optimizers, and their various interconnections. Disparate applications are discussed, including direct encoding, logistics, finance, phase retrieval, machine learning, neural networks, probabilistic graphical models, and image processing, among many others. The main directions of technological advancement and associated challenges in photonics computing are explored, along with an assessment of its efficiency. Finally, the paper discusses prospects and the field of optical quantum computing, providing insights into the potential applications of this technology.
△ Less
Submitted 5 June, 2023; v1 submitted 27 January, 2023;
originally announced January 2023.
-
Complex matter field universal models with optimal scaling for solving combinatorial optimization problems
Authors:
Natalia G. Berloff
Abstract:
We develop a universal model based on the classical complex matter fields that allow the optimal map** of many real-life NP-hard combinatorial optimisation problems into the problem of minimising a spin Hamiltonian. We explicitly formulate one-to-one map** for three famous problems: graph colouring, the travelling salesman, and the modular N-queens problem. We show that such a formulation allo…
▽ More
We develop a universal model based on the classical complex matter fields that allow the optimal map** of many real-life NP-hard combinatorial optimisation problems into the problem of minimising a spin Hamiltonian. We explicitly formulate one-to-one map** for three famous problems: graph colouring, the travelling salesman, and the modular N-queens problem. We show that such a formulation allows for several orders of magnitude improvement in the search for the global minimum compared to the standard Ising formulation. At the same time, the amplitude dynamics escape from the local minima.
△ Less
Submitted 18 January, 2022;
originally announced January 2022.
-
Large-scale Sustainable Search on Unconventional Computing Hardware
Authors:
Kirill P. Kalinin,
Natalia G. Berloff
Abstract:
Since the advent of the Internet, quantifying the relative importance of web pages is at the core of search engine methods. According to one algorithm, PageRank, the worldwide web structure is represented by the Google matrix, whose principal eigenvector components assign a numerical value to web pages for their ranking. Finding such a dominant eigenvector on an ever-growing number of web pages be…
▽ More
Since the advent of the Internet, quantifying the relative importance of web pages is at the core of search engine methods. According to one algorithm, PageRank, the worldwide web structure is represented by the Google matrix, whose principal eigenvector components assign a numerical value to web pages for their ranking. Finding such a dominant eigenvector on an ever-growing number of web pages becomes a computationally intensive task incompatible with Moore's Law. We demonstrate that special-purpose optical machines such as networks of optical parametric oscillators, lasers, and gain-dissipative condensates, may aid in accelerating the reliable reconstruction of principal eigenvectors of real-life web graphs. We discuss the feasibility of simulating the PageRank algorithm on large Google matrices using such unconventional hardware. We offer alternative rankings based on the minimisation of spin Hamiltonians. Our estimates show that special-purpose optical machines may provide dramatic improvements in power consumption over classical computing architectures.
△ Less
Submitted 6 April, 2021;
originally announced April 2021.
-
XY Neural Networks
Authors:
Nikita Stroev,
Natalia G. Berloff
Abstract:
The classical XY model is a lattice model of statistical mechanics notable for its universality in the rich hierarchy of the optical, laser and condensed matter systems. We show how to build complex structures for machine learning based on the XY model's nonlinear blocks. The final target is to reproduce the deep learning architectures, which can perform complicated tasks usually attributed to suc…
▽ More
The classical XY model is a lattice model of statistical mechanics notable for its universality in the rich hierarchy of the optical, laser and condensed matter systems. We show how to build complex structures for machine learning based on the XY model's nonlinear blocks. The final target is to reproduce the deep learning architectures, which can perform complicated tasks usually attributed to such architectures: speech recognition, visual processing, or other complex classification types with high quality. We developed the robust and transparent approach for the construction of such models, which has universal applicability (i.e. does not strongly connect to any particular physical system), allows many possible extensions while at the same time preserving the simplicity of the methodology.
△ Less
Submitted 31 March, 2021;
originally announced March 2021.
-
Complexity continuum within Ising formulation of NP problems
Authors:
Kirill P. Kalinin,
Natalia G. Berloff
Abstract:
A promising approach to achieve computational supremacy over the classical von Neumann architecture explores classical and quantum hardware as Ising machines. The minimisation of the Ising Hamiltonian is known to be NP-hard problem for certain interaction matrix classes, yet not all problem instances are equivalently hard to optimise. We propose to identify computationally simple instances with an…
▽ More
A promising approach to achieve computational supremacy over the classical von Neumann architecture explores classical and quantum hardware as Ising machines. The minimisation of the Ising Hamiltonian is known to be NP-hard problem for certain interaction matrix classes, yet not all problem instances are equivalently hard to optimise. We propose to identify computationally simple instances with an `optimisation simplicity criterion'. Such optimisation simplicity can be found for a wide range of models from spin glasses to k-regular maximum cut problems. Many optical, photonic, and electronic systems are neuromorphic architectures that can naturally operate to optimise problems satisfying this criterion and, therefore, such problems are often chosen to illustrate the computational advantages of new Ising machines. We further probe an intermediate complexity for sparse and dense models by analysing circulant coupling matrices, that can be `rewired' to introduce greater complexity. A compelling approach for distinguishing easy and hard instances within the same NP-hard class of problems can be a starting point in develo** a standardised procedure for the performance evaluation of emerging physical simulators and physics-inspired algorithms.
△ Less
Submitted 2 August, 2020;
originally announced August 2020.
-
Polaritonic XY-Ising Machine
Authors:
Kirill P. Kalinin,
Alberto Amo,
Jacqueline Bloch,
Natalia G. Berloff
Abstract:
Gain-dissipative systems of various physical origin have recently shown the ability to act as analogue minimisers of hard combinatorial optimisation problems. Whether or not these proposals will lead to any advantage in performance over the classical computations depends on the ability to establish controllable couplings for sufficiently dense short- and long-range interactions between the spins.…
▽ More
Gain-dissipative systems of various physical origin have recently shown the ability to act as analogue minimisers of hard combinatorial optimisation problems. Whether or not these proposals will lead to any advantage in performance over the classical computations depends on the ability to establish controllable couplings for sufficiently dense short- and long-range interactions between the spins. Here, we propose a polaritonic XY-Ising machine based on a network of geometrically isolated polariton condensates capable of minimising discrete and continuous spin Hamiltonians. We elucidate the performance of the proposed computing platform for two types of couplings: relative and absolute. The interactions between the network nodes might be controlled by redirecting the emission between the condensates or by sending the phase information between nodes using resonant excitation. We discuss the conditions under which the proposed machine leads to a pure polariton simulator with pre-programmed couplings or results in a hybrid classical polariton simulator. We argue that the proposed architecture for the remote coupling control offers an improvement over geometrically coupled condensates in both accuracy and stability as well as increases versatility, range and connectivity of spin Hamiltonians that can be simulated with polariton networks.
△ Less
Submitted 20 March, 2020;
originally announced March 2020.
-
Nonlinear systems for unconventional computing
Authors:
Kirill P. Kalinin,
Natalia G. Berloff
Abstract:
The search for new computational machines beyond the traditional von Neumann architecture has given rise to a modern area of nonlinear science -- development of unconventional computing -- requiring the efforts of mathematicians, physicists and engineers. Many analogue physical systems including nonlinear oscillator networks, lasers, and condensates were proposed and realised to address hard compu…
▽ More
The search for new computational machines beyond the traditional von Neumann architecture has given rise to a modern area of nonlinear science -- development of unconventional computing -- requiring the efforts of mathematicians, physicists and engineers. Many analogue physical systems including nonlinear oscillator networks, lasers, and condensates were proposed and realised to address hard computational problems from various areas of social and physical sciences and technology. The analogue systems emulate spin Hamiltonians with continuous or discrete degrees of freedom to which actual optimisation problems can be mapped. Understanding the underlying physical process by which the system finds the ground state often leads to new classes of system-inspired or quantum-inspired algorithms for hard optimisation. Together physical platforms and related algorithms can be combined to form a hybrid architecture that may one day compete with conventional computing. In this Chapter, we review some of the systems and physically-inspired algorithms that show such promise.
△ Less
Submitted 26 December, 2019;
originally announced December 2019.
-
Discrete Polynomial Optimization with Coherent Networks of Condensates and Complex Coupling Switching
Authors:
Nikita Stroev,
Natalia G. Berloff
Abstract:
Gain-dissipative platforms consisting of lasers, optical parametric oscillators and nonequilibrium condensates operating at the condensation/coherence threshold have been recently proposed as efficient analog simulators of 2-local spin Hamiltonians with continuous or discrete degrees of freedom. We show that nonequilibrium condensates above the threshold arranged in an interacting network may real…
▽ More
Gain-dissipative platforms consisting of lasers, optical parametric oscillators and nonequilibrium condensates operating at the condensation/coherence threshold have been recently proposed as efficient analog simulators of 2-local spin Hamiltonians with continuous or discrete degrees of freedom. We show that nonequilibrium condensates above the threshold arranged in an interacting network may realise k-local Hamiltonians with k>2 and lead to nontrivial phase configurations. The principle of the operation of such a system lays the ground for physics-inspired computing and the new efficient methods for finding solutions to the higher order binary optimization problems. We show how to facilitate the search for the global solution by invoking complex couplings in the system and demonstrate the efficiency of the method on tensors with million entries. This approach offers a highly flexible new kind of computation based on gain-dissipative simulators with complex coupling switching. g.
△ Less
Submitted 18 May, 2020; v1 submitted 2 October, 2019;
originally announced October 2019.
-
Global optimization of spin Hamiltonians with gain-dissipative systems
Authors:
Kirill P. Kalinin,
Natalia G. Berloff
Abstract:
Recently, several platforms were proposed and demonstrated a proof-of-principle for finding the global minimum of the spin Hamiltonians such as the Ising and XY models using gain-dissipative quantum and classical systems. The implementation of dynamical adjustment of the gain and coupling strengths has been established as a vital feedback mechanism for analog Hamiltonian physical systems that aim…
▽ More
Recently, several platforms were proposed and demonstrated a proof-of-principle for finding the global minimum of the spin Hamiltonians such as the Ising and XY models using gain-dissipative quantum and classical systems. The implementation of dynamical adjustment of the gain and coupling strengths has been established as a vital feedback mechanism for analog Hamiltonian physical systems that aim to simulate spin Hamiltonians. Based on the principle of operation of such simulators we develop a novel class of gain-dissipative algorithms for global optimisation of NP-hard problems and show its performance in comparison with the classical global optimisation algorithms. These systems can be used to study the ground state and statistical properties of spin systems and as a direct benchmark for the performance testing of the gain-dissipative physical simulators. The estimates of the time operation of the physical implementation of the gain-dissipative simulators for large matrices show a possible speed-up of the several orders of magnitude in comparison with classical computations.
△ Less
Submitted 29 June, 2018;
originally announced July 2018.
