-
Mean-Field Assisted Deep Boltzmann Learning with Probabilistic Computers
Authors:
Shuvro Chowdhury,
Shaila Niazi,
Kerem Y. Camsari
Abstract:
Despite their appeal as physics-inspired, energy-based and generative nature, general Boltzmann Machines (BM) are considered intractable to train. This belief led to simplified models of BMs with restricted intralayer connections or layer-by-layer training of deep BMs. Recent developments in domain-specific hardware -- specifically probabilistic computers (p-computer) with probabilistic bits (p-bi…
▽ More
Despite their appeal as physics-inspired, energy-based and generative nature, general Boltzmann Machines (BM) are considered intractable to train. This belief led to simplified models of BMs with restricted intralayer connections or layer-by-layer training of deep BMs. Recent developments in domain-specific hardware -- specifically probabilistic computers (p-computer) with probabilistic bits (p-bit) -- may change established wisdom on the tractability of deep BMs. In this paper, we show that deep and unrestricted BMs can be trained using p-computers generating hundreds of billions of Markov Chain Monte Carlo (MCMC) samples per second, on sparse networks developed originally for use in D-Wave's annealers. To maximize the efficiency of learning the p-computer, we introduce two families of Mean-Field Theory assisted learning algorithms, or xMFTs (x = Naive and Hierarchical). The xMFTs are used to estimate the averages and correlations during the positive phase of the contrastive divergence (CD) algorithm and our custom-designed p-computer is used to estimate the averages and correlations in the negative phase. A custom Field-Programmable-Gate Array (FPGA) emulation of the p-computer architecture takes up to 45 billion flips per second, allowing the implementation of CD-$n$ where $n$ can be of the order of millions, unlike RBMs where $n$ is typically 1 or 2. Experiments on the full MNIST dataset with the combined algorithm show that the positive phase can be efficiently computed by xMFTs without much degradation when the negative phase is computed by the p-computer. Our algorithm can be used in other scalable Ising machines and its variants can be used to train BMs, previously thought to be intractable.
△ Less
Submitted 3 January, 2024;
originally announced January 2024.
-
The galactic tooth-fairy and a cosmic bullet: Amateur discoveries and a call for further research
Authors:
Muhammad Shaheer Niazi
Abstract:
There are countless digital sky surveys and automated scans of the night sky which use computer algorithms to detect and categorize objects. With the advent of Artificial Intelligence such surveys will become even more efficient in the near future. Despite this some objects are missed by surveys or pose no initial interest. At times such missed objects are unique in nature and of decent angular si…
▽ More
There are countless digital sky surveys and automated scans of the night sky which use computer algorithms to detect and categorize objects. With the advent of Artificial Intelligence such surveys will become even more efficient in the near future. Despite this some objects are missed by surveys or pose no initial interest. At times such missed objects are unique in nature and of decent angular sizes, demanding research, unlike the billions of tiny specs of galaxies that would be too tedious to name and study. In this scenario the amateur astronomer and their spirit for old school astronomical discovery steps in, to manually comb the sky and catalogue unique objects as was done in the early days of astronomy. In this paper two unique, previously uncatalogued galaxy candidates, namely Shaheer I and Shaheer II are identified and studied. Both galaxies lay at a distance of 6.67 arc-minutes from each other in the constellation of Camelopardalis. One boasts an unusual morphological profile, akin to a molar tooth, while the other seems to be shooting through space at tremendous velocities. The objects were discovered during visual inspection of digital surveys and then imaged from amateur telescopes at Taqwa observatory, Pakistan's first and only dark sky observatory (bortle 1). We perform photometry using PetroFit to discuss the potential nature of the galaxies and implore further collaborative research to fully uncover their characteristics.
△ Less
Submitted 26 September, 2023;
originally announced September 2023.
-
Cloud-native RStudio on Kubernetes for Hopsworks
Authors:
Gibson Chikafa,
Sina Sheikholeslami,
Salman Niazi,
Jim Dowling,
Vladimir Vlassov
Abstract:
In order to fully benefit from cloud computing, services are designed following the "multi-tenant" architectural model, which is aimed at maximizing resource sharing among users. However, multi-tenancy introduces challenges of security, performance isolation, scaling, and customization. RStudio server is an open-source Integrated Development Environment (IDE) accessible over a web browser for the…
▽ More
In order to fully benefit from cloud computing, services are designed following the "multi-tenant" architectural model, which is aimed at maximizing resource sharing among users. However, multi-tenancy introduces challenges of security, performance isolation, scaling, and customization. RStudio server is an open-source Integrated Development Environment (IDE) accessible over a web browser for the R programming language. We present the design and implementation of a multi-user distributed system on Hopsworks, a data-intensive AI platform, following the multi-tenant model that provides RStudio as Software as a Service (SaaS). We use the most popular cloud-native technologies: Docker and Kubernetes, to solve the problems of performance isolation, security, and scaling that are present in a multi-tenant environment. We further enable secure data sharing in RStudio server instances to provide data privacy and allow collaboration among RStudio users. We integrate our system with Apache Spark, which can scale and handle Big Data processing workloads. Also, we provide a UI where users can provide custom configurations and have full control of their own RStudio server instances. Our system was tested on a Google Cloud Platform cluster with four worker nodes, each with 30GB of RAM allocated to them. The tests on this cluster showed that 44 RStudio servers, each with 2GB of RAM, can be run concurrently. Our system can scale out to potentially support hundreds of concurrently running RStudio servers by adding more resources (CPUs and RAM) to the cluster or system.
△ Less
Submitted 18 July, 2023;
originally announced July 2023.
-
CMOS + stochastic nanomagnets: heterogeneous computers for probabilistic inference and learning
Authors:
Nihal Sanjay Singh,
Keito Kobayashi,
Qixuan Cao,
Kemal Selcuk,
Tianrui Hu,
Shaila Niazi,
Navid Anjum Aadit,
Shun Kanai,
Hideo Ohno,
Shunsuke Fukami,
Kerem Y. Camsari
Abstract:
Extending Moore's law by augmenting complementary-metal-oxide semiconductor (CMOS) transistors with emerging nanotechnologies (X) has become increasingly important. One important class of problems involve sampling-based Monte Carlo algorithms used in probabilistic machine learning, optimization, and quantum simulation. Here, we combine stochastic magnetic tunnel junction (sMTJ)-based probabilistic…
▽ More
Extending Moore's law by augmenting complementary-metal-oxide semiconductor (CMOS) transistors with emerging nanotechnologies (X) has become increasingly important. One important class of problems involve sampling-based Monte Carlo algorithms used in probabilistic machine learning, optimization, and quantum simulation. Here, we combine stochastic magnetic tunnel junction (sMTJ)-based probabilistic bits (p-bits) with Field Programmable Gate Arrays (FPGA) to create an energy-efficient CMOS + X (X = sMTJ) prototype. This setup shows how asynchronously driven CMOS circuits controlled by sMTJs can perform probabilistic inference and learning by leveraging the algorithmic update-order-invariance of Gibbs sampling. We show how the stochasticity of sMTJs can augment low-quality random number generators (RNG). Detailed transistor-level comparisons reveal that sMTJ-based p-bits can replace up to 10,000 CMOS transistors while dissipating two orders of magnitude less energy. Integrated versions of our approach can advance probabilistic computing involving deep Boltzmann machines and other energy-based learning algorithms with extremely high throughput and energy efficiency.
△ Less
Submitted 23 February, 2024; v1 submitted 12 April, 2023;
originally announced April 2023.
-
Training Deep Boltzmann Networks with Sparse Ising Machines
Authors:
Shaila Niazi,
Navid Anjum Aadit,
Masoud Mohseni,
Shuvro Chowdhury,
Yao Qin,
Kerem Y. Camsari
Abstract:
The slowing down of Moore's law has driven the development of unconventional computing paradigms, such as specialized Ising machines tailored to solve combinatorial optimization problems. In this paper, we show a new application domain for probabilistic bit (p-bit) based Ising machines by training deep generative AI models with them. Using sparse, asynchronous, and massively parallel Ising machine…
▽ More
The slowing down of Moore's law has driven the development of unconventional computing paradigms, such as specialized Ising machines tailored to solve combinatorial optimization problems. In this paper, we show a new application domain for probabilistic bit (p-bit) based Ising machines by training deep generative AI models with them. Using sparse, asynchronous, and massively parallel Ising machines we train deep Boltzmann networks in a hybrid probabilistic-classical computing setup. We use the full MNIST and Fashion MNIST (FMNIST) dataset without any downsampling and a reduced version of CIFAR-10 dataset in hardware-aware network topologies implemented in moderately sized Field Programmable Gate Arrays (FPGA). For MNIST, our machine using only 4,264 nodes (p-bits) and about 30,000 parameters achieves the same classification accuracy (90%) as an optimized software-based restricted Boltzmann Machine (RBM) with approximately 3.25 million parameters. Similar results follow for FMNIST and CIFAR-10. Additionally, the sparse deep Boltzmann network can generate new handwritten digits and fashion products, a task the 3.25 million parameter RBM fails at despite achieving the same accuracy. Our hybrid computer takes a measured 50 to 64 billion probabilistic flips per second, which is at least an order of magnitude faster than superficially similar Graphics and Tensor Processing Unit (GPU/TPU) based implementations. The massively parallel architecture can comfortably perform the contrastive divergence algorithm (CD-n) with up to n = 10 million sweeps per update, beyond the capabilities of existing software implementations. These results demonstrate the potential of using Ising machines for traditionally hard-to-train deep generative Boltzmann networks, with further possible improvement in nanodevice-based realizations.
△ Less
Submitted 23 January, 2024; v1 submitted 19 March, 2023;
originally announced March 2023.
-
A full-stack view of probabilistic computing with p-bits: devices, architectures and algorithms
Authors:
Shuvro Chowdhury,
Andrea Grimaldi,
Navid Anjum Aadit,
Shaila Niazi,
Masoud Mohseni,
Shun Kanai,
Hideo Ohno,
Shunsuke Fukami,
Luke Theogarajan,
Giovanni Finocchio,
Supriyo Datta,
Kerem Y. Camsari
Abstract:
The transistor celebrated its 75${}^\text{th}$ birthday in 2022. The continued scaling of the transistor defined by Moore's Law continues, albeit at a slower pace. Meanwhile, computing demands and energy consumption required by modern artificial intelligence (AI) algorithms have skyrocketed. As an alternative to scaling transistors for general-purpose computing, the integration of transistors with…
▽ More
The transistor celebrated its 75${}^\text{th}$ birthday in 2022. The continued scaling of the transistor defined by Moore's Law continues, albeit at a slower pace. Meanwhile, computing demands and energy consumption required by modern artificial intelligence (AI) algorithms have skyrocketed. As an alternative to scaling transistors for general-purpose computing, the integration of transistors with unconventional technologies has emerged as a promising path for domain-specific computing. In this article, we provide a full-stack review of probabilistic computing with p-bits as a representative example of the energy-efficient and domain-specific computing movement. We argue that p-bits could be used to build energy-efficient probabilistic systems, tailored for probabilistic algorithms and applications. From hardware, architecture, and algorithmic perspectives, we outline the main applications of probabilistic computers ranging from probabilistic machine learning and AI to combinatorial optimization and quantum simulation. Combining emerging nanodevices with the existing CMOS ecosystem will lead to probabilistic computers with orders of magnitude improvements in energy efficiency and probabilistic sampling, potentially unlocking previously unexplored regimes for powerful probabilistic algorithms.
△ Less
Submitted 16 March, 2023; v1 submitted 13 February, 2023;
originally announced February 2023.
-
HopsFS: Scaling Hierarchical File System Metadata Using NewSQL Databases
Authors:
Salman Niazi,
Mahmoud Ismail,
Steffen Grohsschmiedt,
Mikael Ronström,
Seif Haridi,
Jim Dowling
Abstract:
Recent improvements in both the performance and scalability of shared-nothing, transactional, in-memory NewSQL databases have reopened the research question of whether distributed metadata for hierarchical file systems can be managed using commodity databases. In this paper, we introduce HopsFS, a next generation distribution of the Hadoop Distributed File System (HDFS) that replaces HDFS' single…
▽ More
Recent improvements in both the performance and scalability of shared-nothing, transactional, in-memory NewSQL databases have reopened the research question of whether distributed metadata for hierarchical file systems can be managed using commodity databases. In this paper, we introduce HopsFS, a next generation distribution of the Hadoop Distributed File System (HDFS) that replaces HDFS' single node in-memory metadata service, with a distributed metadata service built on a NewSQL database. By removing the metadata bottleneck, HopsFS enables an order of magnitude larger and higher throughput clusters compared to HDFS. Metadata capacity has been increased to at least 37 times HDFS' capacity, and in experiments based on a workload trace from Spotify, we show that HopsFS supports 16 to 37 times the throughput of Apache HDFS. HopsFS also has lower latency for many concurrent clients, and no downtime during failover. Finally, as metadata is now stored in a commodity database, it can be safely extended and easily exported to external systems for online analysis and free-text search.
△ Less
Submitted 22 February, 2017; v1 submitted 5 June, 2016;
originally announced June 2016.
