-
Q-S5: Towards Quantized State Space Models
Authors:
Steven Abreu,
Jens E. Pedersen,
Kade M. Heckel,
Alessandro Pierro
Abstract:
In the quest for next-generation sequence modeling architectures, State Space Models (SSMs) have emerged as a potent alternative to transformers, particularly for their computational efficiency and suitability for dynamical systems. This paper investigates the effect of quantization on the S5 model to understand its impact on model performance and to facilitate its deployment to edge and resource-…
▽ More
In the quest for next-generation sequence modeling architectures, State Space Models (SSMs) have emerged as a potent alternative to transformers, particularly for their computational efficiency and suitability for dynamical systems. This paper investigates the effect of quantization on the S5 model to understand its impact on model performance and to facilitate its deployment to edge and resource-constrained platforms. Using quantization-aware training (QAT) and post-training quantization (PTQ), we systematically evaluate the quantization sensitivity of SSMs across different tasks like dynamical systems modeling, Sequential MNIST (sMNIST) and most of the Long Range Arena (LRA). We present fully quantized S5 models whose test accuracy drops less than 1% on sMNIST and most of the LRA. We find that performance on most tasks degrades significantly for recurrent weights below 8-bit precision, but that other components can be compressed further without significant loss of performance. Our results further show that PTQ only performs well on language-based LRA tasks whereas all others require QAT. Our investigation provides necessary insights for the continued development of efficient and hardware-optimized SSMs.
△ Less
Submitted 13 June, 2024;
originally announced June 2024.
-
Automated tomographic assessment of structural defects of freeze-dried pharmaceuticals
Authors:
Patric Müller,
Achim Sack,
Jens Dümler,
Michael Heckel,
Tim Wenzel,
Teresa Siegert,
Sonja Schuldt-Lieb,
Henning Gieseler,
Thorsten Pöschel
Abstract:
The topology and surface characteristics of lyophilisates significantly impact the stability and reconstitutability of freeze-dried pharmaceuticals. Consequently, visual quality control of the product is imperative. However, this procedure is not only time-consuming and labor-intensive but also expensive and prone to errors. In this paper, we present an approach for fully automated, non-destructiv…
▽ More
The topology and surface characteristics of lyophilisates significantly impact the stability and reconstitutability of freeze-dried pharmaceuticals. Consequently, visual quality control of the product is imperative. However, this procedure is not only time-consuming and labor-intensive but also expensive and prone to errors. In this paper, we present an approach for fully automated, non-destructive inspection of freeze-dried pharmaceuticals, leveraging robotics, computed tomography, and machine learning.
△ Less
Submitted 17 April, 2024;
originally announced April 2024.
-
Spyx: A Library for Just-In-Time Compiled Optimization of Spiking Neural Networks
Authors:
Kade M. Heckel,
Thomas Nowotny
Abstract:
As the role of artificial intelligence becomes increasingly pivotal in modern society, the efficient training and deployment of deep neural networks have emerged as critical areas of focus. Recent advancements in attention-based large neural architectures have spurred the development of AI accelerators, facilitating the training of extensive, multi-billion parameter models. Despite their effective…
▽ More
As the role of artificial intelligence becomes increasingly pivotal in modern society, the efficient training and deployment of deep neural networks have emerged as critical areas of focus. Recent advancements in attention-based large neural architectures have spurred the development of AI accelerators, facilitating the training of extensive, multi-billion parameter models. Despite their effectiveness, these powerful networks often incur high execution costs in production environments. Neuromorphic computing, inspired by biological neural processes, offers a promising alternative. By utilizing temporally-sparse computations, Spiking Neural Networks (SNNs) offer to enhance energy efficiency through a reduced and low-power hardware footprint. However, the training of SNNs can be challenging due to their recurrent nature which cannot as easily leverage the massive parallelism of modern AI accelerators. To facilitate the investigation of SNN architectures and dynamics researchers have sought to bridge Python-based deep learning frameworks such as PyTorch or TensorFlow with custom-implemented compute kernels. This paper introduces Spyx, a new and lightweight SNN simulation and optimization library designed in JAX. By pre-staging data in the expansive vRAM of contemporary accelerators and employing extensive JIT compilation, Spyx allows for SNN optimization to be executed as a unified, low-level program on NVIDIA GPUs or Google TPUs. This approach achieves optimal hardware utilization, surpassing the performance of many existing SNN training frameworks while maintaining considerable flexibility.
△ Less
Submitted 29 February, 2024;
originally announced February 2024.
-
Energy Dissipation in Driven Granular Matter in the Absence of Gravity
Authors:
Achim Sack,
Michael Heckel,
Jonathan E. Kollmer,
Fabian Zimber,
Thorsten Poeschel
Abstract:
We experimentally investigate the energy dissipation rate in sinusoidally driven boxes which are partly filled by granular material under conditions of weightlessness. We identify two different modes of granular dynamics, depending on the amplitude of driving, $A$. For intense forcing, A>A_0, the material is found in the collect-and-collide regime where the center of mass of the granulate moves sy…
▽ More
We experimentally investigate the energy dissipation rate in sinusoidally driven boxes which are partly filled by granular material under conditions of weightlessness. We identify two different modes of granular dynamics, depending on the amplitude of driving, $A$. For intense forcing, A>A_0, the material is found in the collect-and-collide regime where the center of mass of the granulate moves synchronously with the driven container while for weak forcing, A<A_0, the granular material exhibits gas-like behavior. Both regimes correspond to different dissipation mechanisms, leading to different scaling with amplitude and frequency of the excitation and with the mass of the granulate. For the collect-and-collide regime, we explain the dependence on frequency and amplitude of the excitation by means of an effective one-particle model. For both regimes, the results may be collapsed to a single curve characterizing the physics of granular dampers.
△ Less
Submitted 5 May, 2013;
originally announced May 2013.
-
Coefficient of Restitution as a Fluctuating Quantity
Authors:
Marina Montaine,
Michael Heckel,
Christof Kruelle,
Thomas Schwager,
Thorsten Poeschel
Abstract:
The coefficient of restitution of a spherical particle in contact with a flat plate is investigated as a function of the impact velocity. As an experimental observation we notice non-trivial (non-Gaussian) fluctuations of the measured values. For a fixed impact velocity, the probability density of the coefficient of restitution, $p(ε)$, is formed by two exponential functions (one increasing, one d…
▽ More
The coefficient of restitution of a spherical particle in contact with a flat plate is investigated as a function of the impact velocity. As an experimental observation we notice non-trivial (non-Gaussian) fluctuations of the measured values. For a fixed impact velocity, the probability density of the coefficient of restitution, $p(ε)$, is formed by two exponential functions (one increasing, one decreasing) of different slope. This behavior may be explained by a certain roughness of the particle which leads to energy transfer between the linear and rotational degrees of freedom.
△ Less
Submitted 31 March, 2011;
originally announced April 2011.
-
Movers and shakers: Granular dam** in microgravity
Authors:
Marcus N. Bannerman,
Jonathan E. Kollmer,
Achim Sack,
Michael Heckel,
Patric Mueller,
Thorsten Poeschel
Abstract:
The response of an oscillating granular damper to an initial perturbation is studied using experiments performed in microgravity and granular dynamics mulations. High-speed video and image processing techniques are used to extract experimental data. An inelastic hard sphere model is developed to perform simulations and the results are in excellent agreement with the experiments. The granular dampe…
▽ More
The response of an oscillating granular damper to an initial perturbation is studied using experiments performed in microgravity and granular dynamics mulations. High-speed video and image processing techniques are used to extract experimental data. An inelastic hard sphere model is developed to perform simulations and the results are in excellent agreement with the experiments. The granular damper behaves like a frictional damper and a linear decay of the amplitude is bserved. This is true even for the simulation model, where friction forces are absent. A simple expression is developed which predicts the optimal dam** conditions for a given amplitude and is independent of the oscillation frequency and particle inelasticities.
△ Less
Submitted 12 October, 2010;
originally announced October 2010.
