-
HASS: Hardware-Aware Sparsity Search for Dataflow DNN Accelerator
Authors:
Zhewen Yu,
Sudarshan Sreeram,
Krish Agrawal,
Junyi Wu,
Alexander Montgomerie-Corcoran,
Cheng Zhang,
Jianyi Cheng,
Christos-Savvas Bouganis,
Yiren Zhao
Abstract:
Deep Neural Networks (DNNs) excel in learning hierarchical representations from raw data, such as images, audio, and text. To compute these DNN models with high performance and energy efficiency, these models are usually deployed onto customized hardware accelerators. Among various accelerator designs, dataflow architecture has shown promising performance due to its layer-pipelined structure and i…
▽ More
Deep Neural Networks (DNNs) excel in learning hierarchical representations from raw data, such as images, audio, and text. To compute these DNN models with high performance and energy efficiency, these models are usually deployed onto customized hardware accelerators. Among various accelerator designs, dataflow architecture has shown promising performance due to its layer-pipelined structure and its scalability in data parallelism.
Exploiting weights and activations sparsity can further enhance memory storage and computation efficiency. However, existing approaches focus on exploiting sparsity in non-dataflow accelerators, which cannot be applied onto dataflow accelerators because of the large hardware design space introduced. As such, this could miss opportunities to find an optimal combination of sparsity features and hardware designs.
In this paper, we propose a novel approach to exploit unstructured weights and activations sparsity for dataflow accelerators, using software and hardware co-optimization. We propose a Hardware-Aware Sparsity Search (HASS) to systematically determine an efficient sparsity solution for dataflow accelerators. Over a set of models, we achieve an efficiency improvement ranging from 1.3$\times$ to 4.2$\times$ compared to existing sparse designs, which are either non-dataflow or non-hardware-aware. Particularly, the throughput of MobileNetV3 can be optimized to 4895 images per second. HASS is open-source: \url{https://github.com/Yu-Zhewen/HASS}
△ Less
Submitted 5 June, 2024;
originally announced June 2024.
-
Probing Multimodal LLMs as World Models for Driving
Authors:
Shiva Sreeram,
Tsun-Hsuan Wang,
Alaa Maalouf,
Guy Rosman,
Sertac Karaman,
Daniela Rus
Abstract:
We provide a sober look at the application of Multimodal Large Language Models (MLLMs) within the domain of autonomous driving and challenge/verify some common assumptions, focusing on their ability to reason and interpret dynamic driving scenarios through sequences of images/frames in a closed-loop control environment. Despite the significant advancements in MLLMs like GPT-4V, their performance i…
▽ More
We provide a sober look at the application of Multimodal Large Language Models (MLLMs) within the domain of autonomous driving and challenge/verify some common assumptions, focusing on their ability to reason and interpret dynamic driving scenarios through sequences of images/frames in a closed-loop control environment. Despite the significant advancements in MLLMs like GPT-4V, their performance in complex, dynamic driving environments remains largely untested and presents a wide area of exploration. We conduct a comprehensive experimental study to evaluate the capability of various MLLMs as world models for driving from the perspective of a fixed in-car camera. Our findings reveal that, while these models proficiently interpret individual images, they struggle significantly with synthesizing coherent narratives or logical sequences across frames depicting dynamic behavior. The experiments demonstrate considerable inaccuracies in predicting (i) basic vehicle dynamics (forward/backward, acceleration/deceleration, turning right or left), (ii) interactions with other road actors (e.g., identifying speeding cars or heavy traffic), (iii) trajectory planning, and (iv) open-set dynamic scene reasoning, suggesting biases in the models' training data. To enable this experimental study we introduce a specialized simulator, DriveSim, designed to generate diverse driving scenarios, providing a platform for evaluating MLLMs in the realms of driving. Additionally, we contribute the full open-source code and a new dataset, "Eval-LLM-Drive", for evaluating MLLMs in driving. Our results highlight a critical gap in the current capabilities of state-of-the-art MLLMs, underscoring the need for enhanced foundation models to improve their applicability in real-world dynamic environments.
△ Less
Submitted 9 May, 2024;
originally announced May 2024.
-
Sculpting Efficiency: Pruning Medical Imaging Models for On-Device Inference
Authors:
Sudarshan Sreeram,
Bernhard Kainz
Abstract:
Leveraging ML advancements to augment healthcare systems can improve patient outcomes. Yet, uninformed engineering decisions in early-stage research inadvertently hinder the feasibility of such solutions for high-throughput, on-device inference, particularly in settings involving legacy hardware and multi-modal gigapixel images. Through a preliminary case study concerning segmentation in cardiolog…
▽ More
Leveraging ML advancements to augment healthcare systems can improve patient outcomes. Yet, uninformed engineering decisions in early-stage research inadvertently hinder the feasibility of such solutions for high-throughput, on-device inference, particularly in settings involving legacy hardware and multi-modal gigapixel images. Through a preliminary case study concerning segmentation in cardiology, we highlight the excess operational complexity in a suboptimally configured ML model from prior work and demonstrate that it can be sculpted away using pruning to meet deployment criteria. Our results show a compression rate of 1148x with minimal loss in quality (~4%) and, at higher rates, achieve faster inference on a CPU than the GPU baseline, stressing the need to consider task complexity and architectural details when using off-the-shelf models. With this, we consider avenues for future research in streamlining workflows for clinical researchers to develop models quicker and better suited for real-world use.
△ Less
Submitted 1 November, 2023; v1 submitted 10 September, 2023;
originally announced September 2023.
-
Infrastructure-based End-to-End Learning and Prevention of Driver Failure
Authors:
Noam Buckman,
Shiva Sreeram,
Mathias Lechner,
Yutong Ban,
Ramin Hasani,
Sertac Karaman,
Daniela Rus
Abstract:
Intelligent intersection managers can improve safety by detecting dangerous drivers or failure modes in autonomous vehicles, warning oncoming vehicles as they approach an intersection. In this work, we present FailureNet, a recurrent neural network trained end-to-end on trajectories of both nominal and reckless drivers in a scaled miniature city. FailureNet observes the poses of vehicles as they a…
▽ More
Intelligent intersection managers can improve safety by detecting dangerous drivers or failure modes in autonomous vehicles, warning oncoming vehicles as they approach an intersection. In this work, we present FailureNet, a recurrent neural network trained end-to-end on trajectories of both nominal and reckless drivers in a scaled miniature city. FailureNet observes the poses of vehicles as they approach an intersection and detects whether a failure is present in the autonomy stack, warning cross-traffic of potentially dangerous drivers. FailureNet can accurately identify control failures, upstream perception errors, and speeding drivers, distinguishing them from nominal driving. The network is trained and deployed with autonomous vehicles in the MiniCity. Compared to speed or frequency-based predictors, FailureNet's recurrent neural network structure provides improved predictive power, yielding upwards of 84% accuracy when deployed on hardware.
△ Less
Submitted 21 March, 2023;
originally announced March 2023.
-
Polydispersity-linked Memory Effects in a Magnetic Nanoparticle System
Authors:
1. S. Chakraverty,
2. A. Frydman,
3. M. Bandyopadhyay,
4. S. Dattagupta,
5. Surajit Sengupta 6. P. A. Sreeram
Abstract:
We have performed a series of measurements on the low temperature behavior of a magnetic nano-particle system. Our results show striking memory effects in the dc magnetization. Dipolar interactions among the nano-particles {\em suppress} the memory effect. We explain this phenomenon by the superposition of different super paramagnetic relaxation times of single domain magnetic nano- particles. M…
▽ More
We have performed a series of measurements on the low temperature behavior of a magnetic nano-particle system. Our results show striking memory effects in the dc magnetization. Dipolar interactions among the nano-particles {\em suppress} the memory effect. We explain this phenomenon by the superposition of different super paramagnetic relaxation times of single domain magnetic nano- particles. Moreover, we observe a crossover in the temperature dependence of coercivity. We show that a dilute dispersion of particles with a flat size distribution yields the best memory.
△ Less
Submitted 8 July, 2004;
originally announced July 2004.
