-
QCQA: Quality and Capacity-aware grouped Query Attention
Authors:
Vinay Joshi,
Prashant Laddha,
Shambhavi Sinha,
Om Ji Omer,
Sreenivas Subramoney
Abstract:
Excessive memory requirements of key and value features (KV-cache) present significant challenges in the autoregressive inference of large language models (LLMs), restricting both the speed and length of text generation. Approaches such as Multi-Query Attention (MQA) and Grouped Query Attention (GQA) mitigate these challenges by grou** query heads and consequently reducing the number of correspo…
▽ More
Excessive memory requirements of key and value features (KV-cache) present significant challenges in the autoregressive inference of large language models (LLMs), restricting both the speed and length of text generation. Approaches such as Multi-Query Attention (MQA) and Grouped Query Attention (GQA) mitigate these challenges by grou** query heads and consequently reducing the number of corresponding key and value heads. However, MQA and GQA decrease the KV-cache size requirements at the expense of LLM accuracy (quality of text generation). These methods do not ensure an optimal tradeoff between KV-cache size and text generation quality due to the absence of quality-aware grou** of query heads. To address this issue, we propose Quality and Capacity-Aware Grouped Query Attention (QCQA), which identifies optimal query head grou**s using an evolutionary algorithm with a computationally efficient and inexpensive fitness function. We demonstrate that QCQA achieves a significantly better tradeoff between KV-cache capacity and LLM accuracy compared to GQA. For the Llama2 $7\,$B model, QCQA achieves $\mathbf{20}$\% higher accuracy than GQA with similar KV-cache size requirements in the absence of fine-tuning. After fine-tuning both QCQA and GQA, for a similar KV-cache size, QCQA provides $\mathbf{10.55}\,$\% higher accuracy than GQA. Furthermore, QCQA requires $40\,$\% less KV-cache size than GQA to attain similar accuracy. The proposed quality and capacity-aware grou** of query heads can serve as a new paradigm for KV-cache optimization in autoregressive LLM inference.
△ Less
Submitted 8 June, 2024;
originally announced June 2024.
-
Convert any android device into a programmable IoT device with the help of IoT Everywhere Framework
Authors:
Vishnu Joshi
Abstract:
The world around us is transforming as the field of the Internet of Things is taking over the world faster than we thought. Everyone in the tech industry is building wonderful things with the help of IoT. Smartwatches, smart coffee machines, smart television, smart homes are some of the examples. Building IoT sensor modules with sensors that connect to the internet can be very intimidating for peo…
▽ More
The world around us is transforming as the field of the Internet of Things is taking over the world faster than we thought. Everyone in the tech industry is building wonderful things with the help of IoT. Smartwatches, smart coffee machines, smart television, smart homes are some of the examples. Building IoT sensor modules with sensors that connect to the internet can be very intimidating for people who have just stepped into the field. Quality components and microcontrollers can be costly too. Components such as proximity sensor, humidity sensor, air pressure sensor, accelerometer, gyroscope, flashlight, microphone, speaker, gsm module, wifi module, Bluetooth modules, and many more. But to program these we need to know java or kotlin and mobile application development. With the use of the IoT Everywhere framework and Origin programming language, one can convert any Android smartphone into an IoT device. This helps students of electrical engineering to grasp the idea of programming since it provides a lot of abstraction through simple function calls it can help to introduce programming to school students, it helps students who are fascinated by IoT and who wants to learn the basic of interfacing components or sensors and helps the student who has no access to an actual personal computer learn to program.
△ Less
Submitted 14 April, 2024;
originally announced May 2024.
-
CiMNet: Towards Joint Optimization for DNN Architecture and Configuration for Compute-In-Memory Hardware
Authors:
Souvik Kundu,
Anthony Sarah,
Vinay Joshi,
Om J Omer,
Sreenivas Subramoney
Abstract:
With the recent growth in demand for large-scale deep neural networks, compute in-memory (CiM) has come up as a prominent solution to alleviate bandwidth and on-chip interconnect bottlenecks that constrain Von-Neuman architectures. However, the construction of CiM hardware poses a challenge as any specific memory hierarchy in terms of cache sizes and memory bandwidth at different interfaces may no…
▽ More
With the recent growth in demand for large-scale deep neural networks, compute in-memory (CiM) has come up as a prominent solution to alleviate bandwidth and on-chip interconnect bottlenecks that constrain Von-Neuman architectures. However, the construction of CiM hardware poses a challenge as any specific memory hierarchy in terms of cache sizes and memory bandwidth at different interfaces may not be ideally matched to any neural network's attributes such as tensor dimension and arithmetic intensity, thus leading to suboptimal and under-performing systems. Despite the success of neural architecture search (NAS) techniques in yielding efficient sub-networks for a given hardware metric budget (e.g., DNN execution time or latency), it assumes the hardware configuration to be frozen, often yielding sub-optimal sub-networks for a given budget. In this paper, we present CiMNet, a framework that jointly searches for optimal sub-networks and hardware configurations for CiM architectures creating a Pareto optimal frontier of downstream task accuracy and execution metrics (e.g., latency). The proposed framework can comprehend the complex interplay between a sub-network's performance and the CiM hardware configuration choices including bandwidth, processing element size, and memory size. Exhaustive experiments on different model architectures from both CNN and Transformer families demonstrate the efficacy of the CiMNet in finding co-optimized sub-networks and CiM hardware configurations. Specifically, for similar ImageNet classification accuracy as baseline ViT-B, optimizing only the model architecture increases performance (or reduces workload execution time) by 1.7x while optimizing for both the model architecture and hardware configuration increases it by 3.1x.
△ Less
Submitted 18 March, 2024; v1 submitted 18 February, 2024;
originally announced February 2024.
-
Next-Generation Teleophthalmology: AI-enabled Quality Assessment Aiding Remote Smartphone-based Consultation
Authors:
Dhruv Srikanth,
Jayang Gurung,
N Satya Deepika,
Vineet Joshi,
Pravin Vaddavalli,
Soumya Jana
Abstract:
Blindness and other eye diseases are a global health concern, particularly in low- and middle-income countries like India. In this regard, during the COVID-19 pandemic, teleophthalmology became a lifeline, and the Grabi attachment for smartphone-based eye imaging gained in use. However, quality of user-captured image often remained inadequate, requiring clinician vetting and delays. In this backdr…
▽ More
Blindness and other eye diseases are a global health concern, particularly in low- and middle-income countries like India. In this regard, during the COVID-19 pandemic, teleophthalmology became a lifeline, and the Grabi attachment for smartphone-based eye imaging gained in use. However, quality of user-captured image often remained inadequate, requiring clinician vetting and delays. In this backdrop, we propose an AI-based quality assessment system with instant feedback mimicking clinicians' judgments and tested on patient-captured images. Dividing the complex problem hierarchically, here we tackle a nontrivial part, and demonstrate a proof of the concept.
△ Less
Submitted 11 February, 2024;
originally announced February 2024.
-
Streaming Bilingual End-to-End ASR model using Attention over Multiple Softmax
Authors:
Aditya Patil,
Vikas Joshi,
Purvi Agrawal,
Rupesh Mehta
Abstract:
Even with several advancements in multilingual modeling, it is challenging to recognize multiple languages using a single neural model, without knowing the input language and most multilingual models assume the availability of the input language. In this work, we propose a novel bilingual end-to-end (E2E) modeling approach, where a single neural model can recognize both languages and also support…
▽ More
Even with several advancements in multilingual modeling, it is challenging to recognize multiple languages using a single neural model, without knowing the input language and most multilingual models assume the availability of the input language. In this work, we propose a novel bilingual end-to-end (E2E) modeling approach, where a single neural model can recognize both languages and also support switching between the languages, without any language input from the user. The proposed model has shared encoder and prediction networks, with language-specific joint networks that are combined via a self-attention mechanism. As the language-specific posteriors are combined, it produces a single posterior probability over all the output symbols, enabling a single beam search decoding and also allowing dynamic switching between the languages. The proposed approach outperforms the conventional bilingual baseline with 13.3%, 8.23% and 1.3% word error rate relative reduction on Hindi, English and code-mixed test sets, respectively.
△ Less
Submitted 21 January, 2024;
originally announced January 2024.
-
TransAct: Transformer-based Realtime User Action Model for Recommendation at Pinterest
Authors:
Xue Xia,
Pong Eksombatchai,
Nikil Pancha,
Dhruvil Deven Badani,
Po-Wei Wang,
Neng Gu,
Saurabh Vishwas Joshi,
Nazanin Farahpour,
Zhiyuan Zhang,
Andrew Zhai
Abstract:
Sequential models that encode user activity for next action prediction have become a popular design choice for building web-scale personalized recommendation systems. Traditional methods of sequential recommendation either utilize end-to-end learning on realtime user actions, or learn user representations separately in an offline batch-generated manner. This paper (1) presents Pinterest's ranking…
▽ More
Sequential models that encode user activity for next action prediction have become a popular design choice for building web-scale personalized recommendation systems. Traditional methods of sequential recommendation either utilize end-to-end learning on realtime user actions, or learn user representations separately in an offline batch-generated manner. This paper (1) presents Pinterest's ranking architecture for Homefeed, our personalized recommendation product and the largest engagement surface; (2) proposes TransAct, a sequential model that extracts users' short-term preferences from their realtime activities; (3) describes our hybrid approach to ranking, which combines end-to-end sequential modeling via TransAct with batch-generated user embeddings. The hybrid approach allows us to combine the advantages of responsiveness from learning directly on realtime user activity with the cost-effectiveness of batch user representations learned over a longer time period. We describe the results of ablation studies, the challenges we faced during productionization, and the outcome of an online A/B experiment, which validates the effectiveness of our hybrid ranking model. We further demonstrate the effectiveness of TransAct on other surfaces such as contextual recommendations and search. Our model has been deployed to production in Homefeed, Related Pins, Notifications, and Search at Pinterest.
△ Less
Submitted 31 May, 2023;
originally announced June 2023.
-
Deep Learning assisted microwave-plasma interaction based technique for plasma density estimation
Authors:
Pratik Ghosh,
Bhaskar Chaudhury,
Shishir Purohit,
Vishv Joshi,
Ashray Kothari,
Devdeep Shetranjiwala
Abstract:
The electron density is a key parameter to characterize any plasma. Most of the plasma applications and research in the area of low-temperature plasmas (LTPs) are based on the accurate estimations of plasma density and plasma temperature. The conventional methods for electron density measurements offer axial and radial profiles for any given linear LTP device. These methods have major disadvantage…
▽ More
The electron density is a key parameter to characterize any plasma. Most of the plasma applications and research in the area of low-temperature plasmas (LTPs) are based on the accurate estimations of plasma density and plasma temperature. The conventional methods for electron density measurements offer axial and radial profiles for any given linear LTP device. These methods have major disadvantages of operational range (not very wide), cumbersome instrumentation, and complicated data analysis procedures. The article proposes a Deep Learning (DL) assisted microwave-plasma interaction-based non-invasive strategy, which can be used as a new alternative approach to address some of the challenges associated with existing plasma density measurement techniques. The electric field pattern due to microwave scattering from plasma is utilized to estimate the density profile. The proof of concept is tested for a simulated training data set comprising a low-temperature, unmagnetized, collisional plasma. Different types of symmetric (Gaussian-shaped) and asymmetrical density profiles, in the range $10^{16}-10^{19}$ m$^{-3}$, addressing a range of experimental configurations have been considered in our study. Real-life experimental issues such as the presence of noise and the amount of measured data (dense vs sparse) have been taken into consideration while preparing the synthetic training data-sets. The DL-based technique has the capability to determine the electron density profile within the plasma. The performance of the proposed deep learning-based approach has been evaluated using three metrics- SSIM, RMSLE, and MAPE. The obtained results show promising performance in estimating the 2D radial profile of the density for the given linear plasma device and affirms the potential of the proposed ML-based approach in plasma diagnostics.
△ Less
Submitted 28 June, 2023; v1 submitted 28 April, 2023;
originally announced April 2023.
-
Can AI Put Gamma-Ray Astrophysicists Out of a Job?
Authors:
Samuel T. Spencer,
Vikas Joshi,
Alison M. W. Mitchell
Abstract:
In what will likely be a litany of generative-model-themed arXiv submissions celebrating April the 1st, we evaluate the capacity of state-of-the-art transformer models to create a paper detailing the detection of a Pulsar Wind Nebula with a non-existent Imaging Atmospheric Cherenkov Telescope (IACT) Array. We do this to evaluate the ability of such models to interpret astronomical observations and…
▽ More
In what will likely be a litany of generative-model-themed arXiv submissions celebrating April the 1st, we evaluate the capacity of state-of-the-art transformer models to create a paper detailing the detection of a Pulsar Wind Nebula with a non-existent Imaging Atmospheric Cherenkov Telescope (IACT) Array. We do this to evaluate the ability of such models to interpret astronomical observations and sources based on language information alone, and to assess potential means by which fraudulently generated scientific papers could be identified during peer review (given that reliable generative model watermarking has yet to be deployed for these tools). We conclude that our jobs as astronomers are safe for the time being. From this point on, prompts given to ChatGPT and Stable Diffusion are shown in orange, text generated by ChatGPT is shown in black, whereas analysis by the (human) authors is in blue.
△ Less
Submitted 4 April, 2023; v1 submitted 31 March, 2023;
originally announced March 2023.
-
Evaluating XGBoost for Balanced and Imbalanced Data: Application to Fraud Detection
Authors:
Gissel Velarde,
Anindya Sudhir,
Sanjay Deshmane,
Anuj Deshmunkh,
Khushboo Sharma,
Vaibhav Joshi
Abstract:
This paper evaluates XGboost's performance given different dataset sizes and class distributions, from perfectly balanced to highly imbalanced. XGBoost has been selected for evaluation, as it stands out in several benchmarks due to its detection performance and speed. After introducing the problem of fraud detection, the paper reviews evaluation metrics for detection systems or binary classifiers,…
▽ More
This paper evaluates XGboost's performance given different dataset sizes and class distributions, from perfectly balanced to highly imbalanced. XGBoost has been selected for evaluation, as it stands out in several benchmarks due to its detection performance and speed. After introducing the problem of fraud detection, the paper reviews evaluation metrics for detection systems or binary classifiers, and illustrates with examples how different metrics work for balanced and imbalanced datasets. Then, it examines the principles of XGBoost. It proposes a pipeline for data preparation and compares a Vanilla XGBoost against a random search-tuned XGBoost. Random search fine-tuning provides consistent improvement for large datasets of 100 thousand samples, not so for medium and small datasets of 10 and 1 thousand samples, respectively. Besides, as expected, XGBoost recognition performance improves as more data is available, and deteriorates detection performance as the datasets become more imbalanced. Tests on distributions with 50, 45, 25, and 5 percent positive samples show that the largest drop in detection performance occurs for the distribution with only 5 percent positive samples. Sampling to balance the training set does not provide consistent improvement. Therefore, future work will include a systematic study of different techniques to deal with data imbalance and evaluating other approaches, including graphs, autoencoders, and generative adversarial methods, to deal with the lack of labels.
△ Less
Submitted 27 March, 2023;
originally announced March 2023.
-
Learning video embedding space with Natural Language Supervision
Authors:
Phani Krishna Uppala,
Abhishek Bamotra,
Shriti Priya,
Vaidehi Joshi
Abstract:
The recent success of the CLIP model has shown its potential to be applied to a wide range of vision and language tasks. However this only establishes embedding space relationship of language to images, not to the video domain. In this paper, we propose a novel approach to map video embedding space to natural langugage. We propose a two-stage approach that first extracts visual features from each…
▽ More
The recent success of the CLIP model has shown its potential to be applied to a wide range of vision and language tasks. However this only establishes embedding space relationship of language to images, not to the video domain. In this paper, we propose a novel approach to map video embedding space to natural langugage. We propose a two-stage approach that first extracts visual features from each frame of a video using a pre-trained CNN, and then uses the CLIP model to encode the visual features for the video domain, along with the corresponding text descriptions. We evaluate our method on two benchmark datasets, UCF101 and HMDB51, and achieve state-of-the-art performance on both tasks.
△ Less
Submitted 7 April, 2023; v1 submitted 25 March, 2023;
originally announced March 2023.
-
A 64-core mixed-signal in-memory compute chip based on phase-change memory for deep neural network inference
Authors:
Manuel Le Gallo,
Riduan Khaddam-Aljameh,
Milos Stanisavljevic,
Athanasios Vasilopoulos,
Benedikt Kersting,
Martino Dazzi,
Geethan Karunaratne,
Matthias Braendli,
Abhairaj Singh,
Silvia M. Mueller,
Julian Buechel,
Xavier Timoneda,
Vinay Joshi,
Urs Egger,
Angelo Garofalo,
Anastasios Petropoulos,
Theodore Antonakopoulos,
Kevin Brew,
Samuel Choi,
Injo Ok,
Timothy Philip,
Victor Chan,
Claire Silvestre,
Ishtiaq Ahsan,
Nicole Saulnier
, et al. (4 additional authors not shown)
Abstract:
The need to repeatedly shuttle around synaptic weight values from memory to processing units has been a key source of energy inefficiency associated with hardware implementation of artificial neural networks. Analog in-memory computing (AIMC) with spatially instantiated synaptic weights holds high promise to overcome this challenge, by performing matrix-vector multiplications (MVMs) directly withi…
▽ More
The need to repeatedly shuttle around synaptic weight values from memory to processing units has been a key source of energy inefficiency associated with hardware implementation of artificial neural networks. Analog in-memory computing (AIMC) with spatially instantiated synaptic weights holds high promise to overcome this challenge, by performing matrix-vector multiplications (MVMs) directly within the network weights stored on a chip to execute an inference workload. However, to achieve end-to-end improvements in latency and energy consumption, AIMC must be combined with on-chip digital operations and communication to move towards configurations in which a full inference workload is realized entirely on-chip. Moreover, it is highly desirable to achieve high MVM and inference accuracy without application-wise re-tuning of the chip. Here, we present a multi-core AIMC chip designed and fabricated in 14-nm complementary metal-oxide-semiconductor (CMOS) technology with backend-integrated phase-change memory (PCM). The fully-integrated chip features 64 256x256 AIMC cores interconnected via an on-chip communication network. It also implements the digital activation functions and processing involved in ResNet convolutional neural networks and long short-term memory (LSTM) networks. We demonstrate near software-equivalent inference accuracy with ResNet and LSTM networks while implementing all the computations associated with the weight layers and the activation functions on-chip. The chip can achieve a maximal throughput of 63.1 TOPS at an energy efficiency of 9.76 TOPS/W for 8-bit input/output matrix-vector multiplications.
△ Less
Submitted 6 December, 2022;
originally announced December 2022.
-
YZR-net : Self-supervised Hidden representations Invariant to Transformations for profanity detection
Authors:
Vedant Sandeep Joshi,
Sivanagaraja Tatinati,
Yubo Wang
Abstract:
On current {\it e-}learning platforms, live classes are an important tool that provides students with an opportunity to get more involved while learning new concepts. In such classes, the element of interaction with teachers and fellow peers helps in removing learning silos and gives each student a chance to experience some aspects relevant to offline learning in this era of virtual classes. One c…
▽ More
On current {\it e-}learning platforms, live classes are an important tool that provides students with an opportunity to get more involved while learning new concepts. In such classes, the element of interaction with teachers and fellow peers helps in removing learning silos and gives each student a chance to experience some aspects relevant to offline learning in this era of virtual classes. One common way of interaction in a class is through the chats / messaging framework, where the teacher can broadcast messages as well as get instant feedback from the students in the live class. This freedom of interaction is a crucial aspect for any student's learning growth but misuse of it can have serious repercussions. Some miscreants use this framework to send profane messages which can have a negative impact on other students as well as the teacher of the class. These rare but high impact situations obviate the need for automatic detection mechanisms that prevent the posting of such chats on any platform. In this work we develop YZR-Net which is a self-supervised framework that is able to robustly detect profane words used in a chat even if the student tries to add clever modifications to fool the system. The matching mechanism on token / word level allows us to maintain a compact as well as dynamic profane vocabulary which can be updated without retraining the underlying model. Our profanity detection framework is language independent and can handle abuses in both English as well as its transliterated counterpart Hinglish (Hindi language words written in English).
△ Less
Submitted 22 November, 2022;
originally announced November 2022.
-
EEG aided boosting of single-lead ECG based sleep staging with Deep Knowledge Distillation
Authors:
Vaibhav Joshi,
Sricharan V,
Preejith SP,
Mohanasankar Sivaprakasam
Abstract:
An electroencephalogram (EEG) signal is currently accepted as a standard for automatic sleep staging. Lately, Near-human accuracy in automated sleep staging has been achievable by Deep Learning (DL) based approaches, enabling multi-fold progress in this area. However, An extensive and expensive clinical setup is required for EEG based sleep staging. Additionally, the EEG setup being obtrusive in n…
▽ More
An electroencephalogram (EEG) signal is currently accepted as a standard for automatic sleep staging. Lately, Near-human accuracy in automated sleep staging has been achievable by Deep Learning (DL) based approaches, enabling multi-fold progress in this area. However, An extensive and expensive clinical setup is required for EEG based sleep staging. Additionally, the EEG setup being obtrusive in nature and requiring an expert for setup adds to the inconvenience of the subject under study, making it adverse in the point of care setting. An unobtrusive and more suitable alternative to EEG is Electrocardiogram (ECG). Unsurprisingly, compared to EEG in sleep staging, its performance remains sub-par. In order to take advantage of both the modalities, transferring knowledge from EEG to ECG is a reasonable approach, ultimately boosting the performance of ECG based sleep staging. Knowledge Distillation (KD) is a promising notion in DL that shares knowledge from a superior performing but usually more complex teacher model to an inferior but compact student model. Building upon this concept, a cross-modality KD framework assisting features learned through models trained on EEG to improve ECG-based sleep staging performance is proposed. Additionally, to better understand the distillation approach, extensive experimentation on the independent modules of the proposed model was conducted. Montreal Archive of Sleep Studies (MASS) dataset consisting of 200 subjects was utilized for this study. The results from the proposed model for weighted-F1-score in 3-class and 4-class sleep staging showed a 13.40 \% and 14.30 \% improvement, respectively. This study demonstrates the feasibility of KD for single-channel ECG based sleep staging's performance enhancement in 3-class (W-R-N) and 4-class (W-R-L-D) classification.
△ Less
Submitted 18 November, 2022;
originally announced November 2022.
-
Looking For A Match: Self-supervised Clustering For Automatic Doubt Matching In e-learning Platforms
Authors:
Vedant Sandeep Joshi,
Sivanagaraja Tatinati,
Yubo Wang
Abstract:
Recently, e-learning platforms have grown as a place where students can post doubts (as a snap taken with smart phones) and get them resolved in minutes. However, the significant increase in the number of student-posted doubts with high variance in quality on these platforms not only presents challenges for teachers' navigation to address them but also increases the resolution time per doubt. Both…
▽ More
Recently, e-learning platforms have grown as a place where students can post doubts (as a snap taken with smart phones) and get them resolved in minutes. However, the significant increase in the number of student-posted doubts with high variance in quality on these platforms not only presents challenges for teachers' navigation to address them but also increases the resolution time per doubt. Both are not acceptable, as high doubt resolution time hinders the students learning progress. This necessitates ways to automatically identify if there exists a similar doubt in repository and then serve it to the teacher as the plausible solution to validate and communicate with the student. Supervised learning techniques (like Siamese architecture) require labels to identify the matches, which is not feasible as labels are scarce and expensive. In this work, we, thus, developed a label-agnostic doubt matching paradigm based on the representations learnt via self-supervised technique. Building on prior theoretical insights of BYOL (bootstrap your own latent space), we propose custom BYOL which combines domain-specific augmentation with contrastive objective over a varied set of appropriately constructed data views. Results highlighted that, custom BYOL improves the top-1 matching accuracy by approximately 6\% and 5\% as compared to both BYOL and supervised learning instances, respectively. We further show that both BYOL-based learning instances performs either on par or better than human labeling.
△ Less
Submitted 20 August, 2022;
originally announced August 2022.
-
WavFT: Acoustic model finetuning with labelled and unlabelled data
Authors:
Utkarsh Chauhan,
Vikas Joshi,
Rupesh R. Mehta
Abstract:
Unsupervised and self-supervised learning methods have leveraged unlabelled data to improve the pretrained models. However, these methods need significantly large amount of unlabelled data and the computational cost of training models with such large amount of data can be prohibitively high. We address this issue by using unlabelled data during finetuning, instead of pretraining. We propose acoust…
▽ More
Unsupervised and self-supervised learning methods have leveraged unlabelled data to improve the pretrained models. However, these methods need significantly large amount of unlabelled data and the computational cost of training models with such large amount of data can be prohibitively high. We address this issue by using unlabelled data during finetuning, instead of pretraining. We propose acoustic model finetuning (FT) using labelled and unlabelled data. The model is jointly trained to learn representations to classify senones, as well as learn contextual acoustic representations. Our training objective is a combination of cross entropy loss, suitable for classification task, and contrastive loss, suitable to learn acoustic representations. The proposed approach outperforms conventional finetuning with 11.2% and 9.19% word error rate relative (WERR) reduction on Gujarati and Bengali languages respectively.
△ Less
Submitted 1 April, 2022;
originally announced April 2022.
-
A Deep Knowledge Distillation framework for EEG assisted enhancement of single-lead ECG based sleep staging
Authors:
Vaibhav Joshi,
Sricharan Vijayarangan,
Preejith SP,
Mohanasankar Sivaprakasam
Abstract:
Automatic Sleep Staging study is presently done with the help of Electroencephalogram (EEG) signals. Recently, Deep Learning (DL) based approaches have enabled significant progress in this area, allowing for near-human accuracy in automated sleep staging. However, EEG based sleep staging requires an extensive as well as an expensive clinical setup. Moreover, the requirement of an expert for setup…
▽ More
Automatic Sleep Staging study is presently done with the help of Electroencephalogram (EEG) signals. Recently, Deep Learning (DL) based approaches have enabled significant progress in this area, allowing for near-human accuracy in automated sleep staging. However, EEG based sleep staging requires an extensive as well as an expensive clinical setup. Moreover, the requirement of an expert for setup and the added inconvenience to the subject under study renders it unfavourable in a point of care context. Electrocardiogram (ECG), an unobtrusive alternative to EEG, is more suitable, but its performance, unsurprisingly, remains sub-par compared to EEG-based sleep staging. Naturally, it would be helpful to transfer knowledge from EEG to ECG, ultimately enhancing the model's performance on ECG based inputs. Knowledge Distillation (KD) is a renowned concept in DL that looks to transfer knowledge from a better but potentially more cumbersome teacher model to a compact student model. Building on this concept, we propose a cross-modal KD framework to improve ECG-based sleep staging performance with assistance from features learned through models trained on EEG. Additionally, we also conducted multiple experiments on the individual components of the proposed model to get better insight into the distillation approach. Data of 200 subjects from the Montreal Archive of Sleep Studies (MASS) was utilized for our study. The proposed model showed a 14.3\% and 13.4\% increase in weighted-F1-score in 4-class and 3-class sleep staging, respectively. This demonstrates the viability of KD for performance improvement of single-channel ECG based sleep staging in 4-class(W-L-D-R) and 3-class(W-N-R) classification.
△ Less
Submitted 5 February, 2022; v1 submitted 14 December, 2021;
originally announced December 2021.
-
Hybrid In-memory Computing Architecture for the Training of Deep Neural Networks
Authors:
Vinay Joshi,
Wangxin He,
Jae-sun Seo,
Bipin Rajendran
Abstract:
The cost involved in training deep neural networks (DNNs) on von-Neumann architectures has motivated the development of novel solutions for efficient DNN training accelerators. We propose a hybrid in-memory computing (HIC) architecture for the training of DNNs on hardware accelerators that results in memory-efficient inference and outperforms baseline software accuracy in benchmark tasks. We intro…
▽ More
The cost involved in training deep neural networks (DNNs) on von-Neumann architectures has motivated the development of novel solutions for efficient DNN training accelerators. We propose a hybrid in-memory computing (HIC) architecture for the training of DNNs on hardware accelerators that results in memory-efficient inference and outperforms baseline software accuracy in benchmark tasks. We introduce a weight representation technique that exploits both binary and multi-level phase-change memory (PCM) devices, and this leads to a memory-efficient inference accelerator. Unlike previous in-memory computing-based implementations, we use a low precision weight update accumulator that results in more memory savings. We trained the ResNet-32 network to classify CIFAR-10 images using HIC. For a comparable model size, HIC-based training outperforms baseline network, trained in floating-point 32-bit (FP32) precision, by leveraging appropriate network width multiplier. Furthermore, we observe that HIC-based training results in about 50% less inference model size to achieve baseline comparable accuracy. We also show that the temporal drift in PCM devices has a negligible effect on post-training inference accuracy for extended periods (year). Finally, our simulations indicate HIC-based training naturally ensures that the number of write-erase cycles seen by the devices is a small fraction of the endurance limit of PCM, demonstrating the feasibility of this architecture for achieving hardware platforms that can learn in the field.
△ Less
Submitted 10 February, 2021;
originally announced February 2021.
-
Learning Explainable Interventions to Mitigate HIV Transmission in Sex Workers Across Five States in India
Authors:
Raghav Awasthi,
Prachi Patel,
Vineet Joshi,
Shama Karkal,
Tavpritesh Sethi
Abstract:
Female sex workers(FSWs) are one of the most vulnerable and stigmatized groups in society. As a result, they often suffer from a lack of quality access to care. Grassroot organizations engaged in improving health services are often faced with the challenge of improving the effectiveness of interventions due to complex influences. This work combines structure learning, discriminative modeling, and…
▽ More
Female sex workers(FSWs) are one of the most vulnerable and stigmatized groups in society. As a result, they often suffer from a lack of quality access to care. Grassroot organizations engaged in improving health services are often faced with the challenge of improving the effectiveness of interventions due to complex influences. This work combines structure learning, discriminative modeling, and grass-root level expertise of designing interventions across five different Indian states to discover the influence of non-obvious factors for improving safe-sex practices in FSWs. A bootstrapped, ensemble-averaged Bayesian Network structure was learned to quantify the factors that could maximize condom usage as revealed from the model. A discriminative model was then constructed using XgBoost and random forest in order to predict condom use behavior The best model achieved 83% sensitivity, 99% specificity, and 99% area under the precision-recall curve for the prediction. Both generative and discriminative modeling approaches revealed that financial literacy training was the primary influence and predictor of condom use in FSWs. These insights have led to a currently ongoing field trial for assessing the real-world utility of this approach. Our work highlights the potential of explainable models for transparent discovery and prioritization of anti-HIV interventions in female sex workers in a resource-limited setting.
△ Less
Submitted 30 November, 2020;
originally announced December 2020.
-
Transfer Learning Approaches for Streaming End-to-End Speech Recognition System
Authors:
Vikas Joshi,
Rui Zhao,
Rupesh R. Mehta,
Kshitiz Kumar,
**yu Li
Abstract:
Transfer learning (TL) is widely used in conventional hybrid automatic speech recognition (ASR) system, to transfer the knowledge from source to target language. TL can be applied to end-to-end (E2E) ASR system such as recurrent neural network transducer (RNN-T) models, by initializing the encoder and/or prediction network of the target language with the pre-trained models from source language. In…
▽ More
Transfer learning (TL) is widely used in conventional hybrid automatic speech recognition (ASR) system, to transfer the knowledge from source to target language. TL can be applied to end-to-end (E2E) ASR system such as recurrent neural network transducer (RNN-T) models, by initializing the encoder and/or prediction network of the target language with the pre-trained models from source language. In the hybrid ASR system, transfer learning is typically done by initializing the target language acoustic model (AM) with source language AM. Several transfer learning strategies exist in the case of the RNN-T framework, depending upon the choice of the initialization model for encoder and prediction networks. This paper presents a comparative study of four different TL methods for RNN-T framework. We show 17% relative word error rate reduction with different TL methods over randomly initialized RNN-T model. We also study the impact of TL with varying amount of training data ranging from 50 hours to 1000 hours and show the efficacy of TL for languages with small amount of training data.
△ Less
Submitted 17 August, 2020; v1 submitted 11 August, 2020;
originally announced August 2020.
-
Image-driven discriminative and generative machine learning algorithms for establishing microstructure-processing relationships
Authors:
Wufei Ma,
Elizabeth Kautz,
Arun Baskaran,
Aritra Chowdhury,
Vineet Joshi,
Bülent Yener,
Daniel Lewis
Abstract:
We investigate methods of microstructure representation for the purpose of predicting processing condition from microstructure image data. A binary alloy (uranium-molybdenum) that is currently under development as a nuclear fuel was studied for the purpose of develo** an improved machine learning approach to image recognition, characterization, and building predictive capabilities linking micros…
▽ More
We investigate methods of microstructure representation for the purpose of predicting processing condition from microstructure image data. A binary alloy (uranium-molybdenum) that is currently under development as a nuclear fuel was studied for the purpose of develo** an improved machine learning approach to image recognition, characterization, and building predictive capabilities linking microstructure to processing conditions. Here, we test different microstructure representations and evaluate model performance based on the F1 score. A F1 score of 95.1% was achieved for distinguishing between micrographs corresponding to ten different thermo-mechanical material processing conditions. We find that our newly developed microstructure representation describes image data well, and the traditional approach of utilizing area fractions of different phases is insufficient for distinguishing between multiple classes using a relatively small, imbalanced original data set of 272 images. To explore the applicability of generative methods for supplementing such limited data sets, generative adversarial networks were trained to generate artificial microstructure images. Two different generative networks were trained and tested to assess performance. Challenges and best practices associated with applying machine learning to limited microstructure image data sets is also discussed. Our work has implications for quantitative microstructure analysis, and development of microstructure-processing relationships in limited data sets typical of metallurgical process design studies.
△ Less
Submitted 27 July, 2020;
originally announced July 2020.
-
Learning not to Discriminate: Task Agnostic Learning for Improving Monolingual and Code-switched Speech Recognition
Authors:
Gurunath Reddy Madhumani,
Sanket Shah,
Basil Abraham,
Vikas Joshi,
Sunayana Sitaram
Abstract:
Recognizing code-switched speech is challenging for Automatic Speech Recognition (ASR) for a variety of reasons, including the lack of code-switched training data. Recently, we showed that monolingual ASR systems fine-tuned on code-switched data deteriorate in performance on monolingual speech recognition, which is not desirable as ASR systems deployed in multilingual scenarios should recognize bo…
▽ More
Recognizing code-switched speech is challenging for Automatic Speech Recognition (ASR) for a variety of reasons, including the lack of code-switched training data. Recently, we showed that monolingual ASR systems fine-tuned on code-switched data deteriorate in performance on monolingual speech recognition, which is not desirable as ASR systems deployed in multilingual scenarios should recognize both monolingual and code-switched speech with high accuracy. Our experiments indicated that this loss in performance could be mitigated by using certain strategies for fine-tuning and regularization, leading to improvements in both monolingual and code-switched ASR. In this work, we present further improvements over our previous work by using domain adversarial learning to train task agnostic models. We evaluate the classification accuracy of an adversarial discriminator and show that it can learn shared layer parameters that are task agnostic. We train end-to-end ASR systems starting with a pooled model that uses monolingual and code-switched data along with the adversarial discriminator. Our proposed technique leads to reductions in Word Error Rates (WER) in monolingual and code-switched test sets across three language pairs.
△ Less
Submitted 9 June, 2020;
originally announced June 2020.
-
Learning to Recognize Code-switched Speech Without Forgetting Monolingual Speech Recognition
Authors:
Sanket Shah,
Basil Abraham,
Gurunath Reddy M,
Sunayana Sitaram,
Vikas Joshi
Abstract:
Recently, there has been significant progress made in Automatic Speech Recognition (ASR) of code-switched speech, leading to gains in accuracy on code-switched datasets in many language pairs. Code-switched speech co-occurs with monolingual speech in one or both languages being mixed. In this work, we show that fine-tuning ASR models on code-switched speech harms performance on monolingual speech.…
▽ More
Recently, there has been significant progress made in Automatic Speech Recognition (ASR) of code-switched speech, leading to gains in accuracy on code-switched datasets in many language pairs. Code-switched speech co-occurs with monolingual speech in one or both languages being mixed. In this work, we show that fine-tuning ASR models on code-switched speech harms performance on monolingual speech. We point out the need to optimize models for code-switching while also ensuring that monolingual performance is not sacrificed. Monolingual models may be trained on thousands of hours of speech which may not be available for re-training a new model. We propose using the Learning Without Forgetting (LWF) framework for code-switched ASR when we only have access to a monolingual model and do not have the data it was trained on. We show that it is possible to train models using this framework that perform well on both code-switched and monolingual test sets. In cases where we have access to monolingual training data as well, we propose regularization strategies for fine-tuning models for code-switching without sacrificing monolingual accuracy. We report improvements in Word Error Rate (WER) in monolingual and code-switched test sets compared to baselines that use pooled data and simple fine-tuning.
△ Less
Submitted 1 June, 2020;
originally announced June 2020.
-
ESSOP: Efficient and Scalable Stochastic Outer Product Architecture for Deep Learning
Authors:
Vinay Joshi,
Geethan Karunaratne,
Manuel Le Gallo,
Irem Boybat,
Christophe Piveteau,
Abu Sebastian,
Bipin Rajendran,
Evangelos Eleftheriou
Abstract:
Deep neural networks (DNNs) have surpassed human-level accuracy in a variety of cognitive tasks but at the cost of significant memory/time requirements in DNN training. This limits their deployment in energy and memory limited applications that require real-time learning. Matrix-vector multiplications (MVM) and vector-vector outer product (VVOP) are the two most expensive operations associated wit…
▽ More
Deep neural networks (DNNs) have surpassed human-level accuracy in a variety of cognitive tasks but at the cost of significant memory/time requirements in DNN training. This limits their deployment in energy and memory limited applications that require real-time learning. Matrix-vector multiplications (MVM) and vector-vector outer product (VVOP) are the two most expensive operations associated with the training of DNNs. Strategies to improve the efficiency of MVM computation in hardware have been demonstrated with minimal impact on training accuracy. However, the VVOP computation remains a relatively less explored bottleneck even with the aforementioned strategies. Stochastic computing (SC) has been proposed to improve the efficiency of VVOP computation but on relatively shallow networks with bounded activation functions and floating-point (FP) scaling of activation gradients. In this paper, we propose ESSOP, an efficient and scalable stochastic outer product architecture based on the SC paradigm. We introduce efficient techniques to generalize SC for weight update computation in DNNs with the unbounded activation functions (e.g., ReLU), required by many state-of-the-art networks. Our architecture reduces the computational cost by re-using random numbers and replacing certain FP multiplication operations by bit shift scaling. We show that the ResNet-32 network with 33 convolution layers and a fully-connected layer can be trained with ESSOP on the CIFAR-10 dataset to achieve baseline comparable accuracy. Hardware design of ESSOP at 14nm technology node shows that, compared to a highly pipelined FP16 multiplier design, ESSOP is 82.2% and 93.7% better in energy and area efficiency respectively for outer product computation.
△ Less
Submitted 25 March, 2020;
originally announced March 2020.
-
Mixed-precision deep learning based on computational memory
Authors:
S. R. Nandakumar,
Manuel Le Gallo,
Christophe Piveteau,
Vinay Joshi,
Giovanni Mariani,
Irem Boybat,
Geethan Karunaratne,
Riduan Khaddam-Aljameh,
Urs Egger,
Anastasios Petropoulos,
Theodore Antonakopoulos,
Bipin Rajendran,
Abu Sebastian,
Evangelos Eleftheriou
Abstract:
Deep neural networks (DNNs) have revolutionized the field of artificial intelligence and have achieved unprecedented success in cognitive tasks such as image and speech recognition. Training of large DNNs, however, is computationally intensive and this has motivated the search for novel computing architectures targeting this application. A computational memory unit with nanoscale resistive memory…
▽ More
Deep neural networks (DNNs) have revolutionized the field of artificial intelligence and have achieved unprecedented success in cognitive tasks such as image and speech recognition. Training of large DNNs, however, is computationally intensive and this has motivated the search for novel computing architectures targeting this application. A computational memory unit with nanoscale resistive memory devices organized in crossbar arrays could store the synaptic weights in their conductance states and perform the expensive weighted summations in place in a non-von Neumann manner. However, updating the conductance states in a reliable manner during the weight update process is a fundamental challenge that limits the training accuracy of such an implementation. Here, we propose a mixed-precision architecture that combines a computational memory unit performing the weighted summations and imprecise conductance updates with a digital processing unit that accumulates the weight updates in high precision. A combined hardware/software training experiment of a multilayer perceptron based on the proposed architecture using a phase-change memory (PCM) array achieves 97.73% test accuracy on the task of classifying handwritten digits (based on the MNIST dataset), within 0.6% of the software baseline. The architecture is further evaluated using accurate behavioral models of PCM on a wide class of networks, namely convolutional neural networks, long-short-term-memory networks, and generative-adversarial networks. Accuracies comparable to those of floating-point implementations are achieved without being constrained by the non-idealities associated with the PCM devices. A system-level study demonstrates 173x improvement in energy efficiency of the architecture when used for training a multilayer perceptron compared with a dedicated fully digital 32-bit implementation.
△ Less
Submitted 31 January, 2020;
originally announced January 2020.
-
Knowledge Enhanced Contextual Word Representations
Authors:
Matthew E. Peters,
Mark Neumann,
Robert L. Logan IV,
Roy Schwartz,
Vidur Joshi,
Sameer Singh,
Noah A. Smith
Abstract:
Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities. We propose a general method to embed multiple knowledge bases (KBs) into large scale models, and thereby enhance their representations with structured, human-curated knowledge. For each KB, we f…
▽ More
Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities. We propose a general method to embed multiple knowledge bases (KBs) into large scale models, and thereby enhance their representations with structured, human-curated knowledge. For each KB, we first use an integrated entity linker to retrieve relevant entity embeddings, then update contextual word representations via a form of word-to-entity attention. In contrast to previous approaches, the entity linkers and self-supervised language modeling objective are jointly trained end-to-end in a multitask setting that combines a small amount of entity linking supervision with a large amount of raw text. After integrating WordNet and a subset of Wikipedia into BERT, the knowledge enhanced BERT (KnowBert) demonstrates improved perplexity, ability to recall facts as measured in a probing task and downstream performance on relationship extraction, entity ty**, and word sense disambiguation. KnowBert's runtime is comparable to BERT's and it scales to large KBs.
△ Less
Submitted 30 October, 2019; v1 submitted 9 September, 2019;
originally announced September 2019.
-
An image-driven machine learning approach to kinetic modeling of a discontinuous precipitation reaction
Authors:
Elizabeth Kautz,
Wufei Ma,
Saumyadeep Jana,
Arun Devaraj,
Vineet Joshi,
Bülent Yener,
Daniel Lewis
Abstract:
Micrograph quantification is an essential component of several materials science studies. Machine learning methods, in particular convolutional neural networks, have previously demonstrated performance in image recognition tasks across several disciplines (e.g. materials science, medical imaging, facial recognition). Here, we apply these well-established methods to develop an approach to microstru…
▽ More
Micrograph quantification is an essential component of several materials science studies. Machine learning methods, in particular convolutional neural networks, have previously demonstrated performance in image recognition tasks across several disciplines (e.g. materials science, medical imaging, facial recognition). Here, we apply these well-established methods to develop an approach to microstructure quantification for kinetic modeling of a discontinuous precipitation reaction in a case study on the uranium-molybdenum system. Prediction of material processing history based on image data (classification), calculation of area fraction of phases present in the micrographs (segmentation), and kinetic modeling from segmentation results were performed. Results indicate that convolutional neural networks represent microstructure image data well, and segmentation using the k-means clustering algorithm yields results that agree well with manually annotated images. Classification accuracies of original and segmented images are both 94\% for a 5-class classification problem. Kinetic modeling results agree well with previously reported data using manual thresholding. The image quantification and kinetic modeling approach developed and presented here aims to reduce researcher bias introduced into the characterization process, and allows for leveraging information in limited image data sets.
△ Less
Submitted 13 June, 2019;
originally announced June 2019.
-
Accurate deep neural network inference using computational phase-change memory
Authors:
Vinay Joshi,
Manuel Le Gallo,
Simon Haefeli,
Irem Boybat,
S. R. Nandakumar,
Christophe Piveteau,
Martino Dazzi,
Bipin Rajendran,
Abu Sebastian,
Evangelos Eleftheriou
Abstract:
In-memory computing is a promising non-von Neumann approach for making energy-efficient deep learning inference hardware. Crossbar arrays of resistive memory devices can be used to encode the network weights and perform efficient analog matrix-vector multiplications without intermediate movements of data. However, due to device variability and noise, the network needs to be trained in a specific w…
▽ More
In-memory computing is a promising non-von Neumann approach for making energy-efficient deep learning inference hardware. Crossbar arrays of resistive memory devices can be used to encode the network weights and perform efficient analog matrix-vector multiplications without intermediate movements of data. However, due to device variability and noise, the network needs to be trained in a specific way so that transferring the digitally trained weights to the analog resistive memory devices will not result in significant loss of accuracy. Here, we introduce a methodology to train ResNet-type convolutional neural networks that results in no appreciable accuracy loss when transferring weights to in-memory computing hardware based on phase-change memory (PCM). We also propose a compensation technique that exploits the batch normalization parameters to improve the accuracy retention over time. We achieve a classification accuracy of 93.7% on the CIFAR-10 dataset and a top-1 accuracy on the ImageNet benchmark of 71.6% after map** the trained weights to PCM. Our hardware results on CIFAR-10 with ResNet-32 demonstrate an accuracy above 93.5% retained over a one day period, where each of the 361,722 synaptic weights of the network is programmed on just two PCM devices organized in a differential configuration.
△ Less
Submitted 11 April, 2020; v1 submitted 7 June, 2019;
originally announced June 2019.
-
Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examples
Authors:
Vidur Joshi,
Matthew Peters,
Mark Hopkins
Abstract:
We revisit domain adaptation for parsers in the neural era. First we show that recent advances in word representations greatly diminish the need for domain adaptation when the target domain is syntactically similar to the source domain. As evidence, we train a parser on the Wall Street Jour- nal alone that achieves over 90% F1 on the Brown corpus. For more syntactically dis- tant domains, we provi…
▽ More
We revisit domain adaptation for parsers in the neural era. First we show that recent advances in word representations greatly diminish the need for domain adaptation when the target domain is syntactically similar to the source domain. As evidence, we train a parser on the Wall Street Jour- nal alone that achieves over 90% F1 on the Brown corpus. For more syntactically dis- tant domains, we provide a simple way to adapt a parser using only dozens of partial annotations. For instance, we increase the percentage of error-free geometry-domain parses in a held-out set from 45% to 73% using approximately five dozen training examples. In the process, we demon- strate a new state-of-the-art single model result on the Wall Street Journal test set of 94.3%. This is an absolute increase of 1.7% over the previous state-of-the-art of 92.6%.
△ Less
Submitted 16 May, 2018;
originally announced May 2018.
-
Modified SPLICE and its Extension to Non-Stereo Data for Noise Robust Speech Recognition
Authors:
D. S. Pavan Kumar,
N. Vishnu Prasad,
Vikas Joshi,
S. Umesh
Abstract:
In this paper, a modification to the training process of the popular SPLICE algorithm has been proposed for noise robust speech recognition. The modification is based on feature correlations, and enables this stereo-based algorithm to improve the performance in all noise conditions, especially in unseen cases. Further, the modified framework is extended to work for non-stereo datasets where clean…
▽ More
In this paper, a modification to the training process of the popular SPLICE algorithm has been proposed for noise robust speech recognition. The modification is based on feature correlations, and enables this stereo-based algorithm to improve the performance in all noise conditions, especially in unseen cases. Further, the modified framework is extended to work for non-stereo datasets where clean and noisy training utterances, but not stereo counterparts, are required. Finally, an MLLR-based computationally efficient run-time noise adaptation method in SPLICE framework has been proposed. The modified SPLICE shows 8.6% absolute improvement over SPLICE in Test C of Aurora-2 database, and 2.93% overall. Non-stereo method shows 10.37% and 6.93% absolute improvements over Aurora-2 and Aurora-4 baseline models respectively. Run-time adaptation shows 9.89% absolute improvement in modified framework as compared to SPLICE for Test C, and 4.96% overall w.r.t. standard MLLR adaptation on HMMs.
△ Less
Submitted 15 July, 2013;
originally announced July 2013.
