-
SeMaScore : a new evaluation metric for automatic speech recognition tasks
Authors:
Zitha Sasindran,
Harsha Yelchuri,
T. V. Prabhakar
Abstract:
In this study, we present SeMaScore, generated using a segment-wise map** and scoring algorithm that serves as an evaluation metric for automatic speech recognition tasks. SeMaScore leverages both the error rate and a more robust similarity score. We show that our algorithm's score generation improves upon the state-of-the-art BERTscore. Our experimental results show that SeMaScore corresponds w…
▽ More
In this study, we present SeMaScore, generated using a segment-wise map** and scoring algorithm that serves as an evaluation metric for automatic speech recognition tasks. SeMaScore leverages both the error rate and a more robust similarity score. We show that our algorithm's score generation improves upon the state-of-the-art BERTscore. Our experimental results show that SeMaScore corresponds well with expert human assessments, signal-to-noise ratio levels, and other natural language metrics. We outperform BERTscore by 41x in metric computation speed. Overall, we demonstrate that SeMaScore serves as a more dependable evaluation metric, particularly in real-world situations involving atypical speech patterns.
△ Less
Submitted 15 January, 2024;
originally announced January 2024.
-
$μ$TAS: Design and implementation of Time Aware Shaper on SmartNICs to achieve bounded latency
Authors:
Joydeep Pal,
Deepak Choudhary,
Nithish Krishnabharathi Gnani,
Chandramani Singh,
T. V. Prabhakar
Abstract:
Time-Aware Shaper (TAS) is a time-triggered scheduling mechanism that ensures bounded latency for time-critical Scheduled Traffic (ST) flows. The Linux kernel implementation (a.k.a TAPRIO) has limited capabilities due to varying CPU workloads and thus does not offer tight latency bound for the ST flows. Also, currently only higher cycle times are possible. Other software implementations are limite…
▽ More
Time-Aware Shaper (TAS) is a time-triggered scheduling mechanism that ensures bounded latency for time-critical Scheduled Traffic (ST) flows. The Linux kernel implementation (a.k.a TAPRIO) has limited capabilities due to varying CPU workloads and thus does not offer tight latency bound for the ST flows. Also, currently only higher cycle times are possible. Other software implementations are limited to simulation studies without physical implementation. In this paper, we present $μ$TAS, a MicroC-based hardware implementation of TAS onto a programmable SmartNIC. $μ$TAS takes advantage of the parallel-processing architecture of the SmartNIC to configure the scheduling behaviour of its queues at runtime. To demonstrate the effectiveness of $μ$TAS, we built a Time-Sensitive Networking (TSN) testbed from scratch. This consists of multiple end-hosts capable of generating ST and Best Effort (BE) flows and TSN switches equipped with SmartNICs running $μ$TAS. Time synchronization is maintained between the switches and hosts. Our experiments demonstrate that the ST flows experience a bounded latency of the order of tens of microseconds.
△ Less
Submitted 11 October, 2023;
originally announced October 2023.
-
EdgeP4: A P4-Programmable Edge Intelligent Ethernet Switch for Tactile Cyber-Physical Systems
Authors:
Nithish Krishnabharathi Gnani,
Joydeep Pal,
Deepak Choudhary,
Himanshu Verma,
Soumya Kanta Rana,
Kaushal Mhapsekar,
T. V. Prabhakar,
Chandramani Singh
Abstract:
Tactile Internet based operations, e.g., telesurgery, rely on end-to-end closed loop control for accuracy and corrections. The feedback and control are subject to network latency and loss. We design two edge intelligence algorithms hosted at P4 programmable end switches. These algorithms locally compute and command corrective signals, thereby dispense the feedback signals from traversing the netwo…
▽ More
Tactile Internet based operations, e.g., telesurgery, rely on end-to-end closed loop control for accuracy and corrections. The feedback and control are subject to network latency and loss. We design two edge intelligence algorithms hosted at P4 programmable end switches. These algorithms locally compute and command corrective signals, thereby dispense the feedback signals from traversing the network to the other ends and save on control loop latency and network load. We implement these algorithms entirely on data plane on Netronome Agilio SmartNICs using P4. Our first algorithm, $\textit{pose correction}$, is placed at the edge switch connected to an industrial robot grip** a tool. The round trip between transmitting force sensor array readings to the edge switch and receiving correct tip coordinates at the robot is shown to be less than $100~μs$. The second algorithm, $\textit{tremor suppression}$, is placed at the edge switch connected to the human operator. It suppresses physiological tremors of amplitudes smaller than $100~μm$ which not only improves the application's performance but also reduces the network load up to $99.9\%$. Our solution allows edge intelligence modules to seamlessly switch between the algorithms based on the tasks being executed at the end hosts.
△ Less
Submitted 19 September, 2023;
originally announced September 2023.
-
PUF Probe: A PUF-based Hardware Authentication Equipment for IEDs
Authors:
Vishal D. Jadhav,
Narahari N. Moudhgalya,
Tapabrata Sen,
T. V. Prabhakar
Abstract:
Intelligent Electronic Devices (IEDs) are vital components in modern electrical substations, collectively responsible for monitoring electrical parameters and performing protective functions. As a result, ensuring the integrity of IEDs is an essential criteria. While standards like IEC 61850 and IEC 60870-5-104 establish cyber-security protocols for secure information exchange in IED-based power s…
▽ More
Intelligent Electronic Devices (IEDs) are vital components in modern electrical substations, collectively responsible for monitoring electrical parameters and performing protective functions. As a result, ensuring the integrity of IEDs is an essential criteria. While standards like IEC 61850 and IEC 60870-5-104 establish cyber-security protocols for secure information exchange in IED-based power systems, the physical integrity of IEDs is often overlooked, leading to a rise in counterfeit and tainted electronic products. This paper proposes a physical unclonable function (PUF)-based device (IEDPUF probe) capable of extracting unique hardware signatures from commercial IEDs. These signatures can serve as identifiers, facilitating the authentication and protection of IEDs against counterfeiting. The paper presents the complete hardware architecture of the IEDPUF probe, along with algorithms for signature extraction and authentication. The process involves the central computer system (CCS) initiating IED authentication requests by sending random challenges to the IEDPUF probe. Based on the challenges, the IEDPUF probe generates responses, which are then verified by the CCS to authenticate the IED. Additionally, a two-way authentication technique is employed to ensure that only verified requests are granted access for signature extraction. Experimental results confirm the efficacy of the proposed IEDPUF probe. The results demonstrate its ability to provide real-time responses possessing randomness while uniquely identifying the IED under investigation. The proposed IEDPUF probe offers a simple, cost-effective, accurate solution with minimal storage requirements, enhancing the authenticity and integrity of IEDs within electrical substations
△ Less
Submitted 28 July, 2023;
originally announced July 2023.
-
Ed-Fed: A generic federated learning framework with resource-aware client selection for edge devices
Authors:
Zitha Sasindran,
Harsha Yelchuri,
T. V. Prabhakar
Abstract:
Federated learning (FL) has evolved as a prominent method for edge devices to cooperatively create a unified prediction model while securing their sensitive training data local to the device. Despite the existence of numerous research frameworks for simulating FL algorithms, they do not facilitate comprehensive deployment for automatic speech recognition tasks on heterogeneous edge devices. This i…
▽ More
Federated learning (FL) has evolved as a prominent method for edge devices to cooperatively create a unified prediction model while securing their sensitive training data local to the device. Despite the existence of numerous research frameworks for simulating FL algorithms, they do not facilitate comprehensive deployment for automatic speech recognition tasks on heterogeneous edge devices. This is where Ed-Fed, a comprehensive and generic FL framework, comes in as a foundation for future practical FL system research. We also propose a novel resource-aware client selection algorithm to optimise the waiting time in the FL settings. We show that our approach can handle the straggler devices and dynamically set the training time for the selected devices in a round. Our evaluation has shown that the proposed approach significantly optimises waiting time in FL compared to conventional random client selection methods.
△ Less
Submitted 14 July, 2023;
originally announced July 2023.
-
MobileASR: A resource-aware on-device learning framework for user voice personalization applications on mobile phones
Authors:
Zitha Sasindran,
Harsha Yelchuri,
Pooja Rao,
T. V. Prabhakar
Abstract:
We describe a comprehensive methodology for develo** user-voice personalized automatic speech recognition (ASR) models by effectively training models on mobile phones, allowing user data and models to be stored and used locally. To achieve this, we propose a resource-aware sub-model-based training approach that considers the RAM, and battery capabilities of mobile phones. By considering the eval…
▽ More
We describe a comprehensive methodology for develo** user-voice personalized automatic speech recognition (ASR) models by effectively training models on mobile phones, allowing user data and models to be stored and used locally. To achieve this, we propose a resource-aware sub-model-based training approach that considers the RAM, and battery capabilities of mobile phones. By considering the evaluation metric and resource constraints of the mobile phones, we are able to perform efficient training and halt the process accordingly. To simulate real users, we use speakers with various accents. The entire on-device training and evaluation framework was then tested on various mobile phones across brands. We show that fine-tuning the models and selecting the right hyperparameter values is a trade-off between the lowest achievable performance metric, on-device training time, and memory consumption. Overall, our methodology offers a comprehensive solution for develo** personalized ASR models while leveraging the capabilities of mobile phones, and balancing the need for accuracy with resource constraints.
△ Less
Submitted 9 November, 2023; v1 submitted 15 June, 2023;
originally announced June 2023.
-
PreMa: Predictive Maintenance of Solenoid Valve in Real-Time at Embedded Edge-Level
Authors:
Prajwal BN,
Harsha Yelchuri,
Vishwanath Shastry,
T. V. Prabhakar
Abstract:
In industrial process automation, sensors (pressure, temperature, etc.), controllers, and actuators (solenoid valves, electro-mechanical relays, circuit breakers, motors, etc.) make sure that production lines are working under the pre-defined conditions. When these systems malfunction or sometimes completely fail, alerts have to be generated in real-time to make sure not only production quality is…
▽ More
In industrial process automation, sensors (pressure, temperature, etc.), controllers, and actuators (solenoid valves, electro-mechanical relays, circuit breakers, motors, etc.) make sure that production lines are working under the pre-defined conditions. When these systems malfunction or sometimes completely fail, alerts have to be generated in real-time to make sure not only production quality is not compromised but also safety of humans and equipment is assured. In this work, we describe the construction of a smart and real-time edge-based electronic product called PreMa, which is basically a sensor for monitoring the health of a Solenoid Valve (SV). PreMa is compact, low power, easy to install, and cost effective. It has data fidelity and measurement accuracy comparable to signals captured using high end equipment. The smart solenoid sensor runs TinyML, a compact version of TensorFlow (a.k.a. TFLite) machine learning framework. While fault detection inferencing is in-situ, model training uses mobile phones to accomplish the `on-device' training. Our product evaluation shows that the sensor is able to differentiate between the distinct types of faults. These faults include: (a) Spool stuck (b) Spring failure and (c) Under voltage. Furthermore, the product provides maintenance personnel, the remaining useful life (RUL) of the SV. The RUL provides assistance to decide valve replacement or otherwise. We perform an extensive evaluation on optimizing metrics related to performance of the entire system (i.e. embedded platform and the neural network model). The proposed implementation is such that, given any electro-mechanical actuator with similar transient response to that of the SV, the system is capable of condition monitoring, hence presenting a first of its kind generic infrastructure.
△ Less
Submitted 21 November, 2022;
originally announced November 2022.
-
H_eval: A new hybrid evaluation metric for automatic speech recognition tasks
Authors:
Zitha Sasindran,
Harsha Yelchuri,
T. V. Prabhakar,
Supreeth Rao
Abstract:
Many studies have examined the shortcomings of word error rate (WER) as an evaluation metric for automatic speech recognition (ASR) systems. Since WER considers only literal word-level correctness, new evaluation metrics based on semantic similarity such as semantic distance (SD) and BERTScore have been developed. However, we found that these metrics have their own limitations, such as a tendency…
▽ More
Many studies have examined the shortcomings of word error rate (WER) as an evaluation metric for automatic speech recognition (ASR) systems. Since WER considers only literal word-level correctness, new evaluation metrics based on semantic similarity such as semantic distance (SD) and BERTScore have been developed. However, we found that these metrics have their own limitations, such as a tendency to overly prioritise keywords. We propose H_eval, a new hybrid evaluation metric for ASR systems that considers both semantic correctness and error rate and performs significantly well in scenarios where WER and SD perform poorly. Due to lighter computation compared to BERTScore, it offers 49 times reduction in metric computation time. Furthermore, we show that H_eval correlates strongly with downstream NLP tasks. Also, to reduce the metric calculation time, we built multiple fast and lightweight models using distillation techniques
△ Less
Submitted 1 December, 2023; v1 submitted 3 November, 2022;
originally announced November 2022.
-
Training end-to-end speech-to-text models on mobile phones
Authors:
Zitha S,
Raghavendra Rao Suresh,
Pooja Rao,
T. V. Prabhakar
Abstract:
Training the state-of-the-art speech-to-text (STT) models in mobile devices is challenging due to its limited resources relative to a server environment. In addition, these models are trained on generic datasets that are not exhaustive in capturing user-specific characteristics. Recently, on-device personalization techniques have been making strides in mitigating the problem. Although many current…
▽ More
Training the state-of-the-art speech-to-text (STT) models in mobile devices is challenging due to its limited resources relative to a server environment. In addition, these models are trained on generic datasets that are not exhaustive in capturing user-specific characteristics. Recently, on-device personalization techniques have been making strides in mitigating the problem. Although many current works have already explored the effectiveness of on-device personalization, the majority of their findings are limited to simulation settings or a specific smartphone. In this paper, we develop and provide a detailed explanation of our framework to train end-to-end models in mobile phones. To make it simple, we considered a model based on connectionist temporal classification (CTC) loss. We evaluated the framework on various mobile phones from different brands and reported the results. We provide enough evidence that fine-tuning the models and choosing the right hyperparameter values is a trade-off between the lowest WER achievable, training time on-device, and memory consumption. Hence, this is vital for a successful deployment of on-device training onto a resource-limited environment like mobile phones. We use training sets from speakers with different accents and record a 7.6% decrease in average word error rate (WER). We also report the associated computational cost measurements with respect to time, memory usage, and cpu utilization in mobile phones in real-time.
△ Less
Submitted 7 December, 2021;
originally announced December 2021.
-
On the Accuracy of Analog Neural Network Inference Accelerators
Authors:
T. Patrick Xiao,
Ben Feinberg,
Christopher H. Bennett,
Venkatraman Prabhakar,
Prashant Saxena,
Vineet Agrawal,
Sapan Agarwal,
Matthew J. Marinella
Abstract:
Specialized accelerators have recently garnered attention as a method to reduce the power consumption of neural network inference. A promising category of accelerators utilizes nonvolatile memory arrays to both store weights and perform $\textit{in situ}$ analog computation inside the array. While prior work has explored the design space of analog accelerators to optimize performance and energy ef…
▽ More
Specialized accelerators have recently garnered attention as a method to reduce the power consumption of neural network inference. A promising category of accelerators utilizes nonvolatile memory arrays to both store weights and perform $\textit{in situ}$ analog computation inside the array. While prior work has explored the design space of analog accelerators to optimize performance and energy efficiency, there is seldom a rigorous evaluation of the accuracy of these accelerators. This work shows how architectural design decisions, particularly in map** neural network parameters to analog memory cells, influence inference accuracy. When evaluated using ResNet50 on ImageNet, the resilience of the system to analog non-idealities - cell programming errors, analog-to-digital converter resolution, and array parasitic resistances - all improve when analog quantities in the hardware are made proportional to the weights in the network. Moreover, contrary to the assumptions of prior work, nearly equivalent resilience to cell imprecision can be achieved by fully storing weights as analog quantities, rather than spreading weight bits across multiple devices, often referred to as bit slicing. By exploiting proportionality, analog system designers have the freedom to match the precision of the hardware to the needs of the algorithm, rather than attempting to guarantee the same level of precision in the intermediate results as an equivalent digital accelerator. This ultimately results in an analog accelerator that is more accurate, more robust to analog errors, and more energy-efficient.
△ Less
Submitted 3 February, 2022; v1 submitted 2 September, 2021;
originally announced September 2021.
-
Device-aware inference operations in SONOS nonvolatile memory arrays
Authors:
Christopher H. Bennett,
T. Patrick Xiao,
Ryan Dellana,
Vineet Agrawal,
Ben Feinberg,
Venkatraman Prabhakar,
Krishnaswamy Ramkumar,
Long Hinh,
Swatilekha Saha,
Vijay Raghavan,
Ramesh Chettuvetty,
Sapan Agarwal,
Matthew J. Marinella
Abstract:
Non-volatile memory arrays can deploy pre-trained neural network models for edge inference. However, these systems are affected by device-level noise and retention issues. Here, we examine damage caused by these effects, introduce a mitigation strategy, and demonstrate its use in fabricated array of SONOS (Silicon-Oxide-Nitride-Oxide-Silicon) devices. On MNIST, fashion-MNIST, and CIFAR-10 tasks, o…
▽ More
Non-volatile memory arrays can deploy pre-trained neural network models for edge inference. However, these systems are affected by device-level noise and retention issues. Here, we examine damage caused by these effects, introduce a mitigation strategy, and demonstrate its use in fabricated array of SONOS (Silicon-Oxide-Nitride-Oxide-Silicon) devices. On MNIST, fashion-MNIST, and CIFAR-10 tasks, our approach increases resilience to synaptic noise and drift. We also show strong performance can be realized with ADCs of 5-8 bits precision.
△ Less
Submitted 2 April, 2020;
originally announced April 2020.
-
PUTWorkbench: Analysing Privacy in AI-intensive Systems
Authors:
Saurabh Srivastava,
Vinay P. Namboodiri,
T. V. Prabhakar
Abstract:
AI intensive systems that operate upon user data face the challenge of balancing data utility with privacy concerns. We propose the idea and present the prototype of an open-source tool called Privacy Utility Trade-off (PUT) Workbench which seeks to aid software practitioners to take such crucial decisions. We pick a simple privacy model that doesn't require any background knowledge in Data Scienc…
▽ More
AI intensive systems that operate upon user data face the challenge of balancing data utility with privacy concerns. We propose the idea and present the prototype of an open-source tool called Privacy Utility Trade-off (PUT) Workbench which seeks to aid software practitioners to take such crucial decisions. We pick a simple privacy model that doesn't require any background knowledge in Data Science and show how even that can achieve significant results over standard and real-life datasets. The tool and the source code is made freely available for extensions and usage.
△ Less
Submitted 5 February, 2019;
originally announced February 2019.
-
Animation and Chirplet-Based Development of a PIR Sensor Array for Intruder Classification in an Outdoor Environment
Authors:
Raviteja Upadrashta,
Tarun Choubisa,
A. Praneeth,
Tony G.,
Aswath V. S.,
P. Vijay Kumar,
Sripad Kowshik,
Hari Prasad Gokul R,
T. V. Prabhakar
Abstract:
This paper presents the development of a passive infra-red sensor tower platform along with a classification algorithm to distinguish between human intrusion, animal intrusion and clutter arising from wind-blown vegetative movement in an outdoor environment. The research was aimed at exploring the potential use of wireless sensor networks as an early-warning system to help mitigate human-wildlife…
▽ More
This paper presents the development of a passive infra-red sensor tower platform along with a classification algorithm to distinguish between human intrusion, animal intrusion and clutter arising from wind-blown vegetative movement in an outdoor environment. The research was aimed at exploring the potential use of wireless sensor networks as an early-warning system to help mitigate human-wildlife conflicts occurring at the edge of a forest. There are three important features to the development. Firstly, the sensor platform employs multiple sensors arranged in the form of a two-dimensional array to give it a key spatial-resolution capability that aids in classification. Secondly, given the challenges of collecting data involving animal intrusion, an Animation-based Simulation tool for Passive Infra-Red sEnsor (ASPIRE) was developed that simulates signals corresponding to human and animal intrusion and some limited models of vegetative clutter. This speeded up the process of algorithm development by allowing us to test different hypotheses in a time-efficient manner. Finally, a chirplet-based model for intruder signal was developed that significantly helped boost classification accuracy despite drawing data from a smaller number of sensors. An SVM-based classifier was used which made use of chirplet, energy and signal cross-correlation-based features. The average accuracy obtained for intruder detection and classification on real-world and simulated data sets was in excess of 97%.
△ Less
Submitted 13 April, 2016;
originally announced April 2016.
-
Examining the Impact of Platform Properties on Quality Attributes
Authors:
Balwinder Sodhi,
T. V. Prabhakar
Abstract:
We examine and bring out the architecturally significant characteristics of various virtualization and cloud oriented platforms. The impact of such characteristics on the ability of guest applications to achieve various quality attributes (QA) has also been determined by examining existing body of architecture knowledge. We observe from our findings that efficiency, resource elasticity and securit…
▽ More
We examine and bring out the architecturally significant characteristics of various virtualization and cloud oriented platforms. The impact of such characteristics on the ability of guest applications to achieve various quality attributes (QA) has also been determined by examining existing body of architecture knowledge. We observe from our findings that efficiency, resource elasticity and security are among the most impacted QAs, and virtualization platforms exhibit the maximum impact on various QAs.
△ Less
Submitted 21 May, 2012;
originally announced May 2012.
-
E-DTN : A Multi-Interface Energy DTN Gateway
Authors:
T. V Prabhakar,
S. N Akshay Uttama Nambi,
H. S Jamadagni
Abstract:
To overcome the problem of unavailability of grid power in rural India, we explore the possibility of powering WSN gateways using a bicycle dynamo. The "Data mule" bicycle generates its own power to ensure a self sustainable data transfer for information dissemination to small and marginal farmers. Our multi-interface WSN gateway is equipped with Bluetooth, Wi-Fi and GPRS technologies. To achieve…
▽ More
To overcome the problem of unavailability of grid power in rural India, we explore the possibility of powering WSN gateways using a bicycle dynamo. The "Data mule" bicycle generates its own power to ensure a self sustainable data transfer for information dissemination to small and marginal farmers. Our multi-interface WSN gateway is equipped with Bluetooth, Wi-Fi and GPRS technologies. To achieve our goal, we exploit the DTN stack in the energy sense and introduce necessary modifications to its configuration.
△ Less
Submitted 17 July, 2011;
originally announced July 2011.
