-
DEMETR: Diagnosing Evaluation Metrics for Translation
Authors:
Marzena Karpinska,
Nishant Raj,
Katherine Thai,
Yixiao Song,
Ankita Gupta,
Mohit Iyyer
Abstract:
While machine translation evaluation metrics based on string overlap (e.g., BLEU) have their limitations, their computations are transparent: the BLEU score assigned to a particular candidate translation can be traced back to the presence or absence of certain words. The operations of newer learned metrics (e.g., BLEURT, COMET), which leverage pretrained language models to achieve higher correlati…
▽ More
While machine translation evaluation metrics based on string overlap (e.g., BLEU) have their limitations, their computations are transparent: the BLEU score assigned to a particular candidate translation can be traced back to the presence or absence of certain words. The operations of newer learned metrics (e.g., BLEURT, COMET), which leverage pretrained language models to achieve higher correlations with human quality judgments than BLEU, are opaque in comparison. In this paper, we shed light on the behavior of these learned metrics by creating DEMETR, a diagnostic dataset with 31K English examples (translated from 10 source languages) for evaluating the sensitivity of MT evaluation metrics to 35 different linguistic perturbations spanning semantic, syntactic, and morphological error categories. All perturbations were carefully designed to form minimal pairs with the actual translation (i.e., differ in only one aspect). We find that learned metrics perform substantially better than string-based metrics on DEMETR. Additionally, learned metrics differ in their sensitivity to various phenomena (e.g., BERTScore is sensitive to untranslated words but relatively insensitive to gender manipulation, while COMET is much more sensitive to word repetition than to aspectual changes). We publicly release DEMETR to spur more informed future development of machine translation evaluation metrics
△ Less
Submitted 24 October, 2022;
originally announced October 2022.
-
EEG-based Image Feature Extraction for Visual Classification using Deep Learning
Authors:
Alankrit Mishra,
Nikhil Raj,
Garima Bajwa
Abstract:
While capable of segregating visual data, humans take time to examine a single piece, let alone thousands or millions of samples. The deep learning models efficiently process sizeable information with the help of modern-day computing. However, their questionable decision-making process has raised considerable concerns. Recent studies have identified a new approach to extract image features from EE…
▽ More
While capable of segregating visual data, humans take time to examine a single piece, let alone thousands or millions of samples. The deep learning models efficiently process sizeable information with the help of modern-day computing. However, their questionable decision-making process has raised considerable concerns. Recent studies have identified a new approach to extract image features from EEG signals and combine them with standard image features. These approaches make deep learning models more interpretable and also enables faster converging of models with fewer samples. Inspired by recent studies, we developed an efficient way of encoding EEG signals as images to facilitate a more subtle understanding of brain signals with deep learning models. Using two variations in such encoding methods, we classified the encoded EEG signals corresponding to 39 image classes with a benchmark accuracy of 70% on the layered dataset of six subjects, which is significantly higher than the existing work. Our image classification approach with combined EEG features achieved an accuracy of 82% compared to the slightly better accuracy of a pure deep learning approach; nevertheless, it demonstrates the viability of the theory.
△ Less
Submitted 26 September, 2022;
originally announced September 2022.
-
SearchMorph:Multi-scale Correlation Iterative Network for Deformable Registration
Authors:
Xiao Fan,
Shuxin Zhuang,
Zhemin Zhuang,
Ye Yuan,
Shunmin Qiu,
Alex Noel Joseph Raj,
Yibiao Rong
Abstract:
Deformable image registration can obtain dynamic information about images, which is of great significance in medical image analysis. The unsupervised deep learning registration method can quickly achieve high registration accuracy without labels. However, these methods generally suffer from uncorrelated features, poor ability to register large deformations and details, and unnatural deformation fi…
▽ More
Deformable image registration can obtain dynamic information about images, which is of great significance in medical image analysis. The unsupervised deep learning registration method can quickly achieve high registration accuracy without labels. However, these methods generally suffer from uncorrelated features, poor ability to register large deformations and details, and unnatural deformation fields. To address the issues above, we propose an unsupervised multi-scale correlation iterative registration network (SearchMorph). In the proposed network, we introduce a correlation layer to strengthen the relevance between features and construct a correlation pyramid to provide multi-scale relevance information for the network. We also design a deformation field iterator, which improves the ability of the model to register details and large deformations through the search module and GRU while ensuring that the deformation field is realistic. We use single-temporal brain MR images and multi-temporal echocardiographic sequences to evaluate the model's ability to register large deformations and details. The experimental results demonstrate that the method in this paper achieves the highest registration accuracy and the lowest folding point ratio using a short elapsed time to state-of-the-art.
△ Less
Submitted 18 July, 2022; v1 submitted 27 June, 2022;
originally announced June 2022.
-
Collaborative Experience between Scientific Software Projects using Agile Scrum Development
Authors:
A. L. Baxter,
S. Y. BenZvi,
W. Bonivento,
A. Brazier,
M. Clark,
A. Coleiro,
D. Collom,
M. Colomer-Molla,
B. Cousins,
A. Delgado Orellana,
D. Dornic,
V. Ekimtcov,
S. ElSayed,
A. Gallo Rosso,
P. Godwin,
S. Griswold,
A. Habig,
S. Horiuchi,
D. A. Howell,
M. W. G. Johnson,
M. Juric,
J. P. Kneller,
A. Kopec,
C. Kopper,
V. Kulikovskiy
, et al. (27 additional authors not shown)
Abstract:
Develo** sustainable software for the scientific community requires expertise in software engineering and domain science. This can be challenging due to the unique needs of scientific software, the insufficient resources for software engineering practices in the scientific community, and the complexity of develo** for evolving scientific contexts. While open-source software can partially addre…
▽ More
Develo** sustainable software for the scientific community requires expertise in software engineering and domain science. This can be challenging due to the unique needs of scientific software, the insufficient resources for software engineering practices in the scientific community, and the complexity of develo** for evolving scientific contexts. While open-source software can partially address these concerns, it can introduce complicating dependencies and delay development. These issues can be reduced if scientists and software developers collaborate. We present a case study wherein scientists from the SuperNova Early Warning System collaborated with software developers from the Scalable Cyberinfrastructure for Multi-Messenger Astrophysics project. The collaboration addressed the difficulties of open-source software development, but presented additional risks to each team. For the scientists, there was a concern of relying on external systems and lacking control in the development process. For the developers, there was a risk in supporting a user-group while maintaining core development. These issues were mitigated by creating a second Agile Scrum framework in parallel with the developers' ongoing Agile Scrum process. This Agile collaboration promoted communication, ensured that the scientists had an active role in development, and allowed the developers to evaluate and implement the scientists' software requirements. The collaboration provided benefits for each group: the scientists actuated their development by using an existing platform, and the developers utilized the scientists' use-case to improve their systems. This case study suggests that scientists and software developers can avoid scientific computing issues by collaborating and that Agile Scrum methods can address emergent concerns.
△ Less
Submitted 2 August, 2022; v1 submitted 19 January, 2021;
originally announced January 2021.
-
A Sui Generis QA Approach using RoBERTa for Adverse Drug Event Identification
Authors:
Harshit Jain,
Nishant Raj,
Suyash Mishra
Abstract:
Extraction of adverse drug events from biomedical literature and other textual data is an important component to monitor drug-safety and this has attracted attention of many researchers in healthcare. Existing works are more pivoted around entity-relation extraction using bidirectional long short term memory networks (Bi-LSTM) which does not attain the best feature representations. In this paper,…
▽ More
Extraction of adverse drug events from biomedical literature and other textual data is an important component to monitor drug-safety and this has attracted attention of many researchers in healthcare. Existing works are more pivoted around entity-relation extraction using bidirectional long short term memory networks (Bi-LSTM) which does not attain the best feature representations. In this paper, we introduce a question answering framework that exploits the robustness, masking and dynamic attention capabilities of RoBERTa by a technique of domain adaptation and attempt to overcome the aforementioned limitations. Our model outperforms the prior work by 9.53% F1-Score.
△ Less
Submitted 30 October, 2020;
originally announced November 2020.
-
Harmony-Search and Otsu based System for Coronavirus Disease (COVID-19) Detection using Lung CT Scan Images
Authors:
V. Ra**ikanth,
Nilanjan Dey,
Alex Noel Joseph Raj,
Aboul Ella Hassanien,
K. C. Santosh,
N. Sri Madhava Raja
Abstract:
Pneumonia is one of the foremost lung diseases and untreated pneumonia will lead to serious threats for all age groups. The proposed work aims to extract and evaluate the Coronavirus disease (COVID-19) caused pneumonia infection in lung using CT scans. We propose an image-assisted system to extract COVID-19 infected sections from lung CT scans (coronal view). It includes following steps: (i) Thres…
▽ More
Pneumonia is one of the foremost lung diseases and untreated pneumonia will lead to serious threats for all age groups. The proposed work aims to extract and evaluate the Coronavirus disease (COVID-19) caused pneumonia infection in lung using CT scans. We propose an image-assisted system to extract COVID-19 infected sections from lung CT scans (coronal view). It includes following steps: (i) Threshold filter to extract the lung region by eliminating possible artifacts; (ii) Image enhancement using Harmony-Search-Optimization and Otsu thresholding; (iii) Image segmentation to extract infected region(s); and (iv) Region-of-interest (ROI) extraction (features) from binary image to compute level of severity. The features that are extracted from ROI are then employed to identify the pixel ratio between the lung and infection sections to identify infection level of severity. The primary objective of the tool is to assist the pulmonologist not only to detect but also to help plan treatment process. As a consequence, for mass screening processing, it will help prevent diagnostic burden.
△ Less
Submitted 6 April, 2020;
originally announced April 2020.
-
Reliable State Machines: A Framework for Programming Reliable Cloud Services
Authors:
Suvam Mukherjee,
Nitin John Raj,
Krishnan Govindraj,
Pantazis Deligiannis,
Chandramouleswaran Ravichandran,
Akash Lal,
Aseem Rastogi,
Raja Krishnaswamy
Abstract:
Building reliable applications for the cloud is challenging because of unpredictable failures during a program's execution. This paper presents a programming framework called Reliable State Machines (RSMs), that offers fault-tolerance by construction. Using our framework, a programmer can build an application as several (possibly distributed) RSMs that communicate with each other via messages, muc…
▽ More
Building reliable applications for the cloud is challenging because of unpredictable failures during a program's execution. This paper presents a programming framework called Reliable State Machines (RSMs), that offers fault-tolerance by construction. Using our framework, a programmer can build an application as several (possibly distributed) RSMs that communicate with each other via messages, much in the style of actor-based programming. Each RSM is additionally fault-tolerant by design and offers the illusion of being "always-alive". An RSM is guaranteed to process each input request exactly once, as one would expect in a failure-free environment. The RSM runtime automatically takes care of persisting state and rehydrating it on a failover. We present the core syntax and semantics of RSMs, along with a formal proof of failure-transparency. We provide an implementation of the RSM framework and runtime on the .NET platform for deploying services to Microsoft Azure. We carried out an extensive performance evaluation on micro-benchmarks to show that one can build high-throughput applications with RSMs. We also present a case study where we rewrote a significant part of a production cloud service using RSMs. The resulting service has simpler code and exhibits production-grade performance.
△ Less
Submitted 27 February, 2019; v1 submitted 25 February, 2019;
originally announced February 2019.
-
Modelling of Human Glottis in VLSI for Low Power Architectures
Authors:
Nikhil Raj,
R. K. Sharma
Abstract:
The Glottal Source is an important component of voice as it can be considered as the excitation signal to the voice apparatus. Nowadays, new techniques of speech processing such as speech recognition and speech synthesis use the glottal closure and opening instants. Current models of the glottal waves derive their shape from approximate information rather than from exactly measured data. General m…
▽ More
The Glottal Source is an important component of voice as it can be considered as the excitation signal to the voice apparatus. Nowadays, new techniques of speech processing such as speech recognition and speech synthesis use the glottal closure and opening instants. Current models of the glottal waves derive their shape from approximate information rather than from exactly measured data. General method concentrate on assessment of the glottis opening using optical, acoustical methods, or on visualization of the larynx position using ultrasound, computer tomography or magnetic resonance imaging techniques. In this work, circuit model of Human Glottis using MOS is designed by exploiting fluid volume velocity to current, fluid pressure to voltage, and linear and nonlinear mechanical impedances to linear and nonlinear electrical impedances. The glottis modeled as current source includes linear, non-linear impedances to represent laminar and turbulent flow respectively, in vocal tract. The MOS modelling and simulation results of glottal circuit has been carried out on BSIM 3v3 model in TSMC 0.18 micrometer technology using ELDO simulator.
△ Less
Submitted 19 April, 2010;
originally announced April 2010.
-
Dpraodv: A Dyanamic Learning System Against Blackhole Attack in Aodv Based Manet
Authors:
Payal N. Raj,
Prashant B. Swadas
Abstract:
Security is an essential requirement in mobile ad hoc networks to provide protected communication between mobile nodes. Due to unique characteristics of MANETS, it creates a number of consequential challenges to its security design. To overcome the challenges, there is a need to build a multifence security solution that achieves both broad protection and desirable network performance. MANETs are…
▽ More
Security is an essential requirement in mobile ad hoc networks to provide protected communication between mobile nodes. Due to unique characteristics of MANETS, it creates a number of consequential challenges to its security design. To overcome the challenges, there is a need to build a multifence security solution that achieves both broad protection and desirable network performance. MANETs are vulnerable to various attacks, blackhole, is one of the possible attacks. Black hole is a type of routing attack where a malicious node advertise itself as having the shortest path to all nodes in the environment by sending fake route reply. By doing this, the malicious node can deprive the traffic from the source node. It can be used as a denial-of-service attack where it can drop the packets later. In this paper, we proposed a DPRAODV (Detection, Prevention and Reactive AODV) to prevent security threats of blackhole by notifying other nodes in the network of the incident. The simulation results in ns2 (ver- 2.33) demonstrate that our protocol not only prevents blackhole attack but consequently improves the overall performance of (normal) AODV in presence of black hole attack.
△ Less
Submitted 12 September, 2009;
originally announced September 2009.
