-
On Evaluating Explanation Utility for Human-AI Decision Making in NLP
Authors:
Fateme Hashemi Chaleshtori,
Atreya Ghosal,
Alexander Gill,
Purbid Bambroo,
Ana Marasović
Abstract:
Is explainability a false promise? This debate has emerged from the insufficient evidence that explanations aid people in situations they are introduced for. More human-centered, application-grounded evaluations of explanations are needed to settle this. Yet, with no established guidelines for such studies in NLP, researchers accustomed to standardized proxy evaluations must discover appropriate m…
▽ More
Is explainability a false promise? This debate has emerged from the insufficient evidence that explanations aid people in situations they are introduced for. More human-centered, application-grounded evaluations of explanations are needed to settle this. Yet, with no established guidelines for such studies in NLP, researchers accustomed to standardized proxy evaluations must discover appropriate measurements, tasks, datasets, and sensible models for human-AI teams in their studies.
To help with this, we first review fitting existing metrics. We then establish requirements for datasets to be suitable for application-grounded evaluations. Among over 50 datasets available for explainability research in NLP, we find that 4 meet our criteria. By finetuning Flan-T5-3B, we demonstrate the importance of reassessing the state of the art to form and study human-AI teams. Finally, we present the exemplar studies of human-AI decision-making for one of the identified suitable tasks -- verifying the correctness of a legal claim given a contract.
△ Less
Submitted 3 July, 2024;
originally announced July 2024.
-
Research trends, challenges, and emerging topics of digital forensics: A review of reviews
Authors:
Fran Casino,
Tom Dasaklis,
Georgios Spathoulas,
Marios Anagnostopoulos,
Amrita Ghosal,
Istvan Borocz,
Agusti Solanas,
Mauro Conti,
Constantinos Patsakis
Abstract:
Due to its critical role in cybersecurity, digital forensics has received significant attention from researchers and practitioners alike. The ever increasing sophistication of modern cyberattacks is directly related to the complexity of evidence acquisition, which often requires the use of several technologies. To date, researchers have presented many surveys and reviews on the field. However, suc…
▽ More
Due to its critical role in cybersecurity, digital forensics has received significant attention from researchers and practitioners alike. The ever increasing sophistication of modern cyberattacks is directly related to the complexity of evidence acquisition, which often requires the use of several technologies. To date, researchers have presented many surveys and reviews on the field. However, such articles focused on the advances of each particular domain of digital forensics individually. Therefore, while each of these surveys facilitates researchers and practitioners to keep up with the latest advances in a particular domain of digital forensics, the global perspective is missing. Aiming to fill this gap, we performed a qualitative review of reviews in the field of digital forensics, determined the main topics on digital forensics topics and identified their main challenges. Our analysis provides enough evidence to prove that the digital forensics community could benefit from closer collaborations and cross-topic research, since it is apparent that researchers and practitioners are trying to find solutions to the same problems in parallel, sometimes without noticing it.
△ Less
Submitted 10 January, 2022; v1 submitted 10 August, 2021;
originally announced August 2021.
-
Is My Model Using The Right Evidence? Systematic Probes for Examining Evidence-Based Tabular Reasoning
Authors:
Vivek Gupta,
Riyaz A. Bhat,
Atreya Ghosal,
Manish Shrivastava,
Maneesh Singh,
Vivek Srikumar
Abstract:
Neural models command state-of-the-art performance across NLP tasks, including ones involving "reasoning". Models claiming to reason about the evidence presented to them should attend to the correct parts of the input avoiding spurious patterns therein, be self-consistent in their predictions across inputs, and be immune to biases derived from their pre-training in a nuanced, context-sensitive fas…
▽ More
Neural models command state-of-the-art performance across NLP tasks, including ones involving "reasoning". Models claiming to reason about the evidence presented to them should attend to the correct parts of the input avoiding spurious patterns therein, be self-consistent in their predictions across inputs, and be immune to biases derived from their pre-training in a nuanced, context-sensitive fashion. {\em Do the prevalent *BERT-family of models do so?} In this paper, we study this question using the problem of reasoning on tabular data. Tabular inputs are especially well-suited for the study -- they admit systematic probes targeting the properties listed above. Our experiments demonstrate that a RoBERTa-based model, representative of the current state-of-the-art, fails at reasoning on the following counts: it (a) ignores relevant parts of the evidence, (b) is over-sensitive to annotation artifacts, and (c) relies on the knowledge encoded in the pre-trained language model rather than the evidence presented in its tabular inputs. Finally, through inoculation experiments, we show that fine-tuning the model on perturbed data does not help it overcome the above challenges.
△ Less
Submitted 5 March, 2022; v1 submitted 1 August, 2021;
originally announced August 2021.
-
Multitask Learning for Citation Purpose Classification
Authors:
Alex Oesterling,
Angikar Ghosal,
Haoyang Yu,
Rui Xin,
Yasa Baig,
Lesia Semenova,
Cynthia Rudin
Abstract:
We present our entry into the 2021 3C Shared Task Citation Context Classification based on Purpose competition. The goal of the competition is to classify a citation in a scientific article based on its purpose. This task is important because it could potentially lead to more comprehensive ways of summarizing the purpose and uses of scientific articles, but it is also difficult, mainly due to the…
▽ More
We present our entry into the 2021 3C Shared Task Citation Context Classification based on Purpose competition. The goal of the competition is to classify a citation in a scientific article based on its purpose. This task is important because it could potentially lead to more comprehensive ways of summarizing the purpose and uses of scientific articles, but it is also difficult, mainly due to the limited amount of available training data in which the purposes of each citation have been hand-labeled, along with the subjectivity of these labels. Our entry in the competition is a multi-task model that combines multiple modules designed to handle the problem from different perspectives, including hand-generated linguistic features, TF-IDF features, and an LSTM-with-attention model. We also provide an ablation study and feature analysis whose insights could lead to future work.
△ Less
Submitted 24 June, 2021;
originally announced June 2021.
-
dame-flame: A Python Library Providing Fast Interpretable Matching for Causal Inference
Authors:
Neha R. Gupta,
Vittorio Orlandi,
Chia-Rui Chang,
Tianyu Wang,
Marco Morucci,
Pritam Dey,
Thomas J. Howell,
Xian Sun,
Angikar Ghosal,
Sudeepa Roy,
Cynthia Rudin,
Alexander Volfovsky
Abstract:
dame-flame is a Python package for performing matching for observational causal inference on datasets containing discrete covariates. This package implements the Dynamic Almost Matching Exactly (DAME) and Fast Large-Scale Almost Matching Exactly (FLAME) algorithms, which match treatment and control units on subsets of the covariates. The resulting matched groups are interpretable, because the matc…
▽ More
dame-flame is a Python package for performing matching for observational causal inference on datasets containing discrete covariates. This package implements the Dynamic Almost Matching Exactly (DAME) and Fast Large-Scale Almost Matching Exactly (FLAME) algorithms, which match treatment and control units on subsets of the covariates. The resulting matched groups are interpretable, because the matches are made on covariates, and high-quality, because machine learning is used to determine which covariates are important to match on. DAME solves an optimization problem that matches units on as many covariates as possible, prioritizing matches on important covariates. FLAME approximates the solution found by DAME via a much faster backward feature selection procedure. The package provides several adjustable parameters to adapt the algorithms to specific applications, and can calculate treatment effect estimates after matching. Descriptions of these parameters, details on estimating treatment effects, and further examples, can be found in the documentation at https://almost-matching-exactly.github.io/DAME-FLAME-Python-Package/
△ Less
Submitted 2 April, 2023; v1 submitted 5 January, 2021;
originally announced January 2021.
-
Robust Quadrupedal Locomotion on Sloped Terrains: A Linear Policy Approach
Authors:
Kartik Paigwar,
Lokesh Krishna,
Sashank Tirumala,
Naman Khetan,
Aditya Sagi,
Ashish Joglekar,
Shalabh Bhatnagar,
Ashitava Ghosal,
Bharadwaj Amrutur,
Shishir Kolathaya
Abstract:
In this paper, with a view toward fast deployment of locomotion gaits in low-cost hardware, we use a linear policy for realizing end-foot trajectories in the quadruped robot, Stoch $2$. In particular, the parameters of the end-foot trajectories are shaped via a linear feedback policy that takes the torso orientation and the terrain slope as inputs. The corresponding desired joint angles are obtain…
▽ More
In this paper, with a view toward fast deployment of locomotion gaits in low-cost hardware, we use a linear policy for realizing end-foot trajectories in the quadruped robot, Stoch $2$. In particular, the parameters of the end-foot trajectories are shaped via a linear feedback policy that takes the torso orientation and the terrain slope as inputs. The corresponding desired joint angles are obtained via an inverse kinematics solver and tracked via a PID control law. Augmented Random Search, a model-free and a gradient-free learning algorithm is used to train this linear policy. Simulation results show that the resulting walking is robust to terrain slope variations and external pushes. This methodology is not only computationally light-weight but also uses minimal sensing and actuation capabilities in the robot, thereby justifying the approach.
△ Less
Submitted 10 November, 2020; v1 submitted 30 October, 2020;
originally announced October 2020.
-
Learning Stable Manoeuvres in Quadruped Robots from Expert Demonstrations
Authors:
Sashank Tirumala,
Sagar Gubbi,
Kartik Paigwar,
Aditya Sagi,
Ashish Joglekar,
Shalabh Bhatnagar,
Ashitava Ghosal,
Bharadwaj Amrutur,
Shishir Kolathaya
Abstract:
With the research into development of quadruped robots picking up pace, learning based techniques are being explored for develo** locomotion controllers for such robots. A key problem is to generate leg trajectories for continuously varying target linear and angular velocities, in a stable manner. In this paper, we propose a two pronged approach to address this problem. First, multiple simpler p…
▽ More
With the research into development of quadruped robots picking up pace, learning based techniques are being explored for develo** locomotion controllers for such robots. A key problem is to generate leg trajectories for continuously varying target linear and angular velocities, in a stable manner. In this paper, we propose a two pronged approach to address this problem. First, multiple simpler policies are trained to generate trajectories for a discrete set of target velocities and turning radius. These policies are then augmented using a higher level neural network for handling the transition between the learned trajectories. Specifically, we develop a neural network-based filter that takes in target velocity, radius and transforms them into new commands that enable smooth transitions to the new trajectory. This transformation is achieved by learning from expert demonstrations. An application of this is the transformation of a novice user's input into an expert user's input, thereby ensuring stable manoeuvres regardless of the user's experience. Training our proposed architecture requires much less expert demonstrations compared to standard neural network architectures. Finally, we demonstrate experimentally these results in the in-house quadruped Stoch 2.
△ Less
Submitted 28 July, 2020;
originally announced July 2020.
-
3D printed cable-driven continuum robots with generally routed cables: modeling and experiments
Authors:
Soumya Kanti Mahapatra,
Ashwin K. P.,
Ashitava Ghosal
Abstract:
Continuum robots are becoming increasingly popular for applications which require the robots to deform and change shape, while also being compliant. A cable-driven continuum robot is one of the most commonly used type. Typical cable driven continuum robots consist of a flexible backbone with spacer disks attached to the backbone and cables passing through the holes in the spacer disks from the fix…
▽ More
Continuum robots are becoming increasingly popular for applications which require the robots to deform and change shape, while also being compliant. A cable-driven continuum robot is one of the most commonly used type. Typical cable driven continuum robots consist of a flexible backbone with spacer disks attached to the backbone and cables passing through the holes in the spacer disks from the fixed base to a free end. In most such robots, the routing of the cables are straight or a smooth helical curve. In this paper, we analyze the experimental and theoretical deformations of a 3D printed continuum robot, for 6 different kinds of cable routings. The results are compared for discrete optimization based kinematic modelling as well as static modelling using Cosserat rod theory. It is shown that the experimental results match the theoretical results with an error margin of 2%. It is also shown that the optimization based approach is faster than the one based on Cosserat rod theory. We also present a three-fingered gripper prototype where each of the fingers are 3D printed continuum robots with general cable routing. It is demonstrated that the prototype can be used for grip** objects and for its manipulation.
△ Less
Submitted 10 March, 2020;
originally announced March 2020.
-
Gait Library Synthesis for Quadruped Robots via Augmented Random Search
Authors:
Sashank Tirumala,
Aditya Sagi,
Kartik Paigwar,
Ashish Joglekar,
Shalabh Bhatnagar,
Ashitava Ghosal,
Bharadwaj Amrutur,
Shishir Kolathaya
Abstract:
In this paper, with a view toward fast deployment of learned locomotion gaits in low-cost hardware, we generate a library of walking trajectories, namely, forward trot, backward trot, side-step, and turn in our custom-built quadruped robot, Stoch 2, using reinforcement learning. There are existing approaches that determine optimal policies for each time step, whereas we determine an optimal policy…
▽ More
In this paper, with a view toward fast deployment of learned locomotion gaits in low-cost hardware, we generate a library of walking trajectories, namely, forward trot, backward trot, side-step, and turn in our custom-built quadruped robot, Stoch 2, using reinforcement learning. There are existing approaches that determine optimal policies for each time step, whereas we determine an optimal policy, in the form of end-foot trajectories, for each half walking step i.e., swing phase and stance phase. The way-points for the foot trajectories are obtained from a linear policy, i.e., a linear function of the states of the robot, and cubic splines are used to interpolate between these points. Augmented Random Search, a model-free and gradient-free learning algorithm is used to learn the policy in simulation. This learned policy is then deployed on hardware, yielding a trajectory in every half walking step. Different locomotion patterns are learned in simulation by enforcing a preconfigured phase shift between the trajectories of different legs. The transition from one gait to another is achieved by using a low-pass filter for the phase, and the sim-to-real transfer is improved by a linear transformation of the states obtained through regression.
△ Less
Submitted 30 December, 2019;
originally announced December 2019.
-
Learning Active Spine Behaviors for Dynamic and Efficient Locomotion in Quadruped Robots
Authors:
Shounak Bhattacharya,
Abhik Singla,
Abhimanyu,
Dhaivat Dholakiya,
Shalabh Bhatnagar,
Bharadwaj Amrutur,
Ashitava Ghosal,
Shishir Kolathaya
Abstract:
In this work, we provide a simulation framework to perform systematic studies on the effects of spinal joint compliance and actuation on bounding performance of a 16-DOF quadruped spined robot Stoch 2. Fast quadrupedal locomotion with active spine is an extremely hard problem, and involves a complex coordination between the various degrees of freedom. Therefore, past attempts at addressing this pr…
▽ More
In this work, we provide a simulation framework to perform systematic studies on the effects of spinal joint compliance and actuation on bounding performance of a 16-DOF quadruped spined robot Stoch 2. Fast quadrupedal locomotion with active spine is an extremely hard problem, and involves a complex coordination between the various degrees of freedom. Therefore, past attempts at addressing this problem have not seen much success. Deep-Reinforcement Learning seems to be a promising approach, after its recent success in a variety of robot platforms, and the goal of this paper is to use this approach to realize the aforementioned behaviors. With this learning framework, the robot reached a bounding speed of 2.1 m/s with a maximum Froude number of 2. Simulation results also show that use of active spine, indeed, increased the stride length, improved the cost of transport, and also reduced the natural frequency to more realistic values.
△ Less
Submitted 15 May, 2019; v1 submitted 15 May, 2019;
originally announced May 2019.
-
Secure OTA Software Updates in Connected Vehicles: A survey
Authors:
Subir Halder,
Amrita Ghosal,
Mauro Conti
Abstract:
This survey highlights and discusses remote OTA software updates in the automotive sector, mainly from the security perspective. In particular, the major objective of this survey is to provide a comprehensive and structured outline of various research directions and approaches in OTA update technologies in vehicles. At first, we discuss the connected car technology and then integrate the relations…
▽ More
This survey highlights and discusses remote OTA software updates in the automotive sector, mainly from the security perspective. In particular, the major objective of this survey is to provide a comprehensive and structured outline of various research directions and approaches in OTA update technologies in vehicles. At first, we discuss the connected car technology and then integrate the relationship of remote OTA update features with the connected car. We also present the benefits of remote OTA updates for cars along with relevant statistics. Then, we emphasize on the security challenges and requirements of remote OTA updates along with use cases and standard road safety regulations followed in different countries. We also provide for a classification of the existing works in literature that deal with implementing different secured techniques for remote OTA updates in vehicles. We further provide an analytical discussion on the present scenario of remote OTA updates with respect to care manufacturers. Finally, we identify possible future research directions of remote OTA updates for automobiles, particularly in the area of security.
△ Less
Submitted 1 April, 2019;
originally announced April 2019.
-
Design, Development and Experimental Realization of a Quadrupedal Research Platform: Stoch
Authors:
Dhaivat Dholakiya,
Shounak Bhattacharya,
Ajay Gunalan,
Abhik Singla,
Shalabh Bhatnagar,
Bharadwaj Amrutur,
Ashitava Ghosal,
Shishir Kolathaya
Abstract:
In this paper, we present a complete description of the hardware design and control architecture of our custom built quadruped robot, called the `Stoch'. Our goal is to realize a robust, modular, and a reliable quadrupedal platform, using which various locomotion behaviors are explored. This platform enables us to explore different research problems in legged locomotion, which use both traditional…
▽ More
In this paper, we present a complete description of the hardware design and control architecture of our custom built quadruped robot, called the `Stoch'. Our goal is to realize a robust, modular, and a reliable quadrupedal platform, using which various locomotion behaviors are explored. This platform enables us to explore different research problems in legged locomotion, which use both traditional and learning based techniques. We discuss the merits and limitations of the platform in terms of exploitation of available behaviours, fast rapid prototy**, reproduction and repair. Towards the end, we will demonstrate trotting, bounding behaviors, and preliminary results in turning. In addition, we will also show various gait transitions i.e., trot-to-turn and trot-to-bound behaviors.
△ Less
Submitted 27 February, 2019; v1 submitted 3 January, 2019;
originally announced January 2019.
-
Realizing Learned Quadruped Locomotion Behaviors through Kinematic Motion Primitives
Authors:
Abhik Singla,
Shounak Bhattacharya,
Dhaivat Dholakiya,
Shalabh Bhatnagar,
Ashitava Ghosal,
Bharadwaj Amrutur,
Shishir Kolathaya
Abstract:
Humans and animals are believed to use a very minimal set of trajectories to perform a wide variety of tasks including walking. Our main objective in this paper is two fold 1) Obtain an effective tool to realize these basic motion patterns for quadrupedal walking, called the kinematic motion primitives (kMPs), via trajectories learned from deep reinforcement learning (D-RL) and 2) Realize a set of…
▽ More
Humans and animals are believed to use a very minimal set of trajectories to perform a wide variety of tasks including walking. Our main objective in this paper is two fold 1) Obtain an effective tool to realize these basic motion patterns for quadrupedal walking, called the kinematic motion primitives (kMPs), via trajectories learned from deep reinforcement learning (D-RL) and 2) Realize a set of behaviors, namely trot, walk, gallop and bound from these kinematic motion primitives in our custom four legged robot, called the `Stoch'. D-RL is a data driven approach, which has been shown to be very effective for realizing all kinds of robust locomotion behaviors, both in simulation and in experiment. On the other hand, kMPs are known to capture the underlying structure of walking and yield a set of derived behaviors. We first generate walking gaits from D-RL, which uses policy gradient based approaches. We then analyze the resulting walking by using principal component analysis. We observe that the kMPs extracted from PCA followed a similar pattern irrespective of the type of gaits generated. Leveraging on this underlying structure, we then realize walking in Stoch by a straightforward reconstruction of joint trajectories from kMPs. This type of methodology improves the transferability of these gaits to real hardware, lowers the computational overhead on-board, and also avoids multiple training iterations by generating a set of derived behaviors from a single learned gait.
△ Less
Submitted 26 February, 2019; v1 submitted 9 October, 2018;
originally announced October 2018.
-
Key Management Systems for Smart Grid Advanced Metering Infrastructure: A Survey
Authors:
Amrita Ghosal,
Mauro Conti
Abstract:
Smart Grids are evolving as the next generation power systems that involve changes in the traditional ways of generation, transmission and distribution of power. Advanced Metering Infrastructure (AMI) is one of the key components in smart grids. An AMI comprises of systems and networks, that collects and analyzes data received from smart meters. In addition, AMI also provides intelligent managemen…
▽ More
Smart Grids are evolving as the next generation power systems that involve changes in the traditional ways of generation, transmission and distribution of power. Advanced Metering Infrastructure (AMI) is one of the key components in smart grids. An AMI comprises of systems and networks, that collects and analyzes data received from smart meters. In addition, AMI also provides intelligent management of various power-related applications and services based on the data collected from smart meters. Thus, AMI plays a significant role in the smooth functioning of smart grids.
AMI is a privileged target for security attacks as it is made up of systems that are highly vulnerable to such attacks. Providing security to AMI is necessary as adversaries can cause potential damage against infrastructures and privacy in smart grid. One of the most effective and challenging topic's identified, is the Key Management System (KMS), for sustaining the security concerns in AMI. Therefore, KMS seeks to be a promising research area for future development of AMI. This survey work highlights the key security issues of advanced metering infrastructures and focuses on how key management techniques can be utilized for safeguarding AMI. First of all, we explore the main features of advanced metering infrastructures and identify the relationship between smart grid and AMI. Then, we introduce the security issues and challenges of AMI. We also provide a classification of the existing works in literature that deal with secure key management system in AMI. Finally, we identify possible future research directions of KMS in AMI.
△ Less
Submitted 31 May, 2018;
originally announced June 2018.
