-
Control Interface Remap** for Bias-Aware Assistive Teleoperation
Authors:
Andrew Thompson,
Larisa Y. C. Loke,
Brenna Argall
Abstract:
Users of assistive devices vary in their extent of motor impairment, and hence their physical interaction with control interfaces can differ. There is the potential for improved utility if control interface actuation is mapped to assistive device control signals in a manner customized to each user. In this paper, we present (1) a method for creating a custom interface to assistive device control m…
▽ More
Users of assistive devices vary in their extent of motor impairment, and hence their physical interaction with control interfaces can differ. There is the potential for improved utility if control interface actuation is mapped to assistive device control signals in a manner customized to each user. In this paper, we present (1) a method for creating a custom interface to assistive device control map** based on the design of a user's bias profile, (2) a procedure and virtual task for gathering interface actuation data from which to build the bias profile and map, and (3) an evaluation of our method on 6 participants with upper limb motor impairments. Our results show that custom interface remap** based on user bias profiles shows promise in providing assistance via an improvement in the reachability of the device control space. This effect was especially pronounced for individuals who had a more limited reachable space.
△ Less
Submitted 17 May, 2022;
originally announced May 2022.
-
The Interface Usage Skills Test: An Open Source Tool for Quantitative Evaluation in Real Time for Clinicians and Researchers
Authors:
Mahdieh Nejati Javaremi,
Sisilia Sinaga,
Yuming **,
Matthew L. Elwin,
Brenna Argall
Abstract:
Assistive machines endow people with limited mobility the opportunity to live more independently. However, operating these machines poses risks to the safety of the human operator as well as the surrounding environment. Thus, proper user training is an essential step towards independent control and use of functionally assistive machines. The human operator can use a variety of control interfaces t…
▽ More
Assistive machines endow people with limited mobility the opportunity to live more independently. However, operating these machines poses risks to the safety of the human operator as well as the surrounding environment. Thus, proper user training is an essential step towards independent control and use of functionally assistive machines. The human operator can use a variety of control interfaces to issue control signals to the device, depending on the residual mobility and level of injury of the human operator. Proficiency in operating the interface of choice precedes the skill in operating the assistive machine. In this systems paper, we present an open source tool for automatically and objectively quantifying user skill in operating various interface devices.
△ Less
Submitted 18 May, 2022; v1 submitted 20 March, 2022;
originally announced March 2022.
-
MAAD: A Model and Dataset for "Attended Awareness" in Driving
Authors:
Deepak Gopinath,
Guy Rosman,
Simon Stent,
Katsuya Terahata,
Luke Fletcher,
Brenna Argall,
John Leonard
Abstract:
We propose a computational model to estimate a person's attended awareness of their environment. We define attended awareness to be those parts of a potentially dynamic scene which a person has attended to in recent history and which they are still likely to be physically aware of. Our model takes as input scene information in the form of a video and noisy gaze estimates, and outputs visual salien…
▽ More
We propose a computational model to estimate a person's attended awareness of their environment. We define attended awareness to be those parts of a potentially dynamic scene which a person has attended to in recent history and which they are still likely to be physically aware of. Our model takes as input scene information in the form of a video and noisy gaze estimates, and outputs visual saliency, a refined gaze estimate, and an estimate of the person's attended awareness. In order to test our model, we capture a new dataset with a high-precision gaze tracker including 24.5 hours of gaze sequences from 23 subjects attending to videos of driving scenes. The dataset also contains third-party annotations of the subjects' attended awareness based on observations of their scan path. Our results show that our model is able to reasonably estimate attended awareness in a controlled setting, and in the future could potentially be extended to real egocentric driving data to help enable more effective ahead-of-time warnings in safety systems and thereby augment driver performance. We also demonstrate our model's effectiveness on the tasks of saliency, gaze calibration, and denoising, using both our dataset and an existing saliency dataset. We make our model and dataset available at https://github.com/ToyotaResearchInstitute/att-aware/.
△ Less
Submitted 16 October, 2021;
originally announced October 2021.
-
Learning to Control Complex Robots Using High-Dimensional Interfaces: Preliminary Insights
Authors:
Jongmin M. Lee,
Temesgen Gebrekristos,
Dalia De Santis,
Mahdieh Nejati-Javaremi,
Deepak Gopinath,
Biraj Parikh,
Ferdinando A. Mussa-Ivaldi,
Brenna D. Argall
Abstract:
Human body motions can be captured as a high-dimensional continuous signal using motion sensor technologies. The resulting data can be surprisingly rich in information, even when captured from persons with limited mobility. In this work, we explore the use of limited upper-body motions, captured via motion sensors, as inputs to control a 7 degree-of-freedom assistive robotic arm. It is possible th…
▽ More
Human body motions can be captured as a high-dimensional continuous signal using motion sensor technologies. The resulting data can be surprisingly rich in information, even when captured from persons with limited mobility. In this work, we explore the use of limited upper-body motions, captured via motion sensors, as inputs to control a 7 degree-of-freedom assistive robotic arm. It is possible that even dense sensor signals lack the salient information and independence necessary for reliable high-dimensional robot control. As the human learns over time in the context of this limitation, intelligence on the robot can be leveraged to better identify key learning challenges, provide useful feedback, and support individuals until the challenges are managed. In this short paper, we examine two uninjured participants' data from an ongoing study, to extract preliminary results and share insights. We observe opportunities for robot intelligence to step in, including the identification of inconsistencies in time spent across all control dimensions, asymmetries in individual control dimensions, and user progress in learning. Machine reasoning about these situations may facilitate novel interface learning in the future.
△ Less
Submitted 9 October, 2021;
originally announced October 2021.
-
An Analysis of Human-Robot Information Streams to Inform Dynamic Autonomy Allocation
Authors:
Christopher X. Miller,
Temesgen Gebrekristos,
Michael Young,
Enid Montague,
Brenna Argall
Abstract:
A dynamic autonomy allocation framework automatically shifts how much control lies with the human versus the robotics autonomy, for example based on factors such as environmental safety or user preference. To investigate the question of which factors should drive dynamic autonomy allocation, we perform a human subject study to collect ground truth data that shifts between levels of autonomy during…
▽ More
A dynamic autonomy allocation framework automatically shifts how much control lies with the human versus the robotics autonomy, for example based on factors such as environmental safety or user preference. To investigate the question of which factors should drive dynamic autonomy allocation, we perform a human subject study to collect ground truth data that shifts between levels of autonomy during shared-control robot operation. Information streams from the human, the interaction between the human and the robot, and the environment are analyzed. Machine learning methods -- both classical and deep learning -- are trained on this data. An analysis of information streams from the human-robot team suggests features which capture the interaction between the human and the robotics autonomy are the most informative in predicting when to shift autonomy levels. Even the addition of data from the environment does little to improve upon this predictive power. The features learned by deep networks, in comparison to the hand-engineered features, prove variable in their ability to represent shift-relevant information. This work demonstrates the classification power of human-only and human-robot interaction information streams for use in the design of shared-control frameworks, and provides insights into the comparative utility of various data streams and methods to extract shift-relevant information from those data.
△ Less
Submitted 3 August, 2021;
originally announced August 2021.
-
Characterization of Assistive Robot Arm Teleoperation: A Preliminary Study to Inform Shared Control
Authors:
Mahdieh Nejati Javaremi,
Brenna D. Argall
Abstract:
Assistive robotic devices can increase the independence of individuals with motor impairments. However, each person is unique in their level of injury, preferences, and skills, which moreover can change over time. Further, the amount of assistance required can vary throughout the day due to pain or fatigue, or over longer periods due to rehabilitation, debilitating conditions, or aging. Therefore,…
▽ More
Assistive robotic devices can increase the independence of individuals with motor impairments. However, each person is unique in their level of injury, preferences, and skills, which moreover can change over time. Further, the amount of assistance required can vary throughout the day due to pain or fatigue, or over longer periods due to rehabilitation, debilitating conditions, or aging. Therefore, in order to become an effective team member, the assistive machine should be able to learn from and adapt to the human user. To do so, we need to be able to characterize the user's control commands to determine when and how autonomy should change to best assist the user. We perform a 20 person pilot study in order to establish a set of meaningful performance measures which can be used to characterize the user's control signals and as cues for the autonomy to modify the level and amount of assistance. Our study includes 8 spinal cord injured and 12 uninjured individuals. The results unveil a set of objective, runtime-computable metrics that are correlated with user-perceived task difficulty, and thus could be used by an autonomy system when deciding whether assistance is required. The results further show that metrics which evaluate the user interaction with the robotic device, robot execution, and the perceived task difficulty show differences among spinal cord injured and uninjured groups, and are affected by the type of control interface used. The results will be used to develop an adaptable, user-centered, and individually customized shared-control algorithms.
△ Less
Submitted 31 July, 2020;
originally announced August 2020.
-
Customized Handling of Unintended Interface Operation in Assistive Robots
Authors:
Deepak Gopinath,
Mahdieh Nejati Javaremi,
Brenna D. Argall
Abstract:
We present an assistance system that reasons about a human's intended actions during robot teleoperation in order to provide appropriate corrections for unintended behavior. We model the human's physical interaction with a control interface during robot teleoperation and distinguish between intended and measured physical actions explicitly. By reasoning over the unobserved intentions using model-b…
▽ More
We present an assistance system that reasons about a human's intended actions during robot teleoperation in order to provide appropriate corrections for unintended behavior. We model the human's physical interaction with a control interface during robot teleoperation and distinguish between intended and measured physical actions explicitly. By reasoning over the unobserved intentions using model-based inference techniques, our assistive system provides customized corrections on a user's issued commands. We validate our algorithm with a 10-person human subject study in which we evaluate the performance of the proposed assistance paradigms. Our results show that the assistance paradigms helped to significantly reduce task completion time, number of mode switches, cognitive workload, and user frustration and improve overall user satisfaction.
△ Less
Submitted 5 November, 2020; v1 submitted 4 July, 2020;
originally announced July 2020.
-
Formalizing and Guaranteeing* Human-Robot Interaction
Authors:
Hadas Kress-Gazit,
Kerstin Eder,
Guy Hoffman,
Henny Admoni,
Brenna Argall,
Ruediger Ehlers,
Christoffer Heckman,
Nils Jansen,
Ross Knepper,
Jan Křetínský,
Shelly Levy-Tzedek,
Jamy Li,
Todd Murphey,
Laurel Riek,
Dorsa Sadigh
Abstract:
Robot capabilities are maturing across domains, from self-driving cars, to bipeds and drones. As a result, robots will soon no longer be confined to safety-controlled industrial settings; instead, they will directly interact with the general public. The growing field of Human-Robot Interaction (HRI) studies various aspects of this scenario - from social norms to joint action to human-robot teams a…
▽ More
Robot capabilities are maturing across domains, from self-driving cars, to bipeds and drones. As a result, robots will soon no longer be confined to safety-controlled industrial settings; instead, they will directly interact with the general public. The growing field of Human-Robot Interaction (HRI) studies various aspects of this scenario - from social norms to joint action to human-robot teams and more. Researchers in HRI have made great strides in develo** models, methods, and algorithms for robots acting with and around humans, but these "computational HRI" models and algorithms generally do not come with formal guarantees and constraints on their operation. To enable human-interactive robots to move from the lab to real-world deployments, we must address this gap.
This article provides an overview of verification, validation and synthesis techniques used to create demonstrably trustworthy systems, describes several HRI domains that could benefit from such techniques, and provides a roadmap for the challenges and the research needed to create formalized and guaranteed human-robot interaction.
△ Less
Submitted 30 June, 2020;
originally announced June 2020.
-
Data-driven Koopman Operators for Model-based Shared Control of Human-Machine Systems
Authors:
Alexander Broad,
Ian Abraham,
Todd Murphey,
Brenna Argall
Abstract:
We present a data-driven shared control algorithm that can be used to improve a human operator's control of complex dynamic machines and achieve tasks that would otherwise be challenging, or impossible, for the user on their own. Our method assumes no a priori knowledge of the system dynamics. Instead, both the dynamics and information about the user's interaction are learned from observation thro…
▽ More
We present a data-driven shared control algorithm that can be used to improve a human operator's control of complex dynamic machines and achieve tasks that would otherwise be challenging, or impossible, for the user on their own. Our method assumes no a priori knowledge of the system dynamics. Instead, both the dynamics and information about the user's interaction are learned from observation through the use of a Koopman operator. Using the learned model, we define an optimization problem to compute the autonomous partner's control policy. Finally, we dynamically allocate control authority to each partner based on a comparison of the user input and the autonomously generated control. We refer to this idea as model-based shared control (MbSC). We evaluate the efficacy of our approach with two human subjects studies consisting of 32 total participants (16 subjects in each study). The first study imposes a linear constraint on the modeling and autonomous policy generation algorithms. The second study explores the more general, nonlinear variant. Overall, we find that model-based shared control significantly improves task and control metrics when compared to a natural learning, or user only, control paradigm. Our experiments suggest that models learned via the Koopman operator generalize across users, indicating that it is not necessary to collect data from each individual user before providing assistance with MbSC. We also demonstrate the data-efficiency of MbSC and consequently, it's usefulness in online learning paradigms. Finally, we find that the nonlinear variant has a greater impact on a user's ability to successfully achieve a defined task than the linear variant.
△ Less
Submitted 12 June, 2020;
originally announced June 2020.
-
Hybrid Control for Learning Motor Skills
Authors:
Ian Abraham,
Alexander Broad,
Allison Pinosky,
Brenna Argall,
Todd D. Murphey
Abstract:
We develop a hybrid control approach for robot learning based on combining learned predictive models with experience-based state-action policy map**s to improve the learning capabilities of robotic systems. Predictive models provide an understanding of the task and the physics (which improves sample-efficiency), while experience-based policy map**s are treated as "muscle memory" that encode fa…
▽ More
We develop a hybrid control approach for robot learning based on combining learned predictive models with experience-based state-action policy map**s to improve the learning capabilities of robotic systems. Predictive models provide an understanding of the task and the physics (which improves sample-efficiency), while experience-based policy map**s are treated as "muscle memory" that encode favorable actions as experiences that override planned actions. Hybrid control tools are used to create an algorithmic approach for combining learned predictive models with experience-based learning. Hybrid learning is presented as a method for efficiently learning motor skills by systematically combining and improving the performance of predictive models and experience-based policies. A deterministic variation of hybrid learning is derived and extended into a stochastic implementation that relaxes some of the key assumptions in the original derivation. Each variation is tested on experience-based learning methods (where the robot interacts with the environment to gain experience) as well as imitation learning methods (where experience is provided through demonstrations and tested in the environment). The results show that our method is capable of improving the performance and sample-efficiency of learning motor skills in a variety of experimental domains.
△ Less
Submitted 5 June, 2020;
originally announced June 2020.
-
Active Intent Disambiguation for Shared Control Robots
Authors:
Deepak E. Gopinath,
Brenna D. Argall
Abstract:
Assistive shared-control robots have the potential to transform the lives of millions of people afflicted with severe motor impairments. The usefulness of shared-control robots typically relies on the underlying autonomy's ability to infer the user's needs and intentions, and the ability to do so unambiguously is often a limiting factor for providing appropriate assistance confidently and accurate…
▽ More
Assistive shared-control robots have the potential to transform the lives of millions of people afflicted with severe motor impairments. The usefulness of shared-control robots typically relies on the underlying autonomy's ability to infer the user's needs and intentions, and the ability to do so unambiguously is often a limiting factor for providing appropriate assistance confidently and accurately. The contributions of this paper are four-fold. First, we introduce the idea of intent disambiguation via control mode selection, and present a mathematical formalism for the same. Second, we develop a control mode selection algorithm which selects the control mode in which the user-initiated motion helps the autonomy to maximally disambiguate user intent. Third, we present a pilot study with eight subjects to evaluate the efficacy of the disambiguation algorithm. Our results suggest that the disambiguation system (a) helps to significantly reduce task effort, as measured by number of button presses, and (b) is of greater utility for more limited control interfaces and more complex tasks. We also observe that (c) subjects demonstrated a wide range of disambiguation request behaviors, with the common thread of concentrating requests early in the execution. As our last contribution, we introduce a novel field-theoretic approach to intent inference inspired by dynamic field theory that works in tandem with the disambiguation scheme.
△ Less
Submitted 7 May, 2020;
originally announced May 2020.
-
Highly Parallelized Data-driven MPC for Minimal Intervention Shared Control
Authors:
Alexander Broad,
Todd Murphey,
Brenna Argall
Abstract:
We present a shared control paradigm that improves a user's ability to operate complex, dynamic systems in potentially dangerous environments without a priori knowledge of the user's objective. In this paradigm, the role of the autonomous partner is to improve the general safety of the system without constraining the user's ability to achieve unspecified behaviors. Our approach relies on a data-dr…
▽ More
We present a shared control paradigm that improves a user's ability to operate complex, dynamic systems in potentially dangerous environments without a priori knowledge of the user's objective. In this paradigm, the role of the autonomous partner is to improve the general safety of the system without constraining the user's ability to achieve unspecified behaviors. Our approach relies on a data-driven, model-based representation of the joint human-machine system to evaluate, in parallel, a significant number of potential inputs that the user may wish to provide. These samples are used to (1) predict the safety of the system over a receding horizon, and (2) minimize the influence of the autonomous partner. The resulting shared control algorithm maximizes the authority allocated to the human partner to improve their sense of agency, while improving safety. We evaluate the efficacy of our shared control algorithm with a human subjects study (n=20) conducted in two simulated environments: a balance bot and a race car. During the experiment, users are free to operate each system however they would like (i.e., there is no specified task) and are only asked to try to avoid unsafe regions of the state space. Using modern computational resources (i.e., GPUs) our approach is able to consider more than 10,000 potential trajectories at each time step in a control loop running at 100Hz for the balance bot and 60Hz for the race car. The results of the study show that our shared control paradigm improves system safety without knowledge of the user's goal, while maintaining high-levels of user satisfaction and low-levels of frustration. Our code is available online at https://github.com/asbroad/mpmi_shared_control.
△ Less
Submitted 5 June, 2019;
originally announced June 2019.
-
Operation and Imitation under Safety-Aware Shared Control
Authors:
Alexander Broad,
Todd Murphey,
Brenna Argall
Abstract:
We describe a shared control methodology that can, without knowledge of the task, be used to improve a human's control of a dynamic system, be used as a training mechanism, and be used in conjunction with Imitation Learning to generate autonomous policies that recreate novel behaviors. Our algorithm introduces autonomy that assists the human partner by enforcing safety and stability constraints. T…
▽ More
We describe a shared control methodology that can, without knowledge of the task, be used to improve a human's control of a dynamic system, be used as a training mechanism, and be used in conjunction with Imitation Learning to generate autonomous policies that recreate novel behaviors. Our algorithm introduces autonomy that assists the human partner by enforcing safety and stability constraints. The autonomous agent has no a priori knowledge of the desired task and therefore only adds control information when there is concern for the safety of the system. We evaluate the efficacy of our approach with a human subjects study consisting of 20 participants. We find that our shared control algorithm significantly improves the rate at which users are able to successfully execute novel behaviors. Experimental results suggest that the benefits of our safety-aware shared control algorithm also extend to the human partner's understanding of the system and their control skill. Finally, we demonstrate how a combination of our safety-aware shared control algorithm and Imitation Learning can be used to autonomously recreate the demonstrated behaviors.
△ Less
Submitted 26 May, 2019;
originally announced May 2019.
-
Learning Models for Shared Control of Human-Machine Systems with Unknown Dynamics
Authors:
Alexander Broad,
Todd Murphey,
Brenna Argall
Abstract:
We present a novel approach to shared control of human-machine systems. Our method assumes no a priori knowledge of the system dynamics. Instead, we learn both the dynamics and information about the user's interaction from observation through the use of the Koopman operator. Using the learned model, we define an optimization problem to compute the optimal policy for a given task, and compare the u…
▽ More
We present a novel approach to shared control of human-machine systems. Our method assumes no a priori knowledge of the system dynamics. Instead, we learn both the dynamics and information about the user's interaction from observation through the use of the Koopman operator. Using the learned model, we define an optimization problem to compute the optimal policy for a given task, and compare the user input to the optimal input. We demonstrate the efficacy of our approach with a user study. We also analyze the individual nature of the learned models by comparing the effectiveness of our approach when the demonstration data comes from a user's own interactions, from the interactions of a group of users and from a domain expert. Positive results include statistically significant improvements on task metrics when comparing a user-only control paradigm with our shared control paradigm. Surprising results include findings that suggest that individualizing the model based on a user's own data does not effect the ability to learn a useful dynamic system. We explore this tension as it relates to develo** human-in-the-loop systems further in the discussion.
△ Less
Submitted 24 August, 2018;
originally announced August 2018.
-
Structured Neural Network Dynamics for Model-based Control
Authors:
Alexander Broad,
Ian Abraham,
Todd Murphey,
Brenna Argall
Abstract:
We present a structured neural network architecture that is inspired by linear time-varying dynamical systems. The network is designed to mimic the properties of linear dynamical systems which makes analysis and control simple. The architecture facilitates the integration of learned system models with gradient-based model predictive control algorithms, and removes the requirement of computing pote…
▽ More
We present a structured neural network architecture that is inspired by linear time-varying dynamical systems. The network is designed to mimic the properties of linear dynamical systems which makes analysis and control simple. The architecture facilitates the integration of learned system models with gradient-based model predictive control algorithms, and removes the requirement of computing potentially costly derivatives online. We demonstrate the efficacy of this modeling technique in computing autonomous control policies through evaluation in a variety of standard continuous control domains.
△ Less
Submitted 3 August, 2018;
originally announced August 2018.
-
Geometry-Based Region Proposals for Real-Time Robot Detection of Tabletop Objects
Authors:
Alexander Broad,
Brenna Argall
Abstract:
We present a novel object detection pipeline for localization and recognition in three dimensional environments. Our approach makes use of an RGB-D sensor and combines state-of-the-art techniques from the robotics and computer vision communities to create a robust, real-time detection system. We focus specifically on solving the object detection problem for tabletop scenes, a common environment fo…
▽ More
We present a novel object detection pipeline for localization and recognition in three dimensional environments. Our approach makes use of an RGB-D sensor and combines state-of-the-art techniques from the robotics and computer vision communities to create a robust, real-time detection system. We focus specifically on solving the object detection problem for tabletop scenes, a common environment for assistive manipulators. Our detection pipeline locates objects in a point cloud representation of the scene. These clusters are subsequently used to compute a bounding box around each object in the RGB space. Each defined patch is then fed into a Convolutional Neural Network (CNN) for object recognition. We also demonstrate that our region proposal method can be used to develop novel datasets that are both large and diverse enough to train deep learning models, and easy enough to collect that end-users can develop their own datasets. Lastly, we validate the resulting system through an extensive analysis of the accuracy and run-time of the full pipeline.
△ Less
Submitted 14 March, 2017;
originally announced March 2017.
