-
An Adaptable, Safe, and Portable Robot-Assisted Feeding System
Authors:
Ethan Kroll Gordon,
Rajat Kumar Jenamani,
Amal Nanavati,
Ziang Liu,
Haya Bolotski,
Raida Karim,
Daniel Stabile,
Atharva Kashyap,
Bernie Hao Zhu,
Xilai Dai,
Tyler Schrenk,
Jonathan Ko,
Taylor Kessler Faulkner,
Tapomayukh Bhattacharjee,
Siddhartha Srinivasa
Abstract:
We demonstrate a robot-assisted feeding system that enables people with mobility impairments to feed themselves. Our system design embodies Safety, Portability, and User Control, with comprehensive full-stack safety checks, the ability to be mounted on and powered by any powered wheelchair, and a custom web-app allowing care-recipients to leverage their own assistive devices for robot control. For…
▽ More
We demonstrate a robot-assisted feeding system that enables people with mobility impairments to feed themselves. Our system design embodies Safety, Portability, and User Control, with comprehensive full-stack safety checks, the ability to be mounted on and powered by any powered wheelchair, and a custom web-app allowing care-recipients to leverage their own assistive devices for robot control. For bite acquisition, we leverage multi-modal online learning to tractably adapt to unseen food types. For bite transfer, we leverage real-time mouth perception and interaction-aware control. Co-designed with community researchers, our system has been validated through multiple end-user studies.
△ Less
Submitted 6 March, 2024;
originally announced March 2024.
-
Multiple Ways of Working with Users to Develop Physically Assistive Robots
Authors:
Amal Nanavati,
Max Pascher,
Vinitha Ranganeni,
Ethan K. Gordon,
Taylor Kessler Faulkner,
Siddhartha S. Srinivasa,
Maya Cakmak,
Patrícia Alves-Oliveira,
Jens Gerken
Abstract:
Despite the growth of physically assistive robotics (PAR) research over the last decade, nearly half of PAR user studies do not involve participants with the target disabilities. There are several reasons for this -- recruitment challenges, small sample sizes, and transportation logistics -- all influenced by systemic barriers that people with disabilities face. However, it is well-established tha…
▽ More
Despite the growth of physically assistive robotics (PAR) research over the last decade, nearly half of PAR user studies do not involve participants with the target disabilities. There are several reasons for this -- recruitment challenges, small sample sizes, and transportation logistics -- all influenced by systemic barriers that people with disabilities face. However, it is well-established that working with end-users results in technology that better addresses their needs and integrates with their lived circumstances. In this paper, we reflect on multiple approaches we have taken to working with people with motor impairments across the design, development, and evaluation of three PAR projects: (a) assistive feeding with a robot arm; (b) assistive teleoperation with a mobile manipulator; and (c) shared control with a robot arm. We discuss these approaches to working with users along three dimensions -- individual vs. community-level insight, logistic burden on end-users vs. researchers, and benefit to researchers vs. community -- and share recommendations for how other PAR researchers can incorporate users into their work.
△ Less
Submitted 7 March, 2024; v1 submitted 1 March, 2024;
originally announced March 2024.
-
Online augmentation of learned grasp sequence policies for more adaptable and data-efficient in-hand manipulation
Authors:
Ethan K. Gordon,
Rana Soltani Zarrin
Abstract:
When using a tool, the grasps used for picking it up, reposing, and holding it in a suitable pose for the desired task could be distinct. Therefore, a key challenge for autonomous in-hand tool manipulation is finding a sequence of grasps that facilitates every step of the tool use process while continuously maintaining force closure and stability. Due to the complexity of modeling the contact dyna…
▽ More
When using a tool, the grasps used for picking it up, reposing, and holding it in a suitable pose for the desired task could be distinct. Therefore, a key challenge for autonomous in-hand tool manipulation is finding a sequence of grasps that facilitates every step of the tool use process while continuously maintaining force closure and stability. Due to the complexity of modeling the contact dynamics, reinforcement learning (RL) techniques can provide a solution in this continuous space subject to highly parameterized physical models. However, these techniques impose a trade-off in adaptability and data efficiency. At test time the tool properties, desired trajectory, and desired application forces could differ substantially from training scenarios. Adapting to this necessitates more data or computationally expensive online policy updates.
In this work, we apply the principles of discrete dynamic programming (DP) to augment RL performance with domain knowledge. Specifically, we first design a computationally simple approximation of our environment. We then demonstrate in physical simulation that performing tree searches (i.e., lookaheads) and policy rollouts with this approximation can improve an RL-derived grasp sequence policy with minimal additional online computation. Additionally, we show that pretraining a deep RL network with the DP-derived solution to the discretized problem can speed up policy training.
△ Less
Submitted 4 April, 2023;
originally announced April 2023.
-
Real Robot Challenge 2021: Cartesian Position Control with Triangle Grasp and Trajectory Interpolation
Authors:
Rishabh Madan,
Harshit Sikchi,
Ethan K. Gordon,
Tapomayukh Bhattacharjee
Abstract:
We present our runner-up approach for the Real Robot Challenge 2021. We build upon our previous approach used in Real Robot Challenge 2020. To solve the task of sequential goal-reaching we focus on two aspects to achieving near-optimal trajectory: Grasp stability and Controller performance. In the RRC 2021 simulated challenge, our method relied on a hand-designed Pinch grasp combined with Trajecto…
▽ More
We present our runner-up approach for the Real Robot Challenge 2021. We build upon our previous approach used in Real Robot Challenge 2020. To solve the task of sequential goal-reaching we focus on two aspects to achieving near-optimal trajectory: Grasp stability and Controller performance. In the RRC 2021 simulated challenge, our method relied on a hand-designed Pinch grasp combined with Trajectory Interpolation for better stability during the motion for fast goal-reaching. In Stage 1, we observe reverting to a Triangular grasp to provide a more stable grasp when combined with Trajectory Interpolation, possibly due to the sim2real gap. The video demonstration for our approach is available at https://youtu.be/dlOueoaRWrM. The code is publicly available at https://github.com/madan96/benchmark-rrc.
△ Less
Submitted 19 March, 2022; v1 submitted 15 March, 2022;
originally announced March 2022.
-
Balancing Efficiency and Comfort in Robot-Assisted Bite Transfer
Authors:
Suneel Belkhale,
Ethan K. Gordon,
Yuxiao Chen,
Siddhartha Srinivasa,
Tapomayukh Bhattacharjee,
Dorsa Sadigh
Abstract:
Robot-assisted feeding in household environments is challenging because it requires robots to generate trajectories that effectively bring food items of varying shapes and sizes into the mouth while making sure the user is comfortable. Our key insight is that in order to solve this challenge, robots must balance the efficiency of feeding a food item with the comfort of each individual bite. We for…
▽ More
Robot-assisted feeding in household environments is challenging because it requires robots to generate trajectories that effectively bring food items of varying shapes and sizes into the mouth while making sure the user is comfortable. Our key insight is that in order to solve this challenge, robots must balance the efficiency of feeding a food item with the comfort of each individual bite. We formalize comfort and efficiency as heuristics to incorporate in motion planning. We present an approach based on heuristics-guided bi-directional Rapidly-exploring Random Trees (h-BiRRT) that selects bite transfer trajectories of arbitrary food item geometries and shapes using our developed bite efficiency and comfort heuristics and a learned constraint model. Real-robot evaluations show that optimizing both comfort and efficiency significantly outperforms a fixed-pose based method, and users preferred our method significantly more than that of a method that maximizes only user comfort. Videos and Appendices are found on our website: https://sites.google.com/view/comfortbitetransfer-icra22/home.
△ Less
Submitted 24 June, 2022; v1 submitted 22 November, 2021;
originally announced November 2021.
-
Real Robot Challenge: A Robotics Competition in the Cloud
Authors:
Stefan Bauer,
Felix Widmaier,
Manuel Wüthrich,
Annika Buchholz,
Sebastian Stark,
Anirudh Goyal,
Thomas Steinbrenner,
Joel Akpo,
Shruti Joshi,
Vincent Berenz,
Vaibhav Agrawal,
Niklas Funk,
Julen Urain De Jesus,
Jan Peters,
Joe Watson,
Claire Chen,
Krishnan Srinivasan,
Junwu Zhang,
Jeffrey Zhang,
Matthew R. Walter,
Rishabh Madan,
Charles Schaff,
Takahiro Maeda,
Takuma Yoneda,
Denis Yarats
, et al. (17 additional authors not shown)
Abstract:
Dexterous manipulation remains an open problem in robotics. To coordinate efforts of the research community towards tackling this problem, we propose a shared benchmark. We designed and built robotic platforms that are hosted at MPI for Intelligent Systems and can be accessed remotely. Each platform consists of three robotic fingers that are capable of dexterous object manipulation. Users are able…
▽ More
Dexterous manipulation remains an open problem in robotics. To coordinate efforts of the research community towards tackling this problem, we propose a shared benchmark. We designed and built robotic platforms that are hosted at MPI for Intelligent Systems and can be accessed remotely. Each platform consists of three robotic fingers that are capable of dexterous object manipulation. Users are able to control the platforms remotely by submitting code that is executed automatically, akin to a computational cluster. Using this setup, i) we host robotics competitions, where teams from anywhere in the world access our platforms to tackle challenging tasks ii) we publish the datasets collected during these competitions (consisting of hundreds of robot hours), and iii) we give researchers access to these platforms for their own projects.
△ Less
Submitted 10 June, 2022; v1 submitted 22 September, 2021;
originally announced September 2021.
-
Benchmarking Structured Policies and Policy Optimization for Real-World Dexterous Object Manipulation
Authors:
Niklas Funk,
Charles Schaff,
Rishabh Madan,
Takuma Yoneda,
Julen Urain De Jesus,
Joe Watson,
Ethan K. Gordon,
Felix Widmaier,
Stefan Bauer,
Siddhartha S. Srinivasa,
Tapomayukh Bhattacharjee,
Matthew R. Walter,
Jan Peters
Abstract:
Dexterous manipulation is a challenging and important problem in robotics. While data-driven methods are a promising approach, current benchmarks require simulation or extensive engineering support due to the sample inefficiency of popular methods. We present benchmarks for the TriFinger system, an open-source robotic platform for dexterous manipulation and the focus of the 2020 Real Robot Challen…
▽ More
Dexterous manipulation is a challenging and important problem in robotics. While data-driven methods are a promising approach, current benchmarks require simulation or extensive engineering support due to the sample inefficiency of popular methods. We present benchmarks for the TriFinger system, an open-source robotic platform for dexterous manipulation and the focus of the 2020 Real Robot Challenge. The benchmarked methods, which were successful in the challenge, can be generally described as structured policies, as they combine elements of classical robotics and modern policy optimization. This inclusion of inductive biases facilitates sample efficiency, interpretability, reliability and high performance. The key aspects of this benchmarking is validation of the baselines across both simulation and the real system, thorough ablation study over the core features of each solution, and a retrospective analysis of the challenge as a manipulation benchmark. The code and demo videos for this work can be found on our website (https://sites.google.com/view/benchmark-rrc).
△ Less
Submitted 8 December, 2021; v1 submitted 5 May, 2021;
originally announced May 2021.
-
Leveraging Post Hoc Context for Faster Learning in Bandit Settings with Applications in Robot-Assisted Feeding
Authors:
Ethan K. Gordon,
Sumegh Roychowdhury,
Tapomayukh Bhattacharjee,
Kevin Jamieson,
Siddhartha S. Srinivasa
Abstract:
Autonomous robot-assisted feeding requires the ability to acquire a wide variety of food items. However, it is impossible for such a system to be trained on all types of food in existence. Therefore, a key challenge is choosing a manipulation strategy for a previously unseen food item. Previous work showed that the problem can be represented as a linear bandit with visual context. However, food ha…
▽ More
Autonomous robot-assisted feeding requires the ability to acquire a wide variety of food items. However, it is impossible for such a system to be trained on all types of food in existence. Therefore, a key challenge is choosing a manipulation strategy for a previously unseen food item. Previous work showed that the problem can be represented as a linear bandit with visual context. However, food has a wide variety of multi-modal properties relevant to manipulation that can be hard to distinguish visually. Our key insight is that we can leverage the haptic context we collect during and after manipulation (i.e., "post hoc") to learn some of these properties and more quickly adapt our visual model to previously unseen food. In general, we propose a modified linear contextual bandit framework augmented with post hoc context observed after action selection to empirically increase learning speed and reduce cumulative regret. Experiments on synthetic data demonstrate that this effect is more pronounced when the dimensionality of the context is large relative to the post hoc context or when the post hoc context model is particularly easy to learn. Finally, we apply this framework to the bite acquisition problem and demonstrate the acquisition of 8 previously unseen types of food with 21% fewer failures across 64 attempts.
△ Less
Submitted 25 March, 2021; v1 submitted 4 November, 2020;
originally announced November 2020.
-
Adaptive Robot-Assisted Feeding: An Online Learning Framework for Acquiring Previously Unseen Food Items
Authors:
Ethan K. Gordon,
Xiang Meng,
Matt Barnes,
Tapomayukh Bhattacharjee,
Siddhartha S. Srinivasa
Abstract:
A successful robot-assisted feeding system requires bite acquisition of a wide variety of food items. It must adapt to changing user food preferences under uncertain visual and physical environments. Different food items in different environmental conditions require different manipulation strategies for successful bite acquisition. Therefore, a key challenge is how to handle previously unseen food…
▽ More
A successful robot-assisted feeding system requires bite acquisition of a wide variety of food items. It must adapt to changing user food preferences under uncertain visual and physical environments. Different food items in different environmental conditions require different manipulation strategies for successful bite acquisition. Therefore, a key challenge is how to handle previously unseen food items with very different success rate distributions over strategy. Combining low-level controllers and planners into discrete action trajectories, we show that the problem can be represented using a linear contextual bandit setting. We construct a simulated environment using a doubly robust loss estimate from previously seen food items, which we use to tune the parameters of off-the-shelf contextual bandit algorithms. Finally, we demonstrate empirically on a robot-assisted feeding system that, even starting with a model trained on thousands of skewering attempts on dissimilar previously seen food items, $ε$-greedy and LinUCB algorithms can quickly converge to the most successful manipulation strategy.
△ Less
Submitted 31 July, 2020; v1 submitted 19 August, 2019;
originally announced August 2019.
-
Robot-Assisted Feeding: Generalizing Skewering Strategies across Food Items on a Realistic Plate
Authors:
Ryan Feng,
Youngsun Kim,
Gilwoo Lee,
Ethan K. Gordon,
Matt Schmittle,
Shivaum Kumar,
Tapomayukh Bhattacharjee,
Siddhartha S. Srinivasa
Abstract:
A robot-assisted feeding system must successfully acquire many different food items. A key challenge is the wide variation in the physical properties of food, demanding diverse acquisition strategies that are also capable of adapting to previously unseen items. Our key insight is that items with similar physical properties will exhibit similar success rates across an action space, allowing the rob…
▽ More
A robot-assisted feeding system must successfully acquire many different food items. A key challenge is the wide variation in the physical properties of food, demanding diverse acquisition strategies that are also capable of adapting to previously unseen items. Our key insight is that items with similar physical properties will exhibit similar success rates across an action space, allowing the robot to generalize its actions to previously unseen items. To better understand which skewering strategy works best for each food item, we collected a dataset of 2450 robot bite acquisition trials for 16 food items with varying properties. Analyzing the dataset provided insights into how the food items' surrounding environment, fork pitch, and fork roll angles affect bite acquisition success. We then developed a bite acquisition framework that takes the image of a full plate as an input, segments it into food items, and then applies our Skewering-Position-Action network (SPANet) to choose a target food item and a corresponding action so that the bite acquisition success rate is maximized. SPANet also uses the surrounding environment features of food items to predict action success rates. We used this framework to perform multiple experiments on uncluttered and cluttered plates. Results indicate that our integrated system can successfully generalize skewering strategies to many previously unseen food items.
△ Less
Submitted 6 September, 2019; v1 submitted 5 June, 2019;
originally announced June 2019.
-
Design and Control of a Photonic Neural Network Applied to High-Bandwidth Classification
Authors:
Ethan K. Gordon
Abstract:
Neural networks can very effectively perform multidimensional nonlinear classification. However, electronic networks suffer from significant bandwidth limitations due to carrier lifetimes and capacitive coupling. This project investigates photonic neural networks that can get around these limitations by performing both the activation function and weighted addition in the optical domain using micro…
▽ More
Neural networks can very effectively perform multidimensional nonlinear classification. However, electronic networks suffer from significant bandwidth limitations due to carrier lifetimes and capacitive coupling. This project investigates photonic neural networks that can get around these limitations by performing both the activation function and weighted addition in the optical domain using microring resonators. These optical microring resonators provide both nonlinearity and superior fan-in without compromising bandwidth. The ability to thermally calibrate networks of cascaded axons and dendrites and train such a network to solve nonlinear classification problems are demonstrated using theory and simulations. The former is also demonstrated experimentally on a two-channel axon cascaded into a two-channel dendrite, showing good agreement between simulation and experiment. In addition, the use of transverse modes to increase the size of each photonic layer is examined. Simulations that determined the optimal waveguide geometry for using these modes were experimentally validated.
△ Less
Submitted 16 September, 2019; v1 submitted 15 October, 2018;
originally announced October 2018.
