-
ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data
Authors:
Zeyi Liu,
Cheng Chi,
Eric Cousineau,
Naveen Kuppuswamy,
Benjamin Burchfiel,
Shuran Song
Abstract:
Audio signals provide rich information for the robot interaction and object properties through contact. These information can surprisingly ease the learning of contact-rich robot manipulation skills, especially when the visual information alone is ambiguous or incomplete. However, the usage of audio data in robot manipulation has been constrained to teleoperated demonstrations collected by either…
▽ More
Audio signals provide rich information for the robot interaction and object properties through contact. These information can surprisingly ease the learning of contact-rich robot manipulation skills, especially when the visual information alone is ambiguous or incomplete. However, the usage of audio data in robot manipulation has been constrained to teleoperated demonstrations collected by either attaching a microphone to the robot or object, which significantly limits its usage in robot learning pipelines. In this work, we introduce ManiWAV: an 'ear-in-hand' data collection device to collect in-the-wild human demonstrations with synchronous audio and visual feedback, and a corresponding policy interface to learn robot manipulation policy directly from the demonstrations. We demonstrate the capabilities of our system through four contact-rich manipulation tasks that require either passively sensing the contact events and modes, or actively sensing the object surface materials and states. In addition, we show that our system can generalize to unseen in-the-wild environments, by learning from diverse in-the-wild human demonstrations. Project website: https://mani-wav.github.io/
△ Less
Submitted 27 June, 2024;
originally announced June 2024.
-
Stretch with Stretch: Physical Therapy Exercise Games Led by a Mobile Manipulator
Authors:
Matthew Lamsey,
You Liang Tan,
Meredith D. Wells,
Madeline Beatty,
Zexuan Liu,
Arjun Majumdar,
Kendra Washington,
Jerry Feldman,
Naveen Kuppuswamy,
Elizabeth Nguyen,
Arielle Wallenstein,
Madeleine E. Hackney,
Charles C. Kemp
Abstract:
Physical therapy (PT) is a key component of many rehabilitation regimens, such as treatments for Parkinson's disease (PD). However, there are shortages of physical therapists and adherence to self-guided PT is low. Robots have the potential to support physical therapists and increase adherence to self-guided PT, but prior robotic systems have been large and immobile, which can be a barrier to use…
▽ More
Physical therapy (PT) is a key component of many rehabilitation regimens, such as treatments for Parkinson's disease (PD). However, there are shortages of physical therapists and adherence to self-guided PT is low. Robots have the potential to support physical therapists and increase adherence to self-guided PT, but prior robotic systems have been large and immobile, which can be a barrier to use in homes and clinics. We present Stretch with Stretch (SWS), a novel robotic system for leading stretching exercise games for older adults with PD. SWS consists of a compact and lightweight mobile manipulator (Hello Robot Stretch RE1) that visually and verbally guides users through PT exercises. The robot's soft end effector serves as a target that users repetitively reach towards and press with a hand, foot, or knee. For each exercise, target locations are customized for the individual via a visually estimated kinematic model, a haptically estimated range of motion, and the person's exercise performance. The system includes sound effects and verbal feedback from the robot to keep users engaged throughout a session and augment physical exercise with cognitive exercise. We conducted a user study for which people with PD (n=10) performed 6 exercises with the system. Participants perceived the SWS to be useful and easy to use. They also reported mild to moderate perceived exertion (RPE).
△ Less
Submitted 21 December, 2023; v1 submitted 20 December, 2023;
originally announced December 2023.
-
Proximity and Visuotactile Point Cloud Fusion for Contact Patches in Extreme Deformation
Authors:
Jessica Yin,
Paarth Shah,
Naveen Kuppuswamy,
Andrew Beaulieu,
Avinash Uttamchandani,
Alejandro Castro,
James Pikul,
Russ Tedrake
Abstract:
Equip** robots with the sense of touch is critical to emulating the capabilities of humans in real world manipulation tasks. Visuotactile sensors are a popular tactile sensing strategy due to data output compatible with computer vision algorithms and accurate, high resolution estimates of local object geometry. However, these sensors struggle to accommodate high deformations of the sensing surfa…
▽ More
Equip** robots with the sense of touch is critical to emulating the capabilities of humans in real world manipulation tasks. Visuotactile sensors are a popular tactile sensing strategy due to data output compatible with computer vision algorithms and accurate, high resolution estimates of local object geometry. However, these sensors struggle to accommodate high deformations of the sensing surface during object interactions, hindering more informative contact with cm-scale objects frequently encountered in the real world. The soft interfaces of visuotactile sensors are often made of hyperelastic elastomers, which are difficult to simulate quickly and accurately when extremely deformed for tactile information. Additionally, many visuotactile sensors that rely on strict internal light conditions or pattern tracking will fail if the surface is highly deformed. In this work, we propose an algorithm that fuses proximity and visuotactile point clouds for contact patch segmentation that is entirely independent from membrane mechanics. This algorithm exploits the synchronous, high-res proximity and visuotactile modalities enabled by an extremely deformable, selectively transmissive soft membrane, which uses visible light for visuotactile sensing and infrared light for proximity depth. We present the hardware design, membrane fabrication, and evaluation of our contact patch algorithm in low (10%), medium (60%), and high (100%+) membrane strain states. We compare our algorithm against three baselines: proximity-only, tactile-only, and a membrane mechanics model. Our proposed algorithm outperforms all baselines with an average RMSE under 2.8mm of the contact patch geometry across all strain ranges. We demonstrate our contact patch algorithm in four applications: varied stiffness membranes, torque and shear-induced wrinkling, closed loop control for whole body manipulation, and pose estimation.
△ Less
Submitted 7 July, 2023;
originally announced July 2023.
-
Punyo-1: Soft tactile-sensing upper-body robot for large object manipulation and physical human interaction
Authors:
Aimee Goncalves,
Naveen Kuppuswamy,
Andrew Beaulieu,
Avinash Uttamchandani,
Katherine M. Tsui,
Alex Alspach
Abstract:
The manipulation of large objects and safe operation in the vicinity of humans are key capabilities of a general purpose domestic robotic assistant. We present the design of a soft, tactile-sensing humanoid upper-body robot and demonstrate whole-body rich-contact manipulation strategies for handling large objects. We demonstrate our hardware design philosophy for outfitting off-the-shelf hard robo…
▽ More
The manipulation of large objects and safe operation in the vicinity of humans are key capabilities of a general purpose domestic robotic assistant. We present the design of a soft, tactile-sensing humanoid upper-body robot and demonstrate whole-body rich-contact manipulation strategies for handling large objects. We demonstrate our hardware design philosophy for outfitting off-the-shelf hard robot arms and other components with soft tactile-sensing modules, including: (i) low-cost, cut-resistant, contact pressure localizing coverings for the arms, (ii) paws based on TRI's Soft-bubble sensors for the end effectors, and (iii) compliant force/geometry sensors for the coarse geometry sensing chest. We leverage the mechanical intelligence and tactile sensing of these modules to develop and demonstrate motion primitives for whole-body gras**. We evaluate the hardware's effectiveness in achieving grasps of varying strengths over a variety of large domestic objects. Our results demonstrate the importance of exploiting softness and tactile sensing in contact-rich manipulation strategies, as well as a path forward for whole-body force-controlled interactions with the world. (The supplemental video is available publicly at https://youtu.be/G8ZYgPRV5LY).
△ Less
Submitted 30 March, 2022; v1 submitted 17 November, 2021;
originally announced November 2021.
-
SEED: Series Elastic End Effectors in 6D for Visuotactile Tool Use
Authors:
H. J. Terry Suh,
Naveen Kuppuswamy,
Tao Pang,
Paul Mitiguy,
Alex Alspach,
Russ Tedrake
Abstract:
We propose the framework of Series Elastic End Effectors in 6D (SEED), which combines a spatially compliant element with visuotactile sensing to grasp and manipulate tools in the wild. Our framework generalizes the benefits of series elasticity to 6-dof, while providing an abstraction of control using visuotactile sensing. We propose an algorithm for relative pose estimation from visuotactile sens…
▽ More
We propose the framework of Series Elastic End Effectors in 6D (SEED), which combines a spatially compliant element with visuotactile sensing to grasp and manipulate tools in the wild. Our framework generalizes the benefits of series elasticity to 6-dof, while providing an abstraction of control using visuotactile sensing. We propose an algorithm for relative pose estimation from visuotactile sensing, and a spatial hybrid force-position controller capable of achieving stable force interaction with the environment. We demonstrate the effectiveness of our framework on tools that require regulation of spatial forces. Video link: https://youtu.be/2-YuIfspDrk
△ Less
Submitted 2 November, 2021;
originally announced November 2021.
-
Variable compliance and geometry regulation of Soft-Bubble grippers with active pressure control
Authors:
Sihah Joonhigh,
Naveen Kuppuswamy,
Andrew Beaulieu,
Alex Alspach,
Russ Tedrake
Abstract:
While compliant grippers have become increasingly commonplace in robot manipulation, finding the right stiffness and geometry for gras** the widest variety of objects remains a key challenge. Adjusting both stiffness and gripper geometry on the fly may provide the versatility needed to manipulate the large range of objects found in domestic environments. We present a system for actively controll…
▽ More
While compliant grippers have become increasingly commonplace in robot manipulation, finding the right stiffness and geometry for gras** the widest variety of objects remains a key challenge. Adjusting both stiffness and gripper geometry on the fly may provide the versatility needed to manipulate the large range of objects found in domestic environments. We present a system for actively controlling the geometry (inflation level) and compliance of Soft-bubble grippers - air filled, highly compliant parallel gripper fingers incorporating visuotactile sensing. The proposed system enables large, controlled changes in gripper finger geometry and grasp stiffness, as well as simple in-hand manipulation. We also demonstrate, despite these changes, the continued viability of advanced perception capabilities such as dense geometry and shear force measurement - we present a straightforward extension of our previously presented approach for measuring shear induced displacements using the internal imaging sensor and taking into account pressure and geometry changes. We quantify the controlled variation of grasp-free geometry, grasp stiffness and contact patch geometry resulting from pressure regulation and we demonstrate new capabilities for the gripper in the home by gras** in constrained spaces, manipulating tools requiring lower and higher stiffness grasps, as well as contact patch modulation.
△ Less
Submitted 15 March, 2021;
originally announced March 2021.
-
Monocular Depth Estimation for Soft Visuotactile Sensors
Authors:
Rares Ambrus,
Vitor Guizilini,
Naveen Kuppuswamy,
Andrew Beaulieu,
Adrien Gaidon,
Alex Alspach
Abstract:
Fluid-filled soft visuotactile sensors such as the Soft-bubbles alleviate key challenges for robust manipulation, as they enable reliable grasps along with the ability to obtain high-resolution sensory feedback on contact geometry and forces. Although they are simple in construction, their utility has been limited due to size constraints introduced by enclosed custom IR/depth imaging sensors to di…
▽ More
Fluid-filled soft visuotactile sensors such as the Soft-bubbles alleviate key challenges for robust manipulation, as they enable reliable grasps along with the ability to obtain high-resolution sensory feedback on contact geometry and forces. Although they are simple in construction, their utility has been limited due to size constraints introduced by enclosed custom IR/depth imaging sensors to directly measure surface deformations. Towards mitigating this limitation, we investigate the application of state-of-the-art monocular depth estimation to infer dense internal (tactile) depth maps directly from the internal single small IR imaging sensor. Through real-world experiments, we show that deep networks typically used for long-range depth estimation (1-100m) can be effectively trained for precise predictions at a much shorter range (1-100mm) inside a mostly textureless deformable fluid-filled sensor. We propose a simple supervised learning process to train an object-agnostic network requiring less than 10 random poses in contact for less than 10 seconds for a small set of diverse objects (mug, wine glass, box, and fingers in our experiments). We show that our approach is sample-efficient, accurate, and generalizes across different objects and sensor configurations unseen at training time. Finally, we discuss the implications of our approach for the design of soft visuotactile sensors and grippers.
△ Less
Submitted 5 January, 2021;
originally announced January 2021.
-
Soft-Bubble grippers for robust and perceptive manipulation
Authors:
Naveen Kuppuswamy,
Alex Alspach,
Avinash Uttamchandani,
Sam Creasey,
Takuya Ikeda,
Russ Tedrake
Abstract:
Manipulation in cluttered environments like homes requires stable grasps, precise placement and robustness against external contact. We present the Soft-Bubble gripper system with a highly compliant grip** surface and dense-geometry visuotactile sensing, capable of multiple kinds of tactile perception. We first present various mechanical design advances and a fabrication technique to deposit cus…
▽ More
Manipulation in cluttered environments like homes requires stable grasps, precise placement and robustness against external contact. We present the Soft-Bubble gripper system with a highly compliant grip** surface and dense-geometry visuotactile sensing, capable of multiple kinds of tactile perception. We first present various mechanical design advances and a fabrication technique to deposit custom patterns to the internal surface of the sensor that enable tracking of shear-induced displacement of the manipuland. The depth maps output by the internal imaging sensor are used in an in-hand proximity pose estimation framework -- the method better captures distances to corners or edges on the manipuland geometry. We also extend our previous work on tactile classification and integrate the system within a robust manipulation pipeline for cluttered home environments. The capabilities of the proposed system are demonstrated through robust execution multiple real-world manipulation tasks. A video of the system in action can be found here: [https://youtu.be/G_wBsbQyBfc].
△ Less
Submitted 28 April, 2020; v1 submitted 7 April, 2020;
originally announced April 2020.
-
A Transition-Aware Method for the Simulation of Compliant Contact with Regularized Friction
Authors:
Alejandro M. Castro,
Ante Qu,
Naveen Kuppuswamy,
Alex Alspach,
Michael Sherman
Abstract:
Multibody simulation with frictional contact has been a challenging subject of research for the past thirty years. Rigid-body assumptions are commonly used to approximate the physics of contact, and together with Coulomb friction, lead to challenging-to-solve nonlinear complementarity problems (NCP). On the other hand, robot grippers often introduce significant compliance. Compliant contact, combi…
▽ More
Multibody simulation with frictional contact has been a challenging subject of research for the past thirty years. Rigid-body assumptions are commonly used to approximate the physics of contact, and together with Coulomb friction, lead to challenging-to-solve nonlinear complementarity problems (NCP). On the other hand, robot grippers often introduce significant compliance. Compliant contact, combined with regularized friction, can be modeled entirely with ODEs, avoiding NCP solves. Unfortunately, regularized friction introduces high-frequency stiff dynamics and even implicit methods struggle with these systems, especially during slip-stick transitions. To improve the performance of implicit integration for these systems we introduce a Transition-Aware Line Search (TALS), which greatly improves the convergence of the Newton-Raphson iterations performed by implicit integrators. We find that TALS works best with semi-implicit integration, but that the explicit treatment of normal compliance can be problematic. To address this, we develop a Transition-Aware Modified Semi-Implicit (TAMSI) integrator that has similar computational cost to semi-implicit methods but implicitly couples compliant contact forces, leading to a more robust method. We evaluate the robustness, accuracy and performance of TAMSI and demonstrate our approach alongside relevant sim-to-real manipulation tasks.
△ Less
Submitted 19 April, 2020; v1 submitted 12 September, 2019;
originally announced September 2019.
-
Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation
Authors:
Alex Alspach,
Kunimatsu Hashimoto,
Naveen Kuppuswamy,
Russ Tedrake
Abstract:
Incorporating effective tactile sensing and mechanical compliance is key towards enabling robust and safe operation of robots in unknown, uncertain and cluttered environments. Towards realizing this goal, we present a lightweight, easy-to-build, highly compliant dense geometry sensor and end effector that comprises an inflated latex membrane with a depth sensor behind it. We present the motivation…
▽ More
Incorporating effective tactile sensing and mechanical compliance is key towards enabling robust and safe operation of robots in unknown, uncertain and cluttered environments. Towards realizing this goal, we present a lightweight, easy-to-build, highly compliant dense geometry sensor and end effector that comprises an inflated latex membrane with a depth sensor behind it. We present the motivations and the hardware design for this Soft-bubble and demonstrate its capabilities through example tasks including tactile-object classification, pose estimation and tracking, and nonprehensile object manipulation. We also present initial experiments to show the importance of high-resolution geometry sensing for tactile tasks and discuss applications in robust manipulation.
△ Less
Submitted 3 April, 2019;
originally announced April 2019.
-
Magneto-mechanical actuation model for fin-based locomotion
Authors:
Juan Pablo Carbajal,
Naveen Kuppuswamy
Abstract:
In this paper, we report the results from the analysis of a numerical model used for the design of a magnetic linear actuator with applications to fin-based locomotion. Most of the current robotic fish generate bending motion using rotary motors which implies at least one mechanical conversion of the motion. We seek a solution that directly bends the fin and, at the same time, is able to exploit t…
▽ More
In this paper, we report the results from the analysis of a numerical model used for the design of a magnetic linear actuator with applications to fin-based locomotion. Most of the current robotic fish generate bending motion using rotary motors which implies at least one mechanical conversion of the motion. We seek a solution that directly bends the fin and, at the same time, is able to exploit the magneto-mechanical properties of the fin material. This strong fin-actuator coupling blends the actuator and the body of the robot, allowing cross optimization of the system's elements.
We study a simplified model of an elastic element, a spring-mass system representing a flexible fin, subjected to nonlinear forcing, emulating magnetic interaction. The dynamics of the system is studied under unforced and periodic forcing conditions. The analysis is focused on the limit cycles present in the system, which allows the periodic bending of the fin and the generation of thrust. The frequency, maximum amplitude and center of the periodic orbits (offset of the bending) depend directly on the stiffness of the fin and the intensity of the forcing; we use this dependency to sketch a simple parameter controller. Although the model is strongly simplified, it provides means to estimate first values of the parameters for this kind of actuator and it is useful to evaluate the feasibility of minimal actuation control of such systems.
△ Less
Submitted 22 August, 2011;
originally announced August 2011.
