Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger
Authors:
Arthur Allshire,
Mayank Mittal,
Varun Lodaya,
Viktor Makoviychuk,
Denys Makoviichuk,
Felix Widmaier,
Manuel Wüthrich,
Stefan Bauer,
Ankur Handa,
Animesh Garg
Abstract:
We present a system for learning a challenging dexterous manipulation task involving moving a cube to an arbitrary 6-DoF pose with only 3-fingers trained with NVIDIA's IsaacGym simulator. We show empirical benefits, both in simulation and sim-to-real transfer, of using keypoints as opposed to position+quaternion representations for the object pose in 6-DoF for policy observations and in reward cal…
▽ More
We present a system for learning a challenging dexterous manipulation task involving moving a cube to an arbitrary 6-DoF pose with only 3-fingers trained with NVIDIA's IsaacGym simulator. We show empirical benefits, both in simulation and sim-to-real transfer, of using keypoints as opposed to position+quaternion representations for the object pose in 6-DoF for policy observations and in reward calculation to train a model-free reinforcement learning agent. By utilizing domain randomization strategies along with the keypoint representation of the pose of the manipulated object, we achieve a high success rate of 83% on a remote TriFinger system maintained by the organizers of the Real Robot Challenge. With the aim of assisting further research in learning in-hand manipulation, we make the codebase of our system, along with trained checkpoints that come with billions of steps of experience available, at https://s2r2-ig.github.io
△ Less
Submitted 20 October, 2022; v1 submitted 22 August, 2021;
originally announced August 2021.
