-
ALOHA 2: An Enhanced Low-Cost Hardware for Bimanual Teleoperation
Authors:
ALOHA 2 Team,
Jorge Aldaco,
Travis Armstrong,
Robert Baruch,
Jeff Bingham,
Sanky Chan,
Kenneth Draper,
Debidatta Dwibedi,
Chelsea Finn,
Pete Florence,
Spencer Goodrich,
Wayne Gramlich,
Torr Hage,
Alexander Herzog,
Jonathan Hoech,
Thinh Nguyen,
Ian Storz,
Baruch Tabanpour,
Leila Takayama,
Jonathan Tompson,
Ayzaan Wahid,
Ted Wahrburg,
Sichun Xu,
Sergey Yaroshenko,
Kevin Zakka
, et al. (1 additional authors not shown)
Abstract:
Diverse demonstration datasets have powered significant advances in robot learning, but the dexterity and scale of such data can be limited by the hardware cost, the hardware robustness, and the ease of teleoperation. We introduce ALOHA 2, an enhanced version of ALOHA that has greater performance, ergonomics, and robustness compared to the original design. To accelerate research in large-scale bim…
▽ More
Diverse demonstration datasets have powered significant advances in robot learning, but the dexterity and scale of such data can be limited by the hardware cost, the hardware robustness, and the ease of teleoperation. We introduce ALOHA 2, an enhanced version of ALOHA that has greater performance, ergonomics, and robustness compared to the original design. To accelerate research in large-scale bimanual manipulation, we open source all hardware designs of ALOHA 2 with a detailed tutorial, together with a MuJoCo model of ALOHA 2 with system identification. See the project website at aloha-2.github.io.
△ Less
Submitted 7 February, 2024;
originally announced May 2024.
-
RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
Authors:
Pierre Sermanet,
Tianli Ding,
Jeffrey Zhao,
Fei Xia,
Debidatta Dwibedi,
Keerthana Gopalakrishnan,
Christine Chan,
Gabriel Dulac-Arnold,
Sharath Maddineni,
Nikhil J Joshi,
Pete Florence,
Wei Han,
Robert Baruch,
Yao Lu,
Suvir Mirchandani,
Peng Xu,
Pannag Sanketi,
Karol Hausman,
Izhak Shafran,
Brian Ichter,
Yuan Cao
Abstract:
We present a scalable, bottom-up and intrinsically diverse data collection scheme that can be used for high-level reasoning with long and medium horizons and that has 2.2x higher throughput compared to traditional narrow top-down step-by-step collection. We collect realistic data by performing any user requests within the entirety of 3 office buildings and using multiple robot and human embodiment…
▽ More
We present a scalable, bottom-up and intrinsically diverse data collection scheme that can be used for high-level reasoning with long and medium horizons and that has 2.2x higher throughput compared to traditional narrow top-down step-by-step collection. We collect realistic data by performing any user requests within the entirety of 3 office buildings and using multiple robot and human embodiments. With this data, we show that models trained on all embodiments perform better than ones trained on the robot data only, even when evaluated solely on robot episodes. We find that for a fixed collection budget it is beneficial to take advantage of cheaper human collection along with robot collection. We release a large and highly diverse (29,520 unique instructions) dataset dubbed RoboVQA containing 829,502 (video, text) pairs for robotics-focused visual question answering. We also demonstrate how evaluating real robot experiments with an intervention mechanism enables performing tasks to completion, making it deployable with human oversight even if imperfect while also providing a single performance metric. We demonstrate a single video-conditioned model named RoboVQA-VideoCoCa trained on our dataset that is capable of performing a variety of grounded high-level reasoning tasks in broad realistic settings with a cognitive intervention rate 46% lower than the zero-shot state of the art visual language model (VLM) baseline and is able to guide real robots through long-horizon tasks. The performance gap with zero-shot state-of-the-art models indicates that a lot of grounded data remains to be collected for real-world deployment, emphasizing the critical need for scalable data collection approaches. Finally, we show that video VLMs significantly outperform single-image VLMs with an average error rate reduction of 19% across all VQA tasks. Data and videos available at https://robovqa.github.io
△ Less
Submitted 1 November, 2023;
originally announced November 2023.
-
Interactive Language: Talking to Robots in Real Time
Authors:
Corey Lynch,
Ayzaan Wahid,
Jonathan Tompson,
Tianli Ding,
James Betker,
Robert Baruch,
Travis Armstrong,
Pete Florence
Abstract:
We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous wo…
▽ More
We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuo-linguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. "make a smiley face out of blocks". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io.
△ Less
Submitted 12 October, 2022;
originally announced October 2022.
