Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots
Authors:
Xavier Puig,
Eric Undersander,
Andrew Szot,
Mikael Dallaire Cote,
Tsung-Yen Yang,
Ruslan Partsey,
Ruta Desai,
Alexander William Clegg,
Michal Hlavac,
So Yeon Min,
Vladimír Vondruš,
Theophile Gervet,
Vincent-Pierre Berges,
John M. Turner,
Oleksandr Maksymets,
Zsolt Kira,
Mrinal Kalakrishnan,
Jitendra Malik,
Devendra Singh Chaplot,
Unnat Jain,
Dhruv Batra,
Akshara Rai,
Roozbeh Mottaghi
Abstract:
We present Habitat 3.0: a simulation platform for studying collaborative human-robot tasks in home environments. Habitat 3.0 offers contributions across three dimensions: (1) Accurate humanoid simulation: addressing challenges in modeling complex deformable bodies and diversity in appearance and motion, all while ensuring high simulation speed. (2) Human-in-the-loop infrastructure: enabling real h…
▽ More
We present Habitat 3.0: a simulation platform for studying collaborative human-robot tasks in home environments. Habitat 3.0 offers contributions across three dimensions: (1) Accurate humanoid simulation: addressing challenges in modeling complex deformable bodies and diversity in appearance and motion, all while ensuring high simulation speed. (2) Human-in-the-loop infrastructure: enabling real human interaction with simulated robots via mouse/keyboard or a VR interface, facilitating evaluation of robot policies with human input. (3) Collaborative tasks: studying two collaborative tasks, Social Navigation and Social Rearrangement. Social Navigation investigates a robot's ability to locate and follow humanoid avatars in unseen environments, whereas Social Rearrangement addresses collaboration between a humanoid and robot while rearranging a scene. These contributions allow us to study end-to-end learned and heuristic baselines for human-robot collaboration in-depth, as well as evaluate them with humans in the loop. Our experiments demonstrate that learned robot policies lead to efficient task completion when collaborating with unseen humanoid agents and human partners that might exhibit behaviors that the robot has not seen before. Additionally, we observe emergent behaviors during collaborative task execution, such as the robot yielding space when obstructing a humanoid agent, thereby allowing the effective completion of the task by the humanoid agent. Furthermore, our experiments using the human-in-the-loop tool demonstrate that our automated evaluation with humanoids can provide an indication of the relative ordering of different policies when evaluated with real human collaborators. Habitat 3.0 unlocks interesting new features in simulators for Embodied AI, and we hope it paves the way for a new frontier of embodied human-AI interaction capabilities.
△ Less
Submitted 19 October, 2023;
originally announced October 2023.
Habitat 2.0: Training Home Assistants to Rearrange their Habitat
Authors:
Andrew Szot,
Alex Clegg,
Eric Undersander,
Erik Wijmans,
Yili Zhao,
John Turner,
Noah Maestre,
Mustafa Mukadam,
Devendra Chaplot,
Oleksandr Maksymets,
Aaron Gokaslan,
Vladimir Vondrus,
Sameer Dharur,
Franziska Meier,
Wojciech Galuba,
Angel Chang,
Zsolt Kira,
Vladlen Koltun,
Jitendra Malik,
Manolis Savva,
Dhruv Batra
Abstract:
We introduce Habitat 2.0 (H2.0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios. We make comprehensive contributions to all levels of the embodied AI stack - data, simulation, and benchmark tasks. Specifically, we present: (i) ReplicaCAD: an artist-authored, annotated, reconfigurable 3D dataset of apartments (matching real spa…
▽ More
We introduce Habitat 2.0 (H2.0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios. We make comprehensive contributions to all levels of the embodied AI stack - data, simulation, and benchmark tasks. Specifically, we present: (i) ReplicaCAD: an artist-authored, annotated, reconfigurable 3D dataset of apartments (matching real spaces) with articulated objects (e.g. cabinets and drawers that can open/close); (ii) H2.0: a high-performance physics-enabled 3D simulator with speeds exceeding 25,000 simulation steps per second (850x real-time) on an 8-GPU node, representing 100x speed-ups over prior work; and, (iii) Home Assistant Benchmark (HAB): a suite of common tasks for assistive robots (tidy the house, prepare groceries, set the table) that test a range of mobile manipulation capabilities. These large-scale engineering contributions allow us to systematically compare deep reinforcement learning (RL) at scale and classical sense-plan-act (SPA) pipelines in long-horizon structured tasks, with an emphasis on generalization to new objects, receptacles, and layouts. We find that (1) flat RL policies struggle on HAB compared to hierarchical ones; (2) a hierarchy with independent skills suffers from 'hand-off problems', and (3) SPA pipelines are more brittle than RL policies.
△ Less
Submitted 1 July, 2022; v1 submitted 28 June, 2021;
originally announced June 2021.
