Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots
Abstract
We present Habitat 3.0: a simulation platform for studying collaborative human-robot tasks in home environments. Habitat 3.0 offers contributions across three dimensions: (1) Accurate humanoid simulation: addressing challenges in modeling complex deformable bodies and diversity in appearance and motion, all while ensuring high simulation speed. (2) Human-in-the-loop infrastructure: enabling real human interaction with simulated robots via mouse/keyboard or a VR interface, facilitating evaluation of robot policies with human input. (3) Collaborative tasks: studying two collaborative tasks, Social Navigation and Social Rearrangement. Social Navigation investigates a robot's ability to locate and follow humanoid avatars in unseen environments, whereas Social Rearrangement addresses collaboration between a humanoid and robot while rearranging a scene. These contributions allow us to study end-to-end learned and heuristic baselines for human-robot collaboration in-depth, as well as evaluate them with humans in the loop. Our experiments demonstrate that learned robot policies lead to efficient task completion when collaborating with unseen humanoid agents and human partners that might exhibit behaviors that the robot has not seen before. Additionally, we observe emergent behaviors during collaborative task execution, such as the robot yielding space when obstructing a humanoid agent, thereby allowing the effective completion of the task by the humanoid agent. Furthermore, our experiments using the human-in-the-loop tool demonstrate that our automated evaluation with humanoids can provide an indication of the relative ordering of different policies when evaluated with real human collaborators. Habitat 3.0 unlocks interesting new features in simulators for Embodied AI, and we hope it paves the way for a new frontier of embodied human-AI interaction capabilities.
Community
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Human-Robot Gym: Benchmarking Reinforcement Learning in Human-Robot Collaboration (2023)
- Active-Perceptive Motion Generation for Mobile Manipulation (2023)
- ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting (2023)
- Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation (2023)
- Unified Human-Scene Interaction via Prompted Chain-of-Contacts (2023)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Human-Robot Gym: Benchmarking Reinforcement Learning in Human-Robot Collaboration (2023)
- Human-in-the-Loop Task and Motion Planning for Imitation Learning (2023)
- ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting (2023)
- Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation (2023)
- Unified Human-Scene Interaction via Prompted Chain-of-Contacts (2023)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
Introduces Habitat 3.0 simulator: Habitat simulator upgraded with humanoid/avatar simulation, human-in-the-loop (HITL) control and sensing infrastructure (VR), and study on (simulated) collaborative tasks (social navigation: locating humanoids in unseen environments, social rearrangement: interaction between robot and humanoid when rearranging objects in a scene). Human simulation has joints with fast collision testing, skin mesh, parameterized body models, motion and behavior generation policy allowing programmatic control, real-time FPS on CPU (thanks to motion caching) - GPU gives even greater FPS. Human-in-loop control tool has keyboard inputs and VR interface inputs. Other embodies AI simulators: iGibson 2.0 (not as good humanoids), VRKitchen (only egocentric view from robot), Overcooked-AI (simple environments), VirtualHome (no interactions), SEAN (limited scenes for social navigation and slower); work allows Multi-Agent Reinforcement Learning (MARL) in 3D embodied settings. Humanoid appearance (realistic skeletons and meshes) using SMPL-X representation (get identity, joints, meshes, and mesh vertex displacements). Motion and behavior from AMASS (walking) and VPoser (picking). Uses a server-client model for HITL tools. Trains an RL agent on social navigation task (find and follow people) using DD-PPO with humanoid GPS (heading), egocentric arm depth, and binary (human in scene) detector. Social rearrangement task involves assisting a humanoid collaborator in moving objects from spawn location to a goal location; two-layer policy (high and low), with low-level having types oracle (map known to agent) and fully-learned; ZSC (zero-shot-collaboration) pop-eval. Also tests HITL coordination for social rearrangement. Appendix has implementation details for social navigation (DD-PPO policy and reward functions), social rearrangement, and low-level skill training; also has more comparison results, analysis, and more information on robot, scenes (Habitat Synthetic Scenes Dataset HSSD-200); comparison with other simulators, HITL evaluation, and limitations. From Meta (Jitendra Malik, Devendra Singh Chaplot, Dhruv Batra).
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper