BEHAVIOR-1K: A Benchmark for Embodied AI with 1,000 Everyday Activities and Realistic Simulation
Abstract: We present BEHAVIOR-1K, a comprehensive simulation benchmark for human-centered robotics.
BEHAVIOR-1K includes two components, guided and motivated by the results of an extensive survey on ‘ what do you want robots to do for you? ’.
The ﬁrst is the deﬁnition of 1,000 everyday activities, grounded inscenes (houses, gardens, restaurants, ofﬁces, etc.) with more than 5,000 objects annotated with rich physical and semantic properties.
The second is O MNI G IBSON , a novel simulation environment that supports these activities via realistic physics simulation and rendering of rigid bodies, deformable bodies, and liquids.
Our experiments indicate that the activities in BEHAVIOR-1K are long-horizon and dependent on complex manipulation skills, both of which remain a challenge for even state-of-the-art robot learning solutions.
To calibrate the simulation-to-reality gap of BEHAVIOR-1K, we provide an initial study on transferring solutions learned with a mobile manipulator in a simulated apartment to its real-world counterpart.
We hope that BEHAVIOR-1K’s human-grounded nature, diversity, and realism make it valuable for embodied AI and robot learning research.
Project website: https://behavior.stanford.edu .
Introduction
Inspired by the progress that benchmarking brought to computer vision [1–11] and natural language processing [12–16], the robotics community has developed several benchmarks in simulation [17–30].
The broader goal of these benchmarks is to fuel the development of general, effective robots that bring major beneﬁts to people’s daily lives – human-centered AI that “serves human needs, goals, and values” [31–34].
Inspiring as they are, the tasks and activities in those benchmarks are designed by researchers; it remains unclear if they are addressing the actual needs of humans.
We observe that a human-centered robotic benchmark should not only be designed for human needs, but also originated from human needs: what everyday activities do humans want robots to do for them?
To this end, we conduct an extensive survey with 1,461 participants (see Sec. 2) to rank a wide
[Figure]
e n c e s c o r e
How much would you benefit if a robot did this for you
[Figure]
range of daily activities based on participants’ desire to delegate these activities to robots.
We also ask layperson annotators to provide deﬁnitions of those activities.
The survey reveals systematicity in what activities people want robots to do, but more importantly, highlights two key factors that we should prioritize when designing robotic benchmarks: diversity in the type of scenes, objects, and activities, and realism of the underlying simulation environments.
The most needed activities indicated by the survey range from ‘wash ﬂoor’ to ‘clean bathtub.’
Clearly, the diversity of these activities is far beyond what real-world robotics challenges may offer [35–42].
Developing simulation environments is a natural alternative: one can train and test robotic agents in many activities with diverse scenes, objects, and conditions efﬁciently and safely.
However, for this paradigm to work, the activities have to be simulated realistically, reproducing accurately the circumstances that a robot may encounter in the real world.
While signiﬁcant progress in realism has been made in speciﬁc domains [43–45], achieving realism for a diverse set of activities remains a tremendous challenge, due to the effort required to provide realistic models and simulation features.
In this work, we present BEHAVIOR-1K , a Benchmark of 1,000 Everyday Household Activities in Virtual, Interactive, and Ecological Environments – the next generation of BEHAVIOR-100 [27].
BEHAVIOR-1K includes two novel components to address the demands for diversity and realism: the diverse BEHAVIOR-1K D ATASET and the realistic O MNI G IBSON simulation environment.
The BEHAVIOR-1K D ATASET is a large-scale dataset comprising 1) a common-sense knowledge base for 1,000 activities with deﬁnitions in predicate logic (initial and goal conditions), as well as the objects involved, their properties, and their state transitions, and 2) high-quality 3D assets including 50 scenes and 5,000+ object models with rich physical and semantic annotations.
All activities in the BEHAVIOR-1K D ATASET are instantiated in a novel simulation environment,
O MNI G IBSON , which we build on top of Nvidia’s Omniverse and PhysX 5 [46] to provide realistic physics simulation and rendering of rigid bodies, deformable bodies, and ﬂuids.
O MNI G IBSON expands beyond Omniverse’s capabilities with a set of extended object states like temperature, toggled, soaked, and dirtiness.
It also includes capabilities to generate valid initial activity conﬁgurations and discriminate valid goal solutions based on activity deﬁnitions.
With all these realistic simulation features, O MNI G IBSON supports the 1,000 diverse activities in the BEHAVIOR-1K D ATASET .
We evaluate state-of-the-art reinforcement learning algorithms [47, 48] in several activities of
BEHAVIOR-1K, both with visuomotor control in the original action space, and with action primitives that leverage sampling-based motion planning [49].
Our analysis indicates that even a single activity in BEHAVIOR-1K is extremely challenging for current AI algorithms, and the baselines can only solve it with a signiﬁcant injection of domain knowledge.
Concretely, the difﬁculties derive in part from the length of BEHAVIOR-1K’s activities and the complexity of the physical manipulation required.
To calibrate the simulation-to-real gap of BEHAVIOR-1K, we provide an initial study on transferring solutions learned with a mobile manipulator in a simulated apartment to its real-world
B E H
A V
I O R - 1 K B E H
A V
I O R - 100 A
I 2 T H O R V i s .
R oo m R ea rr .
T D W T r a n s po r t R ea rr .
T 5 ( H a b it a t ) M a n i
pu
l
a T H
O R
A r m P o i n t N a v
I n t e r ac ti v e G i b s on B e n c h m a r k
V i r t u a l H o m e A L F R E D H a b it a t 2 .
H A B S A P
I E N M a n i S k ill W a t c
[Figure]
- A nd
[Figure]
e l p R F U n i v
[Figure]
r s
[Figure]
R ea rr .
[Figure]
2 ( O C R T O C )
I K E
A F u r n it u r e A ss e m b l y
R L B e n c h M e t a w o r l d R obo s u it e S o f t G y m D ee
p M i nd C on t r o l
S u it e
O p e n A
I G y m H
a b it a t 1 .
G i b s on Mobilemanipulation
Staticmanipulation
Navigation Activitiesfrom humanpreferencesurvey ¸
Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Activities
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
D i v e r
s
it
y
[Table]
¸ ¸
[Table]
¸ ¸ ¸ ¸ ¸ ¸ ¸ Ø ¸ ¸ ¸ Ø ¸ ¸ ¸ ¸ ¸ ¸ ¸ ¸ ¸ ¸ ¸
R ea li s m Continuousextendedstates(temp.
,wetness) ¸ ¸
Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Flexiblematerials ¸
Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø ¸ Ø Ø Ø
Ø Ø ¸ Ø Ø Ø Ø Deformablebodies ¸
Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø ¸ Ø Ø Ø Ø Ø Ø Ø
Ø Ø Ø RealisticFluid ¸ Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø ¸ Ø Ø Ø Ø Ø ¸ Ø Ø Ø Ø Thermaleffects(steam/ﬁre) ¸
Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø Ø ¸ Ø Ø Ø Ø Ø Ø Ø Ø Ø
Ø Realisticactionexecution ¸ ¸ Ø ¸ Ø ¸ ¸ Ø Ø ¸ ¸
Ø ¸ ¸ ¸ ¸ ¸ ¸ ¸ ¸ ¸ ¸ ¸ Benchmarkfocus:Task-Planningand/orControl TP+C TP+C TP TP+C TP+C TP+C C TP TP TP+C C TP TP+C TP+C
C TP+C C C C C C
C
C counterpart.
We hope that the BEHAVIOR-1K benchmark, our survey, and our analysis will serve to support and guide the development of future embodied AI agents and robots.  Creating a Benchmark Grounded in Human Needs: A Survey Study A signiﬁcant amount of robotics research aspires to satisfy human needs, but those needs are typically assumed or speculated. Human-centric development requires direct information about what humans want from autonomous agents [31].
To create a benchmark that reﬂects these needs, we conduct a survey targeting the general U.S. population that asks: what do you want robots to do for you?
The survey sources around 2,000 activities from time-use surveys [50–52], which record how people spend their time, and from WikiHow articles [53].
We conduct the survey on Amazon Mechanical
Turk with a total of 1,461 respondents (demographics in Appendix A.3) and ﬁfty 10-point Likert scale responses per activity.
Survey results are summarized in Fig.
1 (left), in which we rank the activities based on their human preference score.
The full list of ranked activities can be found on our website.
The distribution shows large statistical dispersion (Gini index=0.158): humans want robots to perform a wide range of activities, from cleaning chores to cooking large feasts.
Tedious tasks like “scrubbing the bathroom ﬂoor” score the highest, while recreational activities like game-play score the lowest.
There are around 200 cleaning activities and over 200 cooking activities, among many other categories.
BEHAVIOR-1K activities include the 909 activities with the highest human preference scores and 91 activities from BEHAVIOR-100 [27], altogether the top-ranked 1000 activities.
BEHAVIOR-1K sets itself apart from other embodied AI benchmarks by sourcing from time-use surveys and using survey data to prioritize activities considered most important and useful by humans, and by including a tremendously diverse set of activities.
Related Work: Embodied AI Benchmarks
We provide an extensive comparison between BEHAVIOR-1K, and other embodied AI benchmarks in simulation [17–26] in Table 1.
We include a number of factors that contribute to diversity and realism and observe a signiﬁcant step forward with BEHAVIOR-1K. First, no other benchmark grounds their activity set on the needs of lay people.
Other benchmarks often target a relatively restricted set of activities, and their simulators are realistic only in the relevant aspects for those tasks.
In fact, we often observe a diversity-realism tradeoff.
For instance, instruction-following benchmarks such as VirtualHome [20] and ALFRED [20, 21] are diverse in the number of scenes, objects, and state changes, but offer a limited low-level physical realism.
On the other hand, household rearrangement benchmarks such as Habitat 2.0 HAB [26], TDW Transport [19], and SAPIEN ManiSkill [54, 55] support realistic action execution and accurate physics simulation for rigid bodies, but only include a handful of tasks.
Similarly, SoftGym [45] and RFUniverse [56] have the closest simulation features and hence realism to O MNI G IBSON , but they also lack the task diversity needed to support the development of human-centered general robots. [Figure]
The most similar benchmark to us is the previous generation BEHAVIOR-100 [27].
BEHAVIORbrought forward several beneﬁcial design choices that we inherit in BEHAVIOR-1K such as the activity sources (ATUS [50]), activity deﬁnition logic language, and evaluation metrics.
However, it fell short in the diversity and realism necessary to support a human-centered embodied AI benchmark in simulation, dimensions where BEHAVIOR-1K achieves unmatched levels.
While BEHAVIORcomprises 100 activities selected by researchers, our BEHAVIOR-1K increases diversity by one order of magnitude, to 1,000 activities, that are grounded in human needs thanks to our unique survey.
Furthermore, BEHAVIOR-100 includes only 15 scenes (all houses) and 300+ object categories, while
BEHAVIOR-1K increases to 50 scenes (houses, stores, restaurants, ofﬁces, etc.) and 1,200+ object categories.
In terms of realism, BEHAVIOR-1K extends the simulatable physical states and processes with O MNI G IBSON : ﬂuids, ﬂexible materials, mixing substances, etc.
The realism achieved in rendering by O MNI G IBSON for BEHAVIOR-1K is also signiﬁcantly higher than what was possible in BEHAVIOR-100 and other benchmarks (see Fig. 3).
BEHAVIOR-1K D ATASET
Once activities have been sourced to reﬂect human needs, they need to be concretely deﬁned and instantiated the way they would occur in the real world.
We build the BEHAVIOR-1K D ATASET , which includes a knowledge base of crowdsourced activity deﬁnitions with relevant objects and object states, and a large-scale repository of high-quality, interactive 3D models.
We crowdsource concrete deﬁnitions of activities in the form of BEHAVIOR Domain Deﬁnition
Language (BDDL) [27].
BDDL is based on predicate logic and designed to be accessible for laypeople to describe concrete initial and goal conditions for a given activity.
Unlike geometric, image/video, or experience goal speciﬁcations [17, 18], BDDL deﬁnitions are in terms of objects and object states, allowing annotators to deﬁne at an intuitive semantic level.
The semantic symbols also capture the fact that multiple physical states might be valid initializations and solutions to an activity.
See Listing 1, 2 and 3 in Appendix for example deﬁnitions.
The object and object state spaces that activity deﬁnitions are built upon are annotated to be ecologically plausible.
The object spaces are derived from 5,000 WikiHow articles for the 1,000 activities and mapped to 1,484 WordNet [57] synsets.
Through crowdworkers, students, and GPT-3 [58], we also associate each object with our fully simulatable object states: for example, apple is associated with cooked and sliced , but not toggledOn .
Many object-property pairs are augmented with parameters, e.g. “cooked temperature for apples”, taking advantage of O MNI G IBSON ’s continuous extended states to make activities especially realistic.
Finally, annotators and researchers also create transition rules, e.g. turning tomatoes and salt into sauces, or requiring sandpaper to remove rust.
The result is a knowledge base of tens of thousands of elements underlying 1000 ecologically plausible activity deﬁnitions.
We ensure annotation quality by having ﬁve experienced machine learning annotators [Figure] verify a subset of all types of annotations and receive extremely high approval rates ( > 96.8%).
See
Appendix B for more details about the knowledge base.
The diversity of these activity deﬁnitions requires diverse object and scene models.
On top of the 15 house scenes from BEHAVIOR-100 [27], we acquire 35 fully interactive scenes across diverse scene types, such as gardens, ofﬁces, restaurants, and stores, that are essential for everyday activities.
This is unprecedented compared to other benchmarks (see Table 1).
We also acquire 5,000+ object instances across 1,200+ categories required by the activities, and annotate rich physical (e.g., friction, mass, articulation) and semantic properties (e.g., category) for each object.
Representative scene and object models can be seen in Fig.
More details of 3D models can be found in Appendix D.
O MNI G IBSON : Instantiating BEHAVIOR-1K with Realistic Simulation
BEHAVIOR-100 is implemented in iGibson 2.0 [59]; however, realistic simulation of the diverse activities in BEHAVIOR-1K is beyond the capability of iGibson 2.0.
We present a novel simulation environment, O MNI G IBSON , that provides the necessary functionalities to support and instantiate
BEHAVIOR-1K. O MNI G IBSON is built on top of Nvidia Omniverse and PhysX 5, providing the simulation of not only rigid bodies, but also deformable objects, ﬂuids, and ﬂexible materials (see
Fig. 4), while generating highly realistic ray-traced or path-traced virtual images (see Fig. 3).
These features signiﬁcantly boost the realism of BEHAVIOR-1K compared to other benchmarks.
Similar to BEHAVIOR-100, O MNI G IBSON also simulates additional, non-kinematic extended object states (e.g. temperature, soaked level) based on heuristics (e.g. temperature increases when being next to a heat source that is toggled on).
O MNI G IBSON also implements the functionalities to generate inﬁnite valid physical conﬁgurations that satisfy the activities’ initial conditions as logical predicates (e.g. food is frozen ), and to evaluate their goal conditions (e.g. food is cooked and onTop of a plate , the cloth is folded ) based on the object’s physical states (pose and joint conﬁguration) and extended states.
O MNI G IBSON natively supports randomization during scene initialization, and can sample amongst object models and their poses/states.
The full details of extended object states and logical predicates that O MNI G IBSON supports can be found in Appendix E.1.
Many everyday tasks are difﬁcult to simulate because they require modeling complex physical processes, such as folding a towel or pouring a glass of water.
O MNI G IBSON unlocks them by supporting realistic simulation of ﬂuids, deformable bodies, and cloths (see Fig. 2).
Indeed, without these features, over half of BEHAVIOR-1K activities would not be simulatable, highlighting how crucial these features are for capturing everyday activities.
O MNI G IBSON also captures multiple physical processes that are not natively simulatable by Omniverse, such as baking pies or pureeing vegetables.
Aside from the extended states mentioned above, we also design a modular Transition
Machine , which speciﬁes custom transitions between groups of objects when speciﬁed conditions are met.
For example, a dough placed inside an oven that reaches a certain temperature threshold will turn into a pie.
This further expands O MNI G IBSON ’s capacity to simulate complex, realistic activities that would otherwise be intractable to fully simulate physically.
Experiments: Evaluating Embodied AI Solutions in BEHAVIOR-1K
In our experiments, we aim to answer three questions: How do existing vision-based robot learning algorithms perform in BEHAVIOR-1K, and what assumptions have to be made to improve their success?
What elements of the activities are the most problematic for current AI?
What are the main
Rearrangement Habitat 2.
AI2THOR
TDW SoftGym w a t e r . n .
0b o w l . n .
s a l t .
n .
f a b r i c .
n .
0d i r t .
n .
s o a p .
n .
0s p oo n . n .
0c u p . n .
b a g . n .
t o w e l . n .
0o x t a i l . n .
0s w i v e l _
c
h a
i r .
n .
0q u a i l . n .
0m u s t a r d . n .
0q u i c h e .
n .
Object synset
[Figure]
Capability
[Table]
to-end visuomotor control completely fails to solve any of the activities, whereas RL-Prim. and RL-Prim. Hist. with action primitives are able achieve decent performance.
Memory of observations helps in longer horizon activities (e.g. CollectTrash ).
sources of the sim-real gap in BEHAVIOR-1K/O MNI G IBSON ? Our goal is to indicate promising research directions to improve AI’s performance in BEHAVIOR-1K activities in simulation and, ultimately, in the real world. 6.1 Evaluating BEHAVIOR-1K Solutions in O MNI G IBSON
Experimental Setups.
We selected three paradigmatic activities for our experiments: CollectTrash , where the agent gathers empty bottles and cups, and throws them into a trash bin (rigid body manipulation); StoreDecoration , where the agent stores items into a drawer (articulated object manipulation); and CleanTable , where the agent wipes a dirty table with a soaked piece of cloth (manipulation of ﬂexible materials and ﬂuids).
We evaluate three different baselines based on state-of-the-art reinforcement learning algorithms (RL) [60]:
• RL-VMC, a visuomotor control (from image to low-level joint commands) RL solution based on Soft Actor-Critic (SAC) [48]; • RL-Prim., a RL solution based on PPO [47] that leverages a set of action primitives based on a sampling-based motion planner [61, 62, 49] ( pick , place , push , navigate , dip and wipe ).
The policy outputs a discrete selection of a primitive applied on an object; • RL-Prim.
Hist., a variant of RL-Prim.
that takes in the history observations (3 steps) as additional inputs to help disentangle similar-looking states.
All agents are trained with a sparse task success reward without any reward engineering.
Following the metrics proposed in BEHAVIOR-100 [27], we report the success rate and efﬁciency metrics (distance traveled, time invested, and disarrangement caused) in Table 2 and 3, and the success score
Q in Table A.13 in Appendix.
Grasping is a challenging research topic on its own.
To facilitate our experiments, we adopt an assistive pick primitive that creates a rigid connection between the object and the gripper if grasping is requested when all ﬁngers are in contact with the object, a stricter form of StickyMitten used in prior works [26, 63, 64].
Furthermore, to accelerate training, the action primitives check only the feasibility (e.g., reachability, collisions) of the ﬁnal conﬁguration, e.g. the grasping pose for pick or the desired location for navigate .
If kinematically feasible, the action primitives will directly set the robot state to the ﬁnal conﬁguration, and continue to simulate from then on.
We include an ablation [Table] [Table] analysis of the effect of these assumptions and simpliﬁcations in our evaluation (see Table 4).
Further details about our training and evaluation setup can be found in Appendix F
Results: Task Completion.
Table 2 contains task success rates across our baseline methods.
The extreme long-horizon in our activities causes the visuor-motor control (RL-VMC) policy to fail in all three activities, potentially due to problems such as credit assignment [65], deep exploration [66, 67], and vanishing gradients [68] as reported by prior works.
Our baselines with time-extended action primitives (RL-Prim.
and RL-Prim.
Hist.) obtain better success, achieving over 40% success rates across all three activities.
We observe that longer-horizon activities are more challenging: while CleanTable can be accomplished by executing the optimal sequence of 6 primitive steps,
CollectTrash requires at least 16 .
This supports the idea that some form of action-space abstraction must be necessary to solve long-horizon activities of BEHAVIOR-1K, as others reported [27, 26, 69].
When analyzing the role of memory, we observe a sizable performance gain from RL-Prim.
to
RL-Prim.
Hist., especially in long-horizon activities with aliased observations such as CollectTrash .
In this task, when the robot is looking at the trash bin, it needs additional information to know what location has been cleaned already in order to proceed to other locations.
Our results indicate that memory will play a critical role for embodied AI in long-horizon BEHAVIOR-1K activities.
Results: Efﬁciency.
In addition to success, efﬁciency is also critical in the evaluation of embodied
AI: a successful policy in simulation may be infeasible in the real world if it takes a long time or wastes too much energy.
In Table 3, we report the results with three efﬁciency metrics proposed by Srivastava et al.
We observe that the use of memory (RL-Prim.
Hist.) improves efﬁciency across all metrics: distance navigated (Dist.
Nav.), simulated time (Sim. Time), and kinematic object disarrangement (Kin.
Dis.), i.e., amount of object displacement due to robot motion.
We also evaluate to what extent the simpliﬁcations we introduce in physics and actuation (grasping, motion execution) during training impact the performance of RL-Prim.
during evaluation when these simpliﬁcations are removed.
We report the results in Table 4.
We observe a radical performance drop after enabling fully physics-based grasping during evaluation.
Grasping is thus a critical component of any embodied AI task and researchers should be careful when simplifying its execution during training.
While O MNI G IBSON supports fully physics-based grasping, designing a pick action primitive for arbitrary objects that leverages fully physics-based grasping is by itself an open research problem that we leave for future work.
In contrast, there is much less performance drop after enabling full trajectory motion execution during evaluation.
This result supports our hypothesis that it is reasonable to accelerate the training process by assuming that motion planning is likely to provide viable paths in free space during evaluation.
6.2 Evaluating BEHAVIOR-1K Solutions on a Real Robot
We performed a series of experiments with a real robot to answer the question: what are the main sources of discrepancy between our realistic simulation and the real world?
To that end, we used a real-world counterpart of the simulated scene of a mockup apartment for the CollectTrash activity.
We scanned the apartment and converted it into a virtual, interactive scene.
We use a real bi-manual mobile manipulator Tiago, and leverage the RGB-D images from its onboard sensors and a YOLOv object detector [70, 71] to localize the objects in 3D space for manipulation.
For navigation, the robot localizes with a particle ﬁlter [72] based on two LiDAR sensors and a map of the apartment.
The action primitives are implemented with the same sampling-based motion planning algorithm as in simulation [62, 49] with additional tuning.
We evaluate two strategies for selecting action primitives in the real world: an optimal policy based on human input, and a vision-based policy (RL-Prim.)
trained in O MNI G IBSON .
To facilitate sim-to-real transfer, during training we additionally applied image-based data augmentation to the observations based on a prior work [73] (see Appendix G for S 4% 41% 11% 44% R-TP 19% 38% 6% 38% R-OP
Motion planning
Grasping
Placing Obj. det. Policy further details). With the optimal policy, we evaluate the gap in actuation between the simulated and the real robot; with the learned policy, we also evaluate the gap in visual perception.
We achieve different success rates in simulation (50 runs, ∼ 40% success) and in real world with optimal (27 runs, ∼ 22%) and trained policies (26 runs, 0%), hinting on a sim-real gap that we analyze below.
The failure cases are depicted in Fig.
5 (right).
We observe that the majority of failures in simulation are due to the visual policy (perception), while others are caused by stochasticity in the place primitive and the sampling-based motion planner.
The reason why none of the failures are due to grasping is because in simulation we evaluate with the assistive pick primitive.
Grasping is fairly difﬁcult in the real world, contributing to around 40% of the failures for both the trained and the optimal policy.
For the learned policy, 44% of the errors come from the visual policy selecting the wrong action primitive due to the differences between the simulated and the real images.
The visual discrepancy results from unmodeled effects such as the real camera’s poor dynamic range (see Fig.
5, left and middle) and imperfect object modeling (e.g. the exact wooden texture and the surface reﬂectivity of the tables), which can be alleviated by more targeted domain randomization.
Interestingly, several manipulation failures on the real robot are caused by unfavorable robot base placement resulting from navigation inaccuracies in the previous timestep.
This compounding source of error is not present in simulation because we assume perfect localization and execution.
We believe this analysis provides relevant information about the main sources and severity of the sim-real gap in
BEHAVIOR-1K in O MNI G IBSON , and provide insights for future research avenues.
Our plan is to use some of these insights to create novel sim-real solutions that make progress on BEHAVIOR-1K.
Discussion and Limitations
We presented BEHAVIOR-1K, a benchmark for embodied AI and robotics research with realistic simulation of 1000 diverse activities grounded in human needs.
BEHAVIOR-1K comprises two elements: BEHAVIOR-1K D ATASET , a semantic knowledge base of everyday activities, and a largescale 3D model library; O MNI G IBSON , a simulation environment that provides realistic rendering and physics for rigid/deformable objects, ﬂexible materials and ﬂuids.
In our evaluation, we observed that
BEHAVIOR-1K is an extremely challenging benchmark: solving these 1,000 activities autonomously is beyond the capability of current state-of-the-art AI algorithms.
We studied and attempted to solve a handful of the activities with action primitives in order to gain insights into the most challenging components, providing a starting point for other researchers to work on our benchmark.
Similarly, we explored the sources of the sim-real gap by creating a digital twin of a real-world mock apartment, and by performing rigorous evaluation and analysis of policies in both simulation and the real world with a simulated and real mobile manipulator.
Limitations: We inherit several limitations from our underlying physics and rendering engine,
Nvidia’s Omniverse.
In O MNI G IBSON , we trade off rendering speed for visual realism (ray-traced), reaching around 60 fps for a house scene of around 60 objects (v.s. around 100 fps in iGibson 2.0 [59]).
We are actively working on performance optimization.
Another limitation is that we only include activities that do not require interactions with humans.
Realistic simulation of humans (behavior, motion, appearance) is extremely challenging and an open research area.
We plan to include simulated humans when the technology becomes more mature.
Finally, there is still room for improvement in O MNI G IBSON to further facilitate sim2real transfer, such as incorporating noise models of perception and actuation.
Acknowledgments
This work was done in part when Chengshu Li, Josiah Wong, and Michael Lingelbach were interns at Nvidia Research.
The work is in part supported by the Stanford Institute for Human-Centered
AI (HAI), the Toyota Research Institute (TRI), NSF CCRI #2120095, NSF RI #2211258, NSF
NRI #2024247, ONR MURI N00014-22-1-2740, ONR MURI N00014-21-1-2801, Amazon, Bosch,
Salesforce, and Samsung.
Ruohan Zhang is partially supported by Wu Tsai Human Performance
Alliance Fellowship.
Sanjana Srivastava is partially supported National Science Foundation Graduate
Research Fellowship Program (NSF GRFP).