iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Recent research in embodied AI has been boosted by the use of simulation environments to develop and train robot learning approaches.
However, theuse of simulation has skewed the attention to tasks that only require what roboticssimulators can simulate: motion and physical contact.
We present iGibson 2.0, anopen-source simulation environment that supports the simulation of a more diverseset of household tasks through three key innovations.
First, iGibson 2.0 supportsobject states, including temperature, wetness level, cleanliness level, and toggledand sliced states, necessary to cover a wider range of tasks.
Second, iGibson 2.0implements a set of predicate logic functions that map the simulator states to logicstates like Cooked or Soaked .
Additionally, given a logic state, iGibson 2.0 cansample valid physical states that satisfy it.
This functionality can generate poten- tially inﬁnite instances of tasks with minimal effort from the users.
The samplingmechanism allows our scenes to be more densely populated with small objects insemantically meaningful locations.
Third, iGibson 2.0 includes a virtual reality(VR) interface to immerse humans in its scenes to collect demonstrations.
As aresult, we can collect demonstrations from humans on these new types of tasks, anduse them for imitation learning.
We evaluate the new capabilities of iGibson 2.0 toenable robot learning of novel tasks, in the hope of demonstrating the potential ofthis new simulator to support new research in embodied AI.
iGibson 2.0 and itsnew dataset will be publicly available after the anonymous reviewing process.
Introduction
Recent years, we have seen the emergence of many simulation environments for robotics andembodied AI research [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].
The main function of these simulators is tocompute the motion resulting from the physical contact-interaction between (rigid) bodies, as this isthe main process that allows robots to navigate and manipulate the environment.
This kinodynamicsimulation is sufﬁcient for pick-and-place and rearrangement tasks [11, 12, 13, 14]; however, as theﬁeld advances, researchers are taking on more diverse and complex tasks that cannot be performedin these simulators, e.g., household activities that involve changing the temperature of objects, theirdirtiness and wetness levels.
There is a need for new simulation environments that can maintain andupdate new types of object states to broaden the diversity of activities that can be studied.
We present iGibson 2.0, an open-source extension of the kinodynamic simulator iGibson with severalnovel functionalities.
First and foremost, iGibson 2.0 maintains and updates new extended physicalstates resulting from the approximation of additional physical processes.
These states include notonly kinodynamics (pose, motion, forces), but also object’s temperature, wetness level, cleanlinesslevel, toggled and sliced state (functional states).
These states have a direct effect on the appearanceof the objects, captured by the high-quality virtual sensor signals rendered by the simulator.
Second, iGibson 2.0 provides a set of logical predicates that can be evaluated with a single object (e.g.
Cooked ) or a pair of objects (e.g. InsideOf ).
These logical predicates discriminate the continuousphysical state maintained by the simulator into semantically meaningful logical states (e.g. Cooked is
True if the temperature is above a certain threshold).
Complementary to the discriminative functions,iGibson 2.0 implements generative functions that sample valid simulated physical states based onlogical states.
Scene initialization can then be described as a set of logical states that the simulator can translate into valid physical instances.
This enables faster prototyping and speciﬁcation of scenes iniGibson 2.0, facilitating the training of embodied AI agents in diverse instances of the same tasks.
Wedemonstrate the potential of this generative functionality with a new dataset of home scenes denselypopulated with small objects.
We generate this new dataset by applying a set of hand-designedsemantic-logic rules to the original scenes of iGibson 1.0.
[Figure]
Third, to facilitate the developmentof new embodied AI solutions to newtasks in these new scenes, iGibson 2.0includes a new virtual reality interface (VR) compatible with the twomain commercially available VR systems.
All states are logged during execution and can be replayed deterministically in the simulator, enabling thegeneration a posteriori of additionalvirtual sensor signals or visualizationsof the interactions and the development of imitation learning solutions.
We evaluate the new functionalitiesof iGibson 2.0 on six novel tasks forembodied AI agents, and apply stateof-the-art robot learning algorithmsto solve them.
These tasks were notpossible before in iGibson or in alternative simulation environments.
Additionally, we evaluate the use of thenew iGibson 2.0 VR interface to collect human demonstrations to train an imitation learning policy for bimanual operations.
Whilethe previous version of iGibson and other simulators provide interfaces to control an agent with akeyboard and/or a mouse, these interfaces are insufﬁcient for bimanual manipulation.
In summary, iGibson 2.0 presents the following contributions:
• A set of new physical properties, e g. temperature, wetness and cleanliness level, maintained andupdated by the simulator; and a set of unary and binary logical predicates that map simulated statesto a logical state that have a direct connection to semantics and language,• A set of generative functions associated with the logical predicates to sample valid simulated statesfrom a given logical state, and a new rule-based mechanism exploiting these functions to populate the
iGibson scenes with small objects placed at semantically meaningful locations to increase realism,• A novel virtual reality interface that allows humans to collect demonstrations for robot learning,
We hope that iGibson 2.0 will open new avenues of research and development in embodied AI,
enabling solutions to household activities that have been under-explored before.
Related Work
Simulation environments with (mostly) kinodynamic simulation: In the last years, the roboticsand AI communities have presented several impressive simulation environments and benchmarks:iGibson [3, 1], Habitat AI [5], AI2Thor (and variants) [15, 6], ThreeDWorld [7], Sapien [4], Robosuite [9], VirtualHome [16], RLBench [8], MetaWorld [10], and more.
They are based in physicsengines such as (py)bullet [17], MuJoCo [18], and Nvidia Physx [19, 20], combined with renderingcapabilities and usually enriched with a dataset of objects and/or scenes to use to develop embodied
AI solutions.
While these simulators have fueled research with new possibilities for training, testingand developing robotic solutions, they have skewed, with few exceptions, the exploration towardsactivities related to what they can simulate accurately: changes in kinematic states of (rigid or ﬂexible)objects, i.e. Rearrangement tasks [11].
However, many everyday activities require the simulationof other physical states that can be modiﬁed by the agents, like the temperature of objects and theirlevel of wetness or cleanliness.
Recent simulators have attempted realistic simulation of ﬂuids andﬂexible materials [19, 21, 22, 23] or extended rigid body simulators to approximate the dynamics of soft materials [17, 24].
Compared with these simulators, iGibson 2.0 provides a simple but effectivemechanism to simulate the temperature of objects that change based on proximity to heat sources.
Italso simulates ﬂuids through a system of droplets that can be absorbed by objects and change theirwetness level.
Despite simpler than the accurate simulation of heat transfer, ﬂuids and soft materialsused by a few other simulators, iGibson 2.0 object-centric solution leads to realistic robotic behaviorand motion in tasks involving changing temperature, handling liquids, or soaking objects.
Simulation environments with object-centric representation: In robotics, some simulators haveadopted an object-centric representation with extended physical states, e.g. AI2Thor [15] and
VirtualHome [16].
Both are based on Unity [25] and share a common set of functionalities.
Actionsin these simulators are predeﬁned, discrete and symbolic, and characterized by preconditions (“whatconditions need to be fulﬁlled for this action to be executable?”) and postconditions (“what conditionswill change after the execution of this action?”), similar to actions in the planning domain deﬁnitionlanguage (PDDL [26]) or STRIPS [27] but with the additional link to visual rendering of thepredeﬁned action execution and outcome.
In AI2Thor and VirtualHome, some of the actions changethe temperature of an object between two or more discrete values pre- and post-execution of an action,e.g. raw and cooked .
This is fundamentally different to our approach in iGibson 2.0: instead ofmaintaining only a symbolic state ( raw / cooked ), we provide a simple simulation of the underlyingphysical process (e.g. heat transfer) leading to continuously varying values of temperature and otherextended states.
These states are then mapped into a symbolic representation through predicates(see Sec. 4).
This provides a new level of detail in the execution of actions, where the agent can andshould control the speciﬁc value of the object’s extended states (temperature, wetness, cleanliness) toachieve a task, leading to more complex activities and more realistic execution.
In the appendix, weinclude a detailed comparison between iGibson 2.0 and other simulation environments in Table A.7.
Simulation environments with virtual reality interfaces: Researchers have used virtual realityinterfaces before to develop robotic solutions with real robots [28, 29, 30].
VR has also been used insimulation environments to collect demos.
VRKitchen [31] collected demos for ﬁve cooking tasks insimulation.
While realistic looking, activities in VRKitchen are performed with primitive actionssimilar to the pre-condition/post-condition system of AI2Thor and VirtualHome, falling short ofrealistic motion.
More physically realistic are the VR interactions with UnrealROX/RobotriX
[32, 33]and ThreeDWorld [7], based on Unreal [34] and Nvidia Physx [20].
Our interface also enablesrealistic manipulation of objects, with additional features such as gaze tracking and assistive graspingto bridge the differences between simulation and the real-world.
Extended Physical States for Simulation of Everyday Household Tasks
To perform household tasks, an agent needs to change objects’ states beyond their poses.
iGibson2.0 extends objects with ﬁve additional states: temperature, T , wetness level, w , cleanliness level(dustiness level, d , or stain level, s ), toggled state, TS , and sliced state, SS .
While some of these statescould be different for different parts of an object, in iGibson 2.0 we simplify their simulation andadopt an object-centric representation : the simulator maintains a single value of each extended statefor every simulated object (rigid, ﬂexible, or articulated).
This simpliﬁcation is sufﬁcient to simulaterealistically household tasks such as cooking or cleaning.
We assume that the extended propertiesare latent : agents are not able to observe them directly.
Therefore, iGibson 2.0 implements amechanism to change objects’ appearance based on their latent extended states (see Fig. 2), so thatvisually-guided agents can infer the latent states from sensor signals.
We further impose in iGibson 2.0 that every simulated object should be an instance of an existingobject category in WordNet [35].
This semantic structure allows us to associate characteristics to allinstances of the same category [36, 37].
For example, we further simplify the simulation of extendedstates by annotating what extended states each category need.
Not all object categories need all ﬁveextended states (e.g., the temperature is not necessary/relevant for non-food categories for most tasksof interest).
The extended states required by each object category are determined by a crowdsourcedannotation procedure in the WordNet hierarchy.
In the following, we explain the details of the ﬁve extended states and the way they are updated iniGibson 2.0.
For a full list of object states (kinematics and extended), see Table A.4.
Temperature: To update temperature, iGibson 2.0 needs to approximate the dynamics of heattransfer from heat sources (hot) or sinks (cold).
To that end, we annotate object categories in the
WordNet hierarchy as heat sources or heat sinks .
Heat sources elevate the objects’ temperature over room temperature (23  ) towards the source’s heating temperature, also annotated per category;similarly, heat sinks decrease the temperature under room temperature towards the sink’s coolingtemperature.
The rate the objects change their temperature towards the temperature of the source/sinkis a parameter annotated per category with units  C / s .
We consider two types of heat source/sink:source/sinks that change the temperature of objects if they are proximal enough (e.g., a stove) andsource/sinks that change the temperature of objects if they are inside of them (e.g., a fridge or anoven).
Additionally, some of the sources/sinks need to be toggled on (see Functional State below)before they can change the temperature of other objects (e.g., a microwave).
For each object that canchange temperature, the simulator ﬁrst evaluates if it fulﬁlls the conditions to be heated or cooled by aheat source/sink.
Assuming that an object with temperature T o fulﬁlls the conditions of a source/sinkwith heating/cooling temperature T and changing rate r , the temperature of the object at each step isupdated by T t +1 o =
T to (1 +  sim ⇥ r ⇥ ( T  T to )) , where  sim is the simulated time.iGibson 2.0 maintains also a historical value of the maximum temperature that each object hasreached in the past, T max o = max T to for t 2 [0 , . . .
, t now ] .
This value dictates the appearance of anobject: if the object reached cooking or burning temperature in the past, it will look cooked or burned,even if its current temperature is low.
Fig.
2a depicts the temperature system in action.
Wetness Level: Similar to temperature, iGibson 2.0 maintains the level of wetness for each objectthat can get soaked .
This level corresponds to the number of droplets that have been absorbed bythe object.
In iGibson 2.0, the system of droplets approximates liquid/ﬂuid simulation.
Speciﬁcally,droplets are small particles of liquid that are created in droplet sources (e.g. faucets), destroyed bydroplet sinks (e.g. sinks), and absorbed by soakable objects (e.g. towels).
They can also be containedin receptacles (e.g. cups) and poured later, leading to realistic behavior for the simulation of severalhousehold activities involving liquids, illustrated in Fig. 2b.
Cleanliness – Dustiness and Stain Level:
A common task for robots in homes and ofﬁces is to cleandirt.
This dirt commonly appears in the form of dust or stains.
In iGibson 2.0, the main differencebetween dust and stains is the way they get cleaned: while dust can be cleaned with a dry cleaningtool like a cleaning cloth, stains can only be cleaned with a soaked cleaning tool like a scrubber.
Toclean a particle of dirt (dust or stain), the right part of a cleaning tool should get in physical contactwith the particle.
Once a dirt particle is cleaned, it disappears from objects’ surface.
In iGibson 2.0, objects can be initialized with visible dust or stain particles on its surface.
The numberof particles at initialization corresponds to a 100% level of dustiness, d , or stains, s , as we assumethat dust/stain particles cannot be generated after initialization.
As particles are cleaned, the level
[Figure]
decreases proportionally to the number of particles removed, reaching a level of 0% dustiness, d , or stains when the object is completely clean (no particles left).
This extended state allows simulatingmultiple cleaning tasks in our simulator: the agent needs to exhibit a behavior (motion, use of tools)similar to the one necessary in the real world.
Fig.
3a depicts an example of the cleanliness levelsimulation in iGibson 2.0, for both dust and stain particles.
Toggled State:
Some object categories in iGibson 2.0 can be toggled on and off.
iGibson 2.0maintains and updates an internal binary functional state for those objects.
The functional state canaffect the appearance of an object, but also activate/deactivate other processes, e.g. heating foodinside a microwave requires the microwave to be toggled on.
To toggle the object, a certain areaneeds to be touched.
For object models of categories that can be toggled on/off, we annotate a
TogglingLink , an additional virtual ﬁxed link that needs to be touched by the agent to change thetoggled state.
Fig.
3b depicts an example of an object that can change its toggled state, an oven.
Sliced State:
Many cooking activities require the agent to slice objects, e.g. food items.
Slicing ischallenging in simulation environments where objects are assumed to have a ﬁxed (rigid or ﬂexible)3D structure of vertices and faces.
To approximate the effect of slicing, iGibson 2.0 maintains andupdates a sliced state in instances of object categories that are annotated as sliceable .
When thesliced state transitions to True, the simulator replaces the whole object with two halves.
The twohalves will be placed at the same location and inherit the extended states from the whole object(e.g. temperature).
The transition is not reversible: the object will remain sliced for all upcomingsimulated time steps.
Objects can only be sliced into two halves, with no further division.
The slicedstate changes when the object is contacted with enough force (over a slicing force threshold for theobject) by a slicing tool, e.g. a knife.
Objects of these categories are annotated as SlicingTool .
Ifan object is a slicing tool, it will undergo a second annotation process to obtain a new virtual ﬁxedlink that acts as SlicingLink , the part of the slicing tool that can slice an object, e.g., the sharp edgeof a knife.
Fig.
3c depicts an example of a peach being sliced by a knife.
For more information aboutupdate rules for all extended physical states, please refer to see Sec. A.2.
Logical Representation of Physical States
The new extended object states from iGibson 2.0 are sufﬁcient to simulate a new set of householdactivities in indoor environments.
However, there is a semantic gap between the extended states(e.g.
temperature or wetness level) and the natural description of activities in a household setup (e.g.cooking apples).
To bridge this gap, we deﬁne a set of functions that map the extended object statesto logical states for single objects and pairs of objects.
The logical states are semantically groundedon common natural language representing properties such as cooked or dusty .
The list of logical predicates covers kinematic states between pair of objects ( InsideOf , OnTopOf ,
NextTo , InContactWith , Under , OnFloor ), states related to the internal degrees of freedom of articulated objects ( Open ), states based on the object temperature ( Cooked , Burnt , Frozen ), wetnessand cleanliness level ( Soaked , Dusty , Stained ), and functional state ( ToggledOn , Sliced ).
Acomplete list of the logic predicates with detailed explanation is included in Table A.1.
They allowiGibson 2.0 to map a physical simulated state into an corresponding logical state.
4.1 Generative System based on Logical Predicates
Logical predicates map multiple physically simulated states to the same logical state, e.g., all relativeposes between to objects that correspond to being onTop .
In addition to this discriminative role, weinclude functionalities in iGibson 2.0 to use logical predicates in a generative manner, to describeinitial states symbolically that can be used to initialize the simulator.
iGibson 2.0 includes a samplingmechanism to create valid instances of tasks described with logical predicates.
This mechanismfacilitates the creation of multiple semantically meaningful initial states, without the laborious processof manually annotating the initial distributions per scene.
The process of sampling valid object states is different depending on the nature of the logical predicate.
For predicates based on objects’ extended states such as Cooked , Frozen or ToggledOn , we justsample values of the extended states that satisfy the predicate’s requirements, e.g., a temperaturebelow the annotated freezing point of an object to fullﬁl the predicate Frozen .
Particles for Dusty and
Stained are sampled on the surface of an object following a pseudo-random procedure.
Generatinginitial states to fulﬁll kinematic predicates such as OnTopOf or Inside is a more complex procedureas the underlying physical state (the object pose) must lead to a stationary state (e.g., not falling)
thatdoes not cause penetration between objects.
Each kinematic predicate is implemented differently,combining mechanisms that include ray-casting and analytical methods to verify the validity ofsampled poses.
For example, to sample a state that satisﬁes Inside(A, B) , we implement aprocedure that generates 6D poses inside of object B and we evaluate that 1) a bounding box of thethe size of object
A does not penetrate object B and rays cast from A intersect object B from evaluatedby casting rays from the the internal box.
For more details on the sampling mechanism, see Sec.
A.2.
4.2 iGibson 2.0 Scenes with Realistic Object Distribution Created by Generative System
One common issue that limits the realismof indoor scenes in simulation is that theyare less densely populated than those in thereal world.
Creating highly populated simulated houses is usually a laborious processthat requires manually selecting and placing models of small objects in different locations.
Thanks to the generative system iniGibson 2.0, this process can be extremelysimpliﬁed.
The users only need to specifya list of logical predicates that represent arealistic distribution of objects in a house.
We provide as part of iGibson 2.0 a set of semantic rules to generate more populated scenes, anda new version of the iGibson original 15 fully interactive scenes, populated with additional smallobjects as a result of the application of the rules.
Given an indoor scene with multiple typesof rooms ( kitchen , bathroom , ...) that are populated with furniture containers and appliances( fridge , cabinet , ...), the semantic rules deﬁne the probabilities for object instances of diversecategories to be sampled in certain container type in a given room type, e.g. p ( InsideOf(Beer,
Fridge) ^ InsideOf(Fridge, Kitchen) ) .
To generate the more densely populated versions ofthe 15 scenes, we ﬁrst collect a large number of small 3D object models created by artists and weannotate them with semantic categories (e.g. cereal , apple , bowl ) and realistic dimensions.
Then,we apply in a random sequence the logic predicates using the generative system explained in Sec.
4.1,increasing the number of objects in the scene by over 100.
The result is depicted in Fig.
Virtual Reality Interface
iGibson 2.0’s new functionalities enable modeling new household activities and generating multipleinstances in more densely populated scenes.
To facilitate research in these new, complex tasks,iGibson 2.0 includes a novel virtual reality (VR) interface compatible with major commercially [Figure] available VR headset through OpenVR [38].
One of the goals is to allow researchers to collect humandemonstrations, and use them to develop new solutions via imitation (see Sec.
6).iGibson 2.0’s VR interface creates an immersive experience: humans embody an avatar in the samescene and for the same task as the AI agents.
The virtual reality avatar (see Fig. 1) is composed of amain body, two hands and a head.
The human controls the motion of the head and the two hands viathe VR headset and hand controllers with an optional, additional tracker for control of the main body.
Humans receive stereo images as generated from the point of view of the head of the virtual avatar, atat least 30 fps (up to 90 fps) using the PBR rendering functionalities of iGibson [1].
Grasping in VR: Grasping in the real-world, while natural to adult humans, is a complex experiencethat proves difﬁcult to reproduce with virtual reality controllers.
Our empirical observations revealedthat while using solely physical grasping, user dexterity was signiﬁcantly impaired relative to thereal-world resulting in unnatural behavior when manipulating in VR.
To provide a more naturalgrasping experience, we implement an assistive grasp (AG) mechanism that enables an additionalconstraint between the palm and a target object after the user passes a grasp threshold (50% actuation)and provided the object is in contact with the hand, between the ﬁngers and the palm.
This facilitatesgrasping of small objects, and prevents object slippage.
To not render grasping artiﬁcially trivial, the
AG connection can break if the constraint is violated beyond a set threshold, such as while liftingheavy objects or during intense acceleration, encouraging natural task execution that leverages carefulmotions and bimanual manipulation.
Please refer to Sec. A.1 for additional details about AG.
Navigating in VR: Navigation of the avatar is controlled by the locomotion of the human.
However,the VR space is much smaller than the typical size of iGibson 2.0 scenes.
To navigate between rooms,we conﬁgured a touchpad in the hand controller that humans can use to translate the avatar.
Evaluation
In our evaluation, we test the new functionalities of iGibson 2.0 explained above and that sets it apartfrom other existing environments.
First, we create a set of tasks that showcase the new extendedstates (see Fig. 5a) as their modiﬁcation is required to achieve the tasks.
In our experiments, wemake use of our discriminative and generative logical engine to detect task completion and createmultiple instances of each task for training.
Then, we use our novel VR interface to collect humandemonstrations to train an imitation learning policy for a bimanual task.
Experimental Setup: To evaluate and demonstrate the new capabilities of iGibson 2.0, we createthe following six novel tasks for embodied AI agents: 1) Grasping Book: the agent has to grasp abook OnTopOf a table and lift it.
This task demonstrates the kinematic logical states; 2) Soaking Towel: the agent has to soak a cleaning tool (a towel) with water droplets from a sink.
This taskdemonstrates the liquid/droplets system.
3) Cleaning Stained Shelf: the agent has to clean a stainedshelf using a cleaning tool.
This task showcases the capabilities to update the cleanliness level ofan object.
4) Cooking Meat: the agent has to cook a piece of meat by placing it on a heat source(a burning stove) and waiting for enough time for the temperature to rise.
This task showcases thecapabilities to update the temperature of an object.
5) Slicing Fruit: the agent has to slice a fruitby exerting enough force with the SlicingLink of a slicing tool (a knife).
This task showcasesthe capabilities to simulate sliceable objects and to model the interaction between cutting tools and sliceable objects.
6) Bimanual Pick and Place: The agent has to pick up a heavy object (a cauldron)and place it OnTopOf a table.
Manipulating this object requires bimanual interaction as its massis 50 kg and each hand can only exert at most 300 N .
Demonstrating bimanual manipulation isnatural and easy with our novel VR interface, which is not the case with previous interfaces such askeyboard [1].
We conduct two sets of experiments to evaluate the current robot learning algorithms on these tasks. First, we train agents with a state-of-the-art reinforcement learning (RL) algorithm, Soft-Actor Critic(SAC [39]). For embodiment, we use a bimanual humanoid robot (the one used in VR) and a Fetchrobot.
The agent receives RGB-D images from its onboard sensors and proprioception information asobservation and outputs the desired linear and angular velocities of the right hand (assuming the restof the agent is stationary).
We adopt the “sticky mitten” simpliﬁcation [11] for grasping that createsa ﬁxed constraint between the hand and the object when they get into contact.
We also conductexperiments that remove such simpliﬁcation for the Fetch robot, in which case the action spaceincludes one additional DoF for grasping.
Second, we train agents with a standard imitation learning(IL) algorithm, behavioral cloning [40] on Bimanual Pick and Place . We collected 30 demonstrations,more than 6500 frames in total ( ⇠ 215 s ).
The agent receives ground-truth states of the objects andproprioception information as observation, and outputs the desired linear and angular velocities forboth hands.
Additional information about the experimental setup can be found in Sec.
A.3.
RL experiments: With the simpliﬁed grasping mechanism, the agents trained with SAC for both thebimanual humanoid and the Fetch embodiment achieve 100% success rate for Grasping Book , Soaking
Towel , Cleaning Stained Shelf , Cooking Meat tasks.
For Slicing Fruit , the agents achieve only 15%and 0% for bimanual humanoid and Fetch robot respectively, due to the increased accuracy necessaryto align the knife blade with the fruit.
The agent using bimanual humanoid achieves 0% success inthe Bimanual Pick and Place task because of the difﬁculties of controlling and coordinating bothhands. Additionally, we evaluate the performance with the Fetch robot in more realistic conditions,without any simpliﬁcation for grasping, and observe a signiﬁcant drop in performance, achieving25% success rate for 2 tasks and 0% for the other 3 (see Fig.
A.2 in the Appendix). This indicatesthat successful grasping for diverse objects is a signiﬁcant challenge in these manipulation tasks.
Totest generalization, we conducted an ablation study in which we train policies with three differentlevels of variability in Soaking Towel –no variations, different poses, different objects and poses–and evaluate them on an unseen setup (an unseen object with randomized initial pose).
The policiesachieve success rate of 19% , 79% , and 87% , respectively.
This study shows that it’s essential to trainwith diverse object models and initial states to obtain robust policies, and the generative system ofiGibson 2.0 facilitates it by specifying a few logical states that describe the initial scene.
The attachedvideo shows the policies performing the tasks trained using iGibson 2.0’s new extended physicalstates and logical predicates that help generating task instances and discriminating their completion.
IL experiments in Bimanual Pick and Place with VR demonstrations: The trained policy in thefull task diverges and fails, even after training with 30 demonstrations.
We hypothesize that theagent suffers from covariate shift [41], visiting state space not covered by demonstrations, due to thedifferent strategies demonstrated in VR for this long task (300+ steps).
We then evaluate if the policycan successfully perform the last part of the task.
We take a successful human demonstration andinitialize a few seconds before task completion.
We then query the IL policy from this point onwardto control the agent.
The results of this experiment are shown in Fig 5b.
The policy achieves 19%and 46% success rate when starting 6 s and 3 s away from the goal.
This experiment demonstrates thepotential of the new VR interface of iGibson 2.0 for generating demonstrations for IL.
We believethat our new interface will open new avenues for research in bimanual manipulation, hand-eyecoordination, and coordination of robot base and robot arm.
Conclusion
We presented iGibson 2.0, a new open-source simulation environment for household task with severalkey novel features: 1) an object-centric representation and extended object states (e.g. temperature,wetness and cleanliness level, toggled and sliced state), 2) logical predicates mapping simulationstates to logical states, and generative mechanism to create simulated worlds based on a given logicaldescription, and 3) a virtual reality interface to easily collect human demonstrations for imitation.
We demonstrate in multiple experiments the new avenues for research enabled by iGibson 2.0.
Wehope iGibson 2.0 becomes a useful tool for the community, reducing the burden of creating simulatedworlds for more diverse tasks, and facilitating the development of novel embodied AI solutions.