IFR-E XPLORE : L EARNING I NTER - OBJECT F UNCTIONAL R ELATIONSHIPS IN 3D I NDOOR S CENES
A BSTRACT
Building embodied intelligent agents that can interact with 3D indoor environments has received increasing research attention in recent years.
While most works focus on single-object or agent-object visual functionality and affordances, our work proposes to study a new kind of visual relationship that is also important to perceive and model – inter-object functional relationships ( e.g. , a switch on the wall turns on or off the light, a remote control operates the TV).
Humans often spend little or no effort to infer these relationships, even when entering a new room, by using our strong prior knowledge ( e.g. , we know that buttons control electrical devices) or using only a few exploratory interactions in cases of uncertainty ( e.g. , multiple switches and lights in the same room).
In this paper, we take the ﬁrst step in building AI system learning inter-object functional relationships in 3D indoor environments with key technical contributions of modeling prior knowledge by training over large-scale scenes and designing interactive policies for effectively exploring the training scenes and quickly adapting to novel test scenes.
We create a new benchmark based on the AI2Thor and PartNet datasets and perform extensive experiments that prove the effectiveness of our proposed method.
Results show that our model successfully learns priors and fast-interactive-adaptation strategies for exploring inter-object functional relationships in complex 3D scenes.
Several ablation studies further validate the usefulness of each proposed module.
I NTRODUCTION
When ﬁrst-time entering a hotel room in a foreign country, we may need to search among buttons on the wall to ﬁnd the one that turns on a speciﬁc light, ﬁgure out how to turn on the faucets in the bathroom and adjust water temperature, or perform several operations on the TV and the remotes to switch to our favorite channel.
Many of these may require no effort if we have used the same models of lamp or faucet before, while the others that we are less familiar with will take more time for us to explore and ﬁgure out how things work.
We humans can quickly adapt to a new indoor environment thanks to our strong prior knowledge about how objects could be functionally related in general human environments ( e.g. , the room light can be turned on by pressing a button or toggling a switch) and our amazing capability of using only a few interactions while exploring the environment to learn how things work in cases of uncertainty ( e.g. , multiple lights and buttons in the room).
In this work, we investigate how to equip AI systems with the same capabilities of learning inter-object functional relationships in 3D indoor environments.
With the recent research popularity of embodied AI in perception and interaction, many previous works have studied how to perceive, model, and understand object functionality (Kim et al., 2014;
Hu et al., 2016; 2018; 2020) and agent-object interaction (Montesano & Lopes, 2009;
Do et al., 2018; Nagarajan & Grauman, 2020; Mo et al., 2021a), just to name a few.
However, these works mostly investigate single-object/scene or agent-object scenarios.
A few past works (Jiang et al., 2012;
Sun et al., 2014; Zhu et al., 2015; Mo et al., 2021b; Cheng et al., 2021) have studied inter-object relationships given a pair of objects or for speciﬁc tasks ( e.g. , placement, stacking).
We make a ﬁrst step bringing up research attentions to a new type of visual quantity – inter-object functional relationship, and an interesting yet underexplored challenging problem of how to learn such generic functional relationships in novel 3D indoor environments.
[Figure]
Given a novel indoor scene with multiple objects as , we predict a complete directional graph of inter-object functional relationships (Fig. 1, middle) on how the state change of one object causes functional effects on other objects.
While the semantics of individual objects already suggest some functional priors ( e.g. , buttons can be pressed, electrical devices can be turned on), our system learns to build up the correspondence among them ( e.g. , which button controls which electrical device).
Also, since one may encounter unknown object categories exploring novel scenes, we do not assume semantic labels of objects and only use shape geometry as input.
Our system, therefore, learns inter-object functional geometry-mappings to what to trigger ( e.g. , one button among others on the wall) and what other shapes the interaction effects ( e.g. , one light in the room).
Though seemingly simple as one may interact with all objects in the scene to query the functional scene graph, it is actually very challenging how to efﬁciently and accurately perceive, reason, and explore the scene with as few interactions as possible.
First of all, we do not want to waste time interacting with books and bags as they will not functionally trigger other objects.
Secondly, the semantics of two objects may directly indicate functional relationships.
For example, we know that buttons control electrical devices in general.
Besides, in cases of having two buttons and two lights in the scene, one may guess the pairings according to spatial proximity.
All of these examples indicate that learning some geometric and semantic priors among the objects may help predict many inter-object functional relationships even without any interaction.
However, there are always scenarios where uncertainties exist.
For example, for the stove highlighted in Fig.
1 (right), interactions have to be taken place to ﬁgure out which knob controls which burner, though only two interactions are needed for the four pairs if our agent can smartly infer the other two automatically.
We design a two-staged approach, under a self-supervised interactive learning-from-exploration framework, to tackle the problem.
The ﬁrst stage perceives and reasons over the objects in a novel test scene to predict possible inter-object functional relationships using learned functional priors about the object geometry and scene layout.
Then, in the second stage, our system performs a few interactions in the scene to reduce uncertainties and fastly adapts to a posterior functional scene graph prediction.
We jointly train a reinforcement learning policy to explore large-scale training scenes and use the collected interaction observations to supervise the prior/posterior networks in the system.
Since we are the ﬁrst to explore this task to our best knowledge, we create a new hybrid dataset based on AI2THOR (Kolve et al., 2017) and PartNet (Mo et al., 2019b) as the benchmark, on which we conduct extensive experiments comparing to various baselines and illustrating our interesting ﬁndings.
Experiments validate that both stages contribute to the ﬁnal predictions and prove the training synergy of the exploration policy and the prior/posterior networks.
We also ﬁnd reasonable generalization capabilities when testing over novel scene types with unknown object categories.
In summary, this work makes the following main contributions:
• we ﬁrst formulate an important, interesting, yet underexplored challenging task – learning inter-object functional relationships in novel 3D indoor environments; • we propose a novel self-supervised learning-from-exploration framework combining a prior-inference stage and a fast-interactive-adaptation stage to tackle the problem; • we create a new dataset for benchmarking the proposed task and demonstrating the effectiveness of the proposed approach.
R ELATED W ORKS
Visual Functionality and Affordance.
We humans perceive and interact with the surrounding 3D world to accomplish everyday tasks.
To equip AI systems with the same capabilities, researchers have done many works investigating shape functionality (Kim et al., 2014;
Hu et al., 2016; 2018; 2020;
Lai et al., 2021; Guan et al., 2020), grasp affordance (Lenz et al., 2015;
Redmon & Angelova, 2015;
Montesano & Lopes, 2009; Qin et al., 2020; Kokic et al., 2020; Mandikal & Grauman, 2021; Yang et al., 2020; Corona et al., 2020; Kjellstr¨om et al., 2011; Fang et al., 2018; Nagarajan et al., 2019;
Brahmbhatt et al., 2019; Jiang et al., 2021), manipulation affordance (Do et al., 2018; Nagarajan et al., 2019; 2020; Nagarajan & Grauman, 2020; Mo et al., 2021a), scene affordance (Piyathilaka &
Kodagoda, 2015; Rhinehart & Kitani, 2016; Li et al., 2019b), etc.
While these works mostly study single-object/scene or agent-object interaction scenarios, our work explores inter-object relationships.
Besides, we investigate how to explore functional relationships by interacting in 3D scenes, different from previous works that study interactions given speciﬁc object pairs or tasks (Jiang et al., 2012;
Sun et al., 2014; Zhu et al., 2015; Mo et al., 2021b; Cheng et al., 2021).
Inter-object Relationships.
Besides perceiving individual objects in the environment, understanding the rich relationships among them is also crucial for many downstream applications.
Previous works have attempted to model object supporting and contact relationships (Fisher et al., 2011;
Silberman et al., 2012), spatial relationships (Galleguillos et al., 2008; Gould et al., 2008; Kulkarni et al., 2019), semantic relationships (Sadeghi & Farhadi, 2011; Divvala et al., 2014), physical relationships (Mitash et al., 2019), etc.
More recently, researchers have introduced scene graphs (Liu et al., 2014; Johnson et al., 2015; Krishna et al., 2017; Xu et al., 2017; Li et al., 2017b; Zhu et al., 2020;
Wald et al., 2020) and hierarchies (Armeni et al., 2019;
Li et al., 2019a; 2017a; Mo et al., 2019b;a) to parse the objects, object parts, and their relationships.
Unlike the spatial, geometric, semantic, or physical relationships investigated in these works, we pay more attention to inter-object functional relationships in which the state change of one object causes functional effects on other objects.
Learning from Exploration and Interaction.
While deep learning has demonstrated great success in various application domains (Russakovsky et al., 2015; Silver et al., 2016), large-scale annotated data for supervision inevitably becomes the bottleneck.
Many works thus explore self-supervised learning via active perception (Wilkes & Tsotsos, 1992), interactive perception (Bohg et al., 2017), or interactive exploration (Wyatt et al., 2011) to learn visual representations (Jayaraman & Grauman, 2018;
Yang et al., 2019;
Weihs et al., 2019; Fang et al., 2020; Zakka et al., 2020), objects and poses (Caicedo & Lazebnik, 2015; Han et al., 2019;
Deng et al., 2020; Chaplot et al., 2020b; Choi et al., 2021), segmentation and parts (Katz & Brock, 2008; Kenney et al., 2009; Van Hoof et al., 2014; Pathak et al., 2018; Eitel et al., 2019; Lohmann et al., 2020; Gadre et al., 2021), physics and dynamics (Wu et al., 2015; Mottaghi et al., 2016; Agrawal et al., 2016;
Li et al., 2016; Janner et al., 2018; Xu et al., 2019; Lohmann et al., 2020; Ehsani et al., 2020), manipulation skills (Pinto & Gupta, 2016; Levine et al., 2018;
Agrawal et al., 2016; Pinto et al., 2016; Pinto & Gupta, 2017; Zeng et al., 2018; Batra et al., 2020), navigation policies (Anderson et al., 2018; Chen et al., 2019; Chaplot et al., 2020a; Ramakrishnan et al., 2021), etc.
In this work, we design interactive policies to explore novel 3D indoor rooms and learn our newly proposed inter-object functional relationships.
P ROBLEM F ORMULATION
We formulate a novel problem of inferring inter-object functional relationships (IFRs) among objects in novel 3D indoor environments (Fig. 1).
For the task input, each scene S comprises multiple 3D objects { O 1 , O 2 , ··· , O n } , each of which is presented as a 3D point cloud with 2,048 points depicting the complete shape geometry.
Every object O i may be subject to state changes ( e.g. , water is running or not from a faucet, a button is pressed up/down).
The set of 3D objects describe the complete 3D scene S
= { O 1 , O 2 , ··· , O n } including both the objects with and without possible state changes.
The spatial layout of the objects in the scene is explicitly modeled that each object O i =
( ˆ O i , c i , s i ) can be decomposed into a zero-centered unit-sized shape point cloud ˆ O i ∈ R 2 , 048 × 3 , the object center position in the scene c i ∈ R 3 , and its isotropic scale s i ∈ R .
For the task output, we are interested to ﬁgure out the (IFRs) R S .
In a case that interacting with an object O i ( a trigger ) will cause a functional state change of another object O j ( a responder ), we formally deﬁne an observed IFR ( O i , O j ) ∈ R S .
In the task output R S , there could be one-to-one or many-to-many relationships between two objects, as a trigger object may cause functional state changes of many responder objects and vice versa.
[Figure]
We deﬁne a task scenario that an agent actively explores the environment and interacts with the objects to output the inter-object functional relationships following the popular embodied and real-world settings (Nagarajan & Grauman, 2020; Ramakrishnan et al., 2021).
For each interaction, an agent picks an object, triggers its state change, and directly receives the state changes of the other objects that have been affected in the scene.
We assume that standardized visual quantities ( e.g. , object segmentation, state changes) are provided to the agent as they are well-established research topics by themselves (Geiger et al., 2012; Li et al., 2020), and thus that the functional effect of a relationship is directly observable.
An agent is provided with large-scale scenes to explore for learning in the training stage and is asked to predict the functional scene graph ( S , R S ) for a novel scene at the test time.
We also abstract away complexities on robotic navigation (Anderson et al., 2018; Ramakrishnan et al., 2021) and manipulation (Nagarajan & Grauman, 2020; Mo et al., 2021a), which are orthogonal to our contribution to estimating inter-object functional relationships.
T ECHNICAL A PPROACH
Fig.
2 presents the system overview.
Given a novel indoor scene, the prior-inference stage ﬁrst produces a functional scene graph prior ( S , R sS ) by perceiving and reasoning over the objects in the scene S
= { O 1 , O 2 , ··· , O n } .
Then, in the fast-adaptation stage, the agent performs as few interactions as possible to reduce uncertainties and updates prior into posterior for the ﬁnal functional scene graph prediction ( S , R S ) .
We train an exploration policy that learns to effectively and efﬁciently explore large-scale training scenes and collects interaction data to supervise the prior/posterior networks.
4. T HE P RIOR -I NFERENCE S TAGE
We design two prior networks – BR-Prior-Net and SR-Prior-Net, for the functional scene graph prior learning, where BR-Prior-Net focus on modeling binary relationship priors r bi , j ∈ [ 0 , 1 ] given only two input shape geometry ( O i , O j ) ( ∀ i , j ) while SR-Prior-Net reasons scene-level functional relationship priors R sS = { r si , j ∈ [ 0 , 1 ]
| O i , O j ∈ S } considering all the objects in the scene S
= { O 1 , O 2 , ··· , O n } .
4.1. B INARY -R ELATIONSHIP P RIOR N ETWORK (BR-P RIOR -N ET )
Given two objects in the scene ( e.g. , a button and a light), humans often have binary priors if the two objects are functionally correlated or not ( e.g. , the button may control the light) according to geometry and semantics of the two shapes, regardless of their locations in the scene.
Having this observation, we thus design a binary-relationship prior network (BR-Prior-Net) to model the possibility of functional relationship for every pair of trigger object O i and responder object
O j .
As shown in Fig.
3 (left), our BR-Prior-Net takes two object point clouds
O i , O j ∈ R 2048 as input and outputs the belief of IFR r bi , j ∈ [ 0 , 1 ] .
By allowing O i
=
O j , we also consider self-relationship ( e.g. , triggering the button on the microwave opens its door).
We train a shared PointNet++ network Qi et al.
(2017), with four set abstraction layers with N = [ 512 , 128 , 32 , 1 ] point resolutions, to process the two point clouds and output global features f bi , f bj ∈ R 64 that summarize the two input shapes.
We then concatenate the two obtained features and employ a Multilayer Perceptron (MLP) network to estimate r bi , j ∈ [ 0 , 1 ] for the ﬁnal output.
We obtain the binary functional scene relationship prior
R bS =
{ r bi , j ∈ [ 0 , 1 ] | O i , O j ∈ S } by enumerating all object pairs in the scene.
4.1. S CENE -R ELATIONSHIP P RIOR N ETWORK (SR-P RIOR -N ET )
While the binary-relationship prior network empirically performs well in predicting pairwise functional relationships as it effectively scopes down the problem scale to two input objects only, much
[Figure]
important scene-level information is ignored, such as relative object poses and scene object layout.
Such scene context may provide clues on relationship pairings in cases of uncertainties.
For example, the BR-Prior-Net may connect all buttons to all lights in the room, but given the object layout, one may assign each button to its closest light with higher probabilities.
Therefore, we further introduce scene-relationship prior network (SR-Prior-Net) to take the scene context into consideration.
The SR-Prior-Net is implemented as a three-layer Graph Convolutional Network (GCN) (Kipf &
Welling, 2016) that connects and passes messages among all objects in the scene.
Fig.
3 (right) shows an overview of the design.
Each object O i is represented as a node i in the graph, with a node feature n ( 0 ) i that combines the PointNet++ geometry feature and the scene-contextual information ( e.g. , object position and size).
The edge between two nodes is initialized with w ( 0 ) i , j = r bi , j predicted by the BR-Prior-Net.
To allow better graph message passing throughout the graph, we additionally connect two nodes if the two objects are close enough ( < 0 .
5 m , which covers about 53.1% IFRs in our dataset) because closer objects are more likely to have functional relationships, e.g. , a stove knob is close to the corresponding stove burner.
We use an edge weight of w ( 0 ) i , j = γ = 0 . 6, which is experimentally tuned out (see Table E.5).
After three iterations of graph convolutions
[Equation]
where d ( k − 1 ) i = 1 + ∑ j w ( k − 1 ) j , i and Θ denotes the learnable parameters, the network outputs a scenecontextual embedding f Si = n ( 3 ) i ∈
R 32 for each object O i .
Next, for each object pair, we concatenate their feature embeddings, feed it through an MLP, and predict a likelihood score for their functional relationship r si , j ∈ [ 0 , 1 ] .
The ﬁnal scene-level functional scene relationship prior R sS = { r si , j ∈ [ 0 , 1 ] | O i , O j ∈ S } is obtained by enumerating all object pairs in the scene.
4. T HE F AST -I NTERACTIVE -A DAPTATION S TAGE
While the prior networks do their best to propose functionally related objects by perceiving and reasoning over objects in the scene, there are always uncertain cases that priors do not sufﬁce ( e.g. , the stove example in Fig.
1 (right)), where interactions are needed to reduce such uncertainties.
A smart agent only needs to perform very few interactions as needed and is capable of inferring other pairings.
We therefore design SR-Posterior-Net to learn such posterior reasoning after interactions and propose a test-time fast-adaptation strategy that sequentially picks a few objects to interact.
4.2. S CENE -R ELATIONSHIP P OSTERIOR N ETWORK (SR-P OSTERIOR -N ET )
The scene-relationship posterior network (SR-Posterior-Net) shares the same network architecture and weights to SR-Prior-Net introduced in Sec.
We only repurpose the network to model functional scene graph posteriors by simply changing the network input to the current functional scene relationship belief modulated by the observed interaction outcomes.
In our experiments, we perform interactions over objects one-by-one sequentially and thus feed through SR-Posterior-Net multiple times to evolve the functional scene relationship posteriors R ( t ) S ( t = 1 , 2 , ··· , T S ) after each interaction, where T S denotes the total number of interactions performed in this scene.
Concretely, when t = 0, we use R ( 0 ) S = R sS , which is the functional scene relationship prior estimated by SR-Prior-Net without performing any interaction.
For each timestep t , given the interaction observations { e i t , j ∈ { 0 , 1 }| O j ∈ S } after interacting with object O i t , where e i t , j = 1 if
and only if O i t triggers O j , we update the posterior R ( t ) S with all past observations r ( t ) i , j = e i , j ( i =
i 1 , i 2 , ··· , i t , j = 1 , 2 , ··· , n ), feed the modiﬁed posterior through SR-Posterior-Net, and obtain a functional scene relationship posterior R ( t + 1 ) S for the next timestep t + 1.
We compute the ﬁnal functional scene relationship prediction R S
= {
( O i , O j ) | r ( T S )
i , j ∈ R ( T S ) S > τ , ∀ i , j } with a threshold τ = 0 .
4.2. T EST - TIME F AST -I NTERACTIVE -A DAPTATION S TRATEGY
Now, the only questions left are how to pick the object to interact with at each timestep and when to stop.
We ﬁnd that a simple heuristic to explore the object that is predicted most uncertainly in our posterior works very well in our experiments.
Speciﬁcally, at each step t , the agent picks O i that i
[Equation] [Equation]
We stop the test-time fast-interactive-adaptation procedure when all predictions are certain enough, min
[Equation]
i
[Equation]
[Equation]
with a threshold γ = 0 .
05, the agent explores about 19% of objects in the room on average.
γ can also be enlarged to reduce number of interactions.
Another strategy is to ﬁx the test-time interaction budget, e.g. , 10% or 20% of the total number of objects in each room.
4. S ELF -S UPERVISED L EARNING FROM E XPLORATION
We adopt the embodied paradigm of learning from exploration and interaction (Nagarajan & Grauman, 2020; Ramakrishnan et al., 2021) that an agent actively explores the large-scale training scenes, interacts with the environments, and collects data for supervision.
It is a challenging task how to effectively probe the environments for self-supervised learning and perform as few interactions as possible for efﬁciency.
We formulate the task as a reinforcement learning (RL) problem and learn an exploration policy that collects data for supervising the prior and posterior networks.
4.3. E XPLORATION P OLICY
The RL policy, implemented under the Proximal Policy Optimization (PPO) (Schulman et al., 2017) framework, takes the current functional scene graph representation as the state input state ( t ) =
( S , R ( t ) S ) at each timestep t and picks one object as the action output to interact with action ( t ) =
O i ∈ S .
When t = 0, we use R ( 0 ) S = R sS , which is predicted by the SR-Prior-Net (Sec. 4.1.2).
After each interaction, the agent observes the interaction outcome that some or no objects have functional state changes triggered by O i , namely observation ( t ) =
{ e i , j ∈ { 0 , 1 }| O j ∈ S } where e i , j = 1 if
and only if
O i triggers O j , which will then be passed to the SR-Posterior-Net
(Sec. 4.2.1) to update the functional relationship belief R ( t + 1 ) S for the next timestep t + 1.
The agent may decide to stop exploring the current scene and move on to the next one under a limited interaction budget.
We design the RL reward function as follows.
[Equation]
The ﬁrst term encourages interacting with O i to correct the wrong or uncertain functional relationship beliefs r ( t ) i , j , ∀ j .
There are three cases for this term: 1) the agent’s belief is already correct ( r ( t ) i , j = e i , j , thus | r ( t ) i , j − e i , j | = 0); 2) the belief is wrong ( r ( t ) i , j = 1 − e i , j , thus | r ( t ) i , j − e i , j | = 1); 3) the agent is uncertain ( e.g. , r ( t ) i , j = 0 .
5, thus | r ( t ) i , j − e i , j | = 0 . 5).
We use max instead of avg as the aggregation function because the IFRs are sparse and an average value may conceal the mistake.
The second term rewards exploring trigger objects of interest, e.g. , it encourages the agent to interact with buttons or switches, instead of boxes or cabinets.
The third term simulates the interaction cost to avoid redundant interactions for efﬁcient exploration.
We use α = 2, β = 1 and γ = 1 in our experiments.
The RL network shares the same three-layer GCN backbone as the scene-relationship networks (Sec. 4.1.2 and 4.2.1) for reasoning over the functional scene graph ( S , R ( t ) S ) .
However, instead of predicting IFRs in SR-Prior-Net (Sec. 4.1.2)
and
SR-Posterior-Net (Sec. 4.2.1), the GCN RL policy outputs action probabilities a ( t ) i ’s over object nodes O
i ’s, by ﬁrstly obtaining an embedding n ( t ) i ∈
Rfor each object
O i , then concatenating with a global scene feature n ( t ) S computed by averaging across all object embeddings, and ﬁnally feeding through an MLP to get an action score a ( t ) i .
In addition, we duplicate the scene feature n ( t ) S and input it to the MLP to query the termination score a stop .
All the action scores are ﬁnally passed through a SoftMax layer for the action probability scores.
4.3. T RAINING S TRATEGY AND L OSSES
To supervise the proposed interactive learning-from-exploration framework, we alternate the training of the exploration RL policy and the prior/posterior networks.
We leverage the training synergy that the prior/posterior networks predict helpful information for better exploration of the scenes, while the exploration RL policy collects useful data to better train the two networks.
In our implementation, we train the exploration policy and the prior/posterior networks to make they jointly converge.
In each loop, conditioned on the current prior/posterior predictions, the exploration policy strategically explores the large-scale training scenes with a total interaction budget of 1000 ( e.g. , roughly 32% of the total 3125 objects from 210 scenes).
The exploration policy learns to wisely allocate budgets across different scenes and pick the objects worth interacting with.
Speciﬁcally, the agent can interact with m objects in one scene and explore the rest scenes with the 1000 − m budget left.
For a scene S
= { O 1 , O 2 , ··· , O n } where the agent sequentially performs m interactions over objects ( O i 1 , O i 2 , ··· , O i m ) , we obtain m sets of observations ( { e i 1 , j ∈ { 0 , 1 }| O j ∈ S } , { e i 2 , j ∈ { 0 , 1 }| O j ∈
S } , ··· , { e i m , j ∈ { 0 , 1 }| O j ∈ S } ) .
We then use the collected interaction observations to train the prior/posterior networks.
For BR-PriorNet, we simply train for the observed link predictions r bi , j → e i , j ( i = i 1 , i 2 , ··· , i m , j = 1 , 2 , ··· , n ).
We learn one shared network for SR-Prior-Net and SR-Posterior-Net but train it with two supervision sources.
For learning the SR-Priors, we input the BR-Prior-Net prediction R bS , output the scene-level prior R sS and train for the observed link predictions as well r si , j → e i , j ( i = i 1 , i 2 , ··· , i m , j = 1 , 2 , ··· , n ).
For training the SR-Posteriors, we use the posterior estimate R ( t ) S at each timestep t = 1 , 2 , ··· , m − 1, update it with the past ground-truth observations r ( t ) i , j = e i , j ( i = i 1 , i 2 , ··· , i t , j = 1 , 2 , ··· , n ), input to the network to obtain the posterior estimate R ( t + 1 ) S , and supervise the link predictions r ( t + 1 ) i , j → e i , j ( i = i t + 1 , i t + 2 , ··· , i m , j = 1 , 2 , ··· , n ).
We use the standard binary cross entropy for all loss terms.
E XPERIMENTS
We evaluate our method on a new hybrid dataset and compare it to several baselines that prove the effectiveness of the proposed method and its components.
We also experiment with transferring to novel scene types that may contain novel shape categories.
See appendix for more results.
5. D ATASET
We create a new hybrid dataset based on AI2THOR (Kolve et al., 2017) and PartNet-Mobility (Mo et al., 2019b; Xiang et al., 2020) to support our study.
AI2THOR provides 120 interactive scenes of 4 types ( i.e. , bathroom, bedroom, living room, and kitchen) containing more than 1500 interactive objects covering 102 categories.
The PartNet-Mobility dataset contains 2,346 3D articulated object
CAD models from 46 categories.
We replace 83 AI2THOR objects with 888 PartNet-Mobility objects for 31 categories (12 exist in AI2THOR while 19 are newly added) to further enrich the diversity of the shape geometry.
We use 27 kinds of functional relationships originally built in AI2THOR ( e.g. , the microwave can be toggled on) and also enrich 21 more types of inter-object functional relationships ( e.g. , the pen can be used to write in the book).
In total, our dataset contains 1200 scenes covering 23360 objects from 121 categories.
We enrich the origin 120 AI2THOR scenes by 1) randomly spawn objects into different locations and 2) randomly replace the object geometries of the same type.
Fig.
A.6 shows one example room from each of the four types.
Among these, 27 are trigger objects, 29 are responders , where 11 are both trigger and responders ( e.g. , desk lamp), and 79 are non-interactive background objects.
We split the dataset into non-overlapping 800 training scenes and 400 test scenes.
Table A.3 presents more detailed statistics.
5. B ASELINES AND M ETRICS
Since we are the ﬁrst to study the problem to our best knowledge, there is no external baseline for directly fair comparisons.
However, to validate the effectiveness for each of our proposed
[Table]
[Figure]
[Figure] [Figure] [Figure] modules ( e.g. , BR-Prior-Net, SR-Prior-Net, the fast-interactive-adaptation strategy, the self-supervised exploration policy), we compare Ours-Final to several baselines and ablated versions:
• Random : the agent performs random actions in the scene at the test time; • Abla-NoBinaryPrior : the agent does not use the binary priors and instead initializes the input edges to SR-Prior-Net with w ( 0 ) i , j = 1; • Abla-NoScenePrior : the agent uses the predicted binary-relationship prior R bS instead of the estimated scene-relationship prior R s S as the prior knowledge; • Abla-RandomExplore : the agent does not use the proposed exploration policy and instead randomly explores training scenes during training; • Abla-RandomAdapt : the agent randomly picks objects to interact with during adaptation; • Ours-PriorOnly : the agent directly uses the scene-relationship prior predictions R sS and cuts a threshold 0 .
5 to predict the ﬁnal outputs without any interaction; • Interaction-
Exploration : we try our best to adapt Nagarajan & Grauman (2020) to our setting, though it originally tackles a very different task on learning affordance over individual shapes in a scene and does not propose IFRs.
We use a PointNet++ to encode object features and an MLP to additionally predict a set of objects that each interaction at a object causes.
We use the standard precision, recall, and F1-score for quantitative evaluations and comparisons.
Additionally, we set a per-room interaction budget for fairly comparing all the methods and report their performance when the budgets are 10% and 20% of the object count in each room.
5. R ESULTS AND A NALYSIS
Table 1 presents the quantitative evaluations and comparisons to baselines, where we see that OursFinal achieves the best scores in F1-score and MCC in all comparisons under different interaction budgets.
The Random baseline observes the direct interaction outcomes and thus by design gives a
[Table]
perfect precision but random interaction renders very low recalls and F1 scores.
Compared to the ablated versions, we validate the effectiveness of our learned binary-prior, scene-prior, the proposed fast-interactive-adaptation strategy, and the self-supervised learning-from-exploration policy.
It also meets our expectation that the performance improves with more interactions ( i.e. , from 10% to 20%), which is further validated by Fig.
D.12.
We show qualitative results in Fig.
4 and analyze them below.
Is the Learned Prior Knowledge Useful?
In Table 1, we see that Ours-PriorOnly already gives decent performance even without performing any interaction, outperforming Random by a large margin, which proves the usefulness of the learned prior knowledge.
As shown in Fig.
4 (the left-most ﬁgure in the bottom row), the prior knowledge correctly predicts many relationships (red lines) to the ground-truth ones (blue lines).
In addition, both BR-Prior-Net and SR-Prior-Net are helpful, as observed in the ablated versions Abla-NoBinaryPrior and Abla-NoScenePrior .
We ﬁnd that
the
BR-Prior-Net makes a lot of false-positive predictions and the SR-Prior-Net successfully removes them given the scene context.
The performance drop of Abla-NoBinaryPrior further illustrates the need for binary priors, which initialize SR-Prior-Net with a sparse graph, better than a dense one.
Does Fast-Interactive-Adaptation Help?
The performance of Ours-PriorOnly is further improved given test-time interactions and posterior adaptions as Ours-Final generates better numbers.
In Fig.
4, we see that some redundant red lines are removed observing the interaction outcomes and applying the SR-Posterior-Net predictions.
Compared to Abla-RandomAdapt that performs random interactions, we ﬁnd that the fast-interactive-adaptation strategy performs better.
Qualitatively, besides the stove case mentioned in Fig.
1 (right), another interesting case in Fig.
4 is that from prior knowledge the agent thinks the two knives (ID: 57/67) may cut egg (ID: 50), maybe because it looks like a tomato (ID: 62) in geometry.
The agent only interacts one knife (ID: 57) to correct this wrong belief and then the false-positive edge from the other knife (ID: 67) is automatically removed.
Is Exploration Policy Effective?
We learn an exploration policy for efﬁcient exploration of the training scenes.
Abla-RandomExplore gives a clear performance drop when using randomly collected data for training, proving that the exploration policy can wisely pick objects to explore.
5. T RANSFER TO N OVEL R OOM AND O BJECT T YPES
We further test our model’s generalization capability to objects and rooms of novel types.
In Table 2, we report quantitative evaluations where we train our systems on one type of AI2THOR rooms and test over the other types, as well as additional results of testing over a totally out-ofdistribution RoboTHOR (Deitke et al., 2020) test dataset
(see Fig.
A.7).
We ﬁnd that 1) it works in most cases showing that our learned skills can transfer to novel room types; 2) our system can also generalize to novel object categories.
For example, as shown in Fig. 5, although the agent never sees the television nor remote in the training scenes of bedrooms, it thinks the TV might be controlled by the remote. 3) the prior knowledge is not working well because of the data distribution shift but the posterior interaction helps.
For example, in
Fig.
5, the interaction helps to make accurate predictions despite the prior is not very accurate.
C ONCLUSION
We formulate a novel task of learning inter-object functional relationships in novel 3D indoor environments and propose a two-staged interactive learning-from-exploration approach to tackle the problem.
Using a newly created hybrid dataset, we provide extensive analysis of our method and report quantitative comparisons to baselines.
Experiments comparing to several ablated versions validate the usefulness of each proposed module.
See appendix for limitations and future works.
Ethics Statement.
We explore AI algorithms towards building future home-assistant robots which can aid people with disabilities, senior people, etc.
While we envision that our proposed approach may suffer from the bias in training data, the privacy of using human-designed data, and domain gaps between training on synthetic data and testing in real-world environments, these are the common problems that most if not all learning-based systems have.
We do not see other particular harm or issue of major concerns that our work may raise.
Reproducibility Statement.
We support open-sourced research and will make sure that our results are reproducible.
Thus, we promise to release the code, data, and pre-trained models publicly to the community upon paper acceptance.
Acknowledgements This paper is supported by a Vannevar Bush Faculty fellowship and NSF grant
IIS-
A PPENDIX
[Figure]
M [Equation] D ATASET D ETAILS
Fig.
A.6 shows one example room from each of the four types in AI2THOR (Kolve et al., 2017).
Each
AI2THOR scene is designed manually by 3D artists from reference photos.
There is a randomizer that can be used to change the location of the objects, so as to enrich the scene.
RoboTHOR (Deitke et al., 2020) contains 25 test scenes that have different styles from the AI2THOR scenes (see Fig.
A.6).
Fig.
A.7 shows some RoboThor scenes we are using.
We can see that the scenes are visually different and thus we consider them to be two different distributions of scenes.
Table A.3 presents the detailed data statistics.
We collect all AI2Thor models and scenes ofﬂine and build our own simulation environment with standard OpenAI Gym API (Brockman et al., 2016).
We construct the edges of a scene by grammar based on object classes.
For many-to-many relationships
( e.g. , knives can cut potatoes), all triggers are connected to responders.
For one-to-one relationships ( e.g. , the buttons open lamps), we follow the rules of AI2Thor if possible.
For cases that AI2Thor sufﬁces ( e.g. , there is no ceil lamp instance in the AI2Thor scenes), we manually bind the relationships with heuristic rules based on distances and human knowledge.
Among all the 48 edge types, 19 are one-to-one and 29 are many-to-many.
If the object should have detailed parts ( e.g. , Microwave), we substitute the AI2Thor mesh with the PartNet model.
[Table]
[Figure]
[Table]
[Figure]
B M ORE Q UALITATIVE R ESULTS
We show more qualitative results in this section.
In Fig.
B.9 we show an example inference in the bathroom.
There are many holders used for different purposes.
For example, the bath towel holder (ID: 20) is used for bath towel (ID: 19); the hand towel holder is (ID: 29/30/31/32) used for a hand towel (ID: 18); the toilet paper holder is (ID: 41) used for toilet paper (ID: 23/33).
Those holders look similar and the agent fails to give accurate predictions based on prior knowledge.
However, the agent knows those holders are hard to distinguish and uses the fast-interactive-adaptation policy to reduce the uncertainty.
In Fig.
B.10, we show an example inference in the bedroom.
We can see that the prior knowledge is already good without any interaction.
And with interaction, the performance can be further improved.
For example, the agent learns to turn on the ceiling lamp (ID: 3) by toggling a novel button (ID: 9) that doesn’t appear in the training scenes.
C D IFFERENT RELATIONSHIP ABLATION .
In this section we discuss the agent’s performance over difference types of relationships.
As shown in
Table C.4.
Overall the agent performs better on the Many-to-Many relationship, which is reasonable because One-to-One relationships would suffer from ambiguities.
Another clear trend is that the agent performs much better in the Self relationship compared to the Binary relationship, which indicates the difﬁculties to build IFRs across objects. [Figure] [Table] [Figure]
[Table]
D F AST -I NTERACTIVE - ADAPTATION BEHAVIOR
Here we show the behavior of fast-interactive-adaptation policy.
In Fig.
D.11, we visualize the distribution of objects that the fast-interactive-adaptation policy interacts with.
We can see that the adaptation policy tends to interact with interesting objects with IFR ( e.g. , switches and knobs).
In
Fig.
D.12, we can also see the performance steadily improve with the number of interactions.
E INITIALIZATION OF E DGES IN GCN.
Here we discussed the edge initialization of GCN of SR-Prior-Net and Exploration Policy.
Table E. shows that: 1) the performance drops with only prior knowledge or distance information as edges; 2) a scaling factor can boost the performance.
Our hypothesis is that the scaling of initial edges helps the model to distinguish them from GT edges, thus making it easier to do casual inference.
F F AILURE C ASES AND A NALYSIS
There are cases where the object shapes are very similar, but the IFR is different.
For example,
Fig.
F.13 shows one failure case in a bathroom scene, where the agent confuses the hand towel with the bath towel and predicts two extra edges.
(the hand towel holder is used to hold the bath towel and the bath towel holder is used to hold hand towel.)
G L IMITATIONS AND F UTURE W ORKS .
First, our work uses 3D point clouds as inputs, ignoring the object color, texture, or other nongeometric information.
Besides, our system assumes all the objects in the scene are pre-segmented and many visual quantities are direction observable ( e.g. , the state change of objects). Future [Figure]
[Figure] [Table] works can study how imperfect visual observations may affect our network predictions and how to address such challenges if any.
Lastly, we also abstract away the robotic navigation and detailed manipulation.
Future works may investigate building an end-to-end pipeline considering navigation and manipulation
[Table]