Embodied Semantic Scene Graph Generation
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Semantic scene graph provides an effective way for intelligent agentsto better understand the environment and it has been extensively used in manyrobotic applications.
Existing work mainly focuses on generating the scene graphfrom the sensory information collected from a pre-deﬁned path, while the environment should be exhaustively explored with a carefully designed path in orderto obtain a comprehensive semantic scene graph efﬁciently.
In this paper, we propose a new task of Embodied Semantic Scene Graph Generation, which exploitsthe embodiment of the intelligent agent to autonomously generate an appropriatepath to explore the environment for scene graph generation.
To this end, a learningframework with the paradigms of imitation learning and reinforcement learning isproposed to help the agent generate proper actions to explore the environment andthe scene graph is incrementally constructed.
The proposed method is evaluatedon the AI2Thor environment using both the quantitative and qualitative performance indexes.
Additionally, we implement the proposed method on a streamingvideo captioning task and promising experimental results are achieved.
Introduction
[Figure]
Graph Generation.
The agent moves around and always selects an appropriate action to obtain better semantic scene graph.
Further, the generated semantic scene graph can be helpful to generate the streaming video captioning.
The scene graph is a collection ofnodes in a graph structure wherenodes usually represent scene entities and edges represent geometricaltransformation between the nodes.
Itprovides a ﬂexible way to track objects and their spatial relations withinthe scene.
Recently, the concept ofscene graph is successfully adaptedin many applications in computer vision and robotics [1][2].
For example, the Visual Genome dataset [3],which contains annotations of objects and their relationships in images, has been extensively utilized invarious scene graph generation tasks[4][5][6].
In [1], the author proposes to extract the knowledge graphfrom recorded videos for better videounderstanding.
Xu et al.
[2] provides an extensive survey on the recent progress on the generation andapplications of the scene graph.
Furthermore, 3D semantic scene graph is of great interest for robotic applications such as navigation, mapping, and interaction [7].
To understand the 3D environment, Kim et al. [8] proposes a 3D scene graph construction framework using RGB-D data processing pipeline, which can be usedfor task planning and visual question answering.
Bozcan and Kalkan [9] develops a Boltzmann Machines-based generative scene model bringing objects, their spatial relations and affordancestogether. Armeni et al.
[10] presents a 3D scene graph, which uniﬁes multi-modal semantic information in the 3D space.
Recently, Rosinol et al.
[11] proposes a 3D dynamic scene graph whichextends to represent dynamic scenes with moving agents.
Though great success has been achieved for semantic scene graph, all of the above work focuseson generating scene graph from the sensory information collected from a pre-deﬁned path.
However, to obtain a comprehensive and informative semantic scene graph, the environment should beexhaustively explored, which requires large time consumption.
In many scenarios, the disconnection between the scene graph generation and the exploration path even results in unsatisﬁed results.
Therefore, it is necessary for the agent to have the ability to autonomously explore the environmentto generate a good semantic scene graph.
In this work, we adopt the embodied exploration framework to tackle the Embodied Semantic Scene
Graph Generation problem, which has never been addressed before.
It combines the action andperception abilities of the agent.
The embodied agent generates actions to autonomously explorethe environment and then incrementally constructs a high-quality semantic scene graph using thedetected object instances and contextual information.
The collected sensor data is continuouslyprocessed and incorporated into the scene graph model along with the exploration process.
Theillustration is demonstrated in Fig.1.
Different from the existing work which focuses on generatingscene graphs from acquired sensor information, our work introduces the embodiment of the agentto effectively combine the action, vision and language together for autonomous 3D semantic scenegraph generation.
The main contributions are summarized as follows:
We propose a new framework for the Embodied Semantic Scene Graph Generation problem, which exploits the embodiment characteristic of the intelligent agent to ﬁnd an appropriate path for scene graph generation in an embodied environment.2.
We develop a learning framework with paradigms of imitation learning and reinforcementlearning to help the agent acquire the intelligence to generate high-quality scene graph.3.
We test the proposed method on the AI2Thor dataset and evaluate its effectiveness usingthe quantitative and qualitative performance indexes.
For the rest of the paper, Section II presents the related work.
Problem formulation and architectureare introduced in Section III.
Sections IV and V give details about the scene graph generation andnavigation modules.
Section VI presents the experimental results and Section VII closes this paper.
Related Work
This work falls into the intersection of the embodied exploration and semantic understanding.
Therefore, we present a brief review on both domains.
The embodied exploration lies in the intersection of the computer vision and robotics.
The representative work includes active sensing [12][13], embodied question answering [14], embodied captioning [15], embodied amodal detection [16], multi-agent embodied exploration [17], etc.
In theseworks, one or multiple embodied agents equipped with visual perception modules can autonomouslyexplore the environment to perform various active sensing and perception tasks.
In recent years, researchers have combined deep reinforcement learning with active vision sensing.
Han et al.
[18]builds an active vision system based on DQN to guide the agent generate appropriate actions and getbetter images for detection.
Furthermore, Chaplot et al. [19] studies the task of embodied interactive learning for object detection.
It should be noticed that in [15], the authors propose an embodiedcaptioning task, which is relevant to our work, while our work solves a more fundamental task ofgenerating the 3D scene graph, which provides more comprehensive contextual information of theenvironment.
Actually, the generated scene graph can be useful for many down-streaming tasks suchas image captioning, streaming video captioning, embodied Questions and Answering, etc.
On the other hand, the embodied exploration task can also beneﬁt from the semantic scene graph.
Druon et al.
[20] and Zeng et al. [21] study how to enhance the active object search using the spatialcontext and semantic linking maps respectively.
The hierarchical mechanical search also shows
[Figure]
t+
[Figure]
performance improvement by using the semantic modeling [22].
Du et al.
[23] proposes to learnobject relation graph and tentative policy for visual navigation.
Besides, the scene graph also playsan important role in the visual question answering system for robot manipulation [24].
These worksillustrate the signiﬁcance of semantic scene graph in embodied perception.
However, they all useexisting scene graphs and do not address the generation of the scene graph.
Problem Formulation and Architecture
[Figure]
The goal of this work is to enable the agent to automatically generate a sequence of actions toexplore the environment and build the corresponding semantic scene graph incrementally.
Concretely speaking, the agent starts from any location of a scene with an initial empty global semantic scene graph GSSG 0 .
At time instant t , the agent could construct a local scene graph LSSG t = {N ( l ) t , R ( l ) t } from its observation s t , which could be extracted from observed RGBand depth images.
The local scene graph LSSG t should be merged with the previous global scenegraph GSSG t − 1 = {N ( g ) t − 1 , R ( g ) t − 1 } to get the updated global scene graph GSSG t = {N ( g ) t , R ( g ) t } .
Please note that we follow the conventional symbols with slight modiﬁcations [25], i.e. N ( l ) t and R ( l ) t represent the node and semantic relationships for the local scene graph LSSG t , and N ( g ) t and R ( g ) t represent the node and semantic relationships for the global scene graph GSSG t .
After that,a navigation module should produce an action a t for the agent to obtain the next observation s t +1 .
The above procedure iterates until a satisfactory global semantic scene graph is achieved and the stopaction is triggered.
The obtained semantic scene graph can then be used to describe the environmentcomprehensively and solve some downstream tasks such as captioning, question answering, etc.
We call this problem as Embodied Semantic Scene Graph Generation , which exploits the embodiment capability of the agent to collect data and construct high-quality scene graph.
It is signiﬁcantlydifferent from existing scene graph generation work such as [8][10].
To solve this problem, wedesign an architecture as shown in Fig.2.
It is mainly composed of the scene graph generation andnavigation module.
For the scene graph generation, a 3D semantic point cloud is produced fromthe RGB and depth images and is used for generating the local scene graph, which is then used toincrementally update the global scene graph.
For the navigation module, we utilize the RGB frame,previous action and the scene graph as input and an action is generated for the next step.
Scene Graph Generation
[Figure]
The scene graph generation module can be divided into two parts: local scene graph prediction andglobal scene graph generation, which are introduced as follows.
Local scene graph prediction: we mainly follow the graph convolution network based scene graphgeneration method which is able to pass the information between objects. It is widely used in tasksthat utilize the scene graph to bridge the gap between semantic and visual information [25][26].
Thegenerated local semantic scene graph includes the current in-sight objects and their relations witheach other.
We additionally introduce object class embedding and bounding box coordinates as theinput for each object node, and the training objective is to restore the bounding box coordinates andlabels for each object and edge.
Please see the supplementary material for more details.
Global scene graph generation: At each time step, the agent takes an action to move, and theequipped camera captures the RGB and depth frames of the scene to construct a local scene graph,which is further merged into the global scene graph.
During this period, the detected objects in thelocal scene graph and the existing objects in the previous global scene graph are aligned.
Different from the multi-view consistency considered in [10], which has the 3D mesh and multi-viewvisual information known in advance, we do not have any prior information about the scene and theinformation of each object is incrementally updated.
Considering this situation, we propose a pointcloud based weighted voting mechanism which utilizes the number of the pixels in the point cloudto measure the conﬁdence of the object category.
In practice, for each detected object, we maintain ascore distribution across predeﬁned classes.
The score is incrementally accumulated by the weightedvalue of the conﬁdence provided by the detector and the size of the acquired point cloud.
[Figure]
Then, we align the object point cloud withthe nodes in global scene graph by computingthe fraction of point cloud in the object pointcloud that is inside the node’s 3D boundingbox.
We calculate this fraction between eachobject point cloud and the node in the globalscene graph.
If the highest fraction of objectpoint cloud of the corresponding node is higherthan the preset threshold and this fraction ishigher than those with other objects, the object and the node are considered aligned.
Thisbidirectional alignment procedure could ensurethat each aligned object and the correspondingnode are in one-to-one correspondence.
Fig.3demonstrates the alignment process.
Basedon the aligning result for each object and theweighted voting mechanism, we could align the nodes and edges in the obtained Local Semantic Scene Graph(LSSG) with those in the Global Semantic Scene Graph(GSSG).
The updating operation could involve ADD, UPDATE and REPLACE for both nodes and edges in GSSG.
Please seethe supplementary material for more details.
5.1 Model Structure The goal of the proposed navigation model is to guide the agent to take actions to explore theenvironment and build the semantic scene graph incrementally. The action space considered in thiswork includes Move , where the agent can take nine basic actions corresponding to 8 directions andno move, and Rotate , where the agent can rotate clockwise or counterclockwise with a ﬁxed angleδ r = 90 ° or no rotate.
Especially, if no move and no rotate are selected, a stop action is triggered.
The design of the navigation model is a challenging problem since the agent is expected to build agood semantic scene graph which contains the objects and their relationships accurately.
To tacklethis problem, we construct a navigation architecture as shown in Fig.4.
Considering that the LSSGcould represent the insight semantic observations for the agent, we use an LSTM to embed the
Action
ResNet
LSTM
LSTM Action
ResNet
LSTM LSTM
Double Layer LSTM
MLP Network 𝑎h 1 h
Action
ResNet
LSTM
LSTM … … … … 𝑎 𝑇 − 𝐿𝑆𝑆𝐺 𝑇 RGB 𝐼 𝑇 𝑎
Double Layer LSTM
MLP Network h 2 h𝑎 𝑇 “ STOP ”
Double Layer LSTM
MLP Network 𝑎𝐿 S 𝑆𝐺 2 RGB 𝐼 2 𝑎𝐿 S 𝑆𝐺 1 RGB 𝐼“ START ” GSSG
GSSG
Action feature RGB feature LSSG feature GSSG feature GSSG T
LSSG sequence to help the agent build connections over visited viewpoints on a semantic level. Further, we introduce GSSG vector aiming at help the agent have a complete perception over theglobal semantic information. Since the change of GSSG could reﬂect the adding and updatingof objects and semantic relations in the scene, we utilize an LSTM to make the agent capture thisdynamic process.
Overall, we use a ResNet to extract the visual feature, two LSTMs to encodethe local scene graph and global scene graph sequence respectively, and an embedding module torepresent the previous action.
The three feature vectors are concatenated and fed into a double-layer LSTM model.
Finally, an MLP network is used to generate the next step action.
5.2 Learning for Navigation
Since the direct learning for such an architecture is still challenging, we adopt the imitation learningas the pre-training and the reinforcement learning for ﬁne-tuning.
5.2.1 Imitation Learning
The goal of imitation learning for sequential prediction problem is to train the agent to mimic theexpert’s behaviors.
In this work, it is important to generate some demonstration paths for the agentto imitate.
We adopt a two-stage method to deal with this problem.
In the ﬁrst stage, we try to obtaina way-point set and in the second stage, we perform the interpolation between way-points to getthe whole demonstration path.
To evaluate the way-points, we ﬁrst count the visible object at eachviewpoint, and select the closest next way-point that has the most number of unseen new objects,which can be expressed as:
[Equation]
where O ( v c , k ) represents the feasible viewpoints that are k steps away from v c , and the way-pointset can be updated as W = W ∪ { v ∗ } .
The above procedure is repeated until a maximum distanceis achieved.
In the second step, we implement a beam search over the way-point sequence forinterpolation [27].
The loss function is deﬁned as follows: [Equation] where K is the number of demonstration paths used for training in one batch, T k is the length of thek -th path, ˆ s k,t and ˆ a k,t are the annotated input visual state and action, and θ denotes the parameterof the exploration policy π . The process of minimizing L θ equals to maximize the probability of thedemonstration paths’ action sequence based on the annotated inputs.
5.2.2 Reinforcement Learning
After pre-training the exploration model with imitation learning, we then try to further improve itsperformance using the REINFORCE algorithm.
Since our task aims at formulating a scene graph for the entire scene, we directly measure thesimilarity between the generated scene graph and ground truth one, which can be obtained from theground truth information about the object and the relationships.
Given a global scene graph GSSGwhich is constructed from one path, the similarity score can then be calculated as a weighted sum ofthe precision and recall rate of the nodes and edges: [Equation] where P node and P edge are the precision of the nodes and edges, respectively, while R node and R edge are the recall rate of the nodes and edges, respectively.
On the other hand, we hope to encourage the observation diversity when constructing the scenegraph.
The diversity can be characterized as the number of observation viewpoints of the detectedobjects.
Concretely speaking, given a global scene graph GSSG which is constructed from one path,we use O to denote the set of the detected objects and calculate the diversity as:
[Equation]
where num _ viewpoints ( o ) is the number of the viewpoints about the detected object instance o .
Therefore we may formulate the score at time instant t as:
[Equation] where the third term is used to penally the length of the path, and λ d , ρ are the correspondingweighting parameters. In practice, we set λ node = 0 . 1 , λ p = 0 .
5 , and λ d = ρ = 0 .
According to the above deﬁnition, the immediate reward is designed to be the increment of the scorer ( s t , a t ) =
p t − p t − 1 and the cumulative reward can be computed as: [Equation] t (cid:48) =
t + γ t (cid:48) − t r ( s t (cid:48) , a t (cid:48) ) , (5) where R ( s t , a t ) represents the expected accumulated reward when agent takes action a t at state s t ,the discount parameter γ is set to 0.99, and T is the length of the action and state sequence withupper bound of 40 steps, and we use SGD optimizer with a learning rate of 10 − 4 .
Support
Placement
Hanging
Position
Connecting
Attachment
Component
Statue Counter Top
Stove Burner
Sink
Table
Garbage Can
Floor
Statue Desk Bread Counter Top Sink Towel Towel Holder
Wall Mirror
Painting
Sofa Toaster Coffee Machine
Cabinet Cabinet Drawer Drawer
Faucet
Stove Burner Knob Desk Drawer
Experimental Validation
6.1 Dataset
We choose AI2THOR [28] to generate our dataset since it supports a continuous navigation while
MP3D[29] does not, and its has higher rendering quality of images and more accurate sensors compared with Habitat[30].
We follow the guidelines in [25] to extract 16 semantic relationships, whichcan be clustered into the following seven categories (Fig.5).
Please refer to the supplementary material for more details.
[Table]
[Table]
[Table]
6.2 Method and Metrics
To verify the effectiveness of our framework and the inﬂuence of the input vectors, we introduce thefollowing methods for comparison:
Random
The agent randomly selects an action from action space to perform scene exploration.
PPO+Map[31] The agent utilizes Resnet-18 to encode RGB frame, local map and global maprespectively and concatenates them as the input for the LSTM to predict the next action.
Frontier-based Exploration[32]
A heuristic algorithm that guides the agent to detect and sweepto the closest frontier with path planning.
LSTM+Act+Pose[14, 15] This model is widely used in embodied tasks, it encodes RGB frame,agent position and last step action as the input of LSTM to predict the next step action for the agent.5.
LSSG+GSSG(Ours)
Our framework that takes LSSG and GSSG vectors as the LSTM input.
6. Traverse(Oracle)
Ask the agent to traverse every viewpoint in the scene.
To evaluate the quality of the semantic scene graph, we adopt the precision, recall rate and F1 scoreof the nodes and edges compared to the ground truth scene graph, denoted as P node , R node , F 1 node ,
P edge , R edge and F 1 edge respectively, and the number of steps NoS as quantitative metrics.
Notethat R node could be viewed as an object level coverage, and R edge could represent the semanticrichness of the generated GSSG .
We also propose the following two tasks for evaluation.
RelCls:
In this task, we align the detectingresult from Mask RCNN with ground truth to obtain the correctly detected objects at each viewpoint,and offer the ground truth segmentation of each object to the agent.
Since the detection results andpoint cloud information are always correct, the precision of node is always 1.0.
SGGen:
In thistask, the agent utilizes the raw detecting result from Mask RCNN which is consistent with realworld environment.
The agent is expected to leverage both the global scene graph and navigationpolicy to deal with the imperfection of Mask RCNN and the incompleteness of the collected pointcloud, incrementally generating an accurate and informative scene graph.
[Table]
6.3.1 Quantitative results
The experimental results are shown in Table I and II, from which we have the following observations:
In both RelCls and SGGen, LSSG + GSSG outperforms other baselines on the recall rate and F1score of both the node and edge, and achieves a performance close to Traverse, which demonstratesthe effectiveness of our framework.
Note that Traverse asks the agent to traverse every viewpoint inthe unseen scene which always requires manual assistance in reality.
While our method fully exploitsthe embodied capacity of the robot to explore the scene automatically and reaches a comparableperformance with much fewer NoS .2.
The increase in recall rate is often accompanied by a decrease in precision.
This is mainly causedby the imperfection of the Mask RCNN and Local Scene Graph Prediction Network(LSGPN) sincethe errors at some speciﬁc viewpoints could be hard to update and would accumulate as the agentincreasingly covers the scene.
Our method achieves the highest F1-score on both tasks, whichillustrates that our framework handles the balance of precision and recall rate well.3.
SGGen is much more difﬁcult than RelCls since both the label and the mask of the detecting resultcould be wrong, and the detecting error could further affect the generated LSSG and GSSG .
Thisexplains the drop of all metrics in SGGen compared with those in RelCls.
No matter how difﬁcultthe task is, the baseline methods only implement spatial exploration and adopt the same exploringpath.
While our method explores the scene at a semantic level indicating that the agent is aware of
GSSG and would try to update and add the semantic information of GSSG in subsequent steps.4.
From table II we could discover that in RelCls, both LSSG and GSSG beneﬁt the task, and theimprovement of introducing LSSG is more obvious.
This may be because the LSSG sequence encoded by LSTM contains a certain amount of GSSG information.
While in SGGen, adding GSSGvectors is worse than Base method.
We think this could be caused by the more frequently changingfrequency on GSSG and its vector is very different from that in the training.
LSSG + GSSGperforms best on both tasks, which illustrate the effectiveness and robustness of our framework.5.
Table III shows the performance of our LSSG + GSSG method under different trainingparadigm.
We can see that IL performs better than RL in RelCls.
In SGGen, the performancehas an obvious drop as the behaviour cloning based method utilized by IL has poor generalizationability.
In unseen testing scene, especially the ones in SGGen that have huge differences in aspectof room layout and point cloud accuracy with training scenes, IL trained model tends to performmuch worse.
Meanwhile, while RL trained model has a smaller performance drop in SGGen, it hasthe lowest recall rate over nodes and edges in RelCls and SGGen, which demonstrates that the scenehas not been completely explored.
Therefore, we design the mix training paradigm that utilizes imitation learning to ﬁrst pre-train the model, and then ﬁnetunes it with reinforcement learning.
Ourmethod (IL+RL) achieves the highest score over the recall rate and f1 score of nodes and edges onboth two tasks, demonstrating its efﬁciency.
6.3.2 Qualitative Analysis and Applications
The generated semantic scene graph can be used for various downstream tasks.
And we haveapplied the generated semantic scene graph in a streaming video captioning task in this paper.
The caption of the video is incrementally enriched with the generated scene graph while theagent exploring the environment. Experiment results show that the quality of the generated caption is superior to that generated with a Random exploration policy.
The detailed results andfurther discussion are presented in the supplementary document.
Please also visit our websitehttps://embodiedscenegraph.vercel.app/ for more information.
Conclusion
In this paper, a novel embodied semantic scene graph generation framework is established and thehybrid imitation reinforcement learning method is developed to address this new task.
With thelearned policy, the agent is able to autonomously explore the environment and then incrementallygenerate a high-quality semantic scene graph.
This research exploits the important embodimentcharacteristic of the intelligent agent and paves a new path for the agent to semantically describe theenvironment. Additionally, it provides a fundamental component for some challenging downstreamtasks such as streaming video captioning, robotic manipulation, visual navigation, etc. We
would