Embodied Semantic Scene Graph Generation
Abstract: Semantic scene graph provides an effective way for intelligent agents to better understand the environment and it has been extensively used in many robotic applications.
Existing work mainly focuses on generating the scene graph from the sensory information collected from a pre-deﬁned path, while the environment should be exhaustively explored with a carefully designed path in order to obtain a comprehensive semantic scene graph efﬁciently.
In this paper, we propose a new task of Embodied Semantic Scene Graph Generation, which exploits the embodiment of the intelligent agent to autonomously generate an appropriate path to explore the environment for scene graph generation.
To this end, a learning framework with the paradigms of imitation learning and reinforcement learning is proposed to help the agent generate proper actions to explore the environment and the scene graph is incrementally constructed.
The proposed method is evaluated on the AI2Thor environment using both the quantitative and qualitative performance indexes.
Additionally, we implement the proposed method on a streaming video captioning task and promising experimental results are achieved.
Introduction
[Figure]
The scene graph is a collection of nodes in a graph structure where nodes usually represent scene entities and edges represent geometrical transformation between the nodes.
It provides a ﬂexible way to track objects and their spatial relations within the scene.
Recently, the concept of scene graph is successfully adapted in many applications in computer vision and robotics [1][2].
For example, the Visual Genome dataset [3], which contains annotations of objects and their relationships in images, has been extensively utilized in various scene graph generation tasks [4][5][6].
In [1], the author proposes to extract the knowledge graph from recorded videos for better video understanding.
Xu et al.
[2] provides an extensive survey on the recent progress on the generation and applications of the scene graph.
Furthermore, 3D semantic scene graph is of great interest for robotic applications such as navigation, mapping, and interaction [7].
To understand the 3D environment, Kim et al.
[8] proposes a 3D scene graph construction framework using RGB-D data processing pipeline, which can be used for task planning and visual question answering.
Bozcan and Kalkan [9] develops a Boltzmann Machines-based generative scene model bringing objects, their spatial relations and affordances together.
Armeni et al.
[10] presents a 3D 5th (CoRL London, UK. scene graph, which uniﬁes multi-modal semantic information in the 3D space.
Recently, Rosinol et al.
[11] proposes a 3D dynamic scene graph which extends to represent dynamic scenes with moving agents.
Though great success has been achieved for semantic scene graph, all of the above work focuses on generating scene graph from the sensory information collected from a pre-deﬁned path.
However, to obtain a comprehensive and informative semantic scene graph, the environment should be exhaustively explored, which requires large time consumption.
In many scenarios, the disconnection between the scene graph generation and the exploration path even results in unsatisﬁed results.
Therefore, it is necessary for the agent to have the ability to autonomously explore the environment to generate a good semantic scene graph.
In this work, we adopt the embodied exploration framework to tackle the Embodied Semantic Scene
Graph Generation problem, which has never been addressed before.
It combines the action and perception abilities of the agent.
The embodied agent generates actions to autonomously explore the environment and then incrementally constructs a high-quality semantic scene graph using the detected object instances and contextual information.
The collected sensor data is continuously processed and incorporated into the scene graph model along with the exploration process.
The illustration is demonstrated in Fig.1.
Different from the existing work which focuses on generating scene graphs from acquired sensor information, our work introduces the embodiment of the agent to effectively combine the action, vision and language together for autonomous 3D semantic scene graph generation.
The main contributions are summarized as follows:
We propose a new framework for the Embodied Semantic Scene Graph Generation problem, which exploits the embodiment characteristic of the intelligent agent to ﬁnd an appropriate path for scene graph generation in an embodied environment.
We develop a learning framework with paradigms of imitation learning and reinforcement learning to help the agent acquire the intelligence to generate high-quality scene graph.
We test the proposed method on the AI2Thor dataset and evaluate its effectiveness using the quantitative and qualitative performance indexes.
For the rest of the paper, Section II presents the related work.
Problem formulation and architecture are introduced in Section III.
Sections IV and V give details about the scene graph generation and navigation modules.
Section VI presents the experimental results and Section VII closes this paper.
Related Work
This work falls into the intersection of the embodied exploration and semantic understanding.
Therefore, we present a brief review on both domains.
The embodied exploration lies in the intersection of the computer vision and robotics.
The representative work includes active sensing [12][13], embodied question answering [14], embodied captioning [15], embodied amodal detection [16], multi-agent embodied exploration [17], etc.
In these works, one or multiple embodied agents equipped with visual perception modules can autonomously explore the environment to perform various active sensing and perception tasks.
Zhang and Mei [18] proposes a constructive model for collective intelligence in which each agent would explore and communicate to cooperate with each other.
In recent years, researchers have combined deep reinforcement learning with active vision sensing.
Han et al.
[19] builds an active vision system based on DQN to guide the agent generate appropriate actions and get better images for detection.
Furthermore, Chaplot et al.
[20] studies the task of embodied interactive learning for object detection.
It should be noticed that in [15], the authors propose an embodied captioning task, which is relevant to our work, while our work solves a more fundamental task of generating the 3D scene graph, which provides more comprehensive contextual information of the environment.
Actually, the generated scene graph can be useful for many down-streaming tasks such as image captioning, streaming video captioning, embodied Questions and Answering, etc.
On the other hand, the embodied exploration task can also beneﬁt from the semantic scene graph.
Druon et al.
[21] and Zeng et al.
[22] study how to enhance the active object search using the spatial context and semantic linking maps respectively.
The hierarchical mechanical search also shows performance improvement by using the semantic modeling [23]. Du et al. [24] proposes to learn
[Figure]
Navigation Model
[Figure]
object relation graph and tentative policy for visual navigation.
Besides, the scene graph also plays an important role in the visual question answering system for robot manipulation [25].
These works illustrate the signiﬁcance of semantic scene graph in embodied perception.
However, they all use existing scene graphs and do not address the generation of the scene graph.
Problem Formulation and Architecture
The goal of this work is to enable the agent to automatically generate a sequence of actions to explore the environment and build the corresponding semantic scene graph incrementally.
Concretely speaking, the agent starts from any location of a scene with an initial empty global semantic scene graph GSSG 0 .
At time instant t , the agent could construct a local scene graph
LSSG t = {N ( l ) t , R ( l ) t } from its observation s t , which could be extracted from observed RGB and depth images.
The local scene graph LSSG t should be merged with the previous global scene graph GSSG t − 1 = {N ( g ) t − 1 , R ( g ) t − 1 } to get the updated global scene graph GSSG t = {N ( g ) t , R ( g ) t } .
Please note that we follow the conventional symbols with slight modiﬁcations [26], i.e. N ( l ) t and
R ( l ) t represent the node and semantic relationships for the local scene graph LSSG t , and N ( g ) t and
R ( g ) t represent the node and semantic relationships for the global scene graph GSSG t .
After that, a navigation module should produce an action a t for the agent to obtain the next observation s t +1 .
The above procedure iterates until a satisfactory global semantic scene graph is achieved and the stop action is triggered.
The obtained semantic scene graph can then be used to describe the environment comprehensively and solve some downstream tasks such as captioning, question answering, etc.
We call this problem as Embodied Semantic Scene Graph Generation , which exploits the embodiment capability of the agent to collect data and construct high-quality scene graph.
It is signiﬁcantly different from existing scene graph generation work such as [8][10].
To solve this problem, we design an architecture as shown in Fig.2.
It is mainly composed of the scene graph generation and navigation module.
For the scene graph generation, a 3D semantic point cloud is produced from the RGB and depth images and is used for generating the local scene graph, which is then used to incrementally update the global scene graph.
For the navigation module, we utilize the RGB frame, previous action and the scene graph as input and an action is generated for the next step.
Scene Graph Generation
The scene graph generation module can be divided into two parts: local scene graph prediction and global scene graph generation, which are introduced as follows.
Local scene graph prediction: we mainly follow the graph convolution network based scene graph generation method which is able to pass the information between objects.
It is widely used in tasks that utilize the scene graph to bridge the gap between semantic and visual information [26][27].
The generated local semantic scene graph includes the current in-sight objects and their relations with each other.
We additionally introduce object class embedding and bounding box coordinates as the input for each object node, and the training objective is to restore the bounding box coordinates and labels for each object and edge.
Please see the supplementary material for more details.
Global scene graph generation: At each time step, the agent takes an action to move, and the equipped camera captures the RGB and depth frames of the scene to construct a local scene graph, which is further merged into the global scene graph.
During this period, the detected objects in the local scene graph and the existing objects in the previous global scene graph are aligned.
Different from the multi-view consistency considered in [10], which has the 3D mesh and multi-view visual information known in advance, we do not have any prior information about the scene and the information of each object is incrementally updated.
Considering this situation, we propose a point cloud based weighted voting mechanism which utilizes the number of the pixels in the point cloud to measure the conﬁdence of the object category.
In practice, for each detected object, we maintain a score distribution across predeﬁned classes.
The score is incrementally accumulated by the weighted value of the conﬁdence provided by the detector and the size of the acquired point cloud.
[Figure]
Then, we align the object point cloud with the nodes in global scene graph by computing the fraction of point cloud in the object point cloud that is inside the node’s 3D bounding box.
We calculate this fraction between each object point cloud and the node in the global scene graph.
If the highest fraction of object point cloud of the corresponding node is higher than the preset threshold and this fraction is higher than those with other objects, the object and the node are considered aligned.
This bidirectional alignment procedure could ensure that each aligned object and the corresponding node are in one-to-one correspondence.
Fig. demonstrates the alignment process.
Based on the aligning result for each object and the weighted voting mechanism, we could align the nodes and edges in the obtained Local Semantic
Scene Graph(LSSG) with those in the Global Semantic Scene Graph(GSSG).
The updating operation could involve ADD, UPDATE and REPLACE for both nodes and edges in GSSG.
Please see the supplementary material for more details.
Navigation Model 5.1 Model Structure
The goal of the proposed navigation model is to guide the agent to take actions to explore the environment and build the semantic scene graph incrementally.
The action space considered in this work includes Move , where the agent can take nine basic actions corresponding to 8 directions and no move, and Rotate , where the agent can rotate clockwise or counterclockwise with a ﬁxed angle δ r = 90 ° or no rotate.
Especially, if no move and no rotate are selected, a stop action is triggered.
The design of the navigation model is a challenging problem since the agent is expected to build a good semantic scene graph which contains the objects and their relationships accurately.
To tackle this problem, we construct a navigation architecture as shown in Fig.4.
Considering that the LSSG could represent the insight semantic observations for the agent, we use an LSTM to embed the
Action
ResNet
LSTM
LSTM Action
ResNet
LSTM LSTM
Double Layer LSTM
MLP Network 𝑎h 1 h
Action
ResNet
LSTM
LSTM … … … … 𝑎 𝑇 − 𝐿𝑆𝑆𝐺 𝑇 RGB 𝐼 𝑇 𝑎
Double Layer LSTM
MLP Network h 2 h𝑎 𝑇 “ STOP ”
Double Layer LSTM
MLP Network 𝑎𝐿 S 𝑆𝐺 2 RGB 𝐼 2 𝑎𝐿 S 𝑆𝐺 1 RGB
𝐼“ START ”
GSSG
GSSG
Action feature RGB feature
LSSG feature
GSSG feature
GSSG T
LSSG sequence to help the agent build connections over visited viewpoints on a semantic level.
Further, we introduce GSSG vector aiming at help the agent have a complete perception over the global semantic information.
Since the change of GSSG could reﬂect the adding and updating of objects and semantic relations in the scene, we utilize an LSTM to make the agent capture this dynamic process.
Overall, we use a ResNet to extract the visual feature, two LSTMs to encode the local scene graph and global scene graph sequence respectively, and an embedding module to represent the previous action.
The three feature vectors are concatenated and fed into a double-layer
LSTM model.
Finally, an MLP network is used to generate the next step action.
5.2 Learning for Navigation
Since the direct learning for such an architecture is still challenging, we adopt the imitation learning as the pre-training and the reinforcement learning for ﬁne-tuning.
5.2.1 Imitation Learning
The goal of imitation learning for sequential prediction problem is to train the agent to mimic the expert’s behaviors.
In this work, it is important to generate some demonstration paths for the agent to imitate.
We adopt a two-stage method to deal with this problem.
In the ﬁrst stage, we try to obtain a way-point set and in the second stage, we perform the interpolation between way-points to get the whole demonstration path.
To evaluate the way-points, we ﬁrst count the visible object at each viewpoint, and select the closest next way-point that has the most number of unseen new objects, which can be expressed as:
[Equation]
where O ( v c , k ) represents the feasible viewpoints that are k steps away from v c , and the way-point set can be updated as W = W ∪ { v ∗ } .
The above procedure is repeated until a maximum distance is achieved.
In the second step, we implement a beam search over the way-point sequence for interpolation [28].
The loss function is deﬁned as follows:
[Equation]
where K is the number of demonstration paths used for training in one batch, T k is the length of the k -th path, ˆ s k,t and ˆ a k,t are the annotated input visual state and action, and θ denotes the parameter of the exploration policy π .
The process of minimizing L θ equals to maximize the probability of the demonstration paths’ action sequence based on the annotated inputs.
5.2.2 Reinforcement Learning
After pre-training the exploration model with imitation learning, we then try to further improve its performance using the REINFORCE algorithm.
Since our task aims at formulating a scene graph for the entire scene, we directly measure the similarity between the generated scene graph and ground truth one, which can be obtained from the ground truth information about the object and the relationships.
Given a global scene graph GSSG which is constructed from one path, the similarity score can then be calculated as a weighted sum of the precision and recall rate of the nodes and edges:
[Equation]
where P node and P edge are the precision of the nodes and edges, respectively, while R node and
R edge are the recall rate of the nodes and edges, respectively.
On the other hand, we hope to encourage the observation diversity when constructing the scene graph.
The diversity can be characterized as the number of observation viewpoints of the detected objects.
Concretely speaking, given a global scene graph GSSG which is constructed from one path, we use O to denote the set of the detected objects and calculate the diversity as:
[Equation]
where num _ viewpoints ( o ) is the number of the viewpoints about the detected object instance o .
Therefore we may formulate the score at time instant t as:
[Equation]
where the third term is used to penally the length of the path, and λ d , ρ are the corresponding weighting parameters.
In practice, we set λ node = 0 .
1 , λ p = 0 .
5 , and λ d = ρ = 0 .
According to the above deﬁnition, the immediate reward is designed to be the increment of the score r ( s t , a t ) =
p t − p t − 1 and the cumulative reward can be computed as:
[Equation]
t r ( s t (cid:48) , a t (cid:48) ) , (5) where R ( s t , a t ) represents the expected accumulated reward when agent takes action a t at state s t , the discount parameter γ is set to 0.99, and T is the length of the action and state sequence with upper bound of 40 steps, and we use SGD optimizer with a learning rate of 10 − 4 .
Support
Placement
Hanging
Position
Connecting
Attachment
Component
Statue Counter Top
Stove Burner
Sink
Table
Garbage Can
Floor
Statue Desk
Bread Counter Top Sink
Towel Towel Holder
Wall Mirror
Painting
Sofa
Toaster Coffee Machine
Cabinet Cabinet
Drawer Drawer
Faucet
Stove Burner Knob
Desk Drawer
Experimental Validation 6.1 Dataset
We choose AI2THOR [29] to generate our dataset since it supports a continuous navigation while
MP3D[30] does not, and its has higher rendering quality of images and more accurate sensors compared with Habitat[31].
We follow the guidelines in [26] to extract 16 semantic relationships, which can be clustered into the following seven categories (Fig.5).
Please refer to the supplementary material for more details.
[Table]
[Table] [Table]
6.2 Method and Metrics
To verify the effectiveness of our framework and the inﬂuence of the input vectors, we introduce the following methods for comparison:
Random
The agent randomly selects an action from action space to perform scene exploration.
PPO+Map[32]
The agent utilizes Resnet-18 to encode RGB frame, local map and global map respectively and concatenates them as the input for the LSTM to predict the next action.
Frontier-based Exploration[33]
A heuristic algorithm that guides the agent to detect and sweep to the closest frontier with path planning.
LSTM+Act+Pose[14, 15] This model is widely used in embodied tasks, it encodes RGB frame, agent position and last step action as the input of LSTM to predict the next step action for the agent.
LSSG+GSSG(Ours)
Our framework that takes LSSG and GSSG vectors as the LSTM input.
6. Traverse(Oracle)
Ask the agent to traverse every viewpoint in the scene.
To evaluate the quality of the semantic scene graph, we adopt the precision, recall rate and F1 score of the nodes and edges compared to the ground truth scene graph, denoted as P node , R node , F 1 node ,
P edge , R edge and F 1 edge respectively, and the number of steps NoS as quantitative metrics.
Note that R node could be viewed as an object level coverage, and R edge could represent the semantic richness of the generated GSSG .
We also propose the following two tasks for evaluation.
RelCls:
In this task, we align the detecting result from Mask RCNN with ground truth to obtain the correctly detected objects at each viewpoint, and offer the ground truth segmentation of each object to the agent.
Since the detection results and point cloud information are always correct, the precision of node is always 1.0.
SGGen:
In this task, the agent utilizes the raw detecting result from Mask RCNN which is consistent with realworld environment.
The agent is expected to leverage both the global scene graph and navigation policy to deal with the imperfection of Mask RCNN and the incompleteness of the collected point cloud, incrementally generating an accurate and informative scene graph.
6.3 Results 6.3.1 Quantitative results
The experimental results are shown in Table I and II, from which we have the following observations:
In both RelCls and SGGen, LSSG + GSSG outperforms other baselines on the recall rate and F1score of both the node and edge, and achieves a performance close to Traverse, which demonstrates the effectiveness of our framework.
Note that Traverse asks the agent to traverse every viewpoint in the unseen scene which always requires manual assistance in reality.
While our method fully exploits the embodied capacity of the robot to explore the scene automatically and reaches a comparable performance with much fewer NoS .
The increase in recall rate is often accompanied by a decrease in precision.
This is mainly caused by the imperfection of the Mask RCNN and Local Scene Graph Prediction Network(LSGPN) since the errors at some speciﬁc viewpoints could be hard to update and would accumulate as the agent increasingly covers the scene.
Our method achieves the highest F1-score on both tasks, which illustrates that our framework handles the balance of precision and recall rate well.
SGGen is much more difﬁcult than RelCls since both the label and the mask of the detecting result could be wrong, and the detecting error could further affect the generated LSSG and GSSG .
This explains the drop of all metrics in SGGen compared with those in RelCls.
No matter how difﬁcult the task is, the baseline methods only implement spatial exploration and adopt the same exploring path.
While our method explores the scene at a semantic level indicating that the agent is aware of
GSSG and would try to update and add the semantic information of GSSG in subsequent steps.
From table II we could discover that in RelCls, both LSSG and GSSG beneﬁt the task, and the improvement of introducing LSSG is more obvious.
This may be because the LSSG sequence encoded by LSTM contains a certain amount of GSSG information.
While in SGGen, adding GSSG vectors is worse than Base method.
We think this could be caused by the more frequently changing frequency on GSSG and its vector is very different from that in the training.
LSSG + GSSG performs best on both tasks, which illustrate the effectiveness and robustness of our framework.
Table III shows the performance of our LSSG + GSSG method under different training paradigm.
We can see that IL performs better than RL in RelCls.
In SGGen, the performance has an obvious drop as the behaviour cloning based method utilized by IL has poor generalization ability.
In unseen testing scene, especially the ones in SGGen that have huge differences in aspect of room layout and point cloud accuracy with training scenes, IL trained model tends to perform much worse.
Meanwhile, while RL trained model has a smaller performance drop in SGGen, it has the lowest recall rate over nodes and edges in RelCls and SGGen, which demonstrates that the scene has not been completely explored.
Therefore, we design the mix training paradigm that utilizes imitation learning to ﬁrst pre-train the model, and then ﬁnetunes it with reinforcement learning.
Our method (IL+RL) achieves the highest score over the recall rate and f1 score of nodes and edges on both two tasks, demonstrating its efﬁciency.
6.3.2 Qualitative Analysis and Applications
The generated semantic scene graph can be used for various downstream tasks.
And we have applied the generated semantic scene graph in a streaming video captioning task in this paper.
The caption of the video is incrementally enriched with the generated scene graph while the agent exploring the environment.
Experiment results show that the quality of the generated caption is superior to that generated with a Random exploration policy.
The detailed results and further discussion are presented in the supplementary document.
Please also visit our website https://embodiedscenegraph.vercel.app/ for more information.
Conclusion
In this paper, a novel embodied semantic scene graph generation framework is established and the hybrid imitation reinforcement learning method is developed to address this new task.
With the learned policy, the agent is able to autonomously explore the environment and then incrementally generate a high-quality semantic scene graph.
This research exploits the important embodiment characteristic of the intelligent agent and paves a new path for the agent to semantically describe the environment.
Additionally, it provides a fundamental component for some challenging downstream tasks such as streaming video captioning, robotic manipulation, visual navigation, etc.
We would also like to extend this work to large-scale scene graph generation in more complicated scenarios. Acknowledgements This work was supported in part by the National Natural Science Fund for Distinguished Young Scholars (62025304) and in part by the Seed Fund of Tsinghua University (Department of Computer
Science and Technology)-Siemens Ltd., China Joint Research Center for Industrial Intelligence and Internet of Things.