B UILDING G ENERALIZABLE A GENTS WITH A R EALISTIC AND R ICH 3D E NVIRONMENT
Anonymous authors
Paper under double-blind review
A BSTRACT
Towards bridging the gap between machine and human intelligence, it is of utmost importance to introduce environments that are visually realistic and rich in content.
In this work, we build House3D , a rich, extensible and interactive environment that contains over 45,000 human-designed 3D scenes of houses, ranging from single-room studios to multi-storeyed houses, equipped with a diverse set of fully labeled 3D objects, textures and scene layouts, based on SUNCG dataset (Song et al., 2017).
Using a subset of houses in this environment, we study the task of RoomNav in which an agent navigates towards a target speciﬁed by a high-level instruction.
For this, the agent learns to comprehend the scene it lives in by developing perception, understand the instruction by mapping it to the correct semantics, and navigate to the target by obeying the underlying physical laws.
We tackle this problem by training RL agents with gated-attention networks and show that our trained agent succeeds in new unseen environments, showing generalization capability.
In particular, we observe that (1) training is substantially harder on large house sets but with better generalization, (2) using semantic signals (e.g., segmentation mask) boosts the generalization performance, and (3) gating mechanism helps more about the training but less about the generalization.
We hope House3D, as well as the analysis of RoomNav task, acts as a step towards building a practical intelligent system and can potentially beneﬁt the community.
I NTRODUCTION
Recently, deep reinforcement learning has shown its strength on multiple games, such as Atari (Mnih et al., 2015) and Go (Silver et al., 2016), vastly overpowering human performance.
Underlying these big achievements is a well-deﬁned and efﬁcient simulated environment for the agent to freely explore and learn.
A natural question arises: will such simulated environments extrapolate to our real world?
Till now, many proposed environments have encoded some aspects of human intelligence in the realworld.
This includes 3D understanding (DeepMind Lab Beattie et al.
and Malmo (Johnson et al., 2016)), real-time strategy decision (TorchCraft (Synnaeve et al., 2016) and ELF (Tian et al., 2017)), fast reaction (Atari (Bellemare et al., 2013)), long-term planning (Go, Chess), language and communications (ParlAI (Miller et al., 2017) and (Das et al., 2017)).
However, an intelligence system in real-world requires all these aspects in a comprehensive manner.
For example (Fig. 1b), a workable home robot should be able to understand the perception, execute high-level instructions given by language, and explore and navigate with physical interactions, all at the same time.
Hence, following the aforementioned trend, even if we made substantial progress on each of these environments, what we would get are specially skilled agents, but not almighty humanlike intelligence.
Although transfer learning can be used to transfer skills, a uniﬁed environment that could evaluate all such skills at once remains an open problem.
Moreover, most of the environments miss an important aspect of real-world: the rich vision aspect.
On average, humans can effortlessly recognize and detect 100k objects and deal with their complicated interactions.
Unfortunately, many existing environments only contain a limited number of objects with compositional attributes.
In this paper, we propose House3D, a virtual 3D environment consisting of thousands of indoor scenes equipped with a diverse set of scene types, layouts and objects.
House3D is adapted from the SUNCG dataset (Song et al., 2017) which contains 45K human-designed real-world 3D house models ranging from single studio to houses with gardens.
We enhance the scenes with physics and make it interactive so that an agent could explore the space freely, perceive with a large number of objects with various visual appearance, interact with them via simple physics, and even communicate with other agents via channels like language.
These characteristics create rich visual textures in the observation space and complicated physical interactions, a step towards the real world.
House3D provides multiple modalities for training, including RGB image, ground truth segmentation masks, depth, 2D top-down map and various annotations of rooms and objects, which make it a very desirable choice for a wide range of research purposes.
House3D is orders of magnitude larger and more diverse than other similar environments like AI2-THOR (Zhu et al., 2017) which contains only 30 single rooms for training and testing.
Among all the possible applications enabled by House3D, here we focus on a high-level instruction following task, RoomNav (Fig. 1b).
In this task, an agent is randomly assigned at a location in the house and is asked to navigate to a destination described by a high-level concept (e.g., “kitchen”) without colliding with objects.
From the entire House3D, we manually select 270 houses that are suitable and large enough for navigation tasks, and split them into a small set and a large set for training and a test set for evaluation.
To train an agent on RoomNav, we propose two novel neural models with gated-attention architecture, a gated-CNN and a gated-LSTM, which effectively improve the agent’s sensitivity to different instructions.
Due to the complexity of House3D, we explore both continuous and discrete action space and apply two popular reinforcement learning algorithms, DDPG (Lillicrap et al., 2015) for continuous actions, and A3C (Mnih et al., 2016) for discrete actions.
Further, when evaluating our model’s performance, we emphasize on generalizability .
Note that most of the existing reinforcement learning works focus on the performances on a ﬁxed training environment.
However, a well-trained intelligent robot for real-world applications should be successfully deployed to new scenarios.
Indeed, there is a recent trend in the RL community focusing on the problem of generalization, ranging from learning to plan (Tamar et al., 2016), meta-learning (Duan et al., 2016;
Finn et al., 2017) to zero-shot learning (Oh et al., 2017; Higgins et al., 2017).
But these works either focus on simpliﬁed tasks or test environments which only slightly vary from the training ones.
In this paper, we randomly split and preserve a held-out test set of houses on which we examine the agent’s navigation success rate.
All the environments in the test set are completely different from the training houses and are never presented to the agent during training.
In our experiments, we notice that a larger training set leads to higher generalization ability, which suggests the importance of divergence of simulated scenery.
We also experiment with different visual inputs to the agent and conclude that the visual signal leading to immediate action (depth) or the semantic signal (segmentation) can lead to better generalization.
This empirical observation indicates the signiﬁcance of building a practical vision system for real-world robots and also suggests the separation of vision and robotic learning when handling complicated real-world tasks.
The remaining of the paper is structured as follows.
Section 2 is a summary of relevant work.
Section 3 describes our environment, House3D, in detail and Section 4 describes the task, RoomNav.
Section 5 introduces our gated-networks and the applied algorithms.
Finally, experiment results are shown in Section 6.
R ELATED W ORK
The development of environments largely pushes the frontier of reinforcement learning.
Tablesummarizes the attributes of some of the most relevant environments and compares them to our
House3D. Other than the family of environments closely related to House3D, there are more simulated environments, such as OpenAI Gym (Brockman et al., 2016), ParlAI (Miller et al., 2017) for language communication as well as some strategic game environments (Synnaeve et al., 2016; Tian et al., 2017; Vinyals et al., 2017).
Most of these environments are specialized in one particular aspect of intelligence, such as game or dialogue, and make it hard to facilitate the study of more problems.
On the contrary, we focus on building a ﬂexible platform that intersects with multiple research directions in an efﬁcient manner allowing users to customize the rules and level of complexity to their needs.
3D scenes 3D environment … … … … … … … …
RGB image
Semantic masks
Depth
Top-down 2D view tt t+ tt t+ tt t+1 tt t+ (a) House3D environment
Input at t
Instruction go to kitchen a t
Environment E r t Policy 𝜋𝜋 𝑎𝑎 ; 𝑠𝑠 𝑡𝑡 | 𝜃𝜃
Input at t+
Policy 𝜋𝜋 𝑎𝑎 ; 𝑠𝑠 𝑡𝑡+1 𝜃𝜃 a t+ r t+
Input at t+
Instruction go to kitchen
Agent
Environment go to kitchen
Success ……
Environment E
[Figure]
Environment 3D
Realistic
Large-scale
Fast-speed
Customizable
[Table] [Table]
[Table] [Table]
[Table] [Table]
real-world, Large-Scale : a large set of environments,
Fast-speed : fast rendering speed and Customizable : ﬂexibility to be customized to any desired applications.
There has been noticeable efforts to tackle navigation in 3D environments using reinforcement learning.
Some of the most prominent lines of work include the use of auxiliary tasks to improve training speed (Mirowski et al., 2016; Jaderberg et al., 2016), the use of successor representations (Kulkarni et al., 2016), memory (Parisotto & Salakhutdinov, 2017) and self-supervision (Pathak et al., 2017) in VizDoom (Kempka et al., 2016).
In our work, the goal is not speciﬁed by the environment but is communicated to the agent via a highlevel instruction.
To modulate the behavior of the agent, we encode the instruction as an embedding vector which gates the visual signal.
The idea of gated attention has been used in the past for language grounding (Chaplot et al., 2017), and transfer learning by language grounding (Narasimhan et al., 2017).
Similar to those works, we use language grounding as an attention mechanism.
However, the effects of the grounding are directed to impact the agent’s overall perception and thus future actions towards the goal.
For stronger generalization performance and efﬁcient learning, we rely on multi-task training.
A similar strategy is followed by Tessler et al.
; Andreas et al.
Oh et al.
Teh et al.
In addition and similar to other works which use diverse input modalities (Mirowski et al.,
; Tai & Liu, 2016), we show improved generalization performance in complex 3D scenes when using depth and segmentation masks on top of the raw visual input.
This observation suggests that it is possible to decouple real-world robotics from vision via a vision API.
The vision API trained in the desired scenes, e.g. an object detection or semantic segmentation system, can bridge the gap between simulated environment and real-world (Tobin et al., 2017; Rusu et al., 2016; Christiano et al., 2016).
H OUSE 3D: A N E XTENSIBLE E NVIRONMENT OF 45K 3D H OUSES
Towards building an ultimately practical AI system, there is a need for a realistic environment rich in content and structure which closely resembles the real world.
Such an environment can serve as the testbed for various problems which require visual understanding, language grounding, concept learning and more.
For the environment to be of value, it is important that it obeys the physical laws of the world, carries a complex visual signal and is of large enough scale to enable learning and generalization.
For this, we develop an environment of thousands of indoor scenes, an overview of which is shown in Figure 1a.
An agent is able to navigate inside complex structures which consist of a diverse set of layouts, objects and rooms and are accompanied by useful semantic labels.
3. D ATASET
The 3D scenes are sourced from the SUNCG dataset (Song et al., 2017), which consists of 45, human-designed 3D scenes ranging from single-room studios to multi-ﬂoor houses.
The SUNCG dataset was designed to encourage research on large-scale 3D object recognition problems and thus carries a variety of objects, scene layouts and structures.
On average, there is 8.9 rooms and 1.
ﬂoors per scene with the max being 155 rooms and 3 ﬂoors, respectively.
There is a diverse set of room and object types in each scene.
In total, there is over 20 different room types, such as bedroom, living room, kitchen, bathroom etc., with over 80 different object categories.
On average, there isdifferent objects in each room.
In total, the SUNCG dataset contains 404,508 different rooms and 5,697,217 object instances drawn from 2644 unique object meshes.
3. A NNOTATIONS
SUNCG is accompanied with a diverse set of labels for each scene.
Based on these labels, an agent at every time step t can have access to the following signals in our environment: a) the visual signal of its current ﬁrst-person view, consisting of RGB values, b) semantic segmentation masks for all the objects visible in its current view and c) depth information.
These signals are greatly desired since they enable thorough exploration to determine their practical value for different tasks or can serve as a predictive target when learning models.
For example, for the task of navigation one can easily swap the RGB values with depth information or with semantic masks in order to quantify the importance of these different input signals.
Other than the visual input, each scene in SUNCG is fully annotated with the 3D location and size of all rooms and object instances in the form of a 3D bounding box.
Rooms and objects are also marked with their type, e.g. bedroom or shoe cabinet respectively.
This information allows for a detailed mapping from each 3D location ( x, y, z ) to an object instance (or None if the space is unoccupied), as well as room type.
Furthermore, more features can be built on top of these existing annotations, such as top-down 2D occupancy maps, connectivity analysis and shortest paths, or any arbitrary physics, all potentially helpful for a variety of applications.
3. B UILDING AN I NTERACTIVE E NVIRONMENT 3.3. R ENDERER
To build a realistic 3D environment, we write a renderer for the 3D scenes from SUNCG, and deﬁne actions that obey simple physical rules for an agent to navigate in the scenes.
The renderer is based on OpenGL and can run on Linux and macOS with both CPU and GPU.
The renderer provides not only RGB images, but also segmentation masks and depth maps.
The renderer needs to be efﬁcient to be used for large-scale RL algorithms.
On a NVIDIA Tesla
M40 GPU, it can render 120x90 RGB frames at over 600 fps, while multiple renderers can run in parallel on one or more GPUs.
When rendering multiple houses simultaneously, one M40 GPU can be fully utilized to render at a total of 1800 fps.
The simple physics, currently written in Python, add a small overhead to the compute, which is negligible when used with multi-processing.
The high throughput of our implementation enables efﬁcient learning for a variety of interactive tasks, such as on-policy reinforcement learning.
We will publicly release the environment along with a python API for easy use.
3.3. P HYSICS
For the environment to be realistic, we incorporate physical laws to the SUNCG scenes based on occupancy.
Particularly, an agent can live in any location ( x, y, z ) within a 3D scene, as long as within a small range, i.e., robot’s radius, it is not collided with any object instance (including walls).
Doors, gates and arches are considered passage ways for the agent, meaning that an agent can walk through those structures freely.
These design choices closely follow the behaviour of a real robot navigating inside a house: a robot can not walk through walls or other objects but can pass freely through free space, including doors and gates, in order to reach different parts of the scene.
R OOM N AV : A B ENCHMARK T ASK OF M ULTI -T ARGET R OOM N AVIGATION
Consider a natural use case of a home robot in Figure 1b.
A human may give a high level instruction to the robot, for example, “Go to the kitchen”, so that one can later ask the robot to turn on the oven.
The robot needs to behave appropriately conditioned on the house it is located in as well as the instruction from the human.
Moreover, a practical commercial home robot also needs the ability of generalization : the robot can be intensively trained for an arbitrarily long time by its manufacturer but it needs to be immediately deployed to the user’s home, which can be of a completely different house design and indoor decorations from the training environments.
The key challenges for building real-world home robots are: (1) understanding of the scene from its visual sensors; (2) execution of high-level instructions; (3) safe navigation and exploration in complex indoor environments; (4) generalization to unseen scenes.
To study the aforementioned abilities of a reinforcement learning agent, we develop a new benchmark task, Multi-Target Room Navigation (RoomNav) , based on our House3D environment.
To alleviate the difﬁculty of the task, we restrict the high-level instruction to be of the form “go to
RoomType ”, where RoomType denotes some pre-deﬁned room type.
To ensure fast experimentation cycles, we perform experiments on a subset of House3D. We manually select 270 houses suitable for navigation task and split them into a small set (20 houses), a large set (200 houses) and a test set (50 houses), where the test set is used to evaluate the generalizability of the trained agents.
4. T ASK F ORMULATION
Suppose we have a set of episodic environments E = { E 1 , . . .
, E n } and a set of targeted instructions
I =
{ I 1 , . .
, I m } .
During each episode, the agent is interacting with one environment E ∈ E and is given a targeted instruction I ∈ I .
In the beginning of an episode, the agent will be randomly placed somewhere in E .
At each time step t , the agent receives a visual signal X t from E via its ﬁrst person view sensor.
Let s t = { X 1 , . . .
, X t , I } denote the state of the agent at time t .
The agent needs to propose action a t to navigate and rotate its sensor given s t .
The environment will give out a reward signal r t and terminates when it succeeds or times out.
The objective of this task is to learn an optimal policy π ( a t
| s t , I ) that could lead to the targeted room according to I .
During training, we train the agent in a training set of environments E train .
For measuring the generalization of the learned policy, we will also evaluate the policy in another set of environments E test , such that E test ∩ E train = ∅ .
4. D ETAILED S PECIFICATIONS
Here we provide further details of the task and there are more to ﬁnd in appendix.
[Table]
Environment Statistics: The selected 270 houses are manually veriﬁed to be suitable for navigation task: e.g., they are well connected, contain many targets, and are large enough for exploration (studios excluded).
We split them into 3 disjoint sets, denoted by E small , E large and E test respectively.
For targets, we select the ﬁve most common room types, i.e., kitchen, living room, dining room, bedroom and bathroom.
The detailed statistics are shown in Table 2.
Observation and Action: We utilize three different kinds of visual input signals for X t , including (1) raw pixel values; (2) segmentation mask of the pixel input; and (3) depth information, and experiment with different combinations of them.
For each instruction I , we provide a one-hot vector encoding correspondingly.
The majority of the existing navigation works operate on a discrete action space.
In this work, we predeﬁne 12 discrete actions including rotations and movement.
In addition, due to the complexity of the indoor scenes as well as the fact that real robots may navigate towards any direction with any (bounded) velocity, we also investigate a continuous action space similar to Lowe et al.
See the appendix for more details.
For both the discrete and continuous action space, if the agent hits any obstacles, it will remain still.
Success Measure and Reward Function:
To deﬁne “success” for the task, we want to ensure that the agent identiﬁes the room due to its unique properties (e.g., pan and knives for kitchen and bed for bedroom) instead of merely reaching there by luck.
An episode is considered successful if the agent achieves both of the following two criteria: (1) the agent is located inside the target room; (2) the agent consecutively “sees” a designated object category associated with that target room type for at least 2 time steps.
We assume that an agent “sees” an object if there are at least 4% of pixels in
X t belonging to that object.
Regarding the reward function, ideally two signals are enough to reﬂect the task requirement: (1) a collision penalty when hitting obstacles; and (2) a success reward when completing the instruction.
However, this basic reward function makes it too difﬁcult for an RL agent to learn.
Note that our ultimate goal is to learn a policy that could generalize to unseen houses at test time.
Thus, during training we resort to an informative reward shaping in order to provide additional supervision: we compute the approximate shortest distance from the target room to each location in the house and adopt the difference of shortest distances between the agent’s movement as an additional reward signal.
G ATED -A TTENTION N ETWORKS FOR M ULTI -T ARGET L EARNING
In this section, we propose two baseline models with the gated-attention architecture, similar to Dhingra et al.
(2016) and Chaplot et al.
(2017), for better policy representation with instructions: a gated-CNN network for continuous actions and a gated-LSTM network for discrete actions.
We train the gated-CNN policy using the deep deterministic policy gradient (DDPG) (Lillicrap et al., 2015), while we train the gated-LSTM policy using the asynchronous advantage actor-critic algorithm (A3C) (Mnih et al., 2016).
5. DDPG WITH GATED -CNN POLICY 5.1. D EEP DETERMINISTIC POLICY GRADIENT
DDPG is a widely used ofﬂine-learning algorithm for continuous action space (Lillicrap et al., 2015).
Suppose we have a deterministic policy µ ( s t | θ ) (actor) and the Q-function Q ( s t , a | θ ) (critic) both
[Figure] [Figure]
Gated
[Figure]
Embedding
MLP 𝜇𝜇 ( 𝑠𝑠 𝑡𝑡 | 𝜃𝜃 )
Gated Fusion
MLP 𝑄𝑄 ( 𝑠𝑠 𝑡𝑡 , 𝑎𝑎 𝑡𝑡 | 𝜃𝜃 )
[Figure]
CNN 𝑥𝑥 𝑡𝑡− 𝑥𝑥 𝑡𝑡
Gated Fusion ……
LSTM
MLP 𝑣𝑣 ( 𝑠𝑠 𝑡𝑡 | 𝜃𝜃 ) 𝜋𝜋 ( 𝑎𝑎 ; 𝑠𝑠 𝑡𝑡 | 𝜃𝜃 )
Gated
[Figure]
for DDPG with continuous actions
[Figure] [Figure]
…… 𝑎𝑎 𝑡𝑡 parametrized by θ .
DDPG optimizes the policy µ ( s t | θ ) by maximizing
[Equation]
and updates the Q-function by minimizing
[Equation]
Here, we use a shared network for both actor and critic, which leads to the ﬁnal loss function
L DDPG ( θ ) =
− L µ ( θ ) + α DDPG L Q ( θ ) , where α DDPG is a constant balancing the two objectives.
5.1. G ATED -CNN NETWORK FOR CONTINUOUS POLICY State Encoding:
Given state s t , we ﬁrst channel-wise stack the most recent k frames X
=
[ X t , X t − 1 , . . .
, X t − k +1 ] and apply a convolutional neural network to derive an image representation x = f cnn ( X | θ ) ∈ R d X .
For the instruction, we convert it into an embedding vector y =
f embed ( I | θ ) ∈ R d
I .
Subsequently, we apply a fusion module M ( x, y | θ ) to derive the ﬁnal encoding h s =
M ( x, y | θ ) .
Gated-Attention for Feature Fusion: For the fusion module M ( x, y | θ ) , the straightforward version is concatenation, namely M cat ( x, y |· ) =
[ x, y ] .
In our case, x is always a high-dimensional feature vector (i.e., image feature) while y is a simple low-dimensional conditioning vector (e.g., instruction).
Thus, simple concatenation may result in optimization difﬁculties.
For this reason, we propose to use a gated-attention mechanism.
Suppose x ∈ R d x and y ∈ R
d y where d y < d x .
First, we transform y to y (cid:48)
∈
R d x via an MLP, namely y (cid:48) =
f mlp ( y | θ ) , and then perform a Hadamard product between x and sigmoid ( y (cid:48) ) , which leads to our ﬁnal gated fusion module
M ( x, y | θ ) =
x (cid:12) sigmoid ( f mlp ( y | θ )) .
This gated fusion module could also be interpreted as an attention mechanism over the feature vector, which could help better shape the feature representation.
Policy Representation:
On the state representation h s , we apply an MLP layer followed by a softmax operator (for bounded velocity) to produce the continuous action.
Moreover, in order to produce a stochastic policy for better exploration, we apply the softmax-gumbel trick (Jang et al., 2016), resulting in the ﬁnal policy µ ( s t | θ ) =
gumbel-softmax ( f mlp ( h s | θ )) .
Note that since we add randomness to µ ( s t | θ ) , our DDPG formulation can be also interpreted as the SVG(0) algorithm
(Heess et al., 2015).
Q-function: The Q-function Q ( s, a ) conditions on both state s and action a .
We again apply a gated fusion module to the feature vector x
and the action vector a and derive a hidden representation h Q
= M ( x, a | θ ) .
We eventually apply another MLP to h Q to produce the ﬁnal value Q ( s, a ) .
A model demonstration is shown in the top part of Fig.
2, where each block has its own parameters.
5. A3C WITH GATED -LSTM POLICY 5.2. A SYNCHRONOUS ADVANTAGE ACTOR - CRITIC
A3C is a variant of policy gradient algorithm introduced by Mnih et al.
(2016), which reduces the variance of policy gradient by jointly updating a value function as the baseline.
Suppose we have a discrete policy π ( a ; s | θ ) and a value function v ( s | θ ) .
A3C optimizes the policy by minimizing the loss function
[Equation]
where R t is the discounted accumulative reward deﬁned by R t =
(cid:80) T − t i =
γ i r t
+ i + v ( s T +1 ) .
The value function is updated by minimizing the loss
[Equation]
Finally the overall loss function for A3C is
[Equation]
where α A3C is a constant coefﬁcient.
5.2. G ATED -LSTM NETWORK FOR DISCRETE POLICY State Encoding:
Given state s t , we ﬁrst apply a CNN module to extract image feature x t for each input frame X t .
Similarly, for instruction, we apply a gated fusion module to derive a state representation h t =
M ( x t , I | θ ) at each time step t .
Then, we concatenate h t with the instruction I and the result is fed into the LSTM module (Hochreiter & Schmidhuber, 1997) to obtain a sequence of LSTM outputs { o t } t , so that the LSTM module has direct access to the instruction other than the attended visual feature.
Policy and Value Function:
For each time step t , we concatenate the state vector h t with the output of LSTM o t to obtain a joint hidden vector h joint =
[ h t , o t ] .
Then we apply two MLPs to h joint to obtain the policy distribution π ( a ; s t | θ ) as well as the value function v ( s t | θ ) .
A visualization of the model is shown in the bottom part of Fig.
Note that the parameters of CNN module are shared across time.
E XPERIMENT R ESULTS
We report experimental results for our models on the task of RoomNav under different sized training sets.
We compare models with discrete and continuous action spaces and empirically show the effect of using different input modalities.
6. D ETAILS
We train our CNN model with DDPG and LSTM model with A3C on both the small set ( E small ,houses) and the large set ( E large , 200 houses), and examine both the training performances (success rate) on the training set as well as the generalization performances on the test set ( E test , 50 houses).
All the success rate evaluation uses a ﬁxed random seed for a fair comparison.
In all the cases, if the agent cannot accomplish the goal within 100 steps, we terminate the episode and declare failure.
We use gated-CNN and gated-LSTM to denote the networks with gated-attention and concat-CNN and concat-LSTM for models with simple concatenation.
We also experiment with different visual signals to the agents, including RGB image (RGB Only), RGB image with depth information
[Figure]
(RGB+Depth) and semantics mask with depth information (Mask+Depth).
To preserve the richness of the input visual signal, the resolution of the input image is 120 × 90 .
During each simulated episode, we randomly select a house from the environment set and randomly pick an available target from the house to instruct the agent.
Each episode will be forced to terminate after 100 time steps.
During training, we add an entropy bonus term for both models in addition to the original loss function.
For evaluation, we keep the ﬁnal model for DDPG due to its stable learning curve, while for A3C, we take the model with the highest training success rate.
Optimization is performed via Adam (Kingma & Ba, 2014) while the implementation is done in PyTorch (Paszke et al., 2017).
More model and experiment details can be found in appendix.
6. T RAINING ON THE SMALL SET WITH 20 HOUSES
Training Performances:
For each of the models, we run 2000 evaluation episodes on E small and measure overall the success rate.
The results are summarized in Fig. 3a.
Our empirical results show that the gated models achieve higher success rate than the models without gated-attention architecture.
Especially for the DDPG case, the gated-CNN model outperforms the concat-CNN model with a large margin.
We believe this is due to the fact that there are two gated-fusion modules utilized in gated-CNN model.
Notably, the simpler CNN model with DDPG has stable learning curve and even achieves higher success rate than LSTM with all different input signals, which suggests that simpler
CNN models can be an appropriate candidate for ﬁtting a small set of environments.
Regarding the different combinations of input signals, adding a depth channel to the RGB channel generally improves the training performances.
Furthermore, changing the RGB signal to semantic signal signiﬁcantly boosts the performances for all the models.
Generalization Performances on E test : Here, the models trained on E small are evaluated on the test set E test to measure their generalization ability.
We run 2000 episodes and measure the success rate of each model.
The results are shown in Fig.
3b.
Regarding different types of visual signals, we draw the same conclusions as at training time: depth improves the performances and semantic information contributes to generalizability the most.
More importantly, we observe that the generalization performance is drastically worse than the training performance, especially for the cases with RGB input: the gated-LSTM models achieve even lower success rate than concat-LSTM models at test time despite the fact that they have much better training performances.
This indicates that the neural model tends to overﬁt to the environments when we only have a small training set, while having a semantic input signal could slightly alleviate the issue of overﬁtting.
[Figure]
6. T RAINING ON THE LARGE SET WITH 200 HOUSES
Here we train our models on the large set E large containing 200 different houses.
For visual signals, we focus on “RGB + Depth” and “Mask + Depth”.
For evaluation, we measure the training success rate over 5000 evaluation episodes and the test success rate over 2000 episodes.
Both train and test results are summarized in Fig.
Training Performances:
Compared to the performance on the small training set, the training performances on the large set drops signiﬁcantly, especially for the models using RGB signal.
We also notice that in the case of RGB input, gated models perform similar to concat-models.
This suggests fundamental difﬁculties for reinforcement learning agents to learn within a large set of diverse and visually rich environments.
Whereas, for models with semantic signals, we observe a huge gain on training performances as well as the beneﬁts by having a gated-attention architecture.
This implies that a semantic vision system can be potentially an effective component for building a real-world robotic system.
In addition, we observe that on the large set, even with a relatively unstable algorithm A3C, the models with larger capacity, i.e. LSTMs, considerably outperform the simpler reactive models, i.e.,
CNNs.
We believe this is due to the larger scale and the high complexity of the training set, which makes it almost impossible for an agent to “remember” the optimal actions for every scenario.
Instead, an agent needs to develop high-level abstract strategies, such as exploration, and memory.
This also suggest a potential future direction of persuading the agent to learn generalizable abstractions by introducing more inductive bias into the model.
Generalization Performances: Regarding generalization, as we can see from Fig.
4b, after training on a large number of environments, every model now has a much smaller gap between its training performance and test performance.
For the models with semantic input signals, their test performances are largely improved compared to those trained on the small set.
This again emphasizes the importance of having a large set of diverse environments for training generalizable agents.
Lastly, we also analyze the detailed success rate with respect to each target instruction in appendix.
C ONCLUSION
In this paper, we design a new environment, House3D, which contains 45K of houses with a rich set of objects as well as natural layouts resembling the real-world.
House3D is a ﬂexible and extensible environment that could be used for a variety of applications.
Based on our House3D environment, we further introduce a multi-target navigation task, RoomNav, which tests an agent’s intelligence including understanding the instruction, interpreting the comprehensive visual signal, navigation and, most importantly, generalization.
We develop two novel models, gated-CNN and gated-LSTM, with the gated-attention architecture, which signiﬁcantly helps improve the training performances.
In our experiments we observe that using the semantic signal as the input considerably enhances the agent’s generalizability.
Increasing the size of the training environments is important but at the same time introduces fundamental bottlenecks when training agents to accomplish the RoomNav task due to the higher complexity of the underlying task.
We believe our environment will beneﬁt the community and facilitate the efforts towards building better AI agents.
We also hope that our initial attempts towards addressing generalizability in reinforcement learning will serve as an important step towards building real-world robotic systems
A R OOM N AV T ASK D ETAILS
The location information of an agent can be represented by 4 real numbers: the 3D location ( x, y, z ) and the rotation degree ρ of its ﬁrst person view sensor, which indicates the front direction of the agent.
Note that in RoomNav, the agent is not allowed to change its height z , hence the overall degree of freedom is 3.
An action can be in the form of a triple
a = ( δ x , δ y , δ ρ ) .
After taking the action a , the agent will move to a new 3D location ( x + δ x , y + δ y , z ) with a new rotation ρ + δ ρ .
The physics in House3D will detect collisions with objects under action a and in RoomNav, the agent will remain still in case of a collision.
We also restrict the velocity of the agent such that | δ x | , | δ y | ≤ 0 .
5 and | δ ρ | ≤ 30 to ensure a smooth movement.
Continuous Action
A continuous action a consists of two parts a = [ m, r ] where m = ( m 1 , . .
, m 4 ) is for movement and r = ( r 1 , r 2 ) is for rotation.
Since the velocity of the agent should be bounded, we require m , r to be a valid probability distribution.
Suppose the original location of robot is ( x, y, z ) and the angle of camera is ρ , then after executing a , the new 3D location will be ( x + ( m 1 − m 2 ) ∗ 0 .
5 , y + ( m 3 − m 4 ) ∗ 0 .
5 , z ) and the new angle is ρ
+ ( r 1 − r 2 ) ∗ 30 .
Discrete Actions: We deﬁne 12 different action triples in the form of a i =
( δ x , δ y , δ ρ ) satisfying the velocity constraints.
There are 8 actions for movement and 4 actions for rotation.
In the discrete action setting, we do not allow the agent to move and rotate simultaneously.
Reward Details:
In addition to the reward shaping of difference of shortest distances, we have the following rewards.
When hitting an obstacle, the agent receives a penalty of 0.3.
In the case of success, the winning reward is +10.
In order to encourage exploration (or to prevent eternal rotation), we add a time penalty of 0.1 to the agent for each time step outside the target room.
Note that since we restrict the velocity of the agent, the difference of shortest path after an action will be no more than 0 .
5 × √ 2 ≈ 0 . 7 .
B E XPERIMENT D ETAILS B. N ETWORK ARCHITECTURES
We apply a batch normalization layer after each layer in the CNN module.
The activation function used is ReLU.
The embedding dimension of instruction is 25.
Gated-CNN: In the CNN part, we have 4 convolution layers of 64, 64, 128, 128 channels perspective and with kernel size 5 and stride 2, as well as a fully-connected layer of 512 units.
We use a linear layer to transform the instruction embedding to a 512-dimension vector for gated fusion.
The
MLP for policy has two hidden layers of 128 and 64 units, and the MLP for Q-function has a single hidden layer of 64 units.
Gated-LSTM:
In the CNN module, we have 4 convolution layers of 64, 64, 128, 128 channels each and with kernel size 5 and stride 2, as well as a fully-connected layer of 256 units.
We use a linear layer to convert the instruction embedding to a 256-dimension vector.
The LSTM module hashidden dimensions.
The MLP module for policy contains two layers of 128 and 64 hidden units, and the MLP for value function has two hidden layers of 64 and 32 units.
B. T RAINING PARAMETERS
We normalize each channel of the input frame to [0 , 1] before feeding it into the neural network.
Each of the training procedures includes a weight decay of 10 − 5 and a discounted factor γ = 0 .
DDPG: We stack k = 5 recent frames and use learning rate 10 4 with batch size 128.
We choose α DDPG = 100 for all the settings except for the case with input signal of “RGB+Depth” on E large , where we choose α DDPG = 10 .
We use an entropy bonus term with coefﬁcient 0.001 on E small and 0.01 on E large .
We use exponential average to update the target network with rate 0.001.
A training update is performed every 10 time steps.
The replay buffer size is 7 × 10 5 .
We run training for 80000 episodes in all.
We use a linear exploration strategy in the ﬁrst 30000 episodes.
A3C:
We clip the reward to the range [ − 1 , 1] and use a learning rate 1 e − 3 with batch size 64.
We launch 120 processes on E small and 200 on E large .
During training we estimate the discounted accumulative rewards and back-propagate through time for every 30 time steps unrolled.
We perform a gradient clipping of 1.0 and decay the learning rate by a factor of 1.5 when the difference of KLdivergence becomes larger than 0.01.
For training on E small , we use a entropy bonus term with coefﬁcient 0.1; while on E large , the coefﬁcient is 0.05.
α A3C is 1.0.
We perform 10 5 training updates and keep the best model with the highest training success rate.
B. G ENERALIZATION OVER DIFFERENT TARGETS
[Table]
We illustrate in Table 3 the detailed test success rates of our models trained on E train with respect to each of the 5 targets.
Note that both models have similar behaviour across instructions.
In particular, “dining room” and “living room” are the easiest while “bathroom” is the hardest.
We suspect that this is because dining room and living room are often with large room space and have the best connectivity to other places.
By contrast, bathroom is often very small and harder to ﬁnd in big houses.
Lastly, we also experiment with adding auxiliary tasks of predicting the current room type during training.
We found this does not help the training performance nor the test performance.
We believe it is because our reward shaping has already provided strong supervision signals.