C ROSS -T ASK K NOWLEDGE T RANSFER FOR V ISUALLY -G ROUNDED N AVIGATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent efforts on training visual navigation agents conditioned on language using deep reinforcement learning have been successful in learning policies for two different tasks: learning to follow navigational instructions and embodied question answering.
In this paper, we aim to learn a multitask model capable of jointly learning both tasks, and transferring knowledge of words and their grounding in visual objects across tasks.
The proposed model uses a novel Dual-Attention unit to disentangle the knowledge of words in the textual representations and visual objects in the visual representations, and align them with each other.
This disentangled task-invariant alignment of representations facilitates grounding and knowledge transfer across both tasks.
We show that the proposed model outperforms a range of baselines on both tasks in simulated 3D environments.
We also show that this disentanglement of representations makes our model modular, interpretable, and allows for transfer to instructions containing new words by leveraging object detectors
I NTRODUCTION
Deep reinforcement learning has been shown to be capable of achieving super-human performance in playing games such as Atari 2600 (Mnih et al., 2013) and Go (Silver et al., 2016).
Following the success of deep reinforcement learning in 3D Games such as Doom (Lample & Chaplot, 2017;
Dosovitskiy & Koltun, 2017) and DeepmindLab (Mnih et al., 2016), there has been increased interest in using deep reinforcement learning for training embodied agents, which interact with a 3D environment by receiving ﬁrst-person views of the environment and taking navigational actions.
The simplest navigational agents learn a particular behaviour such as collecting or avoiding particular objects (Kempka et al., 2016; Jaderberg et al., 2016; Mirowski et al., 2016) or playing deathmatches (Lample & Chaplot, 2017; Dosovitskiy & Koltun, 2017).
Subsequently, there have been efforts on training navigational agents whose behaviour is conditioned on a target speciﬁed using images (Zhu et al., 2017) or coordinates (Gupta et al., 2017a; Savva et al., 2017).
More recently, there has been much interest in training agents conditioned on language as it offers several advantages over using images or coordinates.
Firstly, the compositionality of language allows generalization to new tasks without additional learning.
Prior work (Oh et al., 2017; Hermann et al., 2017; Chaplot et al., 2017) has trained navigational agents to follow instructions and shown zero-shot generalization to new instructions which contain unseen composition of words seen in the training instructions.
Secondly, language is also a convenient means for humans to communicate with autonomous agents.
Language not only allows instruction but also interaction.
Gordon et al.
and Das et al.
(2017) train agents to answer questions by navigating in the environment to gather the required information.
These multimodal tasks involve several challenges, such as perception from raw pixels, grounding of words in the instruction or question to visual objects and attributes, reasoning to perform relational tasks, ﬁne-grained navigation in 3D environments with continuous state space, and learning to answer questions.
Training a multi-task model can also facilitate knowledge transfer between the tasks and allow the model to generalize to scenarios which were not possible with single tasks.
For example, if an agent learns to follow the instruction ‘Go to the red pillar’ and answer the question ‘What color is the torch?’
, then it should also be able to follow the instruction ‘Go to the red torch’ and answer the question ‘What color is the pillar?’
without any additional training.
In this paper, we aim to train a multi-task navigation model to follow instructions and answer questions.
To test the generalization of multi-task models, we deﬁne cross-task knowledge transfer as an
[Figure]
evaluation criteria, evaluating zero-shot learning on instructions and questions consisting of unseen composition of words in both tasks.
In order to achieve cross-task knowledge transfer, words in the input space of both tasks need to be aligned with each other and with the answer space while they are being grounded to visual objects and attributes.
Prior models fail to achieve this as they are designed for a single task.
We propose a novel dual-attention model involving sequential Gated- and
Spatial-Attention operations to perform explicit task-invariant alignment between the image representation channels and the words in the input and answer space.
We create datasets and simulation scenarios for testing cross-task knowledge transfer in the Doom environment (Kempka et al., 2016) and show that the proposed model outperforms a range of baselines on both tasks.
Additionally, we demonstrate that the modularity of our model allows easy addition of new objects and attributes to a trained model.
We plan to open-source the implementation of our proposed model as well as the datasets and simulation environments.
R ELATED W ORK
This paper is motivated by a series of works on learning to follow navigation instructions (Oh et al., 2017; Hermann et al., 2017; Chaplot et al., 2017; Wu et al., 2018; Yu et al., 2018a) and learning to answer questions by navigating around the environment (Das et al., 2017; Gordon et al., 2018).
Among methods learning from instructions in 3D environments, Oh et al.
(2017) introduced a hierarchical RL model for learning sequences of instructions by learning skills to solve subtasks.
Chaplot et al.
(2017) introduced a gated-attention model for multimodal fusion of textual and visual representations using multiplicative interactions, whereas Hermann et al.
(2017) introduced auxiliary tasks such as temporal autoencoding and language prediction to improve sample efﬁciency for this task.
Yu et al.
(2018a)
proposed guided feature transformation which involves transformation of visual representations using latent sentence embeddings computed from the language input.
Among models for embodied question answering, Das et al.
(2017) introduced a hierarchical model consisting of 4 modules, each for processing images, encoding questions, navigation, and questionanswering, each of which is pretrained with supervised or imitation learning, followed by ﬁne-tuning of the navigation model using reinforcement learning.
Gordon et al.
(2018) introduced the task of Interactive Question Answering which involves interacting with objects in the environment with non-navigational actions for answering questions.
They proposed Hierarchical Interactive Memory
Network (HIMN), which allows temporal abstraction using a factorized set of controllers.
All of the above methods are designed for a single task, following navigational instructions or answering questions, whereas we aim to train a single model for both tasks.
Yu et al.
(2018b) introduced a model for interactive language acquisition by training on both Visual Question Answering and following instructions in a 2D grid world environment.
We aim to tackle multimodal multitask learning in challenging 3D environments.
Partial observability results in the requirement of learning to navigate for answering the questions, turning visual question answering to embodied question answering.
3D environments also allow us to test on interesting and more challenging instructions based on relative size of the objects, in addition to colors and types.
In addition to the above, there is a huge body of work on multimodal learning in static settings which do not involve navigation or reinforcement learning.
Some relevant works which use attention mechanisms similar to the ones used in our proposed model include Perez
[Figure]
P ROBLEM F ORMULATION
Consider an autonomous agent interacting with an episodic environment as shown in Figure 1.
In the beginning of each episode, the agent receives a textual input T specifying the task that it needs to achieve.
For example, T could be an instruction describing the target object or a question querying about some visual detail of objects in the environment.
At each time step t , the agent observes a state s t
=
( I t , T ) where I t is the ﬁrst-person (egocentric) view of the environment, and takes an action a t , which could be a navigational action or an answer action.
The agent’s objective is to learn a policy π ( a t
| s t ) which leads to successful completion of the task speciﬁed by the textual input T .
Tasks .
We focus on the multi-task learning of two visually-grounded language navigation tasks:
In
Embodied Question Answering (EQA) , the agent is given a question (“What color is the torch?”), and it must navigate around the 3D environment to explore the environment and gather information to answer the question (“red”).
In Semantic Goal Navigation (SGN) , the agent is given a language instruction (“Go to the red torch”) to navigate to a goal location.
Environments .
We adapt the ViZDoom (Kempka et al., 2016)-based language grounding environment proposed by Chaplot et al.
(2017) for visually-grounded multitask learning.
It consists of a single room with 5 objects.
The objects are randomized in each episode based on the textual input.
We use two difﬁculty settings for the Doom domain: Easy : The agent is spawned at a ﬁxed location.
The candidate objects are spawned at ﬁve ﬁxed locations along a single horizontal line in the ﬁeld of view of the agent.
Hard : The candidate objects and the agent are spawned at random locations and the objects may or may not be in the agents ﬁeld of view in the initial conﬁguration.
The agent must explore the map to view all objects.
Datasets .
We use the set of instructions from Chaplot et al.
and create a dataset for questions using the same set of objects and attributes.
We deﬁne cross-task knowledge transfer as an evaluation criteria for testing generalization of multi-task models.
We create train-test splits for both instructions and questions datasets to explicitly test a multitask model’s ability to transfer the knowledge of words across different tasks.
Each instruction in the test set contains a word that is never seen in any instruction in the training set but is seen in some questions in the training set.
Similarly, each question in the test set contains a word never seen in any training set question.
Figure 1 illustrates the train-test split of instructions and questions used in our experiments in the Doom domain.
Note that for the EQA trainset, unseen words can be present in the answer.
The agent can take 4 actions: 3 navigational actions (forward, left, right) and 1 answer action.
When the agent takes the answer action, the answer with the maximum probability in the output answer distribution is used.
Other details such as the train-test splits are deferred to the Appendix.
We also report results on an additional environment based on House3D (Wu et al., 2018) in the Appendix.
P ROPOSED M ETHOD
In this section, we detail our proposed architecture (illustrated in Figure 2).
At the start of each episode, the agent receives a textual input T (an instruction or a question) specifying the task that it needs to achieve.
At each time step, the agent observes an egocentric image I t which is passed through a convolutional neural network (LeCun et al., 1995) with ReLU activations (Glorot et al., 2011) to produce the image representation x I =
f
( I t ; θ conv ) ∈
R V × H × W , where θ conv denotes the parameters of the convolutional network, V is the number of feature maps in the convolutional network output which is equal to the vocabulary size, and H and W are the height and width of each feature map.
We use two representations for the textual input T : (1) the bag-of-words representation denoted by x BoW ∈ R V and (2) a sentence representation x sent =
f ( T ; θ sent ) ∈ R V , which is computed by passing the words in T through a Gated Recurrent Unit (GRU) (Cho et al., 2014) network followed by a linear layer.
Here, θ sent denotes the parameters of the GRU network and the linear layer with ReLU activations.
Next, the Dual-Attention unit f DA combines the image representation with the text representations to get the complete state representation x S and answer prediction x Ans :
[Equation]
Finally, x S and x Ans , along with a time step embedding and a task indicator variable (for whether the task is SGN or EQA), are passed to the policy module to produce an action.
4. D UAL -A TTENTION U NIT
The Dual-Attention unit uses two types of attention mechanisms, Gated-Attention f GA and SpatialAttention f SA , to align representations in different modalities and tasks.
T e x t r ep r e s en t a t i on 𝑦 "
I m age r ep r e s en t a t
i on 𝑦 # 𝑀 % &
H W d d
W H d
Expand d
W H 𝑧 = 𝑓 - .
𝑦 # , 𝑦 " =
𝑀 % & ⨀
𝑦 #
Gated-Attention.
The Gated-Attention unit (Figure 3) was proposed in (Chaplot et al., 2017) for multimodal fusion.
Intuitively, a GA unit attends to the different channels in the image representation based on the text representation.
For example, if the textual input is the instruction ‘Go to the red pillar’, then the GA unit can learn to attend to channels which detect red things and pillars.
More speciﬁcally, the GA unit takes as input a 3-dimensional tensor image representation y
I ∈
R
d × H × W and a text representation y T ∈ R d , and outputs a 3-dimensional tensor z ∈ R d × H × W .
Note that the dimension of y T is equal to the number of feature maps and the size of the ﬁrst dimension of y I .
In the Gated-Attention unit, each element of y T is expanded to a
H × W matrix, resulting in a 3-dimensional tensor M y T ∈ R d × H × W , whose ( i, j, k ) th element is given by M y T [ i, j, k ] =
y T
[ i ] .
This matrix is multiplied element-wise with the image representation: z = f GA ( y I , y T ) =
M y T (cid:12) y
I , where (cid:12) denotes the Hadamard product (Horn, 1990).
[Figure]
Spatial-Attention.
We propose a Spatial-Attention unit (Figure 4) which is analogous to the Gated-Attention unit except that it attends to different pixels in the image representation rather than the channels.
For example, if the textual input is the question ‘Which object is blue in color?’
, then we would like to spatially attend to the parts of the image which contain a blue object in order recognize the type of the blue object.
The Spatial-Attention unit takes as input a 3-dimensional tensor image representation
y I ∈
R d × H × W and a 2-dimensional spatial attention map y S ∈ R H × W , and outputs a tensor z ∈ R d × H × W .
Note that the height and width of the spatial attention map is equal to the height and width of the image representation.
In the spatialattention unit, each element of the spatial attention map is expanded to a d dimensional vector.
This again results in a
3-dimensional tensor M
y S ∈ R d × H × W , whose ( i, j, k ) th element is given by:
M y S [ i, j, k ] =
y S
[ j, k ] .
Just like in the Gated-Attention unit, this matrix is multiplied element-wise with the image representation: z = f SA ( y I , y S )
=
M y S (cid:12) y I .
Similar spatial attention mechanisms have been used for Visual Question Answering (Fukui et al., 2016;
Xu & Saenko, 2016; Hudson &
Manning, 2018; Gupta et al., 2017b) and grounding audio in vision (Zhao et al., 2018).
Dual-Attention .
We now describe the operations in the Dual-Attention unit shown in Figure 5, as well as motivate the intuitions behind each operation.
Given x I , x BoW , and x sent , the Dual-Attention unit ﬁrst computes a Gated-Attention over x I using x BoW :
[Equation]
Intuitively, this ﬁrst Gated-Attention unit associates each word in the vocabulary with a feature map in the image representation.
A particular feature map is activated if and only if the corresponding word occurs in the textual input.
In other words, the feature maps in the convolutional output learns to detect different objects and attributes, and words in the textual input specify which objects and attributes are relevant to the current task.
The Gated-Attention using BoW representation attends to feature maps detecting corresponding objects and attributes, and masks all other feature maps.
We [Figure] use bag-of-words representation for the ﬁrst GA unit as it explicitly aligns the words in textual input irrespective of whether it is a question or an instruction.
Note that bag-of-words representation has been used previously in models trained for learning to follow instructions (Hermann et al., 2017).
Next, the output of the Gated-Attention unit x GA1 is converted to a spatial attention map by summing over all channels followed by a softmax over H × W elements:
[Equation]
where the softmax σ ( z ) j = exp( z j ) / (cid:80) j exp( z j ) ensures that the attention map is normalized.
Summation of x GA1 along the depth dimension gives a spatial attention map which has high activations at spatial locations where relevant objects or attributes are detected.
ReLU activations in the convolutional feature maps makes all elements positive, ensuring that the summation aggregates the activations of relevant feature maps.
x spat and x I are then passed through a Spatial-Attention unit:
[Equation]
The Spatial-Attention unit outputs all attributes present at the locations where relevant objects and attributes are detected.
This is especially helpful for question answering, where a single GatedAttention may not be sufﬁcient.
For example, if the textual input is ‘Which color is the pillar?’
, then the model needs to attend not only to feature maps detecting pillars (done by the Gated-Attention), but also to other attributes at the spatial locations where pillars are seen in order to predict their color.
Note that a single Gated-Attention is sufﬁcient for instruction following, as shown in (Chaplot et al., 2017).
For example, if the textual input is ‘Go to the green pillar’, the ﬁrst Gated-Attention unit can learn to attend to feature maps detecting green objects and pillar, and learn a navigation policy based on the spatial locations of the feature map activations.
x SA is then passed through another Gated-Attention unit with the sentence-level text representation:
[Equation]
This second Gated-Attention unit enables the model to attend to different types of attributes based on the question.
For instance, if the question is asking about the color (‘Which color is the pillar?’), then the model needs to attend to the feature maps corresponding to colors; or if the question is asking about the object type (‘Which object is green in color?’)
, then the model needs to attend to the feature maps corresponding to object types.
The sentence embedding x sent can learn to attend to multiple channels based on the textual input and mask the rest.
Next, the output is transformed to answer prediction by again doing a summation and softmax but this time summing over the height and width instead of the channels:
[Equation]
Summation of x GA2 along each feature map aggregates the activations for relevant attributes spatially.
Again, ReLU activations for sentence embedding ensure aggregation of activations for each attribute or word.
The answer space is identical to the textual input space R V .
Finally, the Dual-Attention unit f DA outputs the answer prediction x Ans and the ﬂattened spatial attention map x S = vec ( x spat ) , where vec ( . ) denotes the ﬂattening operation.
Policy Module .
The policy module takes as input the state representation x S from the DualAttention unit, a time step embedding t , and a task indicator variable I (for whether the task is
SGN or EQA).
The inputs are concatenated then passed through a linear layer, then a recurrent GRU layer, then linear layers to estimate the policy function π ( a t | I t , T ) and the value function V ( I t , T ) .
All above operations are differentiable, making the entire architecture trainable end-to-end.
Note that all attention mechanisms in the Dual-Attention unit only modulate the input image representation, i.e., mask or amplify speciﬁc feature maps or pixels.
This ensures that there is an explicit alignment between the words in the textual input, the feature maps in the image representation, and the words in answer space.
This forces the convolutional network to encode all the information required with respect to a certain word in the corresponding output channel.
This explicit taskinvariant alignment between convolutional feature maps and words in the input and answer space facilitates grounding and allows for cross-task knowledge transfer.
As shown in the results later, this also makes our model modular and allows easy addition of objects and attributes to a trained model.
4. O PTIMIZATION
The entire model is trained to predict both navigational actions and answers jointly.
The policy is trained using Proximal Policy Optimization (PPO) (Schulman et al., 2017).
For training the answer predictions, we use a supervised cross-entropy loss.
Both types of losses have common parameters as the answer prediction is essentially an intermediate representation for the policy.
Auxiliary Task .
As mentioned earlier, the feature maps in the convolutional output are expected to detect different objects and attributes.
Consequently, we add a spatial auxiliary task to detect the object or attribute in the convolutional output channels corresponding to the word in the bag-of-words representation.
A prior work (Gupta et al., 2017b) also explored the use of attribute and object recognition as an auxiliary task for Visual Question Answering.
Rather than doing ﬁne-grained object detection, we keep the size of the auxiliary predictions the same as the convolutional output to avoid increase in number of parameters, and maintain the explicit alignment on the convolutional feature maps with the words.
Consequently, auxiliary labels are ( V × H × W ) -dimensional tensors, where each of the V channels correspond to a word in the vocabulary, and each element in a channel is 1 if the corresponding object or attribute is present in the current frame spatially.
Figure 6 shows examples of auxiliary task labels for the channel corresponding to the word ‘red’.
The auxiliary tasks are also trained with cross-entropy loss.
E XPERIMENTS & R ESULTS
Jointly learning semantic goal navigation and embodied question answering essentially involves a fusion of verbal and visual modalities.
While prior methods are designed for a single task, we adapt several baselines for our environment and tasks by using their multimodal fusion techniques.
We use two naive baselines, Image only and Text only ; two baselines based on prior semantic goal navigation models, Concat (used by Hermann et al.
; Misra et al.
and Gated-Attention (GA) (Chaplot et al., 2017); and two baselines based on Question Answering models, FiLM (Perez et al., 2017) and PACMAN (Das et al., 2017).
For fair comparison, we replace the proposed DualAttention unit with multimodal fusion techniques in the baselines and keep everything else identical to the proposed model.
We provide more implementation details of all baselines in the Appendix.
5. R ESULTS
We train all models for 10 million frames in the Easy setting and 50 million frames in the Hard setting.
We use a +1 reward for reaching the correct object in SGN episodes and predicting the [Table] correct answer in EQA episodes.
We use a small negative reward of -0.001 per time step to encourage shorter paths to target and answering questions as soon as possible.
We also use distance-based reward shaping for SGN episodes, where the agent receives a small reward proportional to decrease in distance to the target.
In the next subsection we evaluate the performance of the proposed model without the reward shaping.
SGN episodes end when the agent reaches any object, and EQA episodes when the agent predicts any answer.
All episodes have a maximum length of 210 time steps.
We train all models with and without the auxiliary tasks using identical reward functions.
All models are trained jointly for both the tasks and tested on each task separately.
In Figure 7, we show the training performance curves for all models trained with Auxiliary tasks in both Easy and
Hard settings.
In Table 1, we report the test performance of all models on both SGN and EQA for both Easy and Hard settings.
During training, the Dual-Attention model learns faster as compared to the baselines in the Easy setting while achieving higher ﬁnal performance in the Hard setting (see
Figure 7).
More interestingly, as shown in Table 1, the Dual-Attention model achieves considerably higher accuracy on the test set for SGN and EQA in both the difﬁculty settings when trained with or without auxiliary tasks.
These results conﬁrm the hypothesis that prior models, which are designed for a single task, lack the ability align the words in both the tasks and transfer knowledge across tasks.
Lower accuracy on EQA for most models (see Table 1) indicates that EQA is more challenging than SGN as it involves alignment between not only input textual and visual representations but also with the answer space.
As expected, using spatial auxiliary tasks lead to better performance for all models Visualization of the attention maps and intermediate representations in the model indicate that the textual and visual representations are aligned as expected (see Appendix for visualizations) 1 .
5. A BLATION TESTS
We perform a series of ablation tests in order to analyze the contribution of each component in the Dual-Attention unit: without Spatial-Attention ( w/o SA ), without the ﬁrst Gated-Attention with x BoW ( w/o GA1 ), and without the second Gated-Attention with x sent ( w/o GA2 ).
We also try removing the task indicator variable ( w/o Indicator Variable ), removing reward shaping ( w/o Reward Shaping ), and training the proposed model on a single task, SGN or EQA ( DA Single-Task ).
Figure 8 shows the training performance curves for the Dual-Attention model along with all ablation models in the Easy Setting.
In Table 2, we report the test set performance of all ablation models.
The results indicate that SA and GA1 contribute the most to the performance of the Dual-Attention [Table] [Table] model.
GA2 is critical for performance on EQA but not SGN (see Table 2).
This is expected as GA2 is designed to attend to different objects and attributes based on the question and is used mainly for answer prediction.
It is not critical for SGN as the spatial attention map consists of locations of relevant objects, which is sufﬁcient for navigating to the correct object.
Reward shaping and indicator variable help with learning speed (see Figure 8), but have little effect on the ﬁnal performance (see Table 2).
Dual-Attention models trained only on single tasks work well on SGN especially with auxiliary tasks.
This is because the auxiliary task for single task models includes object detection labels corresponding to the words in the test set.
This highlights a key advantage of the proposed model.
Due to its modular and interpretable design, the model can used for transferring the policy to new objects and attributes without ﬁne-tuning as discussed in the following subsection.
5. E XTENSION : T RANSFER TO NEW WORDS
Consider a scenario of SGN where the agent is trained to follow instructions of certain objects and attributes.
Suppose that the user wants the agent to follow instructions about a new object such as ‘pillar’ or a new attribute such the color ‘red’ which are never seen in any training instruction.
Prior
SGN models are shown to perform well to unseen combination of object-attribute pairs (Chaplot et al., 2017), but they do not generalize well to instructions containing a new word.
The model retrained only on new instructions will lead to catastrophic forgetting of previous instructions.
In contrast, our model can be used for transfer to new words by training an object detector for each new word and appending it to the image representation x I .
In order to test this, we train a single-task SGN model using the proposed architecture on the training set for instructions.
We use auxiliary tasks but only for words in the vocabulary of the instructions training set.
After training the policy, we would like the agent to follow instructions containing test words ‘red’ and ‘pillar’, which the agent has never seen or received any supervision about how this attribute or object looks visually.
For transferring the policy, we assume access to two object detectors which would give object detections for ‘red’ and ‘pillar’ separately.
We resize the object detections to the size of a feature map in the image representation ( H × W ) and append them as channels to the image representation.
We also append the words ‘red’ and ‘pillar’ to the bag-of-words representations in the same order such that they are aligned with the appended feature maps.
We randomly initialize the embeddings of the new words for computing the sentence embedding.
The results in Tableshow that this policy generalizes well to different types of instructions with unseen words.
This suggests that a trained policy can be scaled to more objects provided the complexity of navigation remains consistent.
C ONCLUSION
We proposed a Dual-Attention model for visually-grounded multitask learning which uses Gatedand Spatial-Attention to disentangle attributes in feature representations and align them with the answer space.
We show that the proposed model is able to transfer the knowledge of words across tasks and outperforms the baselines on both Semantic Goal Navigation and Embodied Question
Answering by a considerable margin.
We showed that disentangled and interpretable representations make our model modular and allows for easy addition of new objects or attributes to a trained model.
In future the model can potentially be extended to transferring knowledge across different domains by using modular interpretable representations of objects which are domain-invariant
B A DDITIONAL E XPERIMENTAL D ETAILS B. H YPERPARAMETERS AND NETWORK DETAILS
The input image is rescaled to size 3 × 168 × 300 .
The convolutional network for processing the image consisted of 3 convolutional layers: conv1 containing 32 8x8 ﬁlters with stride 4, conv containing 64 4x4 ﬁlters with stride 2, and conv3 containing V 3x3 ﬁlters with stride 2.
We use
ReLU activations for conv1 and conv2 and sigmoid for conv3, as its output is used as auxiliary task predictions directly.
We use word embeddings and GRU of size 32 followed by linear layer of size
V to get the sentence-level representation.
The policy module uses hidden dimension 128 for the linear and GRU layers (see Figure 11).
For reinforcement learning, we use Proximal Policy Optimization (PPO) with 8 actors and a time horizon of 128 steps.
We use a single batch with 4 PPO epochs.
The clipping parameter for PPO is set to 0.2.
The discount factor ( γ ) is 0.99.
We used Adam optimizer with learning rate 2.5e-4 for all experiments.
B. B ASELINE DETAILS
Image only : Naive baseline of just using the image representation: x S = vec ( x I ) where vec (
. ) denotes
the ﬂattening operation
Text only : Naive baseline of just using the textual representations: x S =
[ x BoW , x sent
Concat : The image and textual representations are concatenated: x S =
[ vec ( x I ) , x BoW , x
concatenate convolutional image and bag-of-words textual representations for SGN, whereas Misra et al.
(2017) use concatenation with sentence-level textual representations.
Gated-Attention : Adapted from Chaplot et al.
(2017), who used Gated-Attention with sentencelevel textual representations for SGN: x S =
f GA ( x I , x sent ) .
FiLM :
Perez et al.
(2017) introduced a general-purpose conditioning method called Feature-wise
Linear Modulation (FiLM) for Visual Question Answering.
Using FiLM, x S = γ ( x sent ) (cid:12) x I
+ β ( x sent ) where γ ( x sent )
and β ( x sent ) are learnable projections of the sentence representation.
PACMAN : Das et al.
(2017) presented a hierarchical RL model for EQA.
We adapt their method by using the attention mechanism in their QA module, which takes the last 5 frames and the text as input, and computes the similarity of the text with each frame using dot products between image and sentence-level text representations.
These similarities are converted into attention weights using softmax, and the attention-weighted image features are concatenated with question embedding and passed through a softmax classiﬁer to predict the answer distribution.
For this particular baseline, we use the last 5 frames as input at each time step, unlike the proposed model and all other baselines which use a single frame as input.
The attention-weighted image features are used as the state representation.
The PACMAN model used a pretrained QA module, but we train this module jointly with the Navigation model for fair comparison with the proposed model.
For each of the above method except PACMAN, we use a linear layer f with ReLU activations followed by softmax σ to get a V -dimensional answer prediction from the state representations: x Ans = σ ( f ( x S ; θ Lin )) .
x S and x Ans are concatenated and passed to the policy module along with the time step and task indicator variable just as in the proposed model.
C D OOM E XPERIMENT D ETAILS
Additional Results .
We show the training accuracy of all models without auxiliary tasks for Doom
Easy and Hard in Figure 12.
We also show the training accuracy of 3 models across 3 training runs with different seeds with and without auxiliary tasks for Doom Easy environment in Figure 13.
Dataset .
The Doom objects used in our experiments are illustrated in Figure 14.
Instructions and questions used for training and evaluation are listed in Tables 4.
[Table]
D H OUSE 3D E XPERIMENTS
In the House3D domain, we train on one house environment and randomize the colors of each object at the start of each episode.
The agent’s spawn location is ﬁxed.
We create instructions and questions dataset for this house similar to the Doom domain.
The House3D objects used in our experiments are illustrated in Figure 15.
Instructions and questions used for training and evaluation are listed in
Table 6.
[Table]
Each model is trained for 50 million frames jointly on both SGN and EQA, without the auxiliary tasks and using identical reward functions.
Similar to Doom, we use a +1 reward for reaching the correct object in SGN episodes and predicting the correct answer in the EQA episodes.
We use a small negative reward of0.
per time step to encourage shorter paths to target and answering the questions as soon as possible.
We also use distance based reward shaping for both SGN and EQA episodes, where the agent receives a small reward proportional to decrease in distance to the target.
SGN episodes end when the agent reaches any object and EQA episodes when agent predicts any answer.
All episodes have a maximum length of 420 time steps.
In Table 5, we report the train and test performance of all the models on both SGN and EQA.
The results are similar as in Doom: the Dual-Attention model outperforms the baselines by a considerable margin.
[Figure] [Table]