D IAGNOSING THE E NVIRONMENT B IAS IN V ISION - AND -L ANGUAGE N AVIGATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Vision-and-Language Navigation (VLN) requires an agent to follow naturallanguage instructions, explore the given environments, and reach the desired target locations.
These step-by-step navigational instructions are extremely useful in navigating new environments that the agent does not know about previously.
Most recent works that study VLN observe a signiﬁcant performance drop when tested on unseen environments (i.e., environments not used in training), indicating that the neural agent models are highly biased towards training environments.
Although this issue is considered as one of the major challenges in VLN research, it is still under-studied and needs a clearer explanation.
In this work, we design novel diagnosis experiments via environment re-splitting and feature replacement, looking into possible reasons for this environment bias.
We observe that neither the language nor the underlying navigational graph, but the low-level visual appearance conveyed by ResNet features directly affects the agent model and contributes to this environment bias in results.
According to this observation, we explore several kinds of semantic representations which contain less low-level visual information, hence the agent learned with these features could be better generalized to unseen testing environments.
Without modifying the baseline agent model and its training method, our explored semantic features signiﬁcantly decrease the performance gap between seen and unseen on multiple datasets (i.e., 8 . 6% to 0 .
2% on R2R, 23 .
9% to 0 .
1% on R4R, and 3 .
74 to 0 .
17 on CVDN)
and achieve competitive unseen results to previous state-of-the-art models.
I NTRODUCTION
Vision-and-Language Navigation (VLN) tests an agent’s ability to follow complex natural language instructions as well as explore the given environments, so as to be able to reach the desired target locations.
As shown in Fig.
1, the agent is put in an environment and given a detailed step-by-step navigational instruction.
With these inputs, the agent needs to navigate the environment and ﬁnd the correct path to the target location.
In this work, we focus on the instruction-guided navigation (MacMahon et al., 2006; Anderson et al., 2018b; Misra et al., 2018;
Blukis et al., 2018; Chen et al., 2019c) where detailed step-by-step navigational instructions are used (e.g., ‘Go outside the dining room and turn left ...’), in contrast to the target-oriented navigation (Gordon et al., 2018; Das et al., 2018; Mirowski et al., 2018; Yu et al., 2019) where only the target is referred (e.g., ‘Go to the kitchen’ or ‘Tell me the color of the bedroom’).
Although these step-by-step instructions are overdetailed when navigating local areas (e.g., your home), they are actively used in unseen environments (e.g., your friend’s house, a new city) where the desired target is usually unknown to navigational agents.
For this purpose, testing on unseen environments which are not used during agent-training is important and widely accepted by instruction-guided navigation datasets.
Recent works propose different methods to improve generalizability of agents on these unseen testing environments; and most of the existing works (Anderson et al., 2018b; Wang et al., 2018b; Fried et al., 2018;
Wang et al., 2019b; Ma et al., 2019a;b; Tan et al., 2019; Huang et al., 2019;
Hu et al., 2019) observe a signiﬁcant performance drop from seen environments (i.e., the environments used in training) to unseen environments (i.e., the environments not used in training), which indicates a strong bias in the model towards the training environments.
While this performance gap is emphasized as one of the major challenges in current VLN research, the issue is still left unresolved and waits for an explicit explanation.
Thus, in this paper, we aim to answer three questions to this
[Figure]
Unseen Testing Environments
Train
Test Test
Instructions:1.
Exit the bedroom and walk all the way down the hallway.
Stop between the
TV and sofas.
Leave the room and turn right.
Wait in front of the ironing board.
Instructions: 1.
Exit the room.
Turn right and enter the bedroom on the left.
Wait at the foot of the bed.
Instructions: 1.
Exit the room.
Turn right and walk pass the sofa.
Enter the bedroom on the right and wait.
[Figure]
environment bias: 1.
Where (i.e., in which component) is the bias located?
Why does this bias exist?
How to eliminate this bias?
To locate where the bias is, we start by showing that natural-language navigational instructions and underlying navigational graphs are not direct reasons for this performance gap.
We then investigate the effect of environments on the agent’s performance.
In order to conduct a detailed analysis, we resplit the environment and categorize the validation data into three sets based on their visibility to the training set: path-seen data intersecting with the training paths, path-unseen data using the training environments but away from the training paths, and env-unseen data using unseen environments (environments not used in training).
By showing that the results gradually decrease from path-seen data to env-unseen data, we characterize the environment bias at three levels: path level, region level, and environment level.
These three levels of environment biases indicate strong ‘spatial localities’ in the tasks of VLN, which are intuitively reasonable because environments and regions (e.g., houses and cities) usually have their own styles when built or decorated.
We next want to analyze the detailed reason why this locality would further lead to a gap in seen versus unseen results.
Our hypothesis is that the low-level information carried by the ResNet features (He et al., 2016) is the reason.
To keep minimal low-level visual information and promote more high-level semantic information, we replace the ResNet features with the 1000 ImageNet classiﬁcation probabilities.
Although the semantic information encoded by these features is not accurate because of the shifted domain of images and labels, the same model with ImageNet-Labels features performs surprisingly well on various VLN datasets (i.e., Room-to-Room, R4R, and CVDN 1 ).
Most importantly, these noisy semantic features effectively eliminate the performance gap between seen and unseen environments, which suggests that the environment bias is attributed to the ResNet features as our hypothesis.
Following the practice in using ImageNet labels as semantic features, we further provide a discussion on how the environment bias could be eliminated.
For this, we employ advanced high-level semantic features which are more rational for the VLN domain.
We explore three kinds of semantic features: (1) areas of detected object labels (Ren et al., 2015); (2) ground truth semantic views (Chang et al., 2017); and (3) learned semantic view features.
We show that all of these semantic features significantly reduce the environment bias in multiple datasets and also achieve strong results in testing unseen environments.
We hope this work encourages more investigation and research into improving the generalization of vision-language models to unseen real-world scenarios.
R ELATED W ORK
Vision-and-Language Navigation : Vision-and-language navigation is an emerging task in the vision-and-language area.
A lot of datasets have been proposed in recent years, such as Roomto-Room (Anderson et al., 2018b), Room-for-Room (Jain et al., 2019), TouchDown (Chen et al., 2019c), CVDN (Thomason et al., 2019b), RERERE (Qi et al., 2019), House3D (Wu et al., 2018) and EQA (Das et al., 2018).
Recent works (Thomason et al., 2019a; Wang et al., 2018b Fried et al., semantic
, 2019c
; Wang et al., 2019b; Ma et al., 2019a;b; Tan et al., 2019; Hu et al., 2019; Ke et al., 2019; Anderson et al., 2019) focusing on improving the performance of navigation models, especially in unseen testing environments, have helped to increase the navigational success rate.
Domain Adaptation : The general setup of domain adaption contains two sets of data samples { x i } x i ∈ X and { y i }
y i ∈ Y from two domains X and Y .
Based on these samples, we could learn domain invariant feature with adversarial training (Goodfellow et al., 2014; Zhu et al., 2017;
Long et al., 2018; Wang et al., 2019a; Hosseini-Asl et al., 2019; Zhang et al., 2019;
Gong et al., 2019;
Chen et al., 2019b) or learn a transfer function f : X → Y (Wang et al., 2018a; Chen et al., 2019a; Rozantsev et al., 2018).
However, samples from the target domain may not be available (e.g., the testing environments in navigation should not be used in training) in applications.
Thus, we try to give an interpretable explanation to why performance varies in different domains and design a robust feature for it without deliberately considering the target domain.
Two methods in VLN, RCM (Wang et al., 2019b) and EnvDrop (Tan et al., 2019), explore the possibility of domain adaptation.
Both works take the testing environments in training while RCM also uses testing instructions.
Domain Generalization : In domain generalization (Blanchard et al., 2011), the goal is to predict the labels in the previous unseen domain.
Similar to the test setting of VLN tasks, the testing data is unrevealed in training.
Works have been proposed to learn the common features of the training domain (Muandet et al., 2013; Blanchard et al., 2017; Li et al., 2017; 2018; Carlucci et al., 2019;
Deshmukh et al., 2019).
In this paper, we focus on the domain generalization problem in VLN task, and try to ﬁnd the reasons for the failures.
V ISION - AND -L ANGUAGE N AVIGATION AND ITS E NVIRONMENT B IAS
We ﬁrst introduce the task of vision-and-language navigation (VLN) and brieﬂy describe the neural agent models used in our work.
We next survey previous works on multiple indoor navigation datasets to show that the environment bias is widely observed in current VLN research.
Lastly, we claim that this bias also exists in the outdoor navigation tasks, if the agent is tested on unseen regions.
3. V ISION - AND -L ANGUAGE N AVIGATION
Tasks : As shown in Fig.
1, the goal of the VLN task is to train an agent to navigate a certain type of environments { E } (e.g., indoor or outdoor environments) given the instruction I .
Each environment
E is an independent space, such as a room or a house, and consists of a set of viewpoints.
Each viewpoint is represented as a panoramic image and can be decomposed into separate views { o } as inputs to the neural agent models.
The viewpoints and their connectivity form the navigational graph.
In practice, after being placed at a particular viewpoint and given the instruction in the beginning, at each time step, the agent can observe the panoramic image of the viewpoint where it is located, and choose to move along an edge of the graph to the next node (i.e., viewpoint) or stop.
This navigational process produces a path (i.e., a list of viewpoints), and the performance of the agent is evaluated by whether it reaches the target location that the instruction indicates in the end.
Neural Agent Models :
Most instruction-guided navigational agents are built based on attentive encoder-decoder models (Bahdanau et al., 2015).
The encoder reads the instructions while the decoder outputs actions based on the encoded instructions and perceived environments.
Since the main purpose of this work is to understand the environment bias in vision-and-language navigation, we use a minimal representative neural agent model that achieves comparable results to previous works.
Speciﬁcally, we adopt the panoramic-view neural agent model in Fried et al.
(2018) (‘Follower’) with modiﬁcations from Tan et al.
as our baseline model.
We also exclude advanced training techniques (i.e., reinforcement learning and data augmentation) and only train the agent with imitation learning in all our experiments for the same purpose.
More details in original papers.
3. E NVIRONMENT B IAS IN I NDOOR N AVIGATION
In order to evaluate the generalizability of agent models, indoor vision-and-language navigation datasets (e.g., those collected from Matterport3D (Chang et al., 2017)) use disjoint sets of environments in training and testing.
Most of the datasets provide two validation splits to verify the agent’s
[Table]
performance in both sets of environments: validation seen, which takes the data from training environments, and validation unseen, whose data is from new environments apart from the training environments.
In the ﬁrst part of Table 1, we list most of the previous works on the Room-to-Room dataset (Anderson et al., 2018b) and report the success rate under greedy decoding (i.e., without beam-search) on validation seen and validation unseen splits.
The large absolute gaps (from 30 . 9% to 9 . 7% ) between the results of seen and unseen environments show that current neural agent models on R2R suffer from environment bias 2 .
Besides Room-to-Room (R2R), we also analyze two newly-released indoor navigation datasets that were also collected from Matterport3D environments: Room-forRoom (R4R) (Jain et al., 2019) and Cooperative Vision-and-Dialog Navigation (CVDN) (Thomason et al., 2019b).
As shown in the second and third parts of Table.
1, results drop signiﬁcantly from seen to unseen environments (i.e., 26 . 9% on R4R and 3 . 74 on CVDN), indicating that agent models also suffer from the environment bias in these datasets.
Lastly, we show the results (denoted as ‘ours’ in Table.
1) when the environment bias (reason analyzed in Sec. 5) is effectively eliminated by our learned semantic features (described in Sec. 6.3).
As a result, the performance gaps are effectively decreased on all three datasets without changing the model and learning hyper-parameters, compared to our baselines (denoted as ‘Our baseline’) and previous works 3 .
3. E NVIRONMENT B IAS IN O UTDOOR N AVIGATION
Since the three indoor navigational datasets in previous sections are collected from the Matterport3D environments (Chang et al., 2017), in order to show that the environment bias is a general phe-
[Figure]
nomenon also existing in other kinds of environments, we investigate the outdoor navigation task from Touchdown dataset (Chen et al., 2019c), whose environments are taken from New York City.
In the original data splits of Touchdown, the environment is not speciﬁcally divided into seen and unseen and only involved one city.
Thus the trained agent is only tested on the training environments (similar to validation seen split).
To reveal the environment bias in Touchdown dataset, we split the city environment according to latitude and create two sub-environments: ‘training’ and ‘unseen’.
The data are then re-split into training, val-seen, and val-unseen, accordingly.
We adapt our baseline R2R agent model with additional convolutional layers to ﬁt this new task.
As shown in the last part of Table.
1, when experimenting on the original data split, our baseline model achieves state-of-theart results on the original ‘dev’ set and ‘test’ set, proving the validity of our model in this dataset.
However, the results on our re-split data (denoted as ‘Our baseline (seen/unseen split)’) still show a big drop from the ’training’ to the ’unseen’ sub-environment (from 17 .
5% to 5 .
, indicating that environment bias is a broad issue.
W HERE : THE E FFECT OF D IFFERENT T ASK C OMPONENTS
In Sec. 3, we showed that current neural agent models are biased towards the training environments on multiple vision-and-language navigation (VLN) datasets.
In this section, our goal is to locate the component of VLN tasks which this environment bias is attributed to.
As one of the early-released and well-explored datasets of VLN, Room-to-Room (R2R) dataset (Anderson et al., 2018b) is used as the diagnosing dataset in the experiments.
We start by showing that two possible candidates, the natural language instructions and the underlying navigational graph, do not directly contribute to the environment bias.
Then the effect of visual environments is analyzed in detail.
4. T HE E FFECT OF N ATURAL -L ANGUAGE N AVIGATIONAL I NSTRUCTIONS
A common hypothesis is that the navigational instructions for unseen environments (e.g., val unseen) are much different from the training environments (i.e., training and val seen) due to the different objects and layouts in new environments; and this lingual difference thus leads to the performance gap.
In this section, we analyze the distributions of success rate with regard to the relationship between validation data’s instructions and training instructions.
In order to quantitatively evaluate this relationship, we deﬁne the ‘distances’ from a validating instruction to all training instructions as the phrase-matching metric.
Suppose x is a validating datum, T is the training set, and inst( x ) is the instruction of the datum x , we use ROUGE-L (Lin, 2004) and BLEU-4 (Papineni et al., 2002) to
[Figure]
calculate this ‘distance’:
[Equation]
where we consider all the training instructions as references in calculating the BLEU-4 score.
We show the distributions of success rates and distances in Fig.
As opposed to the hypothesis, we do not observe a signiﬁcant difference between the distributions of ‘distances’ (as shown in Fig.(a, b)) on seen validation and unseen validation.
For the success rate distributions (in Fig. 2(c,d)), the performance is better on instructions with smaller ‘distances’ (i.e., higher BLEU-4/ROUGE-L scores w.r.t.
the training instructions) on both validation splits.
However, comparing two splits, with the same ‘distance’ to training instructions, seen validation still signiﬁcantly outperforms the unseen validation set on success rate, which implies the existence of other reasons rather than language attributed to this performance gap.
4. T HE E FFECT OF U NDERLYING N AVIGATIONAL G RAPH
As shown in Fig.
3, an environment could be considered as its underlying navigational graph with visual information (as in Fig. 1).
In order to test whether the agent model could overﬁt to these navigational graphs (and thus be biased towards training environments), we follow the experiments in Hu et al.
(2019) to train the agent without visual information.
Speciﬁcally, we mask out the ResNet features with zero vectors thus the agent could only make the decision based on the instructions and the navigational graph.
With our baseline model, the success rate is 38 .
5% on validation seen and 41 .
0% on validation unseen in this setting, which is consistent with the ﬁnding in Hu et al.
Besides showing the relatively good performance of unseen split without visual contents (similar to Thomason et al. (2019a) and Hu et al.
, we also want to emphasize the low performance gap between seen and unseen environments ( 2 . 5% compared to the > 10% gap in usual).
Hence, we claim that the underlying graph is not a dominant reason for the environment bias.
4. T HE E FFECT OF V ISUAL E NVIRONMENTS
To show how the visual environments affect the agent’s performance, we analyze the results on unseen environments and in different spatial regions of the training environments.
In order to give a detailed characterization of the effect of environments, we are going to reveal the spatial localities which are related to the agent’s performance at three different levels:
• Path-level Locality: Agents are better at paths which intersect with the training paths.
• Region-level Locality: Agents are better in regions which are closer to the training data.
• Environment-level Locality: Agents perform better on training environments than on unseen environments.
[Table]
And the existence of these spatial locality inspires us to ﬁnd the direct cause of the problem in Sec. 3.2.
However, the original split of data is not ﬁne-grained enough to separately reveal these spatial localities.
To better illustrate this, we visualize the data from one environment of the Roomto-Room dataset in Fig.
3, where the vertices are viewpoints with visual information and edges are valid connections between viewpoints.
The vertices highlighted with dark-black indicate the viewpoints which are used in training paths, and the red edges are the connections covered by original val-seen paths.
As shown in Fig.
3, nearly all viewpoints in val-seen paths (vertices connected to red lines) are used as viewpoints in training data (vertices marked by dark-black).
We thus cannot categorize the path-level and region-level localities.
To bypass this, we propose a novel re-splitting method to create our diagnosis data splits.
Structural Data Re-splitting We employ two kinds of structural data splitting methods based on the horizontal or vertical coordinates, denoted as ‘X-split’ and ‘Z-split’, respectively.
The ‘Zsplit’ intuitively separates different ﬂoors in the houses and ‘X-split’ creates separate areas.
When applying to the training environments in R2R dataset, we use one side of the splitting line (see the ‘X-splitting line’ Fig. 3) as the new training ‘environment’, and the other side as the path-unseen ‘environment’.
In addition to this split of environments, we also re-split the original training data and val-seen data while keeping the val-unseen data the same.
The data paths across the splitting line are dropped.
As shown in the right part of Fig.
3, we create three new data splits: training split, val-path-seen split, and val-path-unseen split.
The edges covered by the new val-path-unseen split are highlighted in blue, while the color style of training split and val-path-seen split (‘Black’ for viewpoints in training and ‘Red’ for edges in val path-seen) are the same.
Since the amount of original val-seen data are inadequate to ﬁll two new validation sets (val path-seen and val pathunseen), we bring some (original) training data into our new validation splits.
The overall statistics of original splits and our new splits are shown in Table 2.
Existence of Path-level and Environment-level Localities For both splitting methods, we train our baseline model on the newly-split training set and evaluate on our three validation sets (denoted as ‘X-split’ or ‘Z-split’ rows in Table 2).
The results of our baseline model on the original R2R (denoted as ‘R2R’ rows) splits are listed for comparison.
As shown in Table.
2, the agent performs better on val path-seen than val path-unseen, which suggests that a path-level locality exists in current VLN agent models.
Meanwhile, the results on val path-unseen are further higher than val env-unseen and it indicates the environment-level locality which is independent of the path-level locality.
Existence of the Region-level Locality To further demonstrate region-level locality, we study how the success rate changes in different regions of the environment with respect to their distances to the training data, which is similar to the analysis of language ‘distance’ in Sec. 4.1.
We ﬁrst calculate the point-by-point shortest paths using the Dijkstra’s algorithm
(Dijkstra, 1959), where the shortest distances between viewpoints v and v (cid:48) are denoted as the graph distance dis GRAPH ( v, v (cid:48) ) .
Based on this graph distance, we deﬁne the viewpoint distance dis VIEWPOINT from a viewpoint v to the training
[Figure]
data T as v ’s minimal graph distance to a viewpoint v (cid:48) in training data.
We then deﬁne the path distance dis PATH from a validating data x to the whole training data T as the maximal viewpoint distance in the path of x :
[Equation]
We compute this path distance between paths in the env-seen validation set and training environments in our re-split data.
As shown in Fig.
4, the success rate declines as the path moves further from the training environment on both re-splitting methods (i.e., ‘X-split’ and ‘Z-split’).
As conclusion, the closer the path to the training data, the higher the agent performance is, which suggests the existence of region-level locality.
W HY : W HAT I NSIDE THE E NVIRONMENTS C ONTRIBUTES TO THE B IAS ?
In Sec. 4, we locate the cause of performance gap in visual environments by excluding other potential reasons and categorizing the spatial localities.
However, there are still multiple possible aspects inside the environment which could lead to these spatial localities, e.g., the object layout convention and the room connections.
The agent model could be biased towards the training environments by over-ﬁtting or memorizing these environment-speciﬁc characteristics.
In this section, we want to identify which aspect directly contributes to the bias and draw the following conclusion: the environment bias is attributed to low-level visual information carried by the ResNet features.
We ﬁrst show an experiment that effectively decreases the gap between seen and unseen environments with minimal model modiﬁcations.
We then clarify our conclusions based on the ﬁndings.
5. A N I NVESTIGATION E XPERIMENT : I MAGE N ET L ABELS AS V ISUAL F EATURES
Suspecting that the over-ﬁtting happens when the agent over-learns low-level features, we hope to ﬁnd the replacement of ResNet 2048-features that contain minimal low-level information while preserving distinguishable visual contents.
The most straightforward replacement is that instead of using mean-pooled features, we inject the frozen 1000-way classifying layer in ResNet pre-training, and use the probabilities of ImageNet labels as visual features.
Shown as ‘ImageNet’ in Table.
3, the probability distribution almost closes the gap between seen and unseen.
These results further constrain the reason of environment bias to the low-level ResNet features of image views.
Combining with the ﬁndings of spatial localities, we suggest that environments (i.e., houses) and regions (i.e., rooms) usually have their own ‘style’.
Thus the same semantic label (captured by ImageNetfeatures) has different visual appearances (captured by ResNet features) in different environments or regions.
As a result, ImageNet-1000 features, in spite of being noisy, are not distracted by low-level visual appearance and could generalize to unseen environments, while ResNet features could not.
Although these ImageNet-1000 features decrease the performance gap, it has a disagreement with the VLN domain so that the validation unseen results of R4R and CVDN are slightly worse than baseline (and not much better for R2R).
Hence it motivates us to ﬁnd better semantic representations of environmental features that can both close the seen-unseen gap while also achieving state-of-theart on unseen results (which we discuss next).
[Table]
H OW : M ETHODOLOGY TO F IX THE E NVIRONMENT B IAS
In the previous section (Sec. 5), we found that the environment bias is related to the low-level visual features (i.e., 2048 -dim ResNet features).
Following the ﬁndings we observed in Sec.
5.1, we build our agent on the features which are more correlated to the VLN environmental semantics than the ImageNet label features in Sec. 5.1.
We ﬁrst demonstrate our baseline results on three VLN datasets and then explore the advanced semantic feature replacements.
As shown in Table 3, these advanced semantic features could effectively reduce the performance gap between seen and unseen environments and improve the unseen results compared to our strong baselines.
The effectiveness of these semantic features supports our explanation of the environment bias in Sec. and also suggests that future work in VLN tasks should think about such generalization issues.
6. B ASELINE
In our baseline model, following the previous works we use the standard ResNet features as the representation of environments (Anderson et al., 2018b; Jain et al., 2019; Thomason et al., 2019b).
These features come from the mean-pooled layer after the ﬁnal convolutional layer of ResNet152 (He et al., 2016) pre-trained on ImageNet (Russakovsky et al., 2015).
As shown in ‘Baseline’rows of Table.
3, val-seen results are signiﬁcantly higher than val-unseen results in all three datasets.
Note that our baseline method takes the ‘feature dropout’ technique demonstrated in Tan et al.
(2019) (without back translation): the ResNet features are randomly masked by zero before used as inputs of the agent.
Without this ‘feature dropout’ (denoted as ‘ResNet NoDrop’ in Table. 3), the gaps will increase in R2R and R4R, which suggests that this ‘feature dropout’ technique also helps to eliminate the low-level visual information over-ﬁtting as we discussed in Sec. 5.
However, the performance gap is still large, which leads us to the following discussions of semantic features.
6. D ETECTED O BJECTS A REAS
During navigation, the objects in the environments are crucial since their matchings with the instruction often indicate the locations that can guide the agent, thus object detection results of the environments can provide relevant semantic information.
In our work, we utilize the detection information generated by Faster R-CNN (Ren et al., 2015) to create the feature representations.
Comparing to ImageNet-1000 features (Sec. 5.1), these detection features include more environmental information since the viewing images in VLN usually contain multiple objects.
Instead of directly using classiﬁcation probabilities of the labels from ResNet and different from the approach in Hu et al. (2019) who utilized the embeddings of detected labels, we design our detection features f DETECT of each image view as the sum of the areas of detected objects weighted by detection conﬁdence:
[Equation]
where the c i and a c i are the label and feature of each detected object, Area( ∗ ) and Conf( ∗ ) are the area and conﬁdence of each object.
For implementation details, we use the Faster R-CNN (Ren et al., 2015) trained on Visual Genome (Krishna et al., 2017) provided in Bottom-Up Attention (Anderson et al., 2018a).
To eliminate the labels irrelevant to VLN task, we calculate the total areas of each detection object among all environments and pick the labels that take up a relatively large proportion of the environments, creating features of dimension 152 .
6 Denoted as ‘Detection’ in Table 3, the performance gap is diminished with these detection features compared to baselines in all three datasets, indicating that changing the features to a higher semantic level has a positive effect on alleviating the environment bias.
Meanwhile, the improvement of unseen validation results on R2R an R4R datasets suggests the better efﬁciency in the VLN task than the ImageNet labels.
6. S EMANTIC S EGMENTATION
Although the detection features can provide adequate semantic information for the agent to achieve comparable results as the baseline model, they do not fully utilize the visual information where the content left over from detection may contain useful knowledge for navigation.
A better semantic representation is the semantic segmentation, which segments each view image on the pixel level and gives the label to each segment region, allowing us to utilize the semantics from the entire environment.
Matterport3D (Chang et al., 2017) dataset provides the labeled semantic segmentation information of every scene and we take the rendered images from Tan et al.
A comparison example of RGB images and semantic views is available in the Appendix.
Since the semantic segmentation images are ﬁne-grained and blurry in boundaries, we follow the design of detection features, using the areas of semantic classes in each image view as the semantic features (conﬁdence is excluded since semantic segmentation does not provide this value).
The areas are normalized to [0 , 1] by dividing the area of the whole image region.
We ﬁrst assume that the semantic information is provided as additional environmental information and the results of the model using the ground truth semantic areas are shown in the ‘ground truth’ rows in Table.
We next study the situation where the semantic information is not available in testing environments thus the information needs to be learned from training environments.
Thus we train a separate multi-layer perceptron to predict the areas of these semantic classes (details in Appendix), and the results of the model with these predicted semantics as features are shown in ‘learned’.
As shown in Table.
3, both ‘ground truth’ and ‘learned’ semantic representations bring the performance of seen and unseen closer comparing to the baseline model, and the smallest performance gaps come from learned semantic segmentation features in all three datasets.
The highest validation unseen success rates among all the proposed feature representations are also produced by semantic segmentation features, ‘learned’ semantic for R4R and ‘ground truth’ semantic for R2R and CVDN.
Overall, among all the semantic representations we have explored, the semantic segmentation features are most effective in eliminating the environment bias.
C ONCLUSION
In this paper, we focus on studying the performance gap between seen and unseen environments widely observed in vision-and-language navigation (VLN) tasks, trying to ﬁnd where and why this environment bias exists and provide possible initial solutions.
By designing the diagnosis experiments of environment re-splitting and feature replacement, we locate the environment bias to be in the low-level visual appearance; and we discuss semantic features that decrease the performance gap in three VLN datasets and achieve state-of-the-art results.
E XAMPLES OF RGB I MAGES AND S EMANTIC V IEWS
In Fig.
5, we show a rendered semantic view from Tan et al.
(2019) and its original RGB image.
Different colors indicate different semantic segmentation areas and 40 semantic labels are considered in the Matterport3D dataset Chang et al.
A. D ETAILS OF ‘L EARNED ’ S EMANTIC T RAINING
We use a multi-layer perceptron over the ResNet features to generate the ‘learned’ semantic features.
The multi-layer perceptron includes three fully-connected layers with ReLU activation on the outputs of the ﬁrst two layers.
The input is the 2048 -dim
ResNet feature f of each image view.
The hidden sizes of the ﬁrst two layers are 512 and 128 .
The ﬁnal layer will output the 42 -dim semantic feature y that represents the areas of each semantic class.
After the linear layers, we use the sigmoid function σ to convert the output to the ratio of areas.
[Equation]
The model is trained with ground truth semantic areas y AREA (normalized to [0 , 1] ) and only the views in training environments are used in training.
We minimize the binary cross-entropy loss between the ground truth areas { y ∗ i } and the predicted areas { y i } , where i indicate the i -th semantic class.
[Equation]
Dropout layers with a probability of 0 .
5 are added between fully-connected layers while training.
The sigmoid function σ and the cross-entropy loss are combined to improve numerical stability.
After the model is ﬁtted, we freeze the weight and use it to predict the semantic features of all seen and unseen environments (i.e., environments for training, val-seen, and val-unseen data).
The predicted features are then used as the input of our neural agent model for different datasets (i.e.,
R2R, R4R, and CVDN), and the neural agent models are the same except we change the input dimension from 2048 (the dimension of ResNet features) to 42 (the number of semantic classes).