F OAM : A Follower-aware Speaker Model For Vision-and-Language Navigation
Abstract
The speaker-follower models have proven to be effective in vision-and-language navigation, where a speaker model is used to synthesize new instructions to augment the training data for a follower navigation model.
However, in many of the previous methods, the generated instructions are not directly trained to optimize the performance of the follower.
In this paper, we present FOAM , a F O llower- A ware speaker Model that is constantly updated given the follower feedback, so that the generated instructions can be more suitable to the current learning state of the follower.
Specifically, we optimize the speaker using a bi-level optimization framework and obtain its training signals by evaluating the follower on labeled data.
Experimental results on the Room-to-Room and Room-across-Room datasets demonstrate that our methods can outperform strong baseline models across settings.
Analyses also reveal that our generated instructions are of higher quality than the baselines.
Introduction
The task of vision-and-language navigation (VLN) requires an agent to navigate in a real-world environment given natural language instructions.
In
VLN, one of the major challenges is the lack of training data.
To alleviate the issue, speakerfollower models (Fried et al., 2018b) have been proposed.
Specifically, in the speaker-follower models, an instruction-follower agent is trained to follow a provided natural language instruction to complete a specified goal, and a speaker model learns to model how humans describe routes and synthesize new instructions so as to create more training data for the follower.
While speaker-augmented data is widely used in
VLN (Fried et al., 2018b; Wang et al., 2019; Ma et al., 2019; Tan et al., 2019; Zhu et al., 2020a; aware_speaker .
[Figure]
Hao et al., 2020; Wang et al., 2021; Chen et al., 2021), most of the previous methods focus on improving the follower navigation model.
In contrast, how to improve the speaker model to generate data of higher quality is underexplored.
In this line of research, Fried et al.
(2018a) build a pragmatic speaker that can synthesize instructions based on how the follower may interpret the instructions; Tan et al.
(2019) propose to randomly add noise into the environments when generating instructions, so that the noisy environments can mimic unseen environments and the generated instructions can be more diverse; Kurita and Cho (2021) propose a generative approach for VLN where a speaker model is trained and the actions of the follower are selected by maximizing the probability of generating the given instruction.
In this paper, we propose a follower-aware speaker model (F OAM ) that optimizes the generated instructions by directly obtaining feedback from the follower so that the generated instructions can be more suitable for the follower.
To this end, we frame the idea as a bi-level optimization problem and obtain the feedback signals to improve the speaker based on the follower performance on labeled data.
As illustrated in Figure 1, the follower and speaker are trained in an iterative manner: after updating the follower for one step, it is evaluated on a batch of labeled data and the speaker is updated given the performance of the follower.
In this way, the speaker is trained to directly optimize the performance of the follower.
Experiments on Room-to-Room (Anderson et al., 2018b) and Room-across-Room (Ku et al., 2020) demonstrate strong performance of F OAM over baselines.
Notably, F OAM can achieve comparable performance to a model pre-trained with over millions of text sentences and image-text pairs.
Analyses also reveal that our speaker generates instructions of higher qualities than baselines.
Methods
We first introduce the background before discussing the details of F OAM .
2. Background
Base Settings.
VLN requires an agent to follow a given instruction and find a route in a photorealistic environment ( e.g. navigate in indoor living spaces).
Formally, in an environment e , the follower F parameterized by θ F learns to model the distribution P ( r | i ; θ F ) , where i and r denote instruction and route variables, respectively.
The training data D consists of instruction-route pairs from different environments.
Given a batch of instruction-route pairs ( i k , r k ) from D , we train the follower F to minimize the cross-entropy loss between its prediction F ( i k ; θ F ) =
P (ˆ r | i k ; θ F ) and the ground-truth label r k .
Here, we denote this supervised loss as L l :
[Equation]
Speaker-Follower Models.
Fried et al.
(2018b) propose to train a speaker model S parameterized by θ S that models the distribution of P ( i | r ; θ S ) .
As in Figure 1, with the speaker, we can perform back translation (Sennrich et al., 2016) on randomly sampled routes ˆ r from the training environments E for data augmentation.
Specifically, we first train the speaker S on the same training data as the follower.
Then, given a batch of sampled route ˆ r k ∼ E , we synthesize their human-like textual instructions
ˆ i k
= S (ˆ r k ; θ S ) .
Afterwards, the synthesized training instances (ˆ i k , ˆ r k ) are used to update F .
Here, we denote this loss as L u :
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] 2. Optimizing the Speaker
As we can see from Equation 2, the resulting follower parameters θ ∗ F depends on the speaker parameters θ S , and we can express the dependency as θ ∗ F ( θ S ) .
However, existing speaker-follower models fail to incorporate θ S into the optimization process and θ S is always fixed during training.
Formulation.
In this paper, we propose to optimize the parameters of both the follower and speaker during back translation.
Specifically, taking inspirations from Pham et al.
(2021a,b), we optimize the speaker based on the performance of the follower on the labeled training data, which can be expressed as:
[Equation]
The motivation of Equation 3 is that while the speaker-augmented data can provide additional supervisions for the follower, the main objective of the speaker is to make the follower better follow human instructions, thus we should focus on minimizing follower’s loss on the labeled training data.
Approximation.
Following previous work in bilevel optimization (Finn et al., 2017; Liu et al., 2018; Pham et al., 2021a,b), we can approximate argmin with one-step gradient update and alternatively update the parameters θ F and θ S .
Specifically, at training step t , we first sample a batch of routes and synthesize their instructions using the speaker S .
The generated data is used to update the follower:
[Equation]
where η F is the learning rate.
Then, the speaker is updated to optimize the objective L l ( θ tF )) with
[Equation]
We can approximate the gradient ∇ θ [Equation] L
[Equation]
( θ tF ))
(derivation details in Appendix A) with
[Equation]
We can see that this equation resembles REINFORCE (Williams, 1992) in reinforcement learning.
Therefore, this algorithm can also be interpreted as treating the similarity in the gradients of the follower model on the labeled data and on the augmented data as rewards, and update the speaker model using reinforcement learning.
End-to-End Reconstruction Loss.
In this paper, we also propose to add a reconstruction loss for the speaker.
Concretely, we compute the gradient of Equation 2 with respect to the speaker parameter θ S using straight-through estimator, denoted as ∇ θ S L u ( θ F , θ S ) , and then update the speaker in an end-to-end manner.
To sum up, in F OAM , the final gradient of the speaker is computed based on both the reconstruction loss (Equation 2) and the bi-level optimization loss (Equation 6), and we will perform ablations on the two objectives in the experiment section.
Experiments
Datasets.
We evaluate the models on the Roomto-Room (R2R)
(Anderson et al., 2018b) and
Room-across-Room (RxR) (Ku et al., 2020) datasets.
The R2R dataset consists of 7,189 paths, and each path has 3 English instructions with an average length of 29.
R2R is split into training, validation, and test sets.
The validation set is split into val-seen , where paths are sampled from environments seen during training, and val-unseen , where paths are sampled from environments that are not seen during training.
The paths in the test set are from new environments unseen in the training and validation sets.
The RxR dataset follows the same environment division as R2R and there are 16, paths in total.
The instructions have an average length of 78 and are in three languages, including
English, Hindi, and Telugu.
Evaluation Metrics
Our primary metric is success rate (SR), and we also report navigation error (NE), success rate weighted by path length (SPL) on R2R.
Following the suggestion in Ku et al.
(2020), we also report normalized dynamic time warping (nDTW) and
success rate weighted by dynamic time warping (sDTW)
(Magalhães et al., 2019) on RxR.
Implementation
Details.
Following
EnvDrop (Tan et al., 2019),we build our speaker and follower based on LSTM (Hochreiter and
Schmidhuber, 1997) and environmental dropout is used during back-translation.
The follower is pre-trained with imitation and reinforcement learning, and the speaker is pre-trained with maximum likelihood training.
Here, we refer to this pre-trained follower as base follower .
The two models are pre-trained for 80k steps on
R2R and 200k steps on RxR, and then trained with our method until the 300k-th iteration.
We perform environmental dropout during training as in Tan et al.
(2019), and also use their 176, paths randomly sampled from seen environments for back translation.
Different from Tan et al.
(2019), we use CLIP-ViT-224/
(Radford et al., 2021) to extract vision features as CLIP vision encoders can be beneficial for VLN models (Shen et al., 2022)
and we demonstrate that using CLIP vision encoder can obtain better performance than
ResNet-based models in the following parts.
We compute the cosine similarities between gradients for Equation 6 following Pham et al.
(2021b,a) and also perform the same weighting for the reconstruction loss.
Each training takes aboutdays on 1 NVIDIA V100 GPU to finish.
We report numbers of a single run for evaluations.
3. Main Results Room-to-Room.
We report the main results on
R2R in Table 1.
We can see that our implementation of the baseline EnvDrop model is better than the previous work because of the stronger vision encoder we use.
Based on the strong baseline, our model achieves further improvements on both validation and test sets, outperforming EnvDrop by 2.2% in the success rate on the R2R test dataset, suggesting that our framework is indeed effective
Room-across-Room.
We report the main results on R2R in Table 2.
From the table, we can see that the improvements of our framework are not as good on the RxR dataset, possibly because the instructions are much longer and thus it is hard to train a good speaker.
Specifically, we find that the baseline speaker can only achieve a BLEU score of 7.4 on the English validation set on RxR (compared with over 30 BLEU scores on R2R as in Appendix B), which leads to noisy augmented data and can impact the performance of speaker-follower models.
3. Analysis
We then perform analyses to gain more insights regarding our models: [Table] [Table] EnvDrop 42.4 38.
48.8 F OAM 42.8 38.
[Figure]
Pre-exploration and Beam Search.
We perform experiments in both pre-exploration and beam search settings following previous work (Tan et al., 2019).
Because both the speaker and follower are used in the two settings, the evaluation results can reflect the quality of both of the models.
As shown in Table 3, we find that the best configuration is using our follower and our speaker, suggesting that both our follower and speaker are more suitable for VLN than the baselines.
Notably, in the beam search setting, our model can achieve a success rate of 72.2%, which is comparable to VLN-BERT (Majumdar et al., 2020) that achieves a success rate of 73% and is pre-trained with over millions of text sentences and image-text pairs.
Generated Instructions.
The previous preexploration and beam search results well indicate that our generated instructions are more suitable for our follower, suggesting the effectiveness of our framework.
In this paragraph, we also compare the generated instructions with the reference instructions.
In Figure 2, we plot the histogram of length differences between the reference sentences and the generated instructions using compare-mt (Neubig et al., 2019).
The figure suggests that the baseline model can often generate shorter instructions than the references, but our method can alleviate this issue, indicating that our methods can indeed improve the speaker quality during training.
We also find that our generated instructions are quantitatively and qualitatively better than the baseline using automatic evaluations as in Appendix B. [Table] [Table]
Our reconstruction loss and bi-level optimization loss are complementary to each other and ablating either one of them can lead to degraded performance.
Ablation Studies.
As mentioned in Section 2.2, we perform ablation studies on both of our proposed objectives, namely the bi-level optimization loss (Equation 5) and reconstruction loss.
As shown in Table 4, ablating either of the objectives can lead to degraded performance on the R2R validation sets, indicating that both the objectives can improve the model performance and they are complementary to each other.
Related Work
We overview two lines of related work:
Vision-and-Language Navigation.
Training embodied navigation agents has been an increasingly active research area (Anderson et al., 2018a,b;
Chen et al., 2019; Ku et al., 2020; Shridhar et al., 2020; Padmakumar et al., 2022).
Fried et al.
(2018b) propose to augment the training data with the speaker-follower models, which is improve by Tan et al.
(2019) who add noise into the environments so that the speaker can generate more diverse instructions.
Zhao et al.
(2021) propose methods to measure the quality of the generated instructions and filter noisy samples.
Liu et al.
(2021) propose to adversarially sample the most difficult paths for the follower and translate these paths into instructions using the speaker for data augmentation.
While using the speaker-augmented data has been widely used in VLN, most of the existing work has been focused on improving the follower navigation model (Wang et al., 2018; Li et al., 2019; Zhu et al., 2020b).
For example, the self-monitoring agent (Ma et al., 2019) improves cross-modal alignment through a visual-text co-grounding module and a progress monitor; Zhu et al.
(2020a) propose to utilize four self-supervised auxiliary tasks that can provide additional training signals for the agent.
Most similar to our work, Fried et al.
(2018a) build a speaker that reason about how the instructions may be interpreted; Kurita and Cho (2021) propose a generative approach where a speaker model is trained to model the probability of an instructions given actions, and the follower chooses actions that maximize this probability.
Bi-level Optimization.
Bi-level optimization algorithms have been widely applied in various fields, such as learning initialization parameters (Finn et al., 2017), neural architecture search
(Liu et al., 2018), re-weighting training data (Wang et al., 2020).
Our method takes inspirations from (Pham et al., 2021a), which is applied in pseudo labeling and optimizes the teacher parameters given the student feedback.
Similar techniques have also been used in machine translation (Pham et al., 2021b), where a meta-validation set is constructed to evaluate the model performance and provide feedback.
Conclusions
In this paper, we propose the F OAM model where we improve the speaker-follower model in visionand-language navigation by constantly updating the speaker given the follower feedback during training.
We frame the idea as a bi-level optimization problem and obtain the feedback signal based on the performance of the follower on labeled data.
Experimental results on Room-to-Room and Roomacross-Room datasets demonstrate that our method can outperform strong VLN baselines in different settings.
Analyses also suggest that the quality of our speaker model is indeed improved during training.
Future directions include testing our method on more datasets and investigating more options on the feedback signals.
Acknowledgement
We would like to thank the anonymous reviewers for valuable suggestions and Te-Lin Wu for helpful discussions.
This work is supported in part by the DARPA Machine Common Sense (MCS) program under Cooperative Agreement N66001- 19-2and NIH R01HL152270.
A Derivation of the Speaker Gradient
As shown in Section 2.
at training step t we update the follower according to:
[Equation]
[Table]
We then derive the speaker gradient following previous work (Pham et al., 2021b,a).
We define the expected parameters of the follower as ¯ θ tF :
[Equation]
Then, using the chain rule, we can obtain
[Equation]
where the first term can be approximated with ∂L l ∂θ tF .
Then, for the second term, we have ∂ ¯ θ
[Equation]
We can assume that θ t − 1 F does not depend on θ S with Markov assumption (Pham et al., 2021a), and apply the REINFORCE (Williams, 1992) equation on the second term: ∂
[Equation]
tF
[Equation]
Using Monte Carlo approximation to approximate terms in Equation 11 using a batch of samples and substituting the result into Equation 9, we can get
[Equation]
Note that here η F is a hyper-parameter and can be incorporated into the learning rate of the speaker η S , thus we remove this term in Section 2.2 and our derivation is complete.
B Evaluations of the Generated Instructions
Automatic Evaluations.
As in Table 5, we measure the quality of the generated instructions in BLEU (Papineni et al., 2002) and
BERTScore (Zhang et al., 2020).
We find that our speaker can generate instructions of higher qualities according to the two metrics.
[Table]
As in Table 6, we also find that after training the speaker using our method, the generated instructions are generally longer than the baseline and are more accurate compared with the references.
C License
We evaluate our models on the Room-to-Room (R2R)
(Anderson et al., 2018b) and Room-acrossRoom (RxR) (Ku et al., 2020) datasets based on
Matterport3D (Chang et al., 2017).
The datasets are released under the Matterport3D Terms of Use.
The datasets do not contain any information that names or uniquely identifies individual people or offensive content.
Our code is based on EnvDrop that is released under the MIT license.
3 We use the datasets and code for research purposes, which is consistent with their intended use.
D Limitations and Potential Risks
As in the experiments, our models may not work well when the instructions are long and it is hard to train a reasonable speaker model.
Also, our model requires fine-tuning the speaker during training based on the feedback of the follower, which introduces additional training costs to the model.
In addition, the datasets we use in the paper may make our model biased towards environments of
American buildings.