Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation
Anonymous Author(s)
AffiliationAddress email
Abstract
We address a practical yet challenging problem of training robot agents to navigatein an environment following a path described by some language instructions.
Theinstructions often contain descriptions of objects in the environment and path cuesdefined by humans.
To achieve accurate and efficient navigation, it is criticalto build a map that accurately represents both spatial location and the semanticinformation of the environment objects.
However, enabling a robot to build a mapthat well represents the environment is extremely challenging as the environmentoften involves diverse objects with various attributes.
In this paper, we proposea multi-granularity map, which contains both object fine-grained details ( e.g. ,color, texture) and semantic classes, to represent objects more comprehensively.
Moreover, we propose a weakly-supervised auxiliary task, which requires the agentto localize instruction-relevant objects on the map.
Through this task, the agentnot only learns to localize the instruction-relevant objects for navigation but alsois encouraged to learn a better map representation that reveals object information.
We then feed the learned map and instruction to a waypoint predictor to determinethe next navigation goal.
Experimental results show our method outperforms thestate-of-the-art by 4.0% and 4.6% w.r.t.
success rate both in seen and unseenenvironments, respectively on VLN-CE dataset.
Introduction
Developing a robot that is able to cooperate with humans is one of the goals for embodied artificialintelligence.
An important ability for such a robot is to understand human instructions ( e g. , “ walk togray sofa and stop at wooden table ”) and navigate to the corresponding location.
Toward this goal,
Anderson et al.
[1] propose a vision-and-language navigation (VLN) task, which requires an agentto follow human language instructions to navigate in unseen environments.
In this paper, we focuson its variant task VLN-CE [27], where the agent navigates in a continuous environment.
The agentcould only perform low-level actions and perceive RGB-D with limited field of view.
Current dominant methods for this task are designed in an end-to-end manner [21, 27, 35].
Theyperceive the environment implicitly from raw RGB-D images and predict a sequence of actions usinga recurrent model.
These methods attempt to learn mapping, instruction-vision correspondence, andpath planning implicitly, which increases the learning difficulty.
Recently, modular map-based approaches have been well studied, which equip the embodied agentwith map-like representations to perceive the environment.
Previous methods [6, 14, 22] try to builda semantic map for the agent through projecting RGB image segmentation results to an egocentrictop-down map as shown in Figure 1 (a).
This map reveals the location and semantic class of a part ofenvironment objects.
However, it is hard to represent all object classes ( e.g. , sofa is not represented).
More critically, it can not represent the attributes of objects ( e.g. , shape, color, texture and material),
[Figure] [Figure]
which are critical for localizing an object.
As a result, it provides insufficient information todiscriminate the “ long wooden dining table ” from two table instances shown in Figure 1 (a).
In this paper, we propose to learn a map that represents semantics and spatial location of diverseobjects in a weakly-supervised manner.
We achieve this goal in two steps.
First, we augment existingsemantic map to a multi-granularity map as shown in Figure 1 (b).
This map contains differentgranularity information, namely high-level object semantics which is recognized by a segmentationmodel, and low-level object fine-grained details.
To get the low-level details, we follow existingworks [4] to project high-dimensional segmentation model latent features to a top-down map. Theselatent features proven to contain rich object details such as color, texture, and shape [23, 46].
Second, to make the map better represent instruction objects using the multi-granularity featuresdescribed above, we propose an instruction-relevant object localization auxiliary task as shown in
Figure 1 (b).
Specifically, we feed the map representation and instruction objects to a localizerto predict the location of these objects.
Instead of manually annotating the object location ona map as localization ground-truth, we automatically generate a coarse localization ground-truthfrom instruction-path paired data, considering map regions that are close to instruction path ascoarse ground-truth.
From this task, agents learn to localize instruction-relevant objects for findingan instruction-relevant path.
More critically, to localize instruction objects, the map encoder isencouraged to reason a map representation that reveals precise semantics of each map region.
We then feed the learned map and the instruction to a waypoint predictor to determine the nextnavigation goal.
An existing off-the-shelf local policy [42] is used to determine a low-level action ( e.g. ,go forward, turn left or right) to go for the navigation goal.
We name our method as weakly-supervisedmulti-granularity map ( SS-MGMap ) for VLN.
Experimental results on VLN-CE benchmark datasetshow our proposed method outperforms state-of-the-art methods.
To sum up, our main contributions are as follows:
• We construct a multi-granularity map to represent both fine-grained details and abstract semantics information of environment.
To our best knowledge, it is the first time to introducemulti-granularity knowledge in a map format for VLN task.•
We propose a weakly-supervised object localization auxiliary task, from which agents
learn to leverage multi-granularity information to infer a discriminative map representationwithout the need of manual map annotation.
• With SS-MGMap, our agent robustly localizes objects that are incorrectly recognized by segmentation models.
Experimental results on VLN-CE [27] show our method improves navigation success rate by 4.0% and 4.6% in seen and unseen environments, respectively.
Related Works2.1 Vision-and-language navigation
VLN task [2] has drawn significant attention in embodied AI domain.
Early works focus on thediscrete environments where the agents can only perceive from a sparse set of points, including dataaugmentation methods [12, 39], auxiliary tasks methods [30, 41, 48] and pretraining methods [16,18, 31].
However, these methods can not perform well when the agent navigates in continuous 3Denvironments [20].
To address this issue, [27] exploits Habitat simulator [37] to convert discretetrajectory paths to continuous trajectories and proposes a continuous VLN setting.
Under thesesettings, [27, 35] propose the end-to-end method in which the agent takes a representation of thevisual observation and instructions as input at each time step to predict an action.
Another lineof works [14, 26] tackles the VLN problems by leveraging the language-conditioned waypoint.[8, 14, 22] build a structured semantic top-down map, which is more relevant to our work.
However,those map-based methods are limited in representing all object classes or the attributes of objects,which prevents the agent from effectively grounding the instructions in maps.
In contrast to all theseworks, our method learns a map that represents the semantics and spatial location of diverse objectsin a weakly-supervised manner.
2.2 Map representation for navigation
Perceiving environment through a map representation is helpful for a variety of navigation tasks.
Previous works propose different methods for building a map, each of them focusing on representingdifferent information on the map.
For example, occupancy map indicates whether a point is occupiedand provides location information for point-goal navigation and exploration [24, 25, 34].
Topologicalmaps encode the relationship information among different nodes in the environment and have beenexplored to tackle different navigation tasks [3, 7, 8, 32].
Semantic map represents the semanticinformation of the environment and has been successfully applied in object-goal navigation [6, 9, 44].
Concurrent works [14, 22] attempt to leverage semantic map for VLN task.
Existing works [19,4, 17, 10, 15] try to generate a deep feature map by projecting high-dimensional features encodedfrom neural network to a top-down map, which is used for localizing robot [19], predicting semanticmap [4], navigating to a few target objects [17, 15], and 3D reconstructing objects [10].
However,as compared to object-goal navigation, the instructions in VLN involve more object information.
It is challenging to ground rich object information in the map described above individually due totheir respective limitations.
Unlike these maps, we propose a multi-granularity map which containscomplementary granularity information for representing environment more comprehensively.
2.3 Weakly-supervised learning for object localization
Weakly-supervised object localization is a challenging task that requires to learn localizing objectsgiven solely category information in both image[28, 49, 29, 47, 45, 43] and video[28].
CAM[47,45, 43] has been widely used in weakly-supervised object localization by learning a heat maprepresenting the potential of object location. In contrast to these literature, we aim to utilize theweakly-supervised object localization task to learn discriminative map representation that providesenvironment information for VLN by localizing instruction-relevant objects.
Vision-and-Language Navigation using Multi-Granularity Map
3.1 Problem formulation
We consider the vision-and-language navigation task in a continuous environment [27] (VLN-CE),where an agent is required to follow a specific path described by the natural language instruction
I .
Compared with the original VLN task, in VLN-CE settings, agents can not access to predefinednavigable graph, so the agent is required to perform navigation through a sequence of low-levelactions a ∈ A
= { FORWARD , TURN - LEFT , TURN - RIGHT and STOP } .
We equip the agent with an
RGB-D camera with limited field of view, capturing a 224 × 224 RGB image R and depth image Dat each time step.
Although some current works use predefined navigable graph [8], continuous-spaceactions [26] and panoramic cameras [12, 20] to improve the performance, we strictly keep the samesettings with VLN- [Figure]
[Figure] Existing VLN methods [26, 27, 35] attempt to perceive environment implicitly from RGB-D observation, which often requires a large number of training data.
In contrast, humans will build a map-likeenvironment representation explicitly to provide necessary information for VLN.
Equipping an agentwith the mapping ability like humans is challenging because it is hard to represent diverse environment objects and their attributes in a map.
In this paper, we solve this challenge from two aspects: 1)we augment a semantic map with extra fine-grained environment details ( e.g. , texture, color), whichare extracted from latent features of a pre-trained segmentation model, to build a multi-granularitymap.
These different granularity features provide necessary information for representing diverseobjects and attributes.
To make the multi-granularity map representation more discriminative, wepropose a weakly-supervised localization auxiliary task for learning correspondence between maprepresentation and instruction objects.
With the learned multi-granularity map, we feed it and theinstruction to a waypoint navigator to decide the next navigation action.
The general scheme is shownin Figure 2.
3.2 Perceiving environment via multi-granularity map
To better represent environment objects on a map, we propose to leverage both fine-grained featuresand semantic features about objects for building a multi-granularity map.
We next introduce how tocapture these two types of features for map representation.
Fine-grained map.
To capture fine-grained environment details, we feed RGB images captured ateach time step to a segmentation pre-trained model (a U-Net [36] in our case).
Previous networkinterpretability works [23, 46] show that latent features from different layers of this model containdifferent types of fine-grained details of objects, i.e. , low-level features represent color and texturewhile high-level features represent object parts.
Based on this observation, we use these latentfeatures to represent environment details and project the latent features to an egocentric top-downmap [4].
We name the projected map as fine-grained map M f ∈ R m × m × c f , where m × m is mapsize and c f = 64 is the number of channels of the projected latent features.
Each map pixel represents12 cm × 12 cm space of environments.
We choose to project latent features from the last layer of
U-Net, which contains both low-level and high-level features because of skip connection mechanismin U-Net.
We also evaluate the effect of other layer features in experiments.
Global semantic map.
The fine-grained map aggregates environment details from RGB imageswhich are captured along the navigation path.
Because camera has a limited field of view, thefine-grained map does not contain environmental information that has not been observed.
Moreimportantly, requiring agents to recognize all objects, including some common-seen objects, fromfine-grained details increases learning difficulty.
Thus, to hallucinate environments beyond the fieldof view and to get semantic information about common-seen objects, we feed the fine-grained map toa semantic hallucination module to predict a global semantic map M s ∈ R m × m × c s , where each pixelindicates whether a particular object out of c s = 27 object categories locates in the correspondingspace.
The c s object categories are listed in Appendix.
The hallucination module consists of threeconvolution layers with batch norm and ReLU activations, followed by a U-Net which has twoencoder and two decoder convolutional blocks with skip connections.
We get the ground-truth globalsemantic map ¯ M s from the available 3D semantic information in Matterport3D following existing methods [13, 14].
A semantic loss l s is defined as follows:
[Equation]
Multi-granularity map.
We merge information from both the fine-grained map and predictedsemantic map using a map encoder.
Specifically, these two maps are first processed by a convolutionlayer, respectively.
The concatenation of two processed maps is fed to a convolution layer forproducing a multi-granularity map M ∈ R m × m × c , where c is the number of map channels.
3.3 Weakly-supervised map representation learning via object localization task
To learn the correspondence between multi-granularity map representation and diverse objects, wepropose a weakly-supervised instruction-relevant object localization auxiliary task.
To finish thistask, the agent is required to reason precisely about semantics information for each region of maprepresentation, which makes the map representation more significant.
Formulation of weakly-supervised auxiliary task.
To localize the instruction object on a map, wefeed the instruction and multi-granularity map to an object localization module to predict a grid mapˆ P ∈ R m × m which indicates a 2D distribution of potential locations of instruction-relevant objects.
Because we do not know the exact location of these instruction objects, we use the instruction-relevantpath as guidance to generate a coarse ground-truth grid map P ∈ R m × m .
The principle for generatingcoarse ground-truth is that the regions closer to the path have a higher probability of containinginstruction-relevant objects.
Specifically, we first calculate euclidean distance d k from the k -th mapregion to the path.
A normalized distance map is defined as P ′ =
{ d max − d k d max − d min } m × m k
=1 , where d maxand d min are the maximum and minimum distances among m × m regions, respectively.
For aregion traversed by the path, d k = d min = 0 and P ′ k
= 1 ; for the farthest region, d k = d max and
P ′ k
= 0 .
The coarse ground-truth grid map P is the softmax across all grids in the distance map, i.e. ,
P = Softmax( P ′ ) .
An object localization loss l o is defined as KL divergence between predicted andcoarse ground-truth distributions, i.e. ,
[Equation]
By minimizing the above loss function, the agent is encouraged to learn a better map representationfor localizing instruction objects.
Instruction-relevant object localization module.
The localization module contains two maincomponents, namely a state encoder and a state-instruction attention Att ( · ) .
The state encoderis a gated recurrent unit (GRU) [11] and attention module is a scaled dot-product attention [40].
Specifically, we first track visual history, which consists of RGB image R , depth image D , andmulti-granularity map M , using the state encoder to encode current episode state s t .
Then, thestate s t attends to instruction features using the state-instruction attention to generate attendedinstruction feature ¯ i , where the instruction features are encoded by a bi-directional LSTM [38] frominstruction I .
With the attended instruction feature ¯ i ∈ R c and a multi-granularity map representation
M ∈ R m × m × c , we predict a localization grid map ˆ P ∈ R m × m by calculating the cosine similaritybetween ¯
i and each map region feature in multi-granularity map:
[Equation]
where ¯ i =
Att( s t , BiLSTM
( I )) , s t , h t =
GRU([ M , f R ( R ) , f D ( D )] , h t − 1 ) ,
W q , W k are learnable parameter matrices, h t is hidden state of GRU at time step t , [ · , · ] indicatesconcatenation operation, f R ( · ) and f D ( · ) are off-the-shelf RGB and depth encoders, respectively.
3.4 Waypoint navigator and overall learning objective
With the learned multi-granularity map, we feed it together with the instruction to a waypointnavigator for predicting navigation action.
Specifically, an attended map features ¯ m is produced byweighted averaging multi-granularity map M using predicted localization results ˆ P in the auxiliarytask described in Section 3.3.
The localization results help agents highlight relevant regions that possibly contain instruction-relevant objects.
Then, following CMA [27], another state encoder(GRU) is exploited to predict a state s ′ t from the concatenation of attended map features ¯ m , attendedinstruction features ¯ i and the first state feature s t , which can be formulated as follows:
[Equation]
where h ′ is a hidden state of GRU.
We feed the state s ′ t to a waypoint predictor to predict a waypointˆ w = (∆ x, ∆ y ) indicating the next navigation goal.
We also feed the state to a progress predictor topredict a progress value ˆ p indicating the completeness of navigation process.
Both of these predictorsare a fully-connected layer.
To get the ground-truth waypoint, we first plan the shortest path from thecurrent agent position to the nearest waypoint on the instruction-relevant path following LAW [35].
Then, we draw a circle with 3 meters radius centered at the agent position.
The intersection pointbetween the path and circle is considered as a ground-truth waypoint w .
The ground-truth progress is defined as the normalized distance from the agent current position to thegoal following existing work [27].
The waypoint loss l w and progress loss l p are defined as follows:
[Equation]
With a predicted waypoint, we feed it to an off-the-shelf deep reinforcement learning model DDPPO [42] to determine a low-level action from action space A .
Note that the DD-PPO model do notupdate during trainig.
If the predicted progress is higher than a threshold λ p , the agent executes STOPaction.
We update the waypoint every 3 time steps.
The overall learning objective for our method is as follows:
[Equation]
where α , β and γ are hyper-parameters.
Experiments
4.1 Experimental setups
Dataset and evaluation metrics.
We conduct our experiments on VLN-CE dataset, which contains16,844 path-instruction pairs from 90 scenes in Matterport3D. It also contains about 150k augmenteddata generated by EnvDrop [39].
The dataset is split into the train, seen validation, unseen validation,and test set.
We follow the existing works [14, 27] to evaluate the navigation performance usingsuccess rate (SR), oracle success rate (OS), success weighted by path length (SPL), trajectory length(TL), and navigation error from goal (NE).
Note that an episode is considered successful if the agentcalls STOP action within 3m of the goal.
More details about evaluation metrics are put in Appendix.
Implementation details.
We implement our method based on Pytorch framework [33] and Habitat 1simulator [37].
Following existing work [27, 35], the training process consists of two parts, i.e. ,teacher forcing training on augmented data [39] and fine-tuning models using dagger training.
Wedistribute training over 2 NVIDIA V100 GPUs for 3 days on average.
A FORWARD action movesthe agent forward by 0.25 meters and a TURN action turns by 15 ◦ .
We use the same set of hyperparameters as used in the VLN-CE [27] and show these values in Appendix.
The U-Net used forextracting fine-grained features is pre-trained separately on RGB observations from Matterport3Dscenes following existing work [14] and is frozen during training.
The map size m is set to 100.
α , β ,and γ in Equation (6) are set to 10 such that four reward terms are in the same order of magnitude atinitialization.
Progress threshold λ p is set to 0.8.
4.2 Comparisons with state-of-the-art methods
We compare our SS-MGMap with current state-of-the-art methods in Table 1 on both seen andunseen validation set of VLN-CE dataset.
Our proposed SS-MGMap outperforms all methods thatfollow the same VLN-CE setting ( i.e. no panoramic images) in terms of NE, OS, SR, and SPL, whichdemonstrates the effectiveness of the learned map representation.
Besides, we also get comparable [Table] [Table] results compared with the methods that use panoramic images.
This further verifies the importance ofthe SS-MGMap for helping agents perceive environments comprehensively even just using a camerasensor with a limited field of view.
The detailed analyses are as follows.
We categorize existing methods into two types, i [Table] e. , one for the methods that without building a mapexplicitly and one for that with a map representation.
For the first type of methods ( e.g. , Seq2Seq,
CMA, LAW), our method outperforms them by a large margin.
Specifically, compared with LAW,which is the strongest baseline among them, we increase the success rate from 40.0% to 46.9% andfrom 35.0% to 38.9% on val-seen and val-unseen, respectively.
We attribute the improvement tothe usage of our multi-granularity map, which represents environments explicitly for reducing thelearning difficulty.
As for the concurrent methods that are also with map representation ( e.g. , SASRAand CM2), our SS-MGMap also brings a significant improvement against CM2, increasing successrate by 4.0% and 4.6% on val-seen and val-unseen, respectively.
We suspect this is because thesemantic map used by these methods provides insufficient information for VLN while our SS-MGMapsolves this problem.
All these results show the effectiveness of the proposed SS-MGMap, whichlearns a comprehensive map representation for VLN.
We note that our SS-MGMap even outperforms some baselines ( i.e. , AG-CMTP, R2R-CMTP and
HPN+DN) that use panoramic images on val-unseen, increasing success rate from 23.1%, 26.4% and36.0% to 38.9%, respectively.
This show the potential for replacing expensive panoramic camera witha common camera for VLN.
We also note that our method have a relatively longer trajectory length.
We argue that this is not a good metric for evaluating VLN performance because an unsatisfactoryagent who always stops before it reaches the goal will also get a short trajectory length.
A betteralternative metric is SPL, which considers both episode length and success rate.
In terms of SPL, ourmethod significantly outperforms the existing methods who use the same settings with us (37.0% v.s43.4% on val-seen and 31.0% v.s 34.3% on val-unseen).
VLN-CE leaderboard.
We compare our SS-MGMap with prior work on the held-out test-unseen setused for VLN-CE leaderboard 2 .
In Table 2, our method is leading among those that use the standardobservation (no panoramas) following VLN-CE settings [27].
For a fair comparison, we only reportthe results whose manuscripts are publicly available to ensure that no strong prior knowledge ( e.g. ,exploring the environment in prior) is used.
[Table] [Table]
4.3 Ablation studies
Effectiveness of multi-granularity map.
Our SS-MGMap contains both fine-grained details andsemantic information about environments.
We are interested to evaluate whether these differentgranularity information helps to represent environments for VLN.
To this end, we construct twovariants, i [Table] e. , one only uses the fine-grained map to represent environments and another one only usesa semantic map which is projected from image segmentation results following the current semanticmapping method [14].
In Table [Table] our SS-MGMap significantly outperforms these two variants,increasing success rate from 31.6% and 33.3% to 38.9%, respectively.
These results show that bothgranularity information are important for VLN.
Without the fine-grained map, agents are hard torecognize object attributes.
Without the semantic map, agents are required to recognize all objects,including some common objects that can be represented in the semantic map, from fine-grained objectdetails, which increases learning difficulty.
In comparison, our SS-MGMap leverages complementarymulti-granularity information for representing diverse objects.
We also try a variant that directly takesraw RGB-D as input to perform VLN in an end-to-end manner.
However, this variant drops successrate from 38.9% to 26.5%, indicating the importance of building a map explicitly for VLN.
Effectiveness of weakly-supervised auxiliary task.
For teaching agents to learn a representative mapthat represents diverse objects accurately, we propose a weakly-supervised instruction-relevant objectlocalization auxiliary task.
To evaluate the effectiveness of this task, we remove the object localizationloss in Equation 2, so that there is no auxiliary supervision signal for learning correspondence betweenmaps and instructions.
In Table 4, this variant performs worse than our SS-MGMap, success ratedecreasing significantly from 38.9% to 33.1%.
This shows the importance of the proposed weaklysupervised auxiliary task, which helps to learn map representation from the natural correspondencebetween map and instruction objects.
The coarse ground truth used for this task is soft, i.e. , each mapregion is with a probability from 0 to 1.
In this way, agents are required to figure out the degree ofcorrelation between instructions and all map regions.
An alternative is a hard binary ground-truth,i.e. , map regions whose region-path distance smaller than a threshold are considered positive andother regions are considered negative.
In Table 4, this alternative performs better than the baselinethat does not use our proposed auxiliary task but worse than our soft localization ground-truth.
Effect of different layer features for fine-grained map.
In this paper, we use a U-Net pre-trainedfor segmentation task as an RGB encoder to extract image fine-grained details.
Existing networkinterpretability work [46, 23] points out that latent features from lower layers are dominated by colorand texture concepts while features from higher layers present more object parts.
To evaluate whichtypes of features are more representative for describing environment fine-grained details, we projectdifferent layer latent features ( i.e. , first layer, middle layer, last layer, and classification layer) to buildfine-grained maps.
In Table 5, using last layer features outperforms other variants in terms of almostall metrics.
This is reasonable because the last layer features contain information from both the lowerlayer and higher layer because of the skip connection mechanism in U-Net, which make it morerepresentative for representing environments.
The features produced from the classification layer arelogits for semantic classification.
These features are dominated by semantics information, providingless fine-grained environment details.
We also find that an agent using middle layer features cannot be [Table] [Figure] [Table]
[Figure]
[Table] [Figure] trained to convergence. We suspect it is because the spatial resolution of the middle features ( 9 × 9 )is too low such that the projected fine-grained map is sparse.
4.4 Visualization results
To evaluate whether the proposed multi-granularity map helps to represent diverse objects, wevisualize the localization results ˆ P from Equation 3 in Figure 3.
Compared with the variant that doesnot contain a fine-grained map (b) and the variant without the weakly-supervised auxiliary task (c),our SS-MGMap (a) localizes instruction objects more precisely.
Specifically, red carpet is incorrectlyrecognized as floor by a segmentation model.
With fine-grained map, our SS-MGMap localize itrobustly.
The variant without fine-grained map is confused to localize it because the semantic mapcan not represent red attribute and carpet object.
The variant without the localization auxiliary taskalso performs worse because it is hard to establish the correspondence between map representationand diverse instruction-relevant objects.
More visualization results are shown in Appendix.
Discussion
Limitations and future work.
Although our learned multi-granularity map helps for representingenvironments comprehensively and provides useful information to improve VLN performance, thetraining of our map representation needs additional ground-truth semantic annotation, and our 2Dtop-down map is hard to handle the situation when agents go to another floor.
Future work mayexplore to adapt 3D mapping technique [5] to our multi-granularity map.
Besides, to improve thelanguage-to-object grounding, exploiting commonsense to find relevant area of instruction-mentionobjects could be an interesting future research direction.
In addition, although our method showspromising performance in photo-realistic simulated data, it has not been thoroughly evaluated in realworld.
Directing adapting it to real environments may cause accidents such as breaking somethingsand hitting pedestrian.
Conclusion.
In order to solve the problem that current maps provide insufficient information for VLNtask, we propose to gather different granularity information ( i.e. , fine-grained details and semanticinformation) for map representation.
Moreover, to make the map better represent diverse objectsdescribed in instructions, we propose a weakly-supervised auxiliary task for learning to localizeinstruction-relevant objects on the map with no need of extra localization annotation.
Experimentalresults show that the proposed method significantly outperforms state-of-the-art methods on VLN-CEbenchmark dataset.
Qualitative results also show that the proposed multi-granularity map helps tolocalize instruction objects that are incorrectly classified by current maps.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] We will publish
our code upon acceptance.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)?
[No] Error bars are not reported because it would be too
computationally expensive.(d)
Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.1.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either
[Table]
[Table]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[No]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]