Original: Towards Versatile Embodied Navigation
Revised: Towards Versatile Embodied Navigation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: With the emergence of varied visual navigation tasks ( e.g ., image-/object-/audiogoal and vision-language navigation) that specify the target in different ways, thecommunity has made appealing advances in training specialized agents capable ofhandling individual navigation tasks well.
Revised: With the emergence of varied visual navigation tasks ( e.g ., image-/object-/audiogoal and vision-language navigation) that specify the target in different ways, thecommunity has made appealing advances in training specialized agents capable ofhandling individual navigation tasks well.

Original: Given plenty of embodied navigationtasks and task-specific solutions, we address a more fundamental question: can welearn a single powerful agent that masters not one but multiple navigation tasksconcurrently ?
Revised: Given plenty of embodied navigationtasks and task-specific solutions, we address a more fundamental question: can welearn a single powerful agent that masters not one but multiple navigation tasksconcurrently ?

Original: First, we propose VXN , a large-scale 3D dataset that instantiates fourclassic navigation tasks in standardized, continuous, and audiovisual-rich environments.
Revised: First, we propose VXN , a large-scale 3D dataset that instantiates fourclassic navigation tasks in standardized, continuous, and audiovisual-rich environments.

Original: Second, we propose V IENNA , a versatile embodied navigation agent thatsimultaneously learns to perform the four navigation tasks with one model.
Revised: Second, we propose V IENNA , a versatile embodied navigation agent thatsimultaneously learns to perform the four navigation tasks with one model.

Original: Buildingupon a full-attentive architecture, V IENNA formulates various navigation tasks as aunified, parse-and-query procedure: the target description, augmented with fourtask embeddings, is comprehensively interpreted into a set of diversified goal vectors, which are refined as the navigation progresses, and used as queries to retrievesupportive context from episodic history for decision making.
Revised: Buildingupon a full-attentive architecture, V IENNA formulates various navigation tasks as aunified, parse-and-query procedure: the target description, augmented with fourtask embeddings, is comprehensively interpreted into a set of diversified goal vectors, which are refined as the navigation progresses, and used as queries to retrievesupportive context from episodic history for decision making.

Original: This enables thereuse of knowledge across navigation tasks with varying input domains/modalities.
Revised: This enables thereuse of knowledge across navigation tasks with varying input domains/modalities.

Original: We empirically demonstrate that, compared with learning each visual navigationtask individually, our multitask agent achieves  better performance withimprovedgeneralization and reduced complexity.
Revised: We empirically demonstrate that, compared with learning each visual navigationtask individually, our multitask agent achieves comparable or even better performance with reduced complexity.

Original: Our dataset and code will be released.
Revised: Our dataset and code will be released.

Original:  Introduction
Revised:  Introduction

Original: As a fundamental research topic, visual navigation has attained extensive attention across many disciplines, including robotics [1], computer vision [2,3] and natural language processing [4].
Revised: As a fundamental research topic, visual navigation has attained extensive attention across many disciplines, including robotics [1], computer vision [2,3] and natural language processing [4].

Original: Consider atypical navigation scenario (Fig. 1), in which a human intends to direct a robot agent to navigate to atarget – a buzzing washer.
Revised: Consider atypical navigation scenario (Fig. 1), in which a human intends to direct a robot agent to navigate to atarget – a buzzing washer.

Original: The target can be specified by a photo of the washer ( i e ., image-goalnav.
Revised: The target can be specified by a photo of the washer ( i e ., image-goalnav.

Original: [5]) or the buzzing sound ( i e ., audio-goal nav.
Revised: [5]) or the buzzing sound ( i e ., audio-goal nav.

Original: [6]) or the corresponding semantic tag –washing machine ( i.e ., object-goal nav.
Revised: [6]) or the corresponding semantic tag –washing machine ( i.e ., object-goal nav.

Original: [7]), or linguistic instructions – “ go to the end of this corridor, turn left and enter the laundry-room ” ( i.e ., vision-language nav.
Revised: [7]), or linguistic instructions – “ go to the end of this corridor, turn left and enter the laundry-room ” ( i.e ., vision-language nav.

Original: Naturally, the agent isexpected to be smart enough to execute all these kinds of navigation tasks involving varying modalities/domains ( i.e ., image, audio, semantic tag, text) with different optimal policies.
Revised: Naturally, the agent isexpected to be smart enough to execute all these kinds of navigation tasks involving varying modalities/domains ( i.e ., image, audio, semantic tag, text) with different optimal policies.

Original: Contrary to ourexpectation, almost all existing navigation agents are specifically designed/trained for one specifictask – a “versatile” agent capable of mastering multiple navigation tasks remains far beyond reach.
Revised: Contrary to ourexpectation, almost all existing navigation agents are specifically designed/trained for one specifictask – a “versatile” agent capable of mastering multiple navigation tasks remains far beyond reach.

Original: Besides its great value in practice, investigating embodied navigation in multitask scenarios can helpbetter understand human intelligence.
Revised: Besides its great value in practice, investigating embodied navigation in multitask scenarios can helpbetter understand human intelligence.

Original: First, we humans can learn multiple tasks in a parallel ad hocmanner, and benefit from commonalities across related tasks [9].
Revised: First, we humans can learn multiple tasks in a parallel ad hocmanner, and benefit from commonalities across related tasks [9].

Original: Second, we accomplish tasks byprocessing and combining signals from different modalities.
Revised: Second, we accomplish tasks byprocessing and combining signals from different modalities.

Original: Evidences from cognitive psychologyindicate that our senses are functioning together and multisensory integration is a central tenant ofhuman intelligence [10,11].
Revised: Evidences from cognitive psychologyindicate that our senses are functioning together and multisensory integration is a central tenant ofhuman intelligence [10,11].

Original: Though the idea of multitask learning [12] was widely explored in computer vision field[13], prior attempts are often made in unsupervised and supervised learning settings;
Revised: Though the idea of multitask learning [12] was widely explored in com- puter vision field[13], prior attempts are often made in unsupervised and supervised learning settings

Original: [Figure]
Revised: [Figure]

Original: ision-language nav
Revised: ision-language nav

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: in the context of multitask reinforcement learning (MTRL)
Revised: in the context of multitask reinforcement learning (MTRL)

Original: [14], not much is done for visually-rich,embodied navigation scenarios.
Revised: [14], not much is done for visually-rich,embodied navigation scenarios.

Original: One possible reason is the lack of a suitable dataset, compounded byconsiderable costs involved in data collection, as multiple navigation tasks should be supported.
Revised: One possible reason is the lack of a suitable dataset, compounded byconsiderable costs involved in data collection, as multiple navigation tasks should be supported.

Original: In response, a large-scale 3D dataset, VXN , is established to investigate multitask multimodal embodiednavigation in audiovisual complex indoor environments.
Revised: In response, a large-scale 3D dataset, VXN , is established to investigate multitask multimodal embodiednavigation in audiovisual complex indoor environments.

Original: VXN allows simulated robot agents to concurrently learn four tasks, i.e ., image-goal nav.
Revised: VXN allows simulated robot agents to concurrently learn four tasks, i.e ., image-goal nav.

Original: , audio-goal nav.
Revised: , audio-goal nav.

Original: , object-goal nav. , and vision-languagenav.
Revised: , object-goal nav. , and vision-languagenav.

Original: , in continuous, acoustically-realistic and perceptually-rich world 1 .
Revised: , in continuous, acoustically-realistic and perceptually-rich world 1 .

Original: Based on a high-throughputsimulator [15], VXN instantiates different navigation tasks in unified environments following the samephysical rules.
Revised: Based on a high-throughputsimulator [15], VXN instantiates different navigation tasks in unified environments following the samephysical rules.

Original: It equips the agents with multimodal sensors to gather information from 360 ◦ RGBDand audio observations.
Revised: It equips the agents with multimodal sensors to gather information from 360 ◦ RGBDand audio observations.

Original: Taken all together, VXN provides a realistic testbed for multitask navigation.
Revised: Taken all together, VXN provides a realistic testbed for multitask navigation.

Original: With VXN , we further develop V IENNA , a versatile embodied navigation agent that jointly learns tosolve the four navigation tasks using one single model without switching among different models.
Revised: With VXN , we further develop V IENNA , a versatile embodied navigation agent that jointly learns tosolve the four navigation tasks using one single model without switching among different models.

Original: Based on Transformer encoder-decoder architecture [16], V IENNA encodes the full episode historyof multisensory inputs ( i.e ., RGB, depth, and audio) and navigation actions, and absorbs commonknowledge across different navigation tasks with a shared decoder.
Revised: Based on Transformer encoder-decoder architecture [16], V IENNA encodes the full episode historyof multisensory inputs ( i.e ., RGB, depth, and audio) and navigation actions, and absorbs commonknowledge across different navigation tasks with a shared decoder.

Original: Target signals ( i.e ., goal picture,target class, aural cues, linguistic instruction) are parsed into queries , and the supportive contextretrieved from the encoded history is fed to corresponding policy for task-specific decision making.
Revised: Target signals ( i.e ., goal picture,target class, aural cues, linguistic instruction) are parsed into queries , and the supportive contextretrieved from the encoded history is fed to corresponding policy for task-specific decision making.

Original: With such a fully-attentive model design, V IENNA is able to comprehend multimodal observations,conduct long-term reasoning, and, more essentially, exploit cross-task knowledge.
Revised: With such a fully-attentive model design, V IENNA is able to comprehend multimodal observations,conduct long-term reasoning, and, more essentially, exploit cross-task knowledge.

Original: By contrasting our V IENNA to several single-task counterparts on VXN , we empirically demonstratei) Better performance .
Revised: By contrasting our V IENNA to several single-task counterparts on VXN , we empirically demonstratei) Better performance .

Original: Through exploiting cross-task relatedness, V IENNA outperforms independenttask training.
Revised: Through exploiting cross-task relatedness, V IENNA outperforms independenttask training.

Original: ii)
Revised: ii)

Original: Reduced model-size .
Revised: Reduced model-size .

Original: Training four tasks together using a single V IENNA achievesabout four times model size compression, compared with training them individually.
Revised: Training four tasks together using a single V IENNA achievesabout four times model size compression, compared with training them individually.

Original: iii) Improvedgeneralization .
Revised: iii) Improvedgeneralization .

Original: V IENNA performs robust on unseen environments, through learning task-shared,general representations.
Revised: V IENNA performs robust on unseen environments, through learning task-shared,general representations.

Original: iv) More is better .
Revised: iv) More is better .

Original: The above conclusions are typically true when we train
Revised: The above conclusions are typically true when we train

Original: V IENNA on more navigation tasks.
Revised: V IENNA on more navigation tasks.

Original: v) Multisensory integration does matter .
Revised: v) Multisensory integration does matter .

Original: Both visual (RGB anddepth)
Revised: Both visual (RGB anddepth)

Original: and aural information are crucial building blocks for general-purpose navigation robot creation.
Revised: and aural information are crucial building blocks for general-purpose navigation robot creation.

Original:  Related Work
Revised:  Related Work

Original: Embodied Navigation.
Revised: Embodied Navigation.

Original: As a fundamental element in building intelligent robots, navigation has longbeen the focus of the scientific community[17].
Revised: As a fundamental element in building intelligent robots, navigation has longbeen the focus of the scientific community[17].

Original: The availability of building-scale 3D datasets[18–21]and high-performance simulation platforms [15,22–24] led to a plethora of reproducible research ofnavigation in large-scale, visually-rich environments.
Revised: The availability of building-scale 3D datasets[18–21]and high-performance simulation platforms [15,22–24] led to a plethora of reproducible research ofnavigation in large-scale, visually-rich environments.

Original: Depending on how to specify the target goal,diverse navigation tasks are proposed to let an agent i) navigate to target coordinates ( point-goalnav.
Revised: Depending on how to specify the target goal,diverse navigation tasks are proposed to let an agent i) navigate to target coordinates ( point-goalnav.

Original: [15]), ii) find an instance of a given object category ( object-goal nav.
Revised: [15]), ii) find an instance of a given object category ( object-goal nav.

Original: [7]), iii) search for targetphotos( image-goalnav.
Revised: [7]), iii) search for targetphotos( image-goalnav.

Original: [5]), iv) locatesoundsources( audio-goalnav.
Revised: [5]), iv) locatesoundsources( audio-goalnav.

Original: [6]),or v) follownavigationinstructions ( vision-language nav.
Revised: [6]),or v) follownavigationinstructions ( vision-language nav.

Original: Aside from these battlefields, there are some more complicatedembodied tasks, such as embodied question answering [26], vision-dialog nav.
Revised: Aside from these battlefields, there are some more complicatedembodied tasks, such as embodied question answering [26], vision-dialog nav.

Original: [27–29], and multiagentnav.
Revised: [27–29], and multiagentnav.

Original: The community also made great strides in improving reinforcement learning (RL) algorithms capable of fulfilling specific navigation tasks, by using, for example, recurrent neural networks [8,15,31], map building [3,32–40], path planning [41,42], cross-modal attention [43–46], synthesized orunlabeled data [47–51], and external knowledge [7,52].
Revised: The community also made great strides in improving reinforcement learning (RL) algorithms capable of fulfilling specific navigation tasks, by using, for example, recurrent neural networks [8,15,31], map building [3,32–40], path planning [41,42], cross-modal attention [43–46], synthesized orunlabeled data [47–51], and external knowledge [7,52].

Original: However, though the learning algorithm isgeneral – RL, each solution is not; each navigation agent can only handle the one task it was trained on.
Revised: However, though the learning algorithm isgeneral – RL, each solution is not; each navigation agent can only handle the one task it was trained on.

Original: With various navigation tasks and task-specific navigation solutions, a critical question arises whetherwe can build a single general agent that works well for multiple navigation tasks .
Revised: With various navigation tasks and task-specific navigation solutions, a critical question arises whetherwe can build a single general agent that works well for multiple navigation tasks .

Original: In response, wemake two unique contributions.
Revised: In response, wemake two unique contributions.

Original: First, we build a large-scale 3D dataset that supports four representativenavigation tasks in continuous and realistic environments.
Revised: First, we build a large-scale 3D dataset that supports four representativenavigation tasks in continuous and realistic environments.

Original: In contrast, prior navigation datasets arebuilt upon different platforms and with certain assumptions/configurations ( e.g ., sparse navigationgraphs[8], discrete world representation[6]), making them hard to explore different navigation tasks inunified and standardized environments.
Revised: In contrast, prior navigation datasets arebuilt upon different platforms and with certain assumptions/configurations ( e.g ., sparse navigationgraphs[8], discrete world representation[6]), making them hard to explore different navigation tasks inunified and standardized environments.

Original: Second, we create a generalist agent which is capable ofundertaking a set of navigation tasks of different modalities/domains, and is equipped with multimodalsensors ( i.e ., RGB, depth, audio) to better address real-world scenarios.
Revised: Second, we create a generalist agent which is capable ofundertaking a set of navigation tasks of different modalities/domains, and is equipped with multimodalsensors ( i.e ., RGB, depth, audio) to better address real-world scenarios.

Original: However, existing navigationagents are trained one task at the time, each new task requiring to train a new agent instance.
Revised: However, existing navigationagents are trained one task at the time, each new task requiring to train a new agent instance.

Original: MultitaskLearning(MTL).
Revised: MultitaskLearning(MTL).

Original: MTL[12], inspiredbythe human ability totransferknowledge across different tasks[53], has led to wide success in computer vision[54] and natural language processing[55].
Revised: MTL[12], inspiredbythe human ability totransferknowledge across different tasks[53], has led to wide success in computer vision[54] and natural language processing[55].

Original: Related efforts were made along three directions [13]: i) architecture design ( i.e ., how to partitionthe model into task-specific and shared components) [56–59], ii) optimization ( i.e ., how to balancelearning between different tasks) [60–65], and iii) task relationship learning ( i.e ., how to learn andutilize task relationships to improve learning) [66–68].
Revised: Related efforts were made along three directions [13]: i) architecture design ( i.e ., how to partitionthe model into task-specific and shared components) [56–59], ii) optimization ( i.e ., how to balancelearning between different tasks) [60–65], and iii) task relationship learning ( i.e ., how to learn andutilize task relationships to improve learning) [66–68].

Original: In the field of MTRL[69–75], recent solutionsexplored knowledge transfer [76], modular networks [77,78], and policy distillation [79,80].
Revised: In the field of MTRL[69–75], recent solutionsexplored knowledge transfer [76], modular networks [77,78], and policy distillation [79,80].

Original: A fewrobotics benchmarks [81–83] are also proposed for MTRL.
Revised: A fewrobotics benchmarks [81–83] are also proposed for MTRL.

Original: However, most of these efforts werebased upon low-dimension observations, e.g ., grid-world like or game environments.
Revised: However, most of these efforts werebased upon low-dimension observations, e.g ., grid-world like or game environments.

Original: To the best ofour knowledge, there are two prior work [45,84] that addressed multitask navigation, but they onlyconsider two closely-related, language-guided navigation tasks with the same input modalities.
Revised: To the best ofour knowledge, there are two prior work [45,84] that addressed multitask navigation, but they onlyconsider two closely-related, language-guided navigation tasks with the same input modalities.

Original: Drawing inspiration from these efforts, we seek for a “universal” agent that can complete multiplenavigation tasks with a single agent instance, and distinguish ourselves by i) joint learning of fournavigation tasks with diverse input modalities, ii) visually complex and acoustically realistic operationspace, iii) multisensory integration, and iv) fully-attentive architecture based parse-and-query regime.
Revised: Drawing inspiration from these efforts, we seek for a “universal” agent that can complete multiplenavigation tasks with a single agent instance, and distinguish ourselves by i) joint learning of fournavigation tasks with diverse input modalities, ii) visually complex and acoustically realistic operationspace, iii) multisensory integration, and iv) fully-attentive architecture based parse-and-query regime.

Original: Auxiliary Learning in Embodied Navigation.
Revised: Auxiliary Learning in Embodied Navigation.

Original: There are a group of algorithms that exploit complementary objectives from auxiliary tasks to facilitate navigation policy learning.
Revised: There are a group of algorithms that exploit complementary objectives from auxiliary tasks to facilitate navigation policy learning.

Original: Specifically, supervised auxiliary tasks expose privileged information to the agent ( e.g ., depth[85], surface normals[86],semantics [26], etc .).
Revised: Specifically, supervised auxiliary tasks expose privileged information to the agent ( e.g ., depth[85], surface normals[86],semantics [26], etc .).

Original: Self-supervised auxiliary tasks derive free supervisory signals from the agent’sown experience ( e.g ., next-step visual feature prediction [87], predictive modeling [88], loop closureprediction [85], temporal distance estimation [89], navigation progress estimation [90,91], etc .).
Revised: Self-supervised auxiliary tasks derive free supervisory signals from the agent’sown experience ( e.g ., next-step visual feature prediction [87], predictive modeling [88], loop closureprediction [85], temporal distance estimation [89], navigation progress estimation [90,91], etc .).

Original: Although auxiliary learning based navigation models are also trained on a set of tasks, their ideas arefar away from ours.
Revised: Although auxiliary learning based navigation models are also trained on a set of tasks, their ideas arefar away from ours.

Original: These models still focus on only a single “main” navigation task with extra aidof auxiliary intermediate objectives, while we aim to capture and utilize common knowledge of acollection of different navigation tasks to enhance the performance on all the tasks.
Revised: These models still focus on only a single “main” navigation task with extra aidof auxiliary intermediate objectives, while we aim to capture and utilize common knowledge of acollection of different navigation tasks to enhance the performance on all the tasks.

Original: Moreover, theirauxiliary tasks, in principle, can be utilized by our agent, but they cannot handle our task setting.
Revised: Moreover, theirauxiliary tasks, in principle, can be utilized by our agent, but they cannot handle our task setting.

Original: Transformer in Embodied Navigation and MTL.
Revised: Transformer in Embodied Navigation and MTL.

Original: Inspired by the great success of Transformer[16]in sequence transduction tasks, a few recent methods applied Transformer for certain navigationtasks [40, 92–96].
Revised: Inspired by the great success of Transformer[16]in sequence transduction tasks, a few recent methods applied Transformer for certain navigationtasks [40, 92–96].

Original: Rather than sharing similar advantages in long-term memory and cross-modalinformation fusion, our method further formulates different navigation tasks as a unified process ofactive goal parsing and supportive information query.
Revised: Rather than sharing similar advantages in long-term memory and cross-modalinformation fusion, our method further formulates different navigation tasks as a unified process ofactive goal parsing and supportive information query.

Original: Through cleverly encoding all task-specificembeddings into goal parsing, our agent is able to explicitly leverage cross-task knowledge to boostdifferent navigation tasks.
Revised: Through cleverly encoding all task-specificembeddings into goal parsing, our agent is able to explicitly leverage cross-task knowledge to boostdifferent navigation tasks.

Original: There are also a few notable studies that exploit Transformer-like networkarchitectures for MTL [97–100], while none of them addresses embodied visual tasks.
Revised: There are also a few notable studies that exploit Transformer-like networkarchitectures for MTL [97–100], while none of them addresses embodied visual tasks.

Original:  VXN Dataset for Multitask Multimodal Embodied Navigation
Revised:  VXN Dataset for Multitask Multimodal Embodied Navigation

Original: 3.1 Task Collection and Dataset Acquisition
Revised: 3.1 Task Collection and Dataset Acquisition

Original: VXN includes four famous navigation tasks, i.e ., image-goal nav.
Revised: VXN includes four famous navigation tasks, i.e ., image-goal nav.

Original: [5], audio-goal nav.
Revised: [5], audio-goal nav.

Original: [6], object-goalnav.
Revised: [6], object-goalnav.

Original: [7], and vision-language nav.
Revised: [7], and vision-language nav.

Original: These tasks are with different input modalities/domains ( i.e .,visual, audio, semantic tag, and language); their original datasets adopt different world representations( i.e ., graph based [8] vs discrete [6] vs continuous [15]), environment configurations ( i.e ., visually
Revised: These tasks are with different input modalities/domains ( i.e .,visual, audio, semantic tag, and language); their original datasets adopt different world representations( i.e ., graph based [8] vs discrete [6] vs continuous [15]), environment configurations ( i.e ., visually

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: poor[22] vs perception rich [8] vs audiovisual realistic[6]), and success criteria ( i.e ., 3 m [8] vs 1 m[5]vs 1 m [7] vs 1 m [94]).
Revised: poor[22] vs perception rich [8] vs audiovisual realistic[6]), and success criteria ( i.e ., 3 m [8] vs 1 m[5]vs 1 m [7] vs 1 m [94]).

Original: Hence, to study these four tasks in a single learning system, it is desired tobuild a standardized dataset that initiates them with similar problem settings, e.g ., dynamic transition,world representations, and audiovisual properties, instead of simply combining several single-tasknavigation datasets together.
Revised: Hence, to study these four tasks in a single learning system, it is desired tobuild a standardized dataset that initiates them with similar problem settings, e.g ., dynamic transition,world representations, and audiovisual properties, instead of simply combining several single-tasknavigation datasets together.

Original: On the other hand, it is wise to maximize the reuse of existing datasets,ensuring continuity and compatibility w.r.t.
Revised: On the other hand, it is wise to maximize the reuse of existing datasets,ensuring continuity and compatibility w.r.t.

Original: former research, and reducing data annotation cost.
Revised: former research, and reducing data annotation cost.

Original: Asmanypreviousnavigationdatasets[6,8,101]arebuiltuponMatterport3D (MP3D)[19] environmentsand Habitat [15] simulator, we derive a unified, multitask navigation dataset – VXN – by convertingprevious task-specific datasets to standardized, continuous, audiovisual-rich environments :
Revised: Asmanypreviousnavigationdatasets[6,8,101]arebuiltuponMatterport3D (MP3D)[19] environmentsand Habitat [15] simulator, we derive a unified, multitask navigation dataset – VXN – by convertingprevious task-specific datasets to standardized, continuous, audiovisual-rich environments :

Original: • Our audio-goal nav.
Revised: • Our audio-goal nav.

Original: is built upon SoundSpaces [6], which offers audio renderings for MP3D and
Revised: is built upon SoundSpaces [6], which offers audio renderings for MP3D and

Original: allows to navigate sounding targets, or conduct point navigation with extra aid of audio cues.
Revised: allows to navigate sounding targets, or conduct point navigation with extra aid of audio cues.

Original: Due to
Revised: Due to

Original: heavy acoustic simulation cost, [6] uses a grid-based world model: it samples room impulse res-
Revised: heavy acoustic simulation cost, [6] uses a grid-based world model: it samples room impulse res-

Original: ponse over a discrete, horizontal plane (1.5 m above the floor with 0.5 m × 0.5 m grid size).
Revised: ponse over a discrete, horizontal plane (1.5 m above the floor with 0.5 m × 0.5 m grid size).

Original: We
Revised: We

Original: devise
Revised: devise

Original: an audio simulator to efficiently transfer grid-level audio renderings into continuous setting ( cf . §3.2).
Revised: an audio simulator to efficiently transfer grid-level audio renderings into continuous setting ( cf . §3.2).

Original: • Our vision-language nav.
Revised: • Our vision-language nav.

Original: is built upon R2R [8], which labels MP3D with linguistic navigation ins-
Revised: is built upon R2R [8], which labels MP3D with linguistic navigation ins-

Original: tructions.
Revised: tructions.

Original: R2R is yet bounded to graph-based world representation – each scene can be only observed
Revised: R2R is yet bounded to graph-based world representation – each scene can be only observed

Original: from a few fixed points ( ∼ 117) and environment topologies are pre-given.
Revised: from a few fixed points ( ∼ 117) and environment topologies are pre-given.

Original: We use [25] to convert
Revised: We use [25] to convert

Original: R2R to the continuous setting, and then adopt [6] and our audio simulator for audio rendering.
Revised: R2R to the continuous setting, and then adopt [6] and our audio simulator for audio rendering.

Original: • Our image-goal nav.
Revised: • Our image-goal nav.

Original: is built upon Habitat [15] ImageNav repository [102], which is for photo
Revised: is built upon Habitat [15] ImageNav repository [102], which is for photo

Original: target guided navigation in MP3D environments.
Revised: target guided navigation in MP3D environments.

Original: Again, continuous audio rendering is made.
Revised: Again, continuous audio rendering is made.

Original: • Our object-goalnav. is built upon Habitat2020 ObjectNav challenge[101], which requires an agent
Revised: • Our object-goalnav. is built upon Habitat2020 ObjectNav challenge[101], which requires an agent

Original: to navigate a MP3D environment to find an instance of an object class.
Revised: to navigate a MP3D environment to find an instance of an object class.

Original: A total of 21 visually well
Revised: A total of 21 visually well

Original: defined object categories ( e
Revised: defined object categories ( e

Original: g ., chair ) are considered and audio rendering is also made; but the GPS
Revised: g ., chair ) are considered and audio rendering is also made; but the GPS

Original: + Compass sensor, used in [101], is not adopted in VXN , for formalizing different task settings.
Revised: + Compass sensor, used in [101], is not adopted in VXN , for formalizing different task settings.

Original: 3.2 Task Setting and Dataset Design
Revised: 3.2 Task Setting and Dataset Design

Original: Panoramic Visual Simulator.
Revised: Panoramic Visual Simulator.

Original: With Habitat API, 360 ◦ egocentric RGBD view is rendered at 300 fps.
Revised: With Habitat API, 360 ◦ egocentric RGBD view is rendered at 300 fps.

Original: Audio Simulator.
Revised: Audio Simulator.

Original: With [6], ambisonics are generated at locations sampled in MP3D scenes andconverted to binaural audio [103], i.e ., an agent emulates two human-like ears.
Revised: With [6], ambisonics are generated at locations sampled in MP3D scenes andconverted to binaural audio [103], i.e ., an agent emulates two human-like ears.

Original: To synthesize continuousauditory scenes, we use [104] for real-time binaural room impulse responses (BRIRs) interpolation.
Revised: To synthesize continuousauditory scenes, we use [104] for real-time binaural room impulse responses (BRIRs) interpolation.

Original: We adopt Dynamic Time Wrapping [105] to temporally align left and right ear BRIRs and then mapthe warped interpolated vectors back into the “unwarped” time domain, to get BRIRs at arbitrarylocations and directions.
Revised: We adopt Dynamic Time Wrapping [105] to temporally align left and right ear BRIRs and then mapthe warped interpolated vectors back into the “unwarped” time domain, to get BRIRs at arbitrarylocations and directions.

Original: As in[94], the sounds of the 21 object categories[101] in object-goal nav.
Revised: As in[94], the sounds of the 21 object categories[101] in object-goal nav.

Original: areused for audio rendering.
Revised: areused for audio rendering.

Original: Moreover, the sounds are associated with the objects of same semanticcategories to ensure generating semantically meaningful and contextual audio [94].
Revised: Moreover, the sounds are associated with the objects of same semanticcategories to ensure generating semantically meaningful and contextual audio [94].

Original: For image-goalnav. , object-goal nav. , and vision-language nav.
Revised: For image-goalnav. , object-goal nav. , and vision-language nav.

Original: , the audio is used as background sound, which canreveal the geometry of environment [6], complement the visual cues, and make the tasks closer to thereal-world.
Revised: , the audio is used as background sound, which canreveal the geometry of environment [6], complement the visual cues, and make the tasks closer to thereal-world.

Original: For audio-goal nav.
Revised: For audio-goal nav.

Original: , the navigation target is directly specified by the audio.
Revised: , the navigation target is directly specified by the audio.

Original: Episodes and Dataset Splits.
Revised: Episodes and Dataset Splits.

Original: In VXN , each episode is defined as a tuple: ⟨ scene, audio waveform,agent start location, agent startrotation,goallocation,targetdescription ⟩ .Weusethe standard 58/11/18train / val / test split [106] of MP3D environments.
Revised: In VXN , each episode is defined as a tuple: ⟨ scene, audio waveform,agent start location, agent startrotation,goallocation,targetdescription ⟩ .Weusethe standard 58/11/18train / val / test split [106] of MP3D environments.

Original: Since previous navigation datasets [8,15,101]keep test annotations private, we only use train and val environments to create VXN ( cf .
Revised: Since previous navigation datasets [8,15,101]keep test annotations private, we only use train and val environments to create VXN ( cf .

Original: Table 1).
Revised: Table 1).

Original: Action Space.
Revised: Action Space.

Original: Agents can take four actions: move forward 0.25 m, turn-left or turn-right 15 ◦ , or stop .
Revised: Agents can take four actions: move forward 0.25 m, turn-left or turn-right 15 ◦ , or stop .

Original: Success Criterion.
Revised: Success Criterion.

Original: An episode is considered as successful if the agent i) executes stop action, ii)within 1 m of the goal location, and iii) within a time horizon of 500 actions (like [15,86,107]).
Revised: An episode is considered as successful if the agent i) executes stop action, ii)within 1 m of the goal location, and iii) within a time horizon of 500 actions (like [15,86,107]).

Original: Dataset Features.
Revised: Dataset Features.

Original: As shown in Table 2, VXN poses greater challenges: the agent needs to master fournavigation tasks with various input modalities in continuous, audiovisual complex environments, minecross-task knowledge, and reason intelligently about all the senses available to it (RGB, depth, audio).
Revised: As shown in Table 2, VXN poses greater challenges: the agent needs to master fournavigation tasks with various input modalities in continuous, audiovisual complex environments, minecross-task knowledge, and reason intelligently about all the senses available to it (RGB, depth, audio).

Original: e t -1 W e a te t e 1 panoramic image V t
Revised: e t -1 W e a te t e 1 panoramic image V t

Original: V t panoramic depth D t
Revised: V t panoramic depth D t

Original: D t
Revised: D t

Original: H t panoramic audio H t
Revised: H t panoramic audio H t

Original: o t g
Revised: o t g

Original: G
Revised: G

Original: Q
Revised: Q

Original: C t 0.2 0.
Revised: C t 0.2 0.

Original: p t washing machine 0go to the end of this corridor, turn left and enter laundry-room.
Revised: p t washing machine 0go to the end of this corridor, turn left and enter laundry-room.

Original: ， g I = ∅ g T = ∅ g A = ∅ g L g
Revised: ， g I = ∅ g T = ∅ g A = ∅ g L g

Original: I g T g
Revised: I g T g

Original: A g L τ I τ T
Revised: A g L τ I τ T

Original: τ A τ L ˜ e 1 ˜ e t −˜ e t µµ t − 1 µ t ˜ v t ˜ d t ˜ h t
Revised: τ A τ L ˜ e 1 ˜ e t −˜ e t µµ t − 1 µ t ˜ v t ˜ d t ˜ h t

Original: Target Parser (§4.2)
Revised: Target Parser (§4.2)

Original: Multitask Planner (§4.3)
Revised: Multitask Planner (§4.3)

Original: Episodic Encoder (§4.1) f M HA f M TP
Revised: Episodic Encoder (§4.1) f M HA f M TP

Original: f
Revised: f

Original: E
Revised: E

Original: HE f M SI f
Revised: HE f M SI f

Original: I
Revised: I

Original: MG f D EP
Revised: MG f D EP

Original: f A UD f A
Revised: f A UD f A

Original: TT f A TT f A TT f A VG f
Revised: TT f A TT f A TT f A VG f

Original: A VG
Revised: A VG

Original:  Our Approach
Revised:  Our Approach

Original: Problem Statement.
Revised: Problem Statement.

Original: In single-task navigation , an agent learns to reach a goal position.
Revised: In single-task navigation , an agent learns to reach a goal position.

Original: This is typicallyformulated in a RL framework that solves a partially observable Markov decision process [108]: a tuple( S , A , G , O, P, R, γ ) , where S , A , G are sets of states , actions and targets , o t = O ( s t ) denotes thelocal observation at global state s t ∈ S at epoch (decision step) t , P ( s t +1 | s t , a t ) is the transition probability from s t to s t
Revised: This is typicallyformulated in a RL framework that solves a partially observable Markov decision process [108]: a tuple( S , A , G , O, P, R, γ ) , where S , A , G are sets of states , actions and targets , o t = O ( s t ) denotes thelocal observation at global state s t ∈ S at epoch (decision step) t , P ( s t +1 | s t , a t ) is the transition probability from s t to s t

Original: +1 given action a t ∈A , R ( s, a ) ∈ R gives the reward , and γ ∈ (0 , 1) discounts futurerewards.
Revised: +1 given action a t ∈A , R ( s, a ) ∈ R gives the reward , and γ ∈ (0 , 1) discounts futurerewards.

Original: The agent uses a policy π ( a | o, g ) to produce its action a , conditioned on its local observationo and target goal g ∈ G , and optimizes its accumulated discounted reward J = (cid:80) T t ′ =
Revised: The agent uses a policy π ( a | o, g ) to produce its action a , conditioned on its local observationo and target goal g ∈ G , and optimizes its accumulated discounted reward J = (cid:80) T t ′ =

Original: t γ t ′ − t R ( s t ′ , a t ′ ) .
Revised: t γ t ′ − t R ( s t ′ , a t ′ ) .

Original: In our multitask navigation , a single agent needs to master K =4 tasks, i.e ., { audio-goal , object-goal ,image-goal , vision-language }in VXN environments.
Revised: In our multitask navigation , a single agent needs to master K =4 tasks, i.e ., { audio-goal , object-goal ,image-goal , vision-language }in VXN environments.

Original: WeformalizethisasaMTRLproblem: { ( S , A , G k ,
Revised: WeformalizethisasaMTRLproblem: { ( S , A , G k ,

Original: O, P, R k , γ k ) }
Revised: O, P, R k , γ k ) }

Original: Kk =1 , wheretheagentconcurrentlylearns K task-specificpolicies π 1: K that maximize therewards J 1: K .
Revised: Kk =1 , wheretheagentconcurrentlylearns K task-specificpolicies π 1: K that maximize therewards J 1: K .

Original: The single multitask agent is expected to exploit cross-task knowledge to achieve close orbetter navigation performance on the K tasks, compared with training K single-task agents individually.
Revised: The single multitask agent is expected to exploit cross-task knowledge to achieve close orbetter navigation performance on the K tasks, compared with training K single-task agents individually.

Original: Transformer Preliminary.
Revised: Transformer Preliminary.

Original: The core of Transformer [16] is an attention function (denoted as f A TT ),which takes a query sequence x ∈ R n × d and a context sequence y ∈ R m × d as inputs, and outputs:
Revised: The core of Transformer [16] is an attention function (denoted as f A TT ),which takes a query sequence x ∈ R n × d and a context sequence y ∈ R m × d as inputs, and outputs:

Original: [Equation]
Revised: [Equation]

Original: 1) where ˜ y ∈ R n × d is with the same length n and embedding dimension d as x , and W q,k,v ∈ R d × d arelearnable query , key , and value projection matrices, respectively.
Revised: 1) where ˜ y ∈ R n × d is with the same length n and embedding dimension d as x , and W q,k,v ∈ R d × d arelearnable query , key , and value projection matrices, respectively.

Original: Note that Eq.
Revised: Note that Eq.

Original: 1 is applicable to bothself-attention in Transformer encoder ( i.e ., x ≡ y ), and cross-attention in Transformer decoder ( i.e .,x ̸ = y ).
Revised: 1 is applicable to bothself-attention in Transformer encoder ( i.e ., x ≡ y ), and cross-attention in Transformer decoder ( i.e .,x ̸ = y ).

Original: Further, each Transformer layer block can be given as:
Revised: Further, each Transformer layer block can be given as:

Original: [Equation]
Revised: [Equation]

Original: where f M HA refers to a multi-head attention layer, derived by computing several f A TT in parallel, andf M LP is multi-layer perceptron.
Revised: where f M HA refers to a multi-head attention layer, derived by computing several f A TT in parallel, andf M LP is multi-layer perceptron.

Original: The layer normalization is omitted for brevity.
Revised: The layer normalization is omitted for brevity.

Original: Core Idea.
Revised: Core Idea.

Original: Built upon a Transformer encoder-decoder architecture, our V IENNA unifies the four VXNtasks as an attention-based, parse-and-query framework: the target description g ∈G k is online parsedinto a set of embeddings, which are used to “query” the encoded episode history; the retrieved supportive cues are fed into the corresponding policy π k for decision making.
Revised: Built upon a Transformer encoder-decoder architecture, our V IENNA unifies the four VXNtasks as an attention-based, parse-and-query framework: the target description g ∈G k is online parsedinto a set of embeddings, which are used to “query” the encoded episode history; the retrieved supportive cues are fed into the corresponding policy π k for decision making.

Original: To better handle multiple tasks,
Revised: To better handle multiple tasks,

Original: V IENNA i) learns task-wise context and involves all the task-specific embeddings into target parsing,ii) shares representations among tasks, iii) lets task-specific policies π 1: K reuse knowledge, and iv)trains the polices via a multitask version of Distributed Proximal Policy Optimization (DPPO) [109].
Revised: V IENNA i) learns task-wise context and involves all the task-specific embeddings into target parsing,ii) shares representations among tasks, iii) lets task-specific policies π 1: K reuse knowledge, and iv)trains the polices via a multitask version of Distributed Proximal Policy Optimization (DPPO) [109].

Original: V IENNA has three modules ( cf . Fig.
Revised: V IENNA has three modules ( cf . Fig.

Original: 2): i) an episodic encoder (§4.1) that fuses multisensory cues andencodes the full episode history of navigation; ii) a target parser (§4.2) that actively interprets thetarget specification into several embeddings; and iii) a multitask planner (§4.3) that uses the targetembeddings to query encoded episodic history and leverages the returned context for action prediction.
Revised: 2): i) an episodic encoder (§4.1) that fuses multisensory cues andencodes the full episode history of navigation; ii) a target parser (§4.2) that actively interprets thetarget specification into several embeddings; and iii) a multitask planner (§4.3) that uses the targetembeddings to query encoded episodic history and leverages the returned context for action prediction.

Original: 4.1 Episodic Encoder
Revised: 4.1 Episodic Encoder

Original: At the start of each episode, V IENNA receives a target description g ∈{ goal image, target sound, targetclass, language instruction } and derives an embedding vector g ∈ R d (detailed in§4.2).
Revised: At the start of each episode, V IENNA receives a target description g ∈{ goal image, target sound, targetclass, language instruction } and derives an embedding vector g ∈ R d (detailed in§4.2).

Original: At each epoch t ,
Revised: At each epoch t ,

Original: V IENNA has a 360 ◦ egocentric audiovisual perception o t , i.e ., RGB+depth+audio, of its surrounding.
Revised: V IENNA has a 360 ◦ egocentric audiovisual perception o t , i.e ., RGB+depth+audio, of its surrounding.

Original: Intra-Modal Encoders.
Revised: Intra-Modal Encoders.

Original: A visual encoder f I MG maps perceived panoramic image V t ∈ R 128 × 512 × 3 intovisual features V t =[ v 1 ,t , · · · , v N V ,t ] ∈ R N V × d , where v i,t ∈ R
Revised: A visual encoder f I MG maps perceived panoramic image V t ∈ R 128 × 512 × 3 intovisual features V t =[ v 1 ,t , · · · , v N V ,t ] ∈ R N V × d , where v i,t ∈ R

Original: d is the feature vector of i -th patch in V t .
Revised: d is the feature vector of i -th patch in V t .

Original: Similarly,
Revised: Similarly,

Original: Target-Guided Cross-Modal Encoder.
Revised: Target-Guided Cross-Modal Encoder.

Original: With the target description vector g ∈ R d , cross-attentionf A TT ( cf . Eq.1) is separately applied over V t , D t , and H t to assemble target-related sensory information:
Revised: With the target description vector g ∈ R d , cross-attentionf A TT ( cf . Eq.1) is separately applied over V t , D t , and H t to assemble target-related sensory information:

Original: [Equation]
Revised: [Equation]

Original: Then ˜ v t , ˜ d t , and ˜ h t are concatenated for attention based multisensory information integration (MSI):
Revised: Then ˜ v t , ˜ d t , and ˜ h t are concatenated for attention based multisensory information integration (MSI):

Original: [Equation]
Revised: [Equation]

Original: 4) where f M SI is achieved by stacking two self-attention based Transformer blocks ( cf .
Revised: 4) where f M SI is achieved by stacking two self-attention based Transformer blocks ( cf .

Original: Eq. 2).
Revised: Eq. 2).

Original: Episodic History Encoder.
Revised: Episodic History Encoder.

Original: At epoch t , the multimodal observation embedding o t ∈ R 3 × d and latestnavigation action embedding a t − 1 ∈ R d , are together projected into a compact “ navigation token ”:
Revised: At epoch t , the multimodal observation embedding o t ∈ R 3 × d and latestnavigation action embedding a t − 1 ∈ R d , are together projected into a compact “ navigation token ”:

Original: [Equation]
Revised: [Equation]

Original: All the past navigation tokens, e 1: t , summed with corresponding epoch embedding vectors, µ 1: t ∈ R d ,are collected into a sequence and fed into an episode history encoder (EHE) to get contextualizedhistory representation:
Revised: All the past navigation tokens, e 1: t , summed with corresponding epoch embedding vectors, µ 1: t ∈ R d ,are collected into a sequence and fed into an episode history encoder (EHE) to get contextualizedhistory representation:

Original: [Equation]
Revised: [Equation]

Original: ) where f E HE is implemented as four self-attention based Transformer blocks ( cf .
Revised: ) where f E HE is implemented as four self-attention based Transformer blocks ( cf .

Original: Eq. 2).
Revised: Eq. 2).

Original: In this way,
Revised: In this way,

Original: V IENNA is able to store and access its entire episode history of audiovisual observations and actions,leading to persistent memorization and long-term reasoning.
Revised: V IENNA is able to store and access its entire episode history of audiovisual observations and actions,leading to persistent memorization and long-term reasoning.

Original: The attended history representation ˜ e 1: twill serve as informative context for predicting the navigation action a t at epoch t (detailed in §4.3).
Revised: The attended history representation ˜ e 1: twill serve as informative context for predicting the navigation action a t at epoch t (detailed in §4.3).

Original: 4.2 Target Parser
Revised: 4.2 Target Parser

Original: go to the end of
Revised: go to the end of

Original: V IENNA is equipped with a targetparser that actively interprets thetarget g (no matter it is specified asa photo g
Revised: V IENNA is equipped with a targetparser that actively interprets thetarget g (no matter it is specified asa photo g

Original: I , sound g A , semantic tagg T , or linguistic instruction g L ) as agroup of target embeddings, conditioned on the progress of the navigation episode.
Revised: I , sound g A , semantic tagg T , or linguistic instruction g L ) as agroup of target embeddings, conditioned on the progress of the navigation episode.

Original: Guided by the online created target embeddings,valuable context are selected from episodic experiences ˜
Revised: Guided by the online created target embeddings,valuable context are selected from episodic experiences ˜

Original: e 1: t for flexible decision-making.
Revised: e 1: t for flexible decision-making.

Original: V IENNAthusformulatesvarious navigation tasks in a unified scheme, allowing to exploit cross-task knowledge.
Revised: V IENNAthusformulatesvarious navigation tasks in a unified scheme, allowing to exploit cross-task knowledge.

Original: In image-goal nav. , a goal image g I ∈
Revised: In image-goal nav. , a goal image g I ∈

Original: R 128 × 128 × 3 is given and embedded as g I =
Revised: R 128 × 128 × 3 is given and embedded as g I =

Original: f I MG
Revised: f I MG

Original: ( g I ) ∈
Revised: ( g I ) ∈

Original: R N I × d .
Revised: R N I × d .

Original: Inaudio-goalnav. , the target is signaled by the binaural sound, i.e ., g A
Revised: Inaudio-goalnav. , the target is signaled by the binaural sound, i.e ., g A

Original: = H t ∈ R 24 × 41 × 44 × 2 and g A
Revised: = H t ∈ R 24 × 41 × 44 × 2 and g A

Original: = H t
Revised: = H t

Original: =
Revised: =

Original: f
Revised: f

Original: A UD ( H t ) ∈
Revised: A UD ( H t ) ∈

Original: R N A × d .
Revised: R N A × d .

Original: In object-goal nav.
Revised: In object-goal nav.

Original: , the target is specified by a semantic tag g T ∈{ table , bed , · · · } ,and embedded into a class vector g T ∈ R 1 × d .
Revised: , the target is specified by a semantic tag g T ∈{ table , bed , · · · } ,and embedded into a class vector g T ∈ R 1 × d .

Original: In vision-language nav. , a language-based trajectoryinstruction g L is given and mapped into a sequence of word vectors g L ∈ R N L × d by a bi-LSTM.
Revised: In vision-language nav. , a language-based trajectoryinstruction g L is given and mapped into a sequence of word vectors g L ∈ R N L × d by a bi-LSTM.

Original: Atthe start of each episode, we first build an augmented target description embedding G ∈ R 4 N G × d :
Revised: Atthe start of each episode, we first build an augmented target description embedding G ∈ R 4 N G × d :

Original: [Equation]
Revised: [Equation]

Original: (7) where N G = max ( N I , N A , 1 , N L ) , τ I,A,T,L ∈ R d arelearnable task embedding vectors, and [ · ] N G copiesits input N G times.
Revised: (7) where N G = max ( N I , N A , 1 , N L ) , τ I,A,T,L ∈ R d arelearnable task embedding vectors, and [ · ] N G copiesits input N G times.

Original: Assuming V IENNA is in an image-goal nav.
Revised: Assuming V IENNA is in an image-goal nav.

Original: episode, we have g I =
Revised: episode, we have g I =

Original: f I MG
Revised: f I MG

Original: ( g I ) ∈
Revised: ( g I ) ∈

Original: R N I × d ,and g A =[ 0 ] N A × d , g T =[ 0 ] 1 × d , g L =[ 0 ] N L × d .
Revised: R N I × d ,and g A =[ 0 ] N A × d , g T =[ 0 ] 1 × d , g L =[ 0 ] N L × d .

Original: We pad g
Revised: We pad g

Original: I,A,T,L to a unified length N G , by replication,so as to get g
Revised: I,A,T,L to a unified length N G , by replication,so as to get g

Original: I,A,T,L and make them contribute equally to G .
Revised: I,A,T,L and make them contribute equally to G .

Original: We collect all the task-type embeddingsτ I,A,T,L and current target description g ∈ { g I , g A , g T , g L } into G .
Revised: We collect all the task-type embeddingsτ I,A,T,L and current target description g ∈ { g I , g A , g T , g L } into G .

Original: In §4.3, we will show thisstrategy is essential for making use of cross-task knowledge.
Revised: In §4.3, we will show thisstrategy is essential for making use of cross-task knowledge.

Original: The target description vector g ∈ R dused in Eq.
Revised: The target description vector g ∈ R dused in Eq.

Original: 3 is given as: g = f
Revised: 3 is given as: g = f

Original: A VG ( G ) , where f A VG stands for the average pooling operation.
Revised: A VG ( G ) , where f A VG stands for the average pooling operation.

Original: At epoch t , the target parser comprehends the augmented target description embedding G as a set of
Revised: At epoch t , the target parser comprehends the augmented target description embedding G as a set of

Original: N compact embeddings on-the-fly, conditioned on its episodic, contextualized history encoding ˜
Revised: N compact embeddings on-the-fly, conditioned on its episodic, contextualized history encoding ˜

Original: e 1: t :
Revised: e 1: t :

Original: [Equation]
Revised: [Equation]

Original: where f M HA is a N -head attention layer ( cf .
Revised: where f M HA is a N -head attention layer ( cf .

Original: Eq. 2), i.e ., explain G in different ways, with considerationof current episodic navigation progress.
Revised: Eq. 2), i.e ., explain G in different ways, with considerationof current episodic navigation progress.

Original: Each of the target embedding vectors q t can be viewed as aspecific, time-varying goal, used to guide action selection a t at epoch t .
Revised: Each of the target embedding vectors q t can be viewed as aspecific, time-varying goal, used to guide action selection a t at epoch t .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, given anavigation instruction “ go to the end of this corridor, turn left and · · · ”, the agent focuses more on“ go to the end of this corridor ” at the start of the navigation episode.
Revised: 3, given anavigation instruction “ go to the end of this corridor, turn left and · · · ”, the agent focuses more on“ go to the end of this corridor ” at the start of the navigation episode.

Original: After reaching the end of thecorridor, the agent shifts its attention to “ turn left ”.
Revised: After reaching the end of thecorridor, the agent shifts its attention to “ turn left ”.

Original: Here a collection of N target embeddings q 1: N are generated at each epoch t , allowing the agent to capture different aspects of target-related informationand making the time-varying goal well-planned.
Revised: Here a collection of N target embeddings q 1: N are generated at each epoch t , allowing the agent to capture different aspects of target-related informationand making the time-varying goal well-planned.

Original: For instance, there may exist several essentiallandmarks in a goal image, or multiple discriminative audio clips in target-emitted sound; duringnavigation, the agent should be able to pay attention to all these informative clues simultaneously.
Revised: For instance, there may exist several essentiallandmarks in a goal image, or multiple discriminative audio clips in target-emitted sound; duringnavigation, the agent should be able to pay attention to all these informative clues simultaneously.

Original: 4.3 Multitask Planner
Revised: 4.3 Multitask Planner

Original: At epoch t , a multitask planner (MTP) uses the diversified target embeddings Q t =[ q 1 t , · · · , q Nt ] toquery the episodic history ˜ e 1: t :
Revised: At epoch t , a multitask planner (MTP) uses the diversified target embeddings Q t =[ q 1 t , · · · , q Nt ] toquery the episodic history ˜ e 1: t :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) where f M TP is achieved by a four-layer Transformer decoder; the first two layers are shared amongthe four navigation tasks for capturing task-shared policies, while the last two layers are privatefor each task for task-specific policy learning.
Revised: ) where f M TP is achieved by a four-layer Transformer decoder; the first two layers are shared amongthe four navigation tasks for capturing task-shared policies, while the last two layers are privatefor each task for task-specific policy learning.

Original: We empirically find such shared trunk based MTPdesign yields better performance than learning task-specific policies individually or just training onesingle “universal” policy ( cf .
Revised: We empirically find such shared trunk based MTPdesign yields better performance than learning task-specific policies individually or just training onesingle “universal” policy ( cf .

Original: The retrieved context C t ∈ R N × d is mapped into the probabilitydistribution p t ∈ [0 , 1] 4 of the four navigation actions ( cf . §3.2), i.e ., p t = softmax ( f A VG ( C t ) W p ) .
Revised: The retrieved context C t ∈ R N × d is mapped into the probabilitydistribution p t ∈ [0 , 1] 4 of the four navigation actions ( cf . §3.2), i.e ., p t = softmax ( f A VG ( C t ) W p ) .

Original: As the task embeddings τ I,A,T,L are encoded into Q t , which is used to find supportive cues fromepisodic observations ˜ e 1: t for long-term reasoning and decision-making, τ I,A,T,L are essentiallytrained as task-wise context – they are sensitive to task-related cues.
Revised: As the task embeddings τ I,A,T,L are encoded into Q t , which is used to find supportive cues fromepisodic observations ˜ e 1: t for long-term reasoning and decision-making, τ I,A,T,L are essentiallytrained as task-wise context – they are sensitive to task-related cues.

Original: Thus collecting τ I,A,T,L into G( cf .
Revised: Thus collecting τ I,A,T,L into G( cf .

Original: Eq. 7) enables a clever use of cross-task knowledge.
Revised: Eq. 7) enables a clever use of cross-task knowledge.

Original: For instance, during image-goal nav.
Revised: For instance, during image-goal nav.

Original: τ
Revised: τ

Original: Acan help the agent notice some informative audio signals, τ T can alert the agent to visually essentialsemantics, while τ I can be activated by crucial landmarks.
Revised: Acan help the agent notice some informative audio signals, τ T can alert the agent to visually essentialsemantics, while τ I can be activated by crucial landmarks.

Original: Related experiments can be found in §5.2.
Revised: Related experiments can be found in §5.2.

Original: 4.4 MTRL based Multitask Navigation Training
Revised: 4.4 MTRL based Multitask Navigation Training

Original: Reward Design.
Revised: Reward Design.

Original: With standardized VXN environments,V IENNA adopts a same reward function forthe four navigation tasks, i.e ., R 1 = · · · = R 4 .
Revised: With standardized VXN environments,V IENNA adopts a same reward function forthe four navigation tasks, i.e ., R 1 = · · · = R 4 .

Original: Concretely, R 1:4 has four terms, i.e ., a sparse successreward r success , a progress reward r progress , a slack reward r slack , and an exploration reward r explore .r
Revised: Concretely, R 1:4 has four terms, i.e ., a sparse successreward r success , a progress reward r progress , a slack reward r slack , and an exploration reward r explore .r

Original: success =2 .
Revised: success =2 .

Original: 5 is only received at the end of a successful episode.
Revised: 5 is only received at the end of a successful episode.

Original: r progress = − ∆ geo_dist offers dense signalsindicating the progress that an action contributes: ∆ geo_dist gives the change in geodesic distance to thegoal position by performing the action.
Revised: r progress = − ∆ geo_dist offers dense signalsindicating the progress that an action contributes: ∆ geo_dist gives the change in geodesic distance to thegoal position by performing the action.

Original: r slack = − 10 − 3 , received at each epoch, penalizes redundantactions.
Revised: r slack = − 10 − 3 , received at each epoch, penalizes redundantactions.

Original: r explore [110] divides each environment into a voxel grid with 2 .
Revised: r explore [110] divides each environment into a voxel grid with 2 .

Original: 5 m × 2 . 5 m × 2 .
Revised: 5 m × 2 . 5 m × 2 .

Original: 5 m voxels andrewards the agent for visiting each voxel.
Revised: 5 m voxels andrewards the agent for visiting each voxel.

Original: r explore is defined as 0 .
Revised: r explore is defined as 0 .

Original: 25 η , where η = δ t /ν is a coefficient thatdecays as episode epoch t and visited voxel number ν increase, and δ is a decay constant of 0 . 995 .
Revised: 25 η , where η = δ t /ν is a coefficient thatdecays as episode epoch t and visited voxel number ν increase, and δ is a decay constant of 0 . 995 .

Original: Multitask Distributed Proximal Policy Optimization.
Revised: Multitask Distributed Proximal Policy Optimization.

Original: We present a multitask distributed proximalpolicy optimization (MDPPO) algorithm, which utilizes the power of parallel processing to train
Revised: We present a multitask distributed proximalpolicy optimization (MDPPO) algorithm, which utilizes the power of parallel processing to train

Original: MTRL agents in our continuous and large-scale environments.
Revised: MTRL agents in our continuous and large-scale environments.

Original: MDPPO is built upon (DPPO)
Revised: MDPPO is built upon (DPPO)

Original: [109],a distributed version of proximal policy optimization (PPO)
Revised: [109],a distributed version of proximal policy optimization (PPO)

Original: [111] that bounds parameter updates to atrust region to ensure stability, and distributes the computation over many parallel instances of agentand environment.
Revised: [111] that bounds parameter updates to atrust region to ensure stability, and distributes the computation over many parallel instances of agentand environment.

Original: Similarly, MDPPO has a server-client structure: each client worker has severalagent copies that collect experiences from VXN environments, compute and send PPO’s gradients to theserver; the server worker averages the received gradients, updates the agent, and synchronizes theupdated weights with the clients.
Revised: Similarly, MDPPO has a server-client structure: each client worker has severalagent copies that collect experiences from VXN environments, compute and send PPO’s gradients to theserver; the server worker averages the received gradients, updates the agent, and synchronizes theupdated weights with the clients.

Original: For balanced multitask learning, i.e ., training data in VXN are biasedbetween vision-language nav.
Revised: For balanced multitask learning, i.e ., training data in VXN are biasedbetween vision-language nav.

Original: and other navigation tasks: 10 .
Revised: and other navigation tasks: 10 .

Original: 8 K vs 2 . 

Revised: 8 K vs 2 .

Original: ∼ 5 . 

Revised: ∼ 5 .

Original: M episodes ( cf .
Revised: M episodes ( cf .

Original: Table 1),each client worker is required to build four agent copies corresponding to the four VXN tasks.
Revised: Table 1),each client worker is required to build four agent copies corresponding to the four VXN tasks.

Original: 4.5 Implementation Detail
Revised: 4.5 Implementation Detail

Original: Network Architecture.
Revised: Network Architecture.

Original: The visual encoder f I MG is made as an ImageNet [112]-pretrained ResNet50[113].
Revised: The visual encoder f I MG is made as an ImageNet [112]-pretrained ResNet50[113].

Original: The CNN features are fed into a linear layer for dimension compression and flattened into afeature sequence.
Revised: The CNN features are fed into a linear layer for dimension compression and flattened into afeature sequence.

Original: Similarly, the depth encoder f D EP is a modified ResNet50 CNN.
Revised: Similarly, the depth encoder f D EP is a modified ResNet50 CNN.

Original: The audio encoderf A UD , following [6], is a CNN of conv 8 × 8 , conv 4 × 4 , conv 3 × 3 and a linear layer, interleaved with
Revised: The audio encoderf A UD , following [6], is a CNN of conv 8 × 8 , conv 4 × 4 , conv 3 × 3 and a linear layer, interleaved with

Original: ReLU.
Revised: ReLU.

Original: All the sensory features are combined with orientation embeddings.
Revised: All the sensory features are combined with orientation embeddings.

Original: For the epoch embeddingµ , we use sinusoidal encoding.
Revised: For the epoch embeddingµ , we use sinusoidal encoding.

Original: For the target parser , N =5 target embeddings are generated at eachepoch t .
Revised: For the target parser , N =5 target embeddings are generated at eachepoch t .

Original: We set other hyper-parameters as: d =512 , N V =64 , N A =24 , N I =16 , N L =120 , N G =120 .
Revised: We set other hyper-parameters as: d =512 , N V =64 , N A =24 , N I =16 , N L =120 , N G =120 .

Original: Training and Test.
Revised: Training and Test.

Original: V IENNA is trained on 32 NVIDIA RTX 2080 GPUs, costing 4 , 608 GPU hours.
Revised: V IENNA is trained on 32 NVIDIA RTX 2080 GPUs, costing 4 , 608 GPU hours.

Original: For MDPPO, we use four client workers and set the discounted factor γ as 0 . 99 .
Revised: For MDPPO, we use four client workers and set the discounted factor γ as 0 . 99 .

Original: We use AdamW[114]optimizer with a learning rate of 2 .
Revised: We use AdamW[114]optimizer with a learning rate of 2 .

Original: 5 × 10 − 4 .
Revised: 5 × 10 − 4 .

Original: Casual attention [16] is adopted to prevent the predictionat epoch t from the influence of future tokens after t .
Revised: Casual attention [16] is adopted to prevent the predictionat epoch t from the influence of future tokens after t .

Original: Once trained, a single instance of V IENNA canconduct the four navigation tasks.
Revised: Once trained, a single instance of V IENNA canconduct the four navigation tasks.

Original: As normal, greedy prediction is adopted for action selection.
Revised: As normal, greedy prediction is adopted for action selection.

Original: [Table]
Revised: [Table]

Original:  Experiment
Revised:  Experiment

Original: In §5.
Revised: In §5.

Original: [Table]
Revised: [Table]

Original: we first report comparison results for the four VXN tasks.
Revised: we first report comparison results for the four VXN tasks.

Original: In §5.
Revised: In §5.

Original: [Table]
Revised: [Table]

Original: we conduct diagnosticstudies to examine the efficacy of our core model design.
Revised: we conduct diagnosticstudies to examine the efficacy of our core model design.

Original: More results are put in the supplementary .
Revised: More results are put in the supplementary .

Original: Baseline.
Revised: Baseline.

Original: We test several open-source task-specific navigation methods [2,5,6,25].
Revised: We test several open-source task-specific navigation methods [2,5,6,25].

Original: Note that [2,5,6]are re-trained on VXN , since they use different training data [5], world representation [6] (discretevs continuous), or object categories [2] (6 vs 21).
Revised: Note that [2,5,6]are re-trained on VXN , since they use different training data [5], world representation [6] (discretevs continuous), or object categories [2] (6 vs 21).

Original: For [25], we use its check-point but the successcriteria are different (3 m vs 1 m).
Revised: For [25], we use its check-point but the successcriteria are different (3 m vs 1 m).

Original: Thus their scores on VXN are different from the original ones.
Revised: Thus their scores on VXN are different from the original ones.

Original: Inaddition, we consider a Seq2Seq agent, which also serves as a standard baseline in [8,25]: an LSTMplannerencodes the episode history and predicts navigation actions in a sequential menner.
Revised: Inaddition, we consider a Seq2Seq agent, which also serves as a standard baseline in [8,25]: an LSTMplannerencodes the episode history and predicts navigation actions in a sequential menner.

Original: For all thefour tasks, we provide the performance of both the single-task and multitask versions of our V IENNAand Seq2Seq .
Revised: For all thefour tasks, we provide the performance of both the single-task and multitask versions of our V IENNAand Seq2Seq .

Original: Further, Random policy , i.e ., choosing actions randomly, is included.
Revised: Further, Random policy , i.e ., choosing actions randomly, is included.

Original: Metric.
Revised: Metric.

Original: Four widely-used metrics are adopted for evaluation: i) Success Rate ( SR ); ii) Navigation
Revised: Four widely-used metrics are adopted for evaluation: i) Success Rate ( SR ); ii) Navigation

Original: Error (NE); iii) Oracle success Rate (OR); and iv) Success rate weighted by Path Length (SPL) [106].
Revised: Error (NE); iii) Oracle success Rate (OR); and iv) Success rate weighted by Path Length (SPL) [106].

Original: Reproducibility.
Revised: Reproducibility.

Original: To stimulate follow-up efforts, our data and code will be released.
Revised: To stimulate follow-up efforts, our data and code will be released.

Original: 5.1 Performance Benchmarking
Revised: 5.1 Performance Benchmarking

Original: Table 3 reports the comparison results on the four VXN tasks.
Revised: Table 3 reports the comparison results on the four VXN tasks.

Original: Some key conclusions are list below:
Revised: Some key conclusions are list below:

Original: • V IENNA obtains impressive results, under val seen and unseen sets, across all the tasks and evaluation metrics.
Revised: • V IENNA obtains impressive results, under val seen and unseen sets, across all the tasks and evaluation metrics.

Original: This proves the versatility of V IENNA and the power of our parse-and-query regime.
Revised: This proves the versatility of V IENNA and the power of our parse-and-query regime.

Original: • V IENNA consistently outperforms Seq2Seq baseline, no matter they are trained on single tasksindividually or multiple tasks jointly, in both val seen and unseen sets.
Revised: • V IENNA consistently outperforms Seq2Seq , no matter they are trained on single tasks individuallyor multiple tasks jointly.

Original: V IENNA also beats othertask-specific competitors [2,5,6,25].
Revised: Compared with other task-specific competitors [2,5,6,25], VIENNA gainscomparable results on audio-goal nav . and object-goal nav . , and performs better on image-goalnav . and vision-language nav .

Original:  These results verify the effectiveness of our model design.•
Revised: tasks. These results verify the effectiveness of our model design.

Original:  When considering the performance gain from the single-task setting to multitask, V IENNA givesmuch more impressive results, compared with Seq2Seq .
Revised: • When considering the performance gain from the single-task setting to multitask, V IENNA yieldsmore promising results, compared with Seq2Seq .

Original: For example, in Table 3a, V IENNA MTgreatly outperforms V IENNA ST by 2.2% SR and 2.4% SR, on val seen and unseen , respectively; however, in the same condition, Seq2Seq MT only provides 0.7% and 0.9% SR gains overSeq2Seq ST . 
Revised: For example, in Table 3a, V IENNA MT outperforms V IENNA ST by 2.2% SR and 2.4% SR, on val seen and unseen , respectively; however, inthe same condition, Seq2Seq MT only provides 0.7% and 0.9% SR gains over Seq2Seq ST . These

Original: This demonstrates that V IENNA can make a better use of cross-task knowledge.
Revised: results demonstrates that V IENNA can make a better use of cross-task knowledge.

Original: • When considering the performance gap between seen and unseen environments, V IENNA MT ismore favored than its single-task counterpart, V IENNA ST .
Revised: • When considering the performance gap between seen and unseen environments, V IENNA MT is morefavored than its single-task counterpart, V IENNA ST .

Original: For instance, in Table 3c, V IENNA STsuffers from relatively large performance drop, i e ., 33.0% → 18.6% SR; however,V IENNA MT showsreduceddegradation, i
Revised: For instance, in Table 3c, V IENNA ST suffersfrom relatively large performance drop, i.e ., 33.0% → 18.6%SR;however,V IENNA MT showsreduced

Original: e ., 35.0% → 22.5% SR, in unseen environments.
Revised: degradation, i.e ., 35.0% → 22.5% SR, in unseen environments.

Original: This indicates that V IENNA
Revised: This indicates that investigating

Original: is able to leverage inter-task relatedness  to strengthen its generalizability.
Revised: inter-task relatedness may help to strengthen the generalizability of navigation agents.

Original: • The above results are particularly impressive considering the advantage of V IENNA in efficientparameter utilization i.e
Revised: • The above results are particularly impressive considering the advantage of V IENNA in efficient parameter utilization,

Original:  V IENNA MT (31 M) vs V IENNA ST × 4 (101 M) vs Seq2Seq MT (27 M)vs
Revised: i e ., V IENNA MT (31 M) vs V IENNA ST × 4 (101 M) vs Seq2Seq MT (27 M)

Original: Seq2Seq ST × 4 (93 M) vs [5] + [6] + [2] + [25] (165M = 40 M + 45 M + 38 M + 42 M).
Revised: vsSeq2Seq ST × 4 (93 M) vs [5] + [6] + [2] + [25] (165M = 40 M + 45 M + 38 M + 42 M).

Original: Fig.
Revised: Fig.

Original: 4 plots the training curves of V IENNA ST/MT compared to Seq2Seq ST/MT for the four VXN tasks inunseen envs.
Revised: 4 plots the training curves of V IENNA ST/MT compared to Seq2Seq ST/MT for the four VXN tasks inunseen envs.

Original: Aligning with the results in Table 3, V IENNA outperforms Seq2Seq , and benefits morefrom multiple task learning.
Revised: Aligning with the results in Table 3, V IENNA outperforms Seq2Seq , and benefits morefrom multiple task learning.

Original: This shows that V IENNA makes a better use of cross-task knowledge.
Revised: This shows that V IENNA makes a better use of cross-task knowledge.

Original: 5.2 Ablative Study
Revised: 5.2 Ablative Study

Original: To thoroughly test the efficacy of crucial components of V IENNA , we conduct a series of diagnosticstudies on vision-language nav.
Revised: To thoroughly test the efficacy of crucial components of V IENNA , we conduct a series of diagnosticstudies on vision-language nav.

Original: and audio-goal nav.
Revised: and audio-goal nav.

Original: tasks.
Revised: tasks.

Original: The results are summarized in Table 4.
Revised: The results are summarized in Table 4.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: Multisensory Integration.
Revised: Multisensory Integration.

Original: Agents in VXN are equipped with a multimodal sensor so as to find thetarget by both seeing and hearing and make our navigation setting closer the real-world.
Revised: Agents in VXN are equipped with a multimodal sensor so as to find thetarget by both seeing and hearing and make our navigation setting closer the real-world.

Original: We first studythe influence of different sensory signals ( i.e ., RGB, depth, audio) by training V IENNA with varyingsensory modalities.
Revised: We first studythe influence of different sensory signals ( i.e ., RGB, depth, audio) by training V IENNA with varyingsensory modalities.

Original: As shown in Table 4a, fusing multimodal sensory cues ( i.e ., RGBD + audio) ismore favored.
Revised: As shown in Table 4a, fusing multimodal sensory cues ( i.e ., RGBD + audio) ismore favored.

Original: For example, in VLN, although considering audio alone brings poor performance,supplementing RGBD perception with audio yields notable improvements.
Revised: For example, in VLN, although considering audio alone brings poor performance,supplementing RGBD perception with audio yields notable improvements.

Original: This suggests audio iscomplementary to visual sensory in capturing physical and semantic properties of environments.
Revised: This suggests audio iscomplementary to visual sensory in capturing physical and semantic properties of environments.

Original: Augmented Target Description Embedding.
Revised: Augmented Target Description Embedding.

Original: To better master cross-task knowledge, V IENNAaugments its episodic targets with all the four learnable task embeddings τ
Revised: To better master cross-task knowledge, V IENNAaugments its episodic targets with all the four learnable task embeddings τ

Original: I,A,T,L ( cf.
Revised: I,A,T,L ( cf.

Original: Eq. 7).
Revised: Eq. 7).

Original: Wecompare this design against two variants in Table 4b, and find such  strategy greatly boosts theperformance.
Revised: Wecompare this design against two variants in Table 4b, and find such a strategy is conducive to theperformance.

Original: This is because, through end-to-end training, τ I,A,T,L carry task-specific knowledge,e.g .,
Revised: This is because, through end-to-end training, τ I,A,T,L carry task-specific knowledge,e.g .,

Original: τ A is associated with some discriminative audio clips; τ I focuses on essential visual landmarks.
Revised: τ A is associated with some discriminative audio clips; τ I focuses on essential visual landmarks.

Original: By taking τ I,A,T,L together, V IENNA use key knowledge of different tasks in single task episodes.
Revised: By taking τ I,A,T,L together, V IENNA use key knowledge of different tasks in single task episodes.

Original: Diversified Target Parsing.
Revised: Diversified Target Parsing.

Original: We online parse the augmented target description embedding G into Ntarget embeddings Q t =[ q 1 t , · · · , q Nt ] ( cf.
Revised: We online parse the augmented target description embedding G into Ntarget embeddings Q t =[ q 1 t , · · · , q Nt ] ( cf.

Original: Eq.
Revised: Eq.

Original: 8), to achieve vivid and diversified interpretations of G .
Revised: 8), to achieve vivid and diversified interpretations of G .

Original: In Table 4c, we present evaluation scores with different numbers of generated target embeddings, i.e .,
Revised: In Table 4c, we present evaluation scores with different numbers of generated target embeddings, i.e .,

Original: N = 1 , 3 , 5 , 7 .
Revised: N = 1 , 3 , 5 , 7 .

Original: As can be seen, diversified target parsing indeed boots navigation performance.
Revised: As can be seen, diversified target parsing indeed boots navigation performance.

Original: Multitask Planner.
Revised: Multitask Planner.

Original: Several variants of multitask planner f M TP ( cf .
Revised: Several variants of multitask planner f M TP ( cf .

Original: Eq. 9) are compared in Table 4d.
Revised: Eq. 9) are compared in Table 4d.

Original: The two-layer shared trunk design, which learns both task-shared and task-specific policies, is favored.
Revised: The two-layer shared trunk design is adopted, due to its relatively better performance.

Original: Reward Function.
Revised: Reward Function.

Original: Next we examine the design of our reward function (§4.4).
Revised: Next we examine the design of our reward function (§4.4).

Original: As seen in Table 4e,each reward term is indeed useful and combining all the four terms leads to the best performance.
Revised: As seen in Table 4e,each reward term is indeed useful and combining all the four terms leads to the best performance.

Original: Multitask Learning.
Revised: Multitask Learning.

Original: Table 4f reveals the value of training V IENNA on multiple tasks: training withmore navigation tasks improves both performance and generalizability.
Revised: Table 4f reveals the value of training V IENNA on multiple tasks: training withmore navigation tasks improves both performance and generalizability.

Original: Compared to a compositionof four single-task models, multi-task V IENNA also greatly reduces the model size: 101M → 31M.
Revised: Compared to a compositionof four single-task models, multi-task V IENNA also greatly reduces the model size: 101M → 31M.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we present VXN , a large-scale 3D indoor dataset for multimodal, multitask navigation incontinuous and audiovisual complex environments.
Revised: In this work, we present VXN , a large-scale 3D indoor dataset for multimodal, multitask navigation incontinuous and audiovisual complex environments.

Original: Further, we devise V IENNA , a powerful agentthat simultaneously learns four famous navigation tasks within a single unified parsing-and-queryscheme.
Revised: Further, we devise V IENNA , a powerful agentthat simultaneously learns four famous navigation tasks within a single unified parsing-and-queryscheme.

Original: We empirically show that, through a fully attentive architecture, V IENNA is able to mineand utilize cross-task knowledge to enhance the performance on all the tasks.
Revised: We empirically show that, through a fully attentive architecture, V IENNA is able to mineand utilize cross-task knowledge to enhance the performance on all the tasks.

Original: These efforts move uscloser to a community goal of general-purpose robots capable of fulfilling a multitude of tasks.
Revised: These efforts move uscloser to a community goal of general-purpose robots capable of fulfilling a multitude of tasks.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the pa-
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the pa-

Original: per’s contributions and scope?
Revised: per’s contributions and scope?

Original: [Yes] Our main claims accurately reflect the paper’s
Revised: [Yes] Our main claims accurately reflect the paper’s

Original: contributions and scope.
Revised: contributions and scope.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] The limitations are discussed in the supplemental material.
Revised: [Yes] The limitations are discussed inthe supplemental material.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [Yes]

Original: The
Revised: The

Original: potential negative societal impacts are discussed in the supplemental material.
Revised: potential negative societal impacts are discussed in the supplemental material.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes] We have already read the ethics review guidelines and ensured that
Revised: [Yes] We have already read the ethics review guidelines and ensured that

Original: our
Revised: our

Original: paper conforms to them.
Revised: paper conforms to them.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: Our workdoes not involve any theoretical result.(b)
Revised: Our workdoes not involve any theoretical result.(b)

Original: Did you include complete proofs of all theoretical results?
Revised: Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: Our work does not
Revised: Our work does not

Original: involve any theoretical result.
Revised: involve any theoretical result.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [No] The pseudo-code
Revised: [No] The pseudo-code

Original: is included in the supplemental material.
Revised: is included in the supplemental material.

Original: Our pipeline and implementation details are
Revised: Our pipeline and implementation details are

Original: stated in §4.
Revised: stated in §4.

Original: The samples of our dataset are included in supplemental material.
Revised: The samples of our dataset are included in supplemental material.

Original: Webelieve we have provided sufficient details of our algorithm.
Revised: We believe we have provided sufficient details of our algorithm.

Original: We promise our code and
Revised: We promise our code and

Original: full dataset shall be publicly released right after acceptance.
Revised: full dataset shall be publicly released right after acceptance.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] We specify all the training details in §4.5.
Revised: [Yes] We specify all the training details in §4.5.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] We report the error bars in Fig.
Revised: [Yes] We report the error bars in Fig.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We report our compute hardware
Revised: [Yes] We report our compute hardware

Original: in §4.5.
Revised: in §4.5.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We have cited all the
Revised: [Yes] We have cited all the

Original: used code and data.
Revised: used code and data.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] Habitat Simulator [15] is ob-
Revised: [Yes] Habitat Simulator [15] is ob-

Original: tained ( https://github.com/facebookresearch/habitat-lab ) with MIT li-
Revised: tained ( https://github.com/facebookresearch/habitat-lab ) with MIT li-

Original: cense.
Revised: cense.

Original: The algorithm of Chen et al .
Revised: The algorithm of Chen et al .

Original: [6] is implemented based on the publicly released code ( https
Revised: [6] is implemented based on the publicly released code ( https

Original: //github com facebookresearch/sound-spaces )is with CC-BY-4.0 license.
Revised: //github com/facebookresearch/sound-spaces )is with CC-BY-4.0 license.

Original: The algorithm of Chaplot et al .
Revised: The algorithm of Chaplot et al .

Original: [2] is implementedbased on the publicly released code ( https://github.com/devendrachaplot/
Revised: [2] is implementedbased on the publicly released code ( https://github.com/devendrachaplot/

Original: Object-Goal
Revised: Object-Goal

Original: -Navigation ) with MIT license.
Revised: -Navigation ) with MIT license.

Original: The algorithm of Krantz et
Revised: The algorithm of Krantz et

Original: al .
Revised: al .

Original: [25] is implemented based on the publicly released code ( https://github.com/
Revised: [25] is implemented based on the publicly released code ( https://github.com/

Original: jacobkrantz/VLN-CE ) with MIT license.
Revised: jacobkrantz/VLN-CE ) with MIT license.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: We do not use any new assets excepting the list above.
Revised: We do not use any new assets excepting the list above.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes] We obtain Matterport3D [19] data by signing the terms of use
Revised: [Yes] We obtain Matterport3D [19] data by signing the terms of use

Original: ( http://kaldir.vc.in.tum.de/matterport/MP_TOS.pdf ).
Revised: ( http://kaldir.vc.in.tum.de/matterport/MP_TOS.pdf ).

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] The data used in our paper neither contain any
Revised: [Yes] The data used in our paper neither contain any

Original: personally identifiable information nor offensive content.
Revised: personally identifiable information nor offensive content.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: Our work does not involve any crowdsourcing or conducted researchwith human subjects.
Revised: Our work does not involve any crowdsourcing or conducted research with human subjects.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board

Original: (IRB) approvals, if applicable?
Revised: (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: Our work does not involve any crowdsourcing or
Revised: Our work does not involve any crowdsourcing or

Original: conducted research with human subjects.
Revised: conducted research with human subjects.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: Our work does not involve any crowdsourc-
Revised: Our work does not involve any crowdsourc-

Original: ing or conducted research with human subjects.
Revised: ing or conducted research with human subjects.