Original: Towards Versatile Embodied Navigation
Revised: Towards Versatile Embodied Navigation

Original: Abstract
Revised: Abstract

Original: With the emergence of varied visual navigation tasks ( e.g ., image-/object-/audiogoal and vision-language navigation) that specify the target in different ways, the community has made appealing advances in training specialized agents capable of handling individual navigation tasks well.
Revised: With the emergence of varied visual navigation tasks ( e.g ., image-/object-/audiogoal and vision-language navigation) that specify the target in different ways, the community has made appealing advances in training specialized agents capable of handling individual navigation tasks well.

Original: Given plenty of embodied navigation tasks and task-specific solutions, we address a more fundamental question: can we learn a single powerful agent that masters not one but multiple navigation tasks concurrently ?
Revised: Given plenty of embodied navigation tasks and task-specific solutions, we address a more fundamental question: can we learn a single powerful agent that masters not one but multiple navigation tasks concurrently ?

Original: First, we propose VXN , a large-scale 3D dataset that instantiates four classic navigation tasks in standardized, continuous, and audiovisual-rich environments.
Revised: First, we propose VXN , a large-scale 3D dataset that instantiates four classic navigation tasks in standardized, continuous, and audiovisual-rich environments.

Original: Second, we propose V IENNA , a versatile embodied navigation agent that simultaneously learns to perform the four navigation tasks with one model.
Revised: Second, we propose V IENNA , a versatile embodied navigation agent that simultaneously learns to perform the four navigation tasks with one model.

Original: Building upon a full-attentive architecture, V IENNA formulates various navigation tasks as a unified, parse-and-query procedure: the target description, augmented with four task embeddings, is comprehensively interpreted into a set of diversified goal vectors, which are refined as the navigation progresses, and used as queries to retrieve supportive context from episodic history for decision making.
Revised: Building upon a full-attentive architecture, V IENNA formulates various navigation tasks as a unified, parse-and-query procedure: the target description, augmented with four task embeddings, is comprehensively interpreted into a set of diversified goal vectors, which are refined as the navigation progresses, and used as queries to retrieve supportive context from episodic history for decision making.

Original: This enables the reuse of knowledge across navigation tasks with varying input domains/modalities.
Revised: This enables the reuse of knowledge across navigation tasks with varying input domains/modalities.

Original: We empirically demonstrate that, compared with learning each visual navigation task individually, our multitask agent achieves comparable or even better performance with reduced complexity.
Revised: We empirically demonstrate that, compared with learning each visual navigation task individually, our multitask agent achieves comparable or even better performance with reduced complexity.

Original:  Introduction
Revised:  Introduction

Original: As a fundamental research topic, visual navigation has attained extensive attention across many disciplines, including robotics [1], computer vision [2,3] and natural language processing [4].
Revised: As a fundamental research topic, visual navigation has attained extensive attention across many disciplines, including robotics [1], computer vision [2,3] and natural language processing [4].

Original: Consider a typical navigation scenario (Fig. 1), in which a human intends to direct a robot agent to navigate to a target – a buzzing washer.
Revised: Consider a typical navigation scenario (Fig. 1), in which a human intends to direct a robot agent to navigate to a target – a buzzing washer.

Original: The target can be specified by a photo of the washer ( i.e ., image-goal nav.
Revised: The target can be specified by a photo of the washer ( i.e ., image-goal nav.

Original: [5]), or the buzzing sound ( i.e ., audio-goal nav.
Revised: [5]), or the buzzing sound ( i.e ., audio-goal nav.

Original: [6]), or the corresponding semantic tag – washing machine ( i.e ., object-goal nav.
Revised: [6]), or the corresponding semantic tag – washing machine ( i.e ., object-goal nav.

Original: [7]), or linguistic instructions – “ go to the end of this corridor, turn left and enter the laundry-room ” ( i.e ., vision-language nav.
Revised: [7]), or linguistic instructions – “ go to the end of this corridor, turn left and enter the laundry-room ” ( i.e ., vision-language nav.

Original: Naturally, the agent is expected to be smart enough to execute all these kinds of navigation tasks involving varying modalities/domains ( i.e ., image, audio, semantic tag, text) with different optimal policies.
Revised: Naturally, the agent is expected to be smart enough to execute all these kinds of navigation tasks involving varying modalities/domains ( i.e ., image, audio, semantic tag, text) with different optimal policies.

Original: Contrary to our expectation, almost all existing navigation agents are specifically designed/trained for one specific task – a “versatile” agent capable of mastering multiple navigation tasks remains far beyond reach.
Revised: Contrary to our expectation, almost all existing navigation agents are specifically designed/trained for one specific task – a “versatile” agent capable of mastering multiple navigation tasks remains far beyond reach.

Original: Besides its great value in practice, investigating embodied navigation in multitask scenarios can help better understand human intelligence.
Revised: Besides its great value in practice, investigating embodied navigation in multitask scenarios can help better understand human intelligence.

Original: First, we humans can learn multiple tasks in a parallel ad hoc ∗
Revised: First, we humans can learn multiple tasks in a parallel ad hoc ∗

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: manner, and benefit from commonalities across related tasks [9].
Revised: manner, and benefit from commonalities across related tasks [9].

Original: Second, we accomplish tasks by processing and combining signals from different modalities.
Revised: Second, we accomplish tasks by processing and combining signals from different modalities.

Original: Evidences from cognitive psychology indicate that our senses are functioning together and multisensory integration is a central tenant of human intelligence [10,11].
Revised: Evidences from cognitive psychology indicate that our senses are functioning together and multisensory integration is a central tenant of human intelligence [10,11].

Original: Though the idea of multitask learning [12] was widely explored in computer vision field[13], prior attempts are often made in unsupervised and supervised learning settings; in the context of multitask reinforcement learning (MTRL)
Revised: Though the idea of multitask learning [12] was widely explored in computer vision field[13], prior attempts are often made in unsupervised and supervised learning settings; in the context of multitask reinforcement learning (MTRL)

Original: [14], not much is done for visually-rich, embodied navigation scenarios.
Revised: [14], not much is done for visually-rich, embodied navigation scenarios.

Original: One possible reason is the lack of a suitable dataset, compounded by considerable costs involved in data collection, as multiple navigation tasks should be supported.
Revised: One possible reason is the lack of a suitable dataset, compounded by considerable costs involved in data collection, as multiple navigation tasks should be supported.

Original: In response, a large-scale 3D dataset, VXN , is established to investigate multitask multimodal embodied navigation in audiovisual complex indoor environments.
Revised: In response, a large-scale 3D dataset, VXN , is established to investigate multitask multimodal embodied navigation in audiovisual complex indoor environments.

Original: VXN allows simulated robot agents to concurrently learn four tasks, i.e ., image-goal nav.
Revised: VXN allows simulated robot agents to concurrently learn four tasks, i.e ., image-goal nav.

Original: , audio-goal nav.
Revised: , audio-goal nav.

Original: , object-goal nav. , and vision-language nav.
Revised: , object-goal nav. , and vision-language nav.

Original: , in continuous, acoustically-realistic and perceptually-rich world 2 .
Revised: , in continuous, acoustically-realistic and perceptually-rich world 2 .

Original: Based on a high-throughput simulator [15], VXN instantiates different navigation tasks in unified environments following the same physical rules.
Revised: Based on a high-throughput simulator [15], VXN instantiates different navigation tasks in unified environments following the same physical rules.

Original: It equips the agents with multimodal sensors to gather information from 360 ◦ RGBD and audio observations.
Revised: It equips the agents with multimodal sensors to gather information from 360 ◦ RGBD and audio observations.

Original: Taken all together, VXN provides a realistic testbed for multitask navigation.
Revised: Taken all together, VXN provides a realistic testbed for multitask navigation.

Original: With VXN , we further develop V IENNA , a versatile embodied navigation agent that jointly learns to solve the four navigation tasks using one single model without switching among different models.
Revised: With VXN , we further develop V IENNA , a versatile embodied navigation agent that jointly learns to solve the four navigation tasks using one single model without switching among different models.

Original: Based on Transformer encoder-decoder architecture [16], V IENNA encodes the full episode history of multisensory inputs ( i.e ., RGB, depth, and audio) and navigation actions, and absorbs common knowledge across different navigation tasks with a shared decoder.
Revised: Based on Transformer encoder-decoder architecture [16], V IENNA encodes the full episode history of multisensory inputs ( i.e ., RGB, depth, and audio) and navigation actions, and absorbs common knowledge across different navigation tasks with a shared decoder.

Original: Target signals ( i.e ., goal picture, target class, aural cues, linguistic instruction) are parsed into queries , and the supportive context retrieved from the encoded history is fed to corresponding policy for task-specific decision making.
Revised: Target signals ( i.e ., goal picture, target class, aural cues, linguistic instruction) are parsed into queries , and the supportive context retrieved from the encoded history is fed to corresponding policy for task-specific decision making.

Original: With such a fully-attentive model design, V IENNA is able to comprehend multimodal observations, conduct long-term reasoning, and, more essentially, exploit cross-task knowledge.
Revised: With such a fully-attentive model design, V IENNA is able to comprehend multimodal observations, conduct long-term reasoning, and, more essentially, exploit cross-task knowledge.

Original: By contrasting our V IENNA to several single-task counterparts on VXN , we empirically demonstrate i) Better performance .
Revised: By contrasting our V IENNA to several single-task counterparts on VXN , we empirically demonstrate i) Better performance .

Original: Through exploiting cross-task relatedness, V IENNA outperforms independent task training.
Revised: Through exploiting cross-task relatedness, V IENNA outperforms independent task training.

Original: ii)
Revised: ii)

Original: Reduced model-size .
Revised: Reduced model-size .

Original: Training four tasks together using a single V IENNA achieves about four times model size compression, compared with training them individually.
Revised: Training four tasks together using a single V IENNA achieves about four times model size compression, compared with training them individually.

Original: iii)
Revised: iii)

Original: Improved generalization .
Revised: Improved generalization .

Original: V IENNA performs robust on unseen environments, through learning task-shared, general representations.
Revised: V IENNA performs robust on unseen environments, through learning task-shared, general representations.

Original: iv) More is better .
Revised: iv) More is better .

Original: The above conclusions are typically true when we train
Revised: The above conclusions are typically true when we train

Original: V IENNA on more navigation tasks.
Revised: V IENNA on more navigation tasks.

Original: v) Multisensory integration does matter .
Revised: v) Multisensory integration does matter .

Original: Both visual (RGB and depth) and aural information are crucial building blocks for general-purpose navigation robot creation.
Revised: Both visual (RGB and depth) and aural information are crucial building blocks for general-purpose navigation robot creation.

Original:  Related Work
Revised:  Related Work

Original: Embodied Navigation.
Revised: Embodied Navigation.

Original: As a fundamental element in building intelligent robots, navigation has long been the focus of the scientific community[17].
Revised: As a fundamental element in building intelligent robots, navigation has long been the focus of the scientific community[17].

Original: The availability of building-scale 3D datasets[18–21] and high-performance simulation platforms [15,22–24] led to a plethora of reproducible research of navigation in large-scale, visually-rich environments.
Revised: The availability of building-scale 3D datasets[18–21] and high-performance simulation platforms [15,22–24] led to a plethora of reproducible research of navigation in large-scale, visually-rich environments.

Original: Depending on how to specify the target goal, diverse navigation tasks are proposed to let an agent i) navigate to target coordinates ( point-goal nav.
Revised: Depending on how to specify the target goal, diverse navigation tasks are proposed to let an agent i) navigate to target coordinates ( point-goal nav.

Original: [15]), ii) find an instance of a given object category ( object-goal nav.
Revised: [15]), ii) find an instance of a given object category ( object-goal nav.

Original: [7]), iii) search for target photos( image-goalnav.
Revised: [7]), iii) search for target photos( image-goalnav.

Original: [5]), iv) locatesoundsources( audio-goalnav.
Revised: [5]), iv) locatesoundsources( audio-goalnav.

Original: [6]),or v) follownavigationinstructions ( vision-language nav.
Revised: [6]),or v) follownavigationinstructions ( vision-language nav.

Original: Aside from these battlefields, there are some more complicated embodied tasks, such as embodied question answering [26], vision-dialog nav.
Revised: Aside from these battlefields, there are some more complicated embodied tasks, such as embodied question answering [26], vision-dialog nav.

Original: [27–29], and multiagent nav.
Revised: [27–29], and multiagent nav.

Original: The community also made great strides in improving reinforcement learning (RL) algorithms capable of fulfilling specific navigation tasks, by using, for example, recurrent neural networks [8,15, 31], map building [3,32–41], path planning [42–44], cross-modal attention [41,45–48], synthesized or unlabeled data [49–54], and external knowledge [7,55].
Revised: The community also made great strides in improving reinforcement learning (RL) algorithms capable of fulfilling specific navigation tasks, by using, for example, recurrent neural networks [8,15, 31], map building [3,32–41], path planning [42–45], cross-modal attention [41,46–49], synthesized or unlabeled data [50–55], and external knowledge [7,56].

Original: However, though the learning algorithm is general – RL, each solution is not; each navigation agent can only handle the one task it was trained on.
Revised: However, though the learning algorithm is general – RL, each solution is not; each navigation agent can only handle the one task it was trained on.

Original: With various navigation tasks and task-specific navigation solutions, a critical question arises whether we can build a single general agent that works well for multiple navigation tasks .
Revised: With various navigation tasks and task-specific navigation solutions, a critical question arises whether we can build a single general agent that works well for multiple navigation tasks .

Original: In response, we make two unique contributions.
Revised: In response, we make two unique contributions.

Original: First, we build a large-scale 3D dataset that supports four representative navigation tasks in continuous and realistic environments.
Revised: First, we build a large-scale 3D dataset that supports four representative navigation tasks in continuous and realistic environments.

Original: In contrast, prior navigation datasets are built upon different platforms and with certain assumptions/configurations ( e.g ., sparse navigation graphs[8], discrete world representation[6]), making them hard to explore different navigation tasks in unified and standardized environments.
Revised: In contrast, prior navigation datasets are built upon different platforms and with certain assumptions/configurations ( e.g ., sparse navigation graphs[8], discrete world representation[6]), making them hard to explore different navigation tasks in unified and standardized environments.

Original: Second, we create a generalist agent which is capable of undertaking a set of navigation tasks of different modalities/domains, and is equipped with multimodal sensors ( i.e ., RGB, depth, audio) to better address real-world scenarios.
Revised: Second, we create a generalist agent which is capable of undertaking a set of navigation tasks of different modalities/domains, and is equipped with multimodal sensors ( i.e ., RGB, depth, audio) to better address real-world scenarios.

Original: However, existing navigation agents are trained one task at the time, each new task requiring to train a new agent instance.
Revised: However, existing navigation agents are trained one task at the time, each new task requiring to train a new agent instance.

Original: Multitask Learning (MTL).
Revised: Multitask Learning (MTL).

Original: MTL [12], inspired by the human ability to transfer knowledge across different tasks [56], has led to wide success in computer vision [57–59] and natural language processing [60].
Revised: MTL [12], inspired by the human ability to transfer knowledge across different tasks [57], has led to wide success in computer vision [58–60] and natural language processing [61].

Original: Related efforts were made along three directions [13]: i) architecture design ( i.e ., how to partition the model into task-specific and shared components) [61–64], ii) optimization ( i.e ., how to balancelearningbetweendifferenttasks) [65–70], and iii) task relationship learning ( i.e ., how to learn and utilize task relationships to improve learning) [71–73].
Revised: Related efforts were made along three directions [13]: i) architecture design ( i.e ., how to partition the model into task-specific and shared components) [62–65], ii) optimization ( i.e ., how to balancelearningbetweendifferenttasks) [66–71], and iii) task relationship learning ( i.e ., how to learn and utilize task relationships to improve learning) [72–74].

Original: In the field of MTRL [74–80], recent solutions explored knowledge transfer [81], modular networks [82,83], and policy distillation [84,85].
Revised: In the field of MTRL [75–81], recent solutions explored knowledge transfer [82], modular networks [83,84], and policy distillation [85,86].

Original: A few robotics benchmarks [86–88] are also proposed for MTRL.
Revised: A few robotics benchmarks [87–89] are also proposed for MTRL.

Original: However, most of these efforts were based upon low-dimension observations, e.g ., grid-world like or game environments.
Revised: However, most of these efforts were based upon low-dimension observations, e.g ., grid-world like or game environments.

Original: To the best of our knowledge, there are two prior work [47,89] that addressed multitask navigation, but they only consider two closely-related, language-guided navigation tasks with the same input modalities.
Revised: To the best of our knowledge, there are two prior work [48,90] that addressed multitask navigation, but they only consider two closely-related, language-guided navigation tasks with the same input modalities.

Original: Drawing inspiration from these efforts, we seek for a “universal” agent that can complete multiple navigation tasks with a single agent instance, and distinguish ourselves by i) joint learning of four navigation tasks with diverse input modalities, ii) visually complex and acoustically realistic operation space, iii) multisensory integration, and iv) fully-attentive architecture based parse-and-query regime.
Revised: Drawing inspiration from these efforts, we seek for a “universal” agent that can complete multiple navigation tasks with a single agent instance, and distinguish ourselves by i) joint learning of four navigation tasks with diverse input modalities, ii) visually complex and acoustically realistic operation space, iii) multisensory integration, and iv) fully-attentive architecture based parse-and-query regime.

Original: Auxiliary Learning in Embodied Navigation.
Revised: Auxiliary Learning in Embodied Navigation.

Original: There are a group of algorithms that exploit complementary objectives from auxiliary tasks to facilitate navigation policy learning.
Revised: There are a group of algorithms that exploit complementary objectives from auxiliary tasks to facilitate navigation policy learning.

Original: Specifically, supervised auxiliary tasks expose privileged information to the agent ( e.g ., depth[90], surface normals[91], semantics [26], etc .).
Revised: Specifically, supervised auxiliary tasks expose privileged information to the agent ( e.g ., depth[91], surface normals[92], semantics [26], etc .).

Original: Self-supervised auxiliary tasks derive free supervisory signals from the agent’s own experience ( e.g ., next-step visual feature prediction [92], predictive modeling [93], loop closure prediction [90], temporal distance estimation [94], navigation progress estimation [95,96], etc .).
Revised: Self-supervised auxiliary tasks derive free supervisory signals from the agent’s own experience ( e.g ., next-step visual feature prediction [93], predictive modeling [94], loop closure prediction [91], temporal distance estimation [95], navigation progress estimation [96,97], etc .).

Original: Although auxiliary learning based navigation models are also trained on a set of tasks, their ideas are far away from ours.
Revised: Although auxiliary learning based navigation models are also trained on a set of tasks, their ideas are far away from ours.

Original: These models still focus on only a single “main” navigation task with extra aid of auxiliary intermediate objectives, while we aim to capture and utilize common knowledge of a collection of different navigation tasks to enhance the performance on all the tasks.
Revised: These models still focus on only a single “main” navigation task with extra aid of auxiliary intermediate objectives, while we aim to capture and utilize common knowledge of a collection of different navigation tasks to enhance the performance on all the tasks.

Original: Moreover, their auxiliary tasks, in principle, can be utilized by our agent, but they cannot handle our task setting.
Revised: Moreover, their auxiliary tasks, in principle, can be utilized by our agent, but they cannot handle our task setting.

Original: Transformer in Embodied Navigation and MTL.
Revised: Transformer in Embodied Navigation and MTL.

Original: Inspired by the great success of Transformer[16] in sequence transduction tasks, a few recent methods applied Transformer for certain navigation tasks [40,97–101].
Revised: Inspired by the great success of Transformer[16] in sequence transduction tasks, a few recent methods applied Transformer for certain navigation tasks [40,98–102].

Original: Rather than sharing similar advantages in long-term memory and cross-modal information fusion, our method further formulates different navigation tasks as a unified process of active goal parsing and supportive information query.
Revised: Rather than sharing similar advantages in long-term memory and cross-modal information fusion, our method further formulates different navigation tasks as a unified process of active goal parsing and supportive information query.

Original: Through cleverly encoding all task-specific embeddings into goal parsing, our agent is able to explicitly leverage cross-task knowledge to boost different navigation tasks.
Revised: Through cleverly encoding all task-specific embeddings into goal parsing, our agent is able to explicitly leverage cross-task knowledge to boost different navigation tasks.

Original: There are also a few notable studies that exploit Transformer-like network architectures for MTL [102–105], while none of them addresses embodied visual tasks.
Revised: There are also a few notable studies that exploit Transformer-like network architectures for MTL [103–106], while none of them addresses embodied visual tasks.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Pretraining in Embodied Navigation.
Revised: Pretraining in Embodied Navigation.

Original: A series of methods decompose embodied tasks into visual (and linguistic) representation learning and policy training [52,91,106–108].
Revised: A series of methods decompose embodied tasks into visual (and linguistic) representation learning and policy training [53,92,107–109].

Original: They pretrain a general model on easily-acquired viual or multimodal data ( e
Revised: They pretrain a general model on easily-acquired viual or multimodal data ( e

Original: [Table]
Revised: [Table]

Original: g ., image captions) and fine-tune the policy for “downstream” navigation tasks.
Revised: g ., image captions) and fine-tune the policy for “downstream” navigation tasks.

Original: Though showing improved generalization and transfer abilities, the result is still a collection of independent task-specific models rather than a single agent instance
Revised: Though showing improved generalization and transfer abilities, the result is still a collection of independent task-specific models rather than a single agent instance

Original: [Table]
Revised: [Table]

Original:  VXN Dataset for Multitask Multimodal Embodied Navigation 3.1 Task Collection and Dataset Acquisition
Revised:  VXN Dataset for Multitask Multimodal Embodied Navigation 3.1 Task Collection and Dataset Acquisition

Original: VXN includes four famous navigation tasks, i
Revised: VXN includes four famous navigation tasks, i

Original: [Table]
Revised: [Table]

Original: e ., image-goal nav.
Revised: e ., image-goal nav.

Original: [5], audio-goal nav.
Revised: [5], audio-goal nav.

Original: [6], object-goal nav.
Revised: [6], object-goal nav.

Original: [Table]
Revised: [Table]

Original: and vision-language nav.
Revised: and vision-language nav.

Original: These tasks are with different input modalities/domains ( i.e ., visual, audio, semantic tag
Revised: These tasks are with different input modalities/domains ( i.e ., visual, audio, semantic tag

Original: [Table]
Revised: [Table]

Original: and language)
Revised: and language)

Original: ; their original datasets adopt different world representations ( i.e ., graph based [8] vs discrete [6] vs continuous [15]), environment configurations ( i.e ., visually poor[22] vs perception rich [8] vs audiovisual realistic[6]), and success criteria ( i.e ., 3 m [8] vs 1 m[5] vs 1 m [7] vs 1 m [99]).
Revised: ; their original datasets adopt different world representations ( i.e ., graph based [8] vs discrete [6] vs continuous [15]), environment configurations ( i.e ., visually poor[22] vs perception rich [8] vs audiovisual realistic[6]), and success criteria ( i.e ., 3 m [8] vs 1 m[5] vs 1 m [7] vs 1 m [100]).

Original: Hence, to study these four tasks in a single learning system, it is desired to build a standardized dataset that initiates them with similar problem settings, e.g ., dynamic transition, world representations, and audiovisual properties, instead of simply combining several single-task navigation datasets together.
Revised: Hence, to study these four tasks in a single learning system, it is desired to build a standardized dataset that initiates them with similar problem settings, e.g ., dynamic transition, world representations, and audiovisual properties, instead of simply combining several single-task navigation datasets together.

Original: On the other hand, it is wise to maximize the reuse of existing datasets, ensuring continuity and compatibility w.r.t.
Revised: On the other hand, it is wise to maximize the reuse of existing datasets, ensuring continuity and compatibility w.r.t.

Original: former research, and reducing data annotation cost.
Revised: former research, and reducing data annotation cost.

Original: Asmanypreviousnavigationdatasets[6,8,109]arebuiltuponMatterport3D (MP3D)[19] environments and Habitat [15] simulator, we derive a unified, multitask navigation dataset – VXN – by converting previous task-specific datasets to standardized, continuous, audiovisual-rich environments :
Revised: Asmanypreviousnavigationdatasets[6,8,110]arebuiltuponMatterport3D (MP3D)[19] environments and Habitat [15] simulator, we derive a unified, multitask navigation dataset – VXN – by converting previous task-specific datasets to standardized, continuous, audiovisual-rich environments :

Original: • Our audio-goal nav.
Revised: • Our audio-goal nav.

Original: is built upon SoundSpaces [6], which offers audio renderings for MP3D and allows to navigate sounding targets, or conduct point navigation with extra aid of audio cues.
Revised: is built upon SoundSpaces [6], which offers audio renderings for MP3D and allows to navigate sounding targets, or conduct point navigation with extra aid of audio cues.

Original: Due to heavy acoustic simulation cost, [6] uses a grid-based world model: it samples room impulse res- ponse overadiscrete,horizontalplane(1.5mabovethefloorwith0.5m × 0.5mgridsize).Wedevise an audio simulator to efficiently transfer grid-level audio renderings into continuous setting ( cf . §3.2).
Revised: Due to heavy acoustic simulation cost, [6] uses a grid-based world model: it samples room impulse res- ponse overadiscrete,horizontalplane(1.5mabovethefloorwith0.5m × 0.5mgridsize).Wedevise an audio simulator to efficiently transfer grid-level audio renderings into continuous setting ( cf . §3.2).

Original: • Our vision-language nav.
Revised: • Our vision-language nav.

Original: is built upon R2R [8], which labels MP3D with linguistic navigation instructions.
Revised: is built upon R2R [8], which labels MP3D with linguistic navigation instructions.

Original: R2R is yet bounded to graph-based world representation – each scene can be only observed from a few fixed points ( ∼ 117) and environment topologies are pre-given.
Revised: R2R is yet bounded to graph-based world representation – each scene can be only observed from a few fixed points ( ∼ 117) and environment topologies are pre-given.

Original: We use [25] to convert R2R to the continuous setting, and then adopt [6] and our audio simulator for audio rendering.
Revised: We use [25] to convert R2R to the continuous setting, and then adopt [6] and our audio simulator for audio rendering.

Original: • Our image-goalnav. is built upon Habitat[15] ImageNav repository[110], which is for photo target guided navigation in MP3D environments.
Revised: • Our image-goalnav. is built upon Habitat[15] ImageNav repository[111], which is for photo target guided navigation in MP3D environments.

Original: Again, continuous audio rendering is made.
Revised: Again, continuous audio rendering is made.

Original: • Our object-goal nav.
Revised: • Our object-goal nav.

Original: is built upon
Revised: is built upon

Original: Habitat2020 ObjectNav challenge [109], which requires an agent to navigate a MP3D environment to find an instance of an object class.
Revised: Habitat2020 ObjectNav challenge [110], which requires an agent to navigate a MP3D environment to find an instance of an object class.

Original: A total of 21 visually well defined object categories ( e.g ., chair ) are considered and audio rendering is also made; but the GPS + Compass sensor, used in [109], is not adopted in VXN , for formalizing different task settings.
Revised: A total of 21 visually well defined object categories ( e.g ., chair ) are considered and audio rendering is also made; but the GPS + Compass sensor, used in [110], is not adopted in VXN , for formalizing different task settings.

Original: 3.2 Task Setting and Dataset Design
Revised: 3.2 Task Setting and Dataset Design

Original: Panoramic Visual Simulator.
Revised: Panoramic Visual Simulator.

Original: With Habitat API, 360 ◦ egocentric RGBD view is rendered at 300 fps.
Revised: With Habitat API, 360 ◦ egocentric RGBD view is rendered at 300 fps.

Original: Audio Simulator.
Revised: Audio Simulator.

Original: With [6], ambisonics are generated at locations sampled in MP3D scenes and converted to binaural audio [111], i.e ., an agent emulates two human-like ears.
Revised: With [6], ambisonics are generated at locations sampled in MP3D scenes and converted to binaural audio [112], i.e ., an agent emulates two human-like ears.

Original: To synthesize continuous auditory scenes, we use [112] for real-time binaural room impulse responses (BRIRs) interpolation.
Revised: To synthesize continuous auditoryscenes,weuse[113] for real-time binaural room impulse responses (BRIRs) interpolation.

Original: We adopt Dynamic Time Wrapping [113] to temporally align left and right ear BRIRs and then map the warped interpolated vectors back into the “unwarped” time domain, to get BRIRs at arbitrary locations and directions.
Revised: We adopt Dynamic Time Wrapping [114] to temporally align left and right ear BRIRs and then map the warped interpolated vectors back into the “unwarped” time domain, to get BRIRs at arbitrary locations and directions.

Original: As in[99], the sounds of the 21 object categories[109] in object-goal nav.
Revised: As in [100], the sounds of the 21 object categories [110] in object-goal nav.

Original: are used for audio rendering.
Revised: are used for audio rendering.

Original: Moreover, the sounds are associated with the objects of same semantic categories to ensure generating semantically meaningful and contextual audio [99].
Revised: Moreover, the sounds are associated with the objects of same semantic categories to ensure generating semantically meaningful and contextual audio [100].

Original: For image-goal nav.
Revised: For image-goal nav.

Original: , object-goal nav. , and vision-language nav.
Revised: , object-goal nav. , and vision-language nav.

Original: , the audio is used as background sound, which can reveal the geometry of environment [6], complement the visual cues, and make the tasks closer to the real-world.
Revised: , the audio is used as background sound, which can reveal the geometry of environment [6], complement the visual cues, and make the tasks closer to the real-world.

Original: For audio-goal nav.
Revised: For audio-goal nav.

Original: , the navigation target is directly specified by the audio.
Revised: , the navigation target is directly specified by the audio.

Original: Episodes and Dataset Splits.
Revised: Episodes and Dataset Splits.

Original: In VXN , each episode is defined as a tuple: ⟨ scene, audio waveform, agent start location, agent startrotation,goallocation,targetdescription ⟩ .Weusethe standard 58/11/ train / val / test split [114] of MP3D environments.
Revised: In VXN , each episode is defined as a tuple: ⟨ scene, audio waveform, agent start location, agent startrotation,goallocation,targetdescription ⟩ .Weusethe standard 58/11/ train / val / test split [115] of MP3D environments.

Original: Since previous navigation datasets [8,15,109] keep test annotations private, we only use train and val environments to create VXN ( cf .
Revised: Since previous navigation datasets [8,15,110] keep test annotations private, we only use train and val environments to create VXN ( cf .

Original: Table 1).
Revised: Table 1).

Original: Action Space.
Revised: Action Space.

Original: We adopt a panoramic action space, which is widely used in recent embodied robotic tasks [3,43].
Revised: We adopt a panoramic action space, which is widely used in recent embodied robotic tasks [3,43].

Original: Specifically, the panoramic view is horizontally discreted into a total of 12 sub-views.
Revised: Specifically, the panoramic view is horizontally discreted into a total of 12 sub-views.

Original: Agents can move towards a sub-view 0.25 m or stop .
Revised: Agents can move towards a sub-view 0.25 m or stop .

Original: Success Criterion.
Revised: Success Criterion.

Original: An episode is considered as successful if the agent i) executes stop action, ii) within 1 m of the goal location, and iii) within a time horizon of 500 actions (as in [15,91,115]).
Revised: An episode is considered as successful if the agent i) executes stop action, ii) within 1 m of the goal location, and iii) within a time horizon of 500 actions (as in [15,92,116]).

Original: Dataset Features.
Revised: Dataset Features.

Original: As shown in Table 2, VXN poses greater challenges: the agent needs to master four navigation tasks with various input modalities in continuous, audiovisual complex environments, mine cross-task knowledge, and reason intelligently about all the senses available to it (RGB, depth, audio).
Revised: As shown in Table 2, VXN poses greater challenges: the agent needs to master four navigation tasks with various input modalities in continuous, audiovisual complex environments, mine cross-task knowledge, and reason intelligently about all the senses available to it (RGB, depth, audio).

Original:  Our Approach
Revised:  Our Approach

Original: Problem Statement.
Revised: Problem Statement.

Original: In single-task navigation , an agent learns to reach a goal position.
Revised: In single-task navigation , an agent learns to reach a goal position.

Original: This is typically formulated in a RL framework that solves a partially observable Markov decision process [116]: a tuple ( S , A , G , O, P, R, γ ) , where S , A , G are sets of states , actions and targets , o t
Revised: This is typically formulated in a RL framework that solves a partially observable Markov decision process [117]: a tuple ( S , A , G , O, P, R, γ ) , where S , A , G are sets of states , actions and targets , o t

Original: = O ( s t ) denotes the local observation at global state s t ∈ S at epoch (decision step) t , P ( s t +1 | s t , a t ) is the transition probability from s t to s t
Revised: = O ( s t ) denotes the local observation at global state s t ∈ S at epoch (decision step) t , P ( s t +1 | s t , a t ) is the transition probability from s t to s t

Original: +1 given action a t ∈A , R ( s, a ) ∈ R gives the reward , and γ ∈ (0 , 1) discounts future rewards.
Revised: +1 given action a t ∈A , R ( s, a ) ∈ R gives the reward , and γ ∈ (0 , 1) discounts future rewards.

Original: The agent uses a policy π ( a | o, g ) to produce its action a , conditioned on its local observation o and target goal g ∈ G , and optimizes its accumulated discounted reward J = (cid:80)
Revised: The agent uses a policy π ( a | o, g ) to produce its action a , conditioned on its local observation o and target goal g ∈ G , and optimizes its accumulated discounted reward J = (cid:80)

Original: Tt ′
Revised: Tt ′

Original: =
Revised: =

Original: t γ t ′ − t R ( s t ′ , a t ′ ) .
Revised: t γ t ′ − t R ( s t ′ , a t ′ ) .

Original: In our multitask navigation , a single agent needs to master K =4 tasks, i.e ., { audio-goal , object-goal , image-goal , vision-language }in VXN environments.
Revised: In our multitask navigation , a single agent needs to master K =4 tasks, i.e ., { audio-goal , object-goal , image-goal , vision-language }in VXN environments.

Original: WeformalizethisasaMTRLproblem: { ( S , A , G k ,
Revised: WeformalizethisasaMTRLproblem: { ( S , A , G k ,

Original: O, P, R k , γ k ) }
Revised: O, P, R k , γ k ) }

Original: Kk =1 , wheretheagentconcurrentlylearns K task-specificpolicies π 1: K that maximize the rewards J 1: K .
Revised: Kk =1 , wheretheagentconcurrentlylearns K task-specificpolicies π 1: K that maximize the rewards J 1: K .

Original: The single multitask agent is expected to exploit cross-task knowledge to achieve close or better navigation performance on the K tasks, compared with training K single-task agents individually.
Revised: The single multitask agent is expected to exploit cross-task knowledge to achieve close or better navigation performance on the K tasks, compared with training K single-task agents individually.

Original: Transformer Preliminary.
Revised: Transformer Preliminary.

Original: The core of Transformer [16] is an attention function (denoted as f A TT ), which takes a query sequence x ∈ R n × d and a context sequence y ∈ R m × d as inputs, and outputs:
Revised: The core of Transformer [16] is an attention function (denoted as f A TT ), which takes a query sequence x ∈ R n × d and a context sequence y ∈ R m × d as inputs, and outputs:

Original: [Equation]
Revised: [Equation]

Original: (1) where ˜ y ∈ R n × d is with the same length n and embedding dimension d as x , and W q,k,v
Revised: (1) where ˜ y ∈ R n × d is with the same length n and embedding dimension d as x , and W q,k,v

Original: ∈ R d
Revised: ∈ R d

Original: × d are learnable query , key , and value projection matrices, respectively.
Revised: × d are learnable query , key , and value projection matrices, respectively.

Original: Note that Eq.
Revised: Note that Eq.

Original: 1 is applicable to both self-attention in Transformer encoder ( i.e ., x ≡ y ), and cross-attention in Transformer decoder ( i.e ., x ̸ = y ).
Revised: 1 is applicable to both self-attention in Transformer encoder ( i.e ., x ≡ y ), and cross-attention in Transformer decoder ( i.e ., x ̸ = y ).

Original: Further, each Transformer layer block can be given as:
Revised: Further, each Transformer layer block can be given as:

Original: [Equation]
Revised: [Equation]

Original: where f M HA refers to a multi-head attention layer, derived by computing several f A TT in parallel, and f M LP is multi-layer perceptron.
Revised: where f M HA refers to a multi-head attention layer, derived by computing several f A TT in parallel, and f M LP is multi-layer perceptron.

Original: The layer normalization is omitted for brevity.
Revised: The layer normalization is omitted for brevity.

Original: Core Idea.
Revised: Core Idea.

Original: Built upon a Transformer encoder-decoder architecture, our V IENNA unifies the four VXN tasks as an attention-based, parse-and-query framework: the target description g ∈G k is online parsed into a set of embeddings, which are used to “query” the encoded episode history; the retrieved supportive cues are fed into the corresponding policy π k for decision making.
Revised: Built upon a Transformer encoder-decoder architecture, our V IENNA unifies the four VXN tasks as an attention-based, parse-and-query framework: the target description g ∈G k is online parsed into a set of embeddings, which are used to “query” the encoded episode history; the retrieved supportive cues are fed into the corresponding policy π k for decision making.

Original: To better handle multiple tasks,
Revised: To better handle multiple tasks,

Original: V IENNA i) learns task-wise context and involves all the task-specific embeddings into target parsing, ii) shares representations among tasks, iii) lets task-specific policies π 1: K reuse knowledge, and iv) trains the polices via a multitask version of Distributed Proximal Policy Optimization (DPPO)
Revised: V IENNA i) learns task-wise context and involves all the task-specific embeddings into target parsing, ii) shares representations among tasks, iii) lets task-specific policies π 1: K reuse knowledge, and iv) trains the polices via a multitask version of Distributed Proximal Policy Optimization (DPPO)

Original: V IENNA has three modules ( cf . Fig.
Revised: V IENNA has three modules ( cf . Fig.

Original: 2): i) an episodic encoder (§4.1) that fuses multisensory cues and encodes the full episode history of navigation; ii) a target parser (§4.2) that actively interprets the e t -1 W
Revised: 2): i) an episodic encoder (§4.1) that fuses multisensory cues and encodes the full episode history of navigation; ii) a target parser (§4.2) that actively interprets the e t -1 W

Original: e a te t e 1 panoramic image V t
Revised: e a te t e 1 panoramic image V t

Original: V t panoramic depth D t
Revised: V t panoramic depth D t

Original: D t
Revised: D t

Original: H t panoramic audio H t
Revised: H t panoramic audio H t

Original: o t g
Revised: o t g

Original: G
Revised: G

Original: Q
Revised: Q

Original: C t p t 0washing machine 0go to the end of this corridor, turn left and enter laundry-room.
Revised: C t p t 0washing machine 0go to the end of this corridor, turn left and enter laundry-room.

Original: ， g I = ∅ g T = ∅ g A = ∅ g L g
Revised: ， g I = ∅ g T = ∅ g A = ∅ g L g

Original: I g T g
Revised: I g T g

Original: A g L τ I τ T
Revised: A g L τ I τ T

Original: τ A τ L ˜ e 1 ˜ e t −˜ e t µµ t − 1 µ t ˜ v t ˜ d t ˜ h t
Revised: τ A τ L ˜ e 1 ˜ e t −˜ e t µµ t − 1 µ t ˜ v t ˜ d t ˜ h t

Original: Target Parser (§4.2)
Revised: Target Parser (§4.2)

Original: Multitask Planner (§4.3)
Revised: Multitask Planner (§4.3)

Original: Episodic Encoder (§4.1) f M HA f M TP
Revised: Episodic Encoder (§4.1) f M HA f M TP

Original: f
Revised: f

Original: E
Revised: E

Original: HE f M SI f
Revised: HE f M SI f

Original: I
Revised: I

Original: MG f D EP
Revised: MG f D EP

Original: f A UD f A
Revised: f A UD f A

Original: TT f A TT f A TT f A VG f
Revised: TT f A TT f A TT f A VG f

Original: A VG
Revised: A VG

Original: 4.1 Episodic Encoder
Revised: 4.1 Episodic Encoder

Original: At the start of each episode, V IENNA receives a target description g ∈{ goal image, target sound, target class, language instruction } and derives an embedding vector g ∈ R d (detailed in§4.2).
Revised: At the start of each episode, V IENNA receives a target description g ∈{ goal image, target sound, target class, language instruction } and derives an embedding vector g ∈ R d (detailed in§4.2).

Original: At each epoch t ,
Revised: At each epoch t ,

Original: V IENNA has a 360 ◦ egocentric audiovisual perception o t , i.e ., RGB+depth+audio, of its surrounding.
Revised: V IENNA has a 360 ◦ egocentric audiovisual perception o t , i.e ., RGB+depth+audio, of its surrounding.

Original: Intra-Modal Encoders.
Revised: Intra-Modal Encoders.

Original: A visual encoder f I MG maps perceived panoramic image V t ∈ R 12 × 224 × 224 ×into visual features V t =[ v 1 ,t , · · · , v 12 ,t ] ∈ R 12 × d , where v i,t ∈ R
Revised: A visual encoder f I MG maps perceived panoramic image V t ∈ R 12 × 224 × 224 ×into visual features V t =[ v 1 ,t , · · · , v 12 ,t ] ∈ R 12 × d , where v i,t ∈ R

Original: d is the feature vector of i -th sub-view in V t .
Revised: d is the feature vector of i -th sub-view in V t .

Original: Similarly,
Revised: Similarly,

Original: Target-Guided Cross-Modal Encoder.
Revised: Target-Guided Cross-Modal Encoder.

Original: With the target description vector g ∈ R d , cross-attention f
Revised: With the target description vector g ∈ R d , cross-attention f

Original: A TT ( cf . Eq.1) is separately applied over V t , D t , and H t to assemble target-related sensory information:
Revised: A TT ( cf . Eq.1) is separately applied over V t , D t , and H t to assemble target-related sensory information:

Original: [Equation]
Revised: [Equation]

Original: Then ˜ v t , ˜ d t , and ˜ h t are concatenated for attention based multisensory information integration (MSI):
Revised: Then ˜ v t , ˜ d t , and ˜ h t are concatenated for attention based multisensory information integration (MSI):

Original: [Equation]
Revised: [Equation]

Original: where f M SI is achieved by stacking two self-attention based Transformer blocks ( cf .
Revised: where f M SI is achieved by stacking two self-attention based Transformer blocks ( cf .

Original: Eq. 2).
Revised: Eq. 2).

Original: Episodic History Encoder.
Revised: Episodic History Encoder.

Original: At epoch t , the multimodal observation embedding o t ∈ R 3 × d and latest navigation action embedding a t − 1 ∈ R d , are together projected into a compact “ navigation token ”:
Revised: At epoch t , the multimodal observation embedding o t ∈ R 3 × d and latest navigation action embedding a t − 1 ∈ R d , are together projected into a compact “ navigation token ”:

Original: [Equation]
Revised: [Equation]

Original: All the past navigation tokens, e 1: t , summed with corresponding epoch embedding vectors, µ 1: t ∈ R d , are collected into a sequence and fed into an episode history encoder (EHE) to get contextualized history representation:
Revised: All the past navigation tokens, e 1: t , summed with corresponding epoch embedding vectors, µ 1: t ∈ R d , are collected into a sequence and fed into an episode history encoder (EHE) to get contextualized history representation:

Original: [Equation]
Revised: [Equation]

Original: ) , (6) where f E HE is implemented as four self-attention based Transformer blocks ( cf .
Revised: ) , (6) where f E HE is implemented as four self-attention based Transformer blocks ( cf .

Original: Eq. 2).
Revised: Eq. 2).

Original: In this way, V IENNA is able to store and access its entire episode history of audiovisual observations and actions, leading to persistent memorization and long-term reasoning.
Revised: In this way, V IENNA is able to store and access its entire episode history of audiovisual observations and actions, leading to persistent memorization and long-term reasoning.

Original: The attended history representation ˜ e 1: t will serve as informative context for predicting the navigation action a t at epoch t (detailed in §4.3).
Revised: The attended history representation ˜ e 1: t will serve as informative context for predicting the navigation action a t at epoch t (detailed in §4.3).

Original: 4.2 Target Parser
Revised: 4.2 Target Parser

Original: V IENNA is equipped with a target parser that actively interprets the target g (no matter it is specified as a photo
Revised: V IENNA is equipped with a target parser that actively interprets the target g (no matter it is specified as a photo

Original: g I , sound g A , semantic tag g T , or linguistic instruction g L ) as a group of target embeddings, conditioned on the progress of the navigation episode.
Revised: g I , sound g A , semantic tag g T , or linguistic instruction g L ) as a group of target embeddings, conditioned on the progress of the navigation episode.

Original: Guided by the online created target embeddings, valuable context are selected from episodic experiences ˜
Revised: Guided by the online created target embeddings, valuable context are selected from episodic experiences ˜

Original: e 1: t for flexible decision-making.
Revised: e 1: t for flexible decision-making.

Original: V IENNA thus formulates various navigation tasks in a unified scheme, allowing to exploit cross-task knowledge.
Revised: V IENNA thus formulates various navigation tasks in a unified scheme, allowing to exploit cross-task knowledge.

Original: In image-goal nav. , a goal image g I ∈
Revised: In image-goal nav. , a goal image g I ∈

Original: R 224 × 224 × 3 is given and embedded as g I =
Revised: R 224 × 224 × 3 is given and embedded as g I =

Original: f I MG
Revised: f I MG

Original: ( g I ) ∈
Revised: ( g I ) ∈

Original: R N I × d .
Revised: R N I × d .

Original: In audio-goalnav. , the target is signaled by the binaural sound, i.e ., g A
Revised: In audio-goalnav. , the target is signaled by the binaural sound, i.e ., g A

Original: = H t ∈ R 12 × 41 × 44 × 2 and g A
Revised: = H t ∈ R 12 × 41 × 44 × 2 and g A

Original: = H t
Revised: = H t

Original: =
Revised: =

Original: f
Revised: f

Original: A UD ( H t ) ∈
Revised: A UD ( H t ) ∈

Original: R N A × d .
Revised: R N A × d .

Original: In object-goal nav.
Revised: In object-goal nav.

Original: , the target is specified by a semantic tag g T ∈{ table , bed , · · · } , and embedded into a class vector g T ∈ R 1 × d .
Revised: , the target is specified by a semantic tag g T ∈{ table , bed , · · · } , and embedded into a class vector g T ∈ R 1 × d .

Original: In vision-language nav. , a language-based trajectory instruction g L is given and mapped into a sequence of word vectors g L ∈ R N L × d by a bi-LSTM.
Revised: In vision-language nav. , a language-based trajectory instruction g L is given and mapped into a sequence of word vectors g L ∈ R N L × d by a bi-LSTM.

Original: At the start of each episode, we first build an augmented target description embedding G ∈ R 4 N G × d :
Revised: At the start of each episode, we first build an augmented target description embedding G ∈ R 4 N G × d :

Original: [Equation]
Revised: [Equation]

Original: ) where N G = max ( N I , N A , 1 , N L ) ,
Revised: ) where N G = max ( N I , N A , 1 , N L ) ,

Original: τ I,A,T,L ∈ R d arelearnable task embedding vectors, and [ · ] N G copies its input N G times.
Revised: τ I,A,T,L ∈ R d arelearnable task embedding vectors, and [ · ] N G copies its input N G times.

Original: Assuming V IENNA is in an image-goal nav.
Revised: Assuming V IENNA is in an image-goal nav.

Original: episode, we have g I =
Revised: episode, we have g I =

Original: f I MG
Revised: f I MG

Original: ( g I ) ∈
Revised: ( g I ) ∈

Original: R N I × d , and g A =[ 0 ] N A × d , g T =[ 0 ] 1 × d , g L =[ 0 ] N L × d .
Revised: R N I × d , and g A =[ 0 ] N A × d , g T =[ 0 ] 1 × d , g L =[ 0 ] N L × d .

Original: We pad g
Revised: We pad g

Original: I,A,T,L to a unified length N G , by replication, so as to get g
Revised: I,A,T,L to a unified length N G , by replication, so as to get g

Original: I,A,T,L and make them contribute equally to G .
Revised: I,A,T,L and make them contribute equally to G .

Original: We collect all the task-type embeddings τ
Revised: We collect all the task-type embeddings τ

Original: I,A,T,L and current target description g ∈ { g I , g A , g T , g L } into G .
Revised: I,A,T,L and current target description g ∈ { g I , g A , g T , g L } into G .

Original: In §4.3, we will show this strategy is essential for making use of cross-task knowledge.
Revised: In §4.3, we will show this strategy is essential for making use of cross-task knowledge.

Original: The target description vector g ∈ R d used in Eq.
Revised: The target description vector g ∈ R d used in Eq.

Original: 3 is given as: g = f
Revised: 3 is given as: g = f

Original: A VG ( G ) , where f A VG stands for the average pooling operation.
Revised: A VG ( G ) , where f A VG stands for the average pooling operation.

Original: At epoch t , the target parser comprehends the augmented target description embedding G as a set of
Revised: At epoch t , the target parser comprehends the augmented target description embedding G as a set of

Original: N compact embeddings on-the-fly, conditioned on its episodic, contextualized history encoding ˜
Revised: N compact embeddings on-the-fly, conditioned on its episodic, contextualized history encoding ˜

Original: e 1: t :
Revised: e 1: t :

Original: [Equation]
Revised: [Equation]

Original: where f M HA is a N -head attention layer ( cf .
Revised: where f M HA is a N -head attention layer ( cf .

Original: Eq. 2), i.e ., explain G in different ways, with consideration of current episodic navigation progress.
Revised: Eq. 2), i.e ., explain G in different ways, with consideration of current episodic navigation progress.

Original: Each of the target embedding vectors q t can be viewed as a specific, time-varying goal, used to guide action selection a t at epoch t .
Revised: Each of the target embedding vectors q t can be viewed as a specific, time-varying goal, used to guide action selection a t at epoch t .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, given a navigation instruction “ go to the end of this corridor, turn left and · · · ”, the agent focuses more on “ go to the end of this corridor ” at the start of the navigation episode.
Revised: 3, given a navigation instruction “ go to the end of this corridor, turn left and · · · ”, the agent focuses more on “ go to the end of this corridor ” at the start of the navigation episode.

Original: After reaching the end of the corridor, the agent shifts its attention to “ turn left ”.
Revised: After reaching the end of the corridor, the agent shifts its attention to “ turn left ”.

Original: Here a collection of N target embeddings q 1: N are generated at each epoch t , allowing the agent to capture different aspects of target-related information and making the time-varying goal well-planned.
Revised: Here a collection of N target embeddings q 1: N are generated at each epoch t , allowing the agent to capture different aspects of target-related information and making the time-varying goal well-planned.

Original: For instance, there may exist several essential landmarks in a goal image, or multiple discriminative audio clips in target-emitted sound; during navigation, the agent should be able to pay attention to all these informative clues simultaneously.
Revised: For instance, there may exist several essential landmarks in a goal image, or multiple discriminative audio clips in target-emitted sound; during navigation, the agent should be able to pay attention to all these informative clues simultaneously.

Original: 4.3 Multitask Planner
Revised: 4.3 Multitask Planner

Original: At epoch t , a multitask planner (MTP) uses the diversified target embeddings Q t =[ q 1 t , · · · , q Nt ] to query the episodic history ˜
Revised: At epoch t , a multitask planner (MTP) uses the diversified target embeddings Q t =[ q 1 t , · · · , q Nt ] to query the episodic history ˜

Original: e 1: t :
Revised: e 1: t :

Original: [Equation]
Revised: [Equation]

Original: where f M TP is achieved by a four-layer Transformer decoder; the first two layers are shared among the four navigation tasks for capturing task-shared policies, while the last two layers are private for each task for task-specific policy learning.
Revised: where f M TP is achieved by a four-layer Transformer decoder; the first two layers are shared among the four navigation tasks for capturing task-shared policies, while the last two layers are private for each task for task-specific policy learning.

Original: We empirically find such shared trunk based MTP design yields better performance than learning task-specific policies individually or just training one single “universal” policy ( cf .
Revised: We empirically find such shared trunk based MTP design yields better performance than learning task-specific policies individually or just training one single “universal” policy ( cf .

Original: The decision-making is conditioned on the retrieved context C t ∈ R N × d , and the presentations of current multi-modal observations ( cf . §3.2), including V t =
Revised: The decision-making is conditioned on the retrieved context C t ∈ R N × d , and the presentations of current multi-modal observations ( cf . §3.2), including V t =

Original: [ v 1 ,t , · · · , v 12 ,t ] ∈ R 12 × d , D t =
Revised: [ v 1 ,t , · · · , v 12 ,t ] ∈ R 12 × d , D t =

Original: [ d 1 ,t , · · · , d 12 ,t ] ∈ R 12 × d , and A t =[ a 1 ,t , · · · , a 12 ,t ] ∈ R 12 × d .
Revised: [ d 1 ,t , · · · , d 12 ,t ] ∈ R 12 × d , and A t =[ a 1 ,t , · · · , a 12 ,t ] ∈ R 12 × d .

Original: Specifically, at epoch t , V IENNA makes navigate decision by choosing between the 12 current sub-views, as well as an extra STOP action.
Revised: Specifically, at epoch t , V IENNA makes navigate decision by choosing between the 12 current sub-views, as well as an extra STOP action.

Original: Given 12 subview action embeddings { b i,t ∈ R 3 × d } 12 i =1 , i.e ., o i,t =[ v i,t , d i,t , h i,t ] as well as a STOP action embedding, i.e ., b 13 ,t = ⃗ 0 , represented as an all-zero vector, V IENNA predicts a probability distribution p t =[ p 1 ,t , · · · , p 13 ,t ] :
Revised: Given 12 subview action embeddings { b i,t ∈ R 3 × d } 12 i =1 , i.e ., o i,t =[ v i,t , d i,t , h i,t ] as well as a STOP action embedding, i.e ., b 13 ,t = ⃗ 0 , represented as an all-zero vector, V IENNA predicts a probability distribution p t =[ p 1 ,t , · · · , p 13 ,t ] :

Original: [Equation]
Revised: [Equation]

Original: where i ∈ { 1 , · · · , 13 } .
Revised: where i ∈ { 1 , · · · , 13 } .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: As the task embeddings τ I,A,T,L are encoded into Q t , which is used to find supportive cues from episodic observations ˜ e 1: t for long-term reasoning and decision-making, τ I,A,T,L are essentially trained as task-wise context – they are sensitive to task-related cues.
Revised: As the task embeddings τ I,A,T,L are encoded into Q t , which is used to find supportive cues from episodic observations ˜ e 1: t for long-term reasoning and decision-making, τ I,A,T,L are essentially trained as task-wise context – they are sensitive to task-related cues.

Original: Thus collecting τ I,A,T,L into G ( cf .
Revised: Thus collecting τ I,A,T,L into G ( cf .

Original: Eq. 7) enables a clever use of cross-task knowledge.
Revised: Eq. 7) enables a clever use of cross-task knowledge.

Original: For instance, during image-goal nav.
Revised: For instance, during image-goal nav.

Original: , τ A can help the agent notice some informative audio signals, τ T can alert the agent to visually essential semantics, while τ I can be activated by crucial landmarks.
Revised: , τ A can help the agent notice some informative audio signals, τ T can alert the agent to visually essential semantics, while τ I can be activated by crucial landmarks.

Original: Related experiments can be found in §5.2.
Revised: Related experiments can be found in §5.2.

Original: 4.4 MTRL based Multitask Navigation Training
Revised: 4.4 MTRL based Multitask Navigation Training

Original: Reward Design.
Revised: Reward Design.

Original: With standardized VXN environments,V IENNA adopts a same reward function for the four navigation tasks, i.e ., R 1 = · · · = R 4 .
Revised: With standardized VXN environments,V IENNA adopts a same reward function for the four navigation tasks, i.e ., R 1 = · · · = R 4 .

Original: Concretely, R 1:4 has four terms, i.e ., a sparse success reward r success , a progress reward r progress , a slack reward r slack , and an exploration reward r explore .
Revised: Concretely, R 1:4 has four terms, i.e ., a sparse success reward r success , a progress reward r progress , a slack reward r slack , and an exploration reward r explore .

Original: r success =2 .
Revised: r success =2 .

Original: 5 is only received at the end of a successful episode.
Revised: 5 is only received at the end of a successful episode.

Original: r progress = − ∆ geo_dist offers dense signals indicating the progress that an action contributes: ∆ geo_dist gives the change in geodesic distance to the goal position by performing the action.
Revised: r progress = − ∆ geo_dist offers dense signals indicating the progress that an action contributes: ∆ geo_dist gives the change in geodesic distance to the goal position by performing the action.

Original: r slack = − 10 − 3 , received at each epoch, penalizes redundant actions.
Revised: r slack = − 10 − 3 , received at each epoch, penalizes redundant actions.

Original: r explore [118] divides each environment into a voxel grid with 2 .
Revised: r explore [119] divides each environment into a voxel grid with 2 .

Original: 5 m × 2 . 5 m × 2 .
Revised: 5 m × 2 . 5 m × 2 .

Original: 5 m voxels and rewards the agent for visiting each voxel.
Revised: 5 m voxels and rewards the agent for visiting each voxel.

Original: r explore is defined as 0 .
Revised: r explore is defined as 0 .

Original: 25 η , where η = δ t /ν is a coefficient that decays as episode epoch t and visited voxel number ν increase, and δ is a decay constant of 0 . 995 .
Revised: 25 η , where η = δ t /ν is a coefficient that decays as episode epoch t and visited voxel number ν increase, and δ is a decay constant of 0 . 995 .

Original: Multitask Distributed Proximal Policy Optimization.
Revised: Multitask Distributed Proximal Policy Optimization.

Original: We present a multitask distributed proximal policy optimization (MDPPO) algorithm, which utilizes the power of parallel processing to train
Revised: We present a multitask distributed proximal policy optimization (MDPPO) algorithm, which utilizes the power of parallel processing to train

Original: MTRL agents in our continuous and large-scale environments.
Revised: MTRL agents in our continuous and large-scale environments.

Original: MDPPO is built upon (DPPO)
Revised: MDPPO is built upon (DPPO)

Original: [117], a distributed version of proximal policy optimization (PPO) [119] that bounds parameter updates to a trust region to ensure stability, and distributes the computation over many parallel instances of agent and environment.
Revised: [118], a distributed version of proximal policy optimization (PPO) [120] that bounds parameter updates to a trust region to ensure stability, and distributes the computation over many parallel instances of agent and environment.

Original: Similarly, MDPPO has a server-client structure: each client worker has several agent copies that collect experiences from VXN environments, compute and send PPO’s gradients to the server; the server worker averages the received gradients, updates the agent, and synchronizes the updated weights with the clients.
Revised: Similarly, MDPPO has a server-client structure: each client worker has several agent copies that collect experiences from VXN environments, compute and send PPO’s gradients to the server; the server worker averages the received gradients, updates the agent, and synchronizes the updated weights with the clients.

Original: For balanced multitask learning, i.e ., training data in VXN are biased between vision-language nav.
Revised: For balanced multitask learning, i.e ., training data in VXN are biased between vision-language nav.

Original: and other navigation tasks: 10 .
Revised: and other navigation tasks: 10 .

Original: 8 K vs 2 . 

Revised: 8 K vs 2 .

Original: ∼ 5 . 

Revised: ∼ 5 .

Original: M episodes ( cf .
Revised: M episodes ( cf .

Original: Table 1), each client worker is required to build four agent copies corresponding to the four VXN tasks.
Revised: Table 1), each client worker is required to build four agent copies corresponding to the four VXN tasks.

Original: 4.5 Implementation Detail
Revised: 4.5 Implementation Detail

Original: Network Architecture.
Revised: Network Architecture.

Original: The visual encoder f I MG is made as an ImageNet [120]-pretrained ResNet [121].
Revised: The visual encoder f I MG is made as an ImageNet [121]-pretrained ResNet [122].

Original: The CNN features are fed into a linear layer for dimension compression and flattened into a feature sequence.
Revised: The CNN features are fed into a linear layer for dimension compression and flattened into a feature sequence.

Original: Similarly, the depth encoder f D EP is a modified ResNet50 CNN.
Revised: Similarly, the depth encoder f D EP is a modified ResNet50 CNN.

Original: The audio encoder f
Revised: The audio encoder f

Original: A UD , following [6], is a CNN of conv 8 × 8 , conv 4 × 4 , conv 3 × 3 and a linear layer, interleaved with
Revised: A UD , following [6], is a CNN of conv 8 × 8 , conv 4 × 4 , conv 3 × 3 and a linear layer, interleaved with

Original: ReLU.
Revised: ReLU.

Original: All the sensory features are combined with orientation embeddings.
Revised: All the sensory features are combined with orientation embeddings.

Original: For the epoch embedding µ , we use sinusoidal encoding.
Revised: For the epoch embedding µ , we use sinusoidal encoding.

Original: For the target parser , N =5 target embeddings are generated at each epoch t .
Revised: For the target parser , N =5 target embeddings are generated at each epoch t .

Original: We set other hyper-parameters as: d =512 , N I =16 , N L =120 , N G =120 .
Revised: We set other hyper-parameters as: d =512 , N I =16 , N L =120 , N G =120 .

Original: Training and Test.
Revised: Training and Test.

Original: V IENNA is trained on 32 RTX 2080 GPUs for 180 M frames, costing 4 ,
Revised: V IENNA is trained on 32 RTX 2080 GPUs for 180 M frames, costing 4 ,

Original: GPU hours.
Revised: GPU hours.

Original: As in [25], we select the checkpoint for evaluation with the best SR on val unseen .
Revised: As in [25], we select the checkpoint for evaluation with the best SR on val unseen .

Original: For
Revised: For

Original: MDPPO, we use four client workers and set the discounted factor γ as 0 . 99 .
Revised: MDPPO, we use four client workers and set the discounted factor γ as 0 . 99 .

Original: We use AdamW [122] optimizer with a learning rate of 2 .
Revised: We use AdamW [123] optimizer with a learning rate of 2 .

Original: 5 × 10 − 4 .
Revised: 5 × 10 − 4 .

Original: Casual attention [16] is adopted to prevent the prediction at epoch t from the influence of future tokens after t .
Revised: Casual attention [16] is adopted to prevent the prediction at epoch t from the influence of future tokens after t .

Original: Once trained, a single instance of V IENNA can conduct the four navigation tasks.
Revised: Once trained, a single instance of V IENNA can conduct the four navigation tasks.

Original: As normal, greedy prediction is adopted for action selection.
Revised: As normal, greedy prediction is adopted for action selection.

Original:  Experiment
Revised:  Experiment

Original: In §5.1, we first report comparison results for the four VXN tasks.
Revised: In §5.1, we first report comparison results for the four VXN tasks.

Original: In §5.2, we conduct diagnostic studies to examine the efficacy of our core model design.
Revised: In §5.2, we conduct diagnostic studies to examine the efficacy of our core model design.

Original: More results are put in the supplementary .
Revised: More results are put in the supplementary .

Original: Baseline.
Revised: Baseline.

Original: We test several open-source task-specific navigation methods [2,5,6,25].
Revised: We test several open-source task-specific navigation methods [2,5,6,25].

Original: Note that [2,5,6] are re-trained on VXN , since they use different training data [5], world representation [6] (discrete vs continuous), or object categories [2] (6 vs 21).
Revised: Note that [2,5,6] are re-trained on VXN , since they use different training data [5], world representation [6] (discrete vs continuous), or object categories [2] (6 vs 21).

Original: For [25], we use its check-point but the success criteria are different (3 m vs 1 m).
Revised: For [25], we use its check-point but the success criteria are different (3 m vs 1 m).

Original: Thus their scores on VXN are different from the original ones.
Revised: Thus their scores on VXN are different from the original ones.

Original: In addition, we consider a Seq2Seq agent, which also serves as a standard baseline in [8,25]: an LSTM plannerencodes the episode history and predicts navigation actions in a sequential menner.
Revised: In addition, we consider a Seq2Seq agent, which also serves as a standard baseline in [8,25]: an LSTM plannerencodes the episode history and predicts navigation actions in a sequential menner.

Original: For all the four tasks, we provide the performance of both the single-task and multitask versions of our V IENNA and Seq2Seq .
Revised: For all the four tasks, we provide the performance of both the single-task and multitask versions of our V IENNA and Seq2Seq .

Original: Further, Random policy , i.e ., choosing actions randomly, is included.
Revised: Further, Random policy , i.e ., choosing actions randomly, is included.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Metric.
Revised: Metric.

Original: Four widely-used metrics are adopted for evaluation: i) Success Rate ( SR ); ii) Navigation
Revised: Four widely-used metrics are adopted for evaluation: i) Success Rate ( SR ); ii) Navigation

Original: Error (NE); iii) Oracle success Rate (OR); and iv) Success rate weighted by Path Length (SPL
Revised: Error (NE); iii) Oracle success Rate (OR); and iv) Success rate weighted by Path Length SPL

Original: 5.1 Performance Benchmarking
Revised: 5.1 Performance Benchmarking

Original: Table 3 reports the comparison results on the four VXN tasks.
Revised: Table 3 reports the comparison results on the four VXN tasks.

Original: Some key conclusions are list below:
Revised: Some key conclusions are list below:

Original: • V IENNA obtains impressive results, under val seen and unseen sets, across all the tasks and evaluation metrics.
Revised: • V IENNA obtains impressive results, under val seen and unseen sets, across all the tasks and evaluation metrics.

Original: This proves the versatility of V IENNA and the power of our parse-and-query regime.
Revised: This proves the versatility of V IENNA and the power of our parse-and-query regime.

Original: • V IENNA consistently outperforms Seq2Seq , no matter they are trained on single tasks individually or multiple tasks jointly.
Revised: • V IENNA consistently outperforms Seq2Seq , no matter they are trained on single tasks individually or multiple tasks jointly.

Original: Compared with other task-specific competitors [2,5,6,25], VIENNA gains comparable results on audio-goal nav .
Revised: Compared with other task-specific competitors [2,5,6,25], VIENNA gains comparable results on audio-goal nav .

Original: and object-goal nav .
Revised: and object-goal nav .

Original: , and performs better on image-goal nav .
Revised: , and performs better on image-goal nav .

Original: and vision-language nav .
Revised: and vision-language nav .

Original: tasks.
Revised: tasks.

Original: These results verify the effectiveness of our model design.
Revised: These results verify the effectiveness of our model design.

Original: • When considering the performance gain from the single-task setting to multitask, V IENNA yields more promising results, compared with Seq2Seq .
Revised: • When considering the performance gain from the single-task setting to multitask, V IENNA yields more promising results, compared with Seq2Seq .

Original: For example, in Table 3a, V IENNA MT outperforms V IENNA ST by 2.2% SR and 1.7% SR, on val seen and unseen , respectively; however, in the same condition, Seq2Seq MT only provides 0.7% and 0.9% SR gains over Seq2Seq ST .
Revised: For example, in Table 3a, V IENNA MT outperforms V IENNA ST by 2.2% SR and 1.7% SR, on val seen and unseen , respectively; however, in the same condition, Seq2Seq MT only provides 0.7% and 0.9% SR gains over Seq2Seq ST .

Original: These results demonstrates that V IENNA can make a better use of cross-task knowledge.
Revised: These results demonstrates that V IENNA can make a better use of cross-task knowledge.

Original: • When considering the performance gap between seen and unseen environments, V IENNA MT is more favored than its single-task counterpart, V IENNA ST .
Revised: • When considering the performance gap between seen and unseen environments, V IENNA MT is more favored than its single-task counterpart, V IENNA ST .

Original: For instance, in Table 3c, V IENNA ST suffers from relatively large performance drop, i.e ., 33.2% → 18.5%SR;however,V IENNA MT showsreduced degradation, i.e ., 33.3%
Revised: For instance, in Table 3c, V IENNA ST suffers from relatively large performance drop, i.e ., 33.2% → 18.5%SR;however,V IENNA MT showsreduced degradation, i.e ., 33.3%

Original: → 19.4% SR, in unseen environments.
Revised: → 19.4% SR, in unseen environments.

Original: This indicates that investigating inter-task relatedness may help to strengthen the generalizability of navigation agents.
Revised: This indicates that investigating inter-task relatedness may help to strengthen the generalizability of navigation agents.

Original: • The above results are particularly impressive considering the advantage of V IENNA in efficient parameter utilization, i.e ., V IENNA MT (31 M) vs V IENNA ST × 4 (101 M) vs Seq2Seq MT (27 M) vs Seq2Seq ST × 4 (93 M) vs [5] + [6] + [2] + [25] (165M = 40 M + 45 M + 38 M + 42 M).
Revised: • The above results are particularly impressive considering the advantage of V IENNA in efficient parameter utilization, i.e ., V IENNA MT (31 M) vs V IENNA ST × 4 (101 M) vs Seq2Seq MT (27 M) vs Seq2Seq ST × 4 (93 M) vs [5] + [6] + [2] + [25] (165M = 40 M + 45 M + 38 M + 42 M).

Original: Fig.
Revised: Fig.

Original: 4 plots the training curves of V IENNA ST/MT compared to Seq2Seq ST/MT for the four VXN tasks in unseen envs.
Revised: 4 plots the training curves of V IENNA ST/MT compared to Seq2Seq ST/MT for the four VXN tasks in unseen envs.

Original: Aligning with the results in Table 3, V IENNA outperforms Seq2Seq , and benefits more from multiple task learning.
Revised: Aligning with the results in Table 3, V IENNA outperforms Seq2Seq , and benefits more from multiple task learning.

Original: This shows that V IENNA makes a better use of cross-task knowledge
Revised: This shows that V IENNA makes a better use of cross-task knowledge

Original: [Table]
Revised: [Table]

Original: 5.2 Ablative Study
Revised: 5.2 Ablative Study

Original: To thoroughly test the efficacy of crucial components of V IENNA , we conduct a series of diagnostic studies on vision-language nav.
Revised: To thoroughly test the efficacy of crucial components of V IENNA , we conduct a series of diagnostic studies on vision-language nav.

Original: and audio-goal nav.
Revised: and audio-goal nav.

Original: tasks.
Revised: tasks.

Original: The results are summarized in Table 4.
Revised: The results are summarized in Table 4.

Original: Multisensory Integration.
Revised: Multisensory Integration.

Original: Agents in VXN are equipped with a multimodal sensor so as to find the target by both seeing and hearing and make our navigation setting closer the real-world.
Revised: Agents in VXN are equipped with a multimodal sensor so as to find the target by both seeing and hearing and make our navigation setting closer the real-world.

Original: We first study the influence of different sensory signals ( i.e ., RGB, depth, audio) by training V IENNA with varying sensory modalities.
Revised: We first study the influence of different sensory signals ( i.e ., RGB, depth, audio) by training V IENNA with varying sensory modalities.

Original: As shown in Table 4a, fusing multimodal sensory cues ( i.e ., RGBD + audio) is more favored.
Revised: As shown in Table 4a, fusing multimodal sensory cues ( i.e ., RGBD + audio) is more favored.

Original: For example, in VLN, although considering audio alone brings poor performance, supplementing RGBD perception with audio yields notable improvements.
Revised: For example, in VLN, although considering audio alone brings poor performance, supplementing RGBD perception with audio yields notable improvements.

Original: This suggests audio is complementary to visual sensory in capturing physical and semantic properties of environments.
Revised: This suggests audio is complementary to visual sensory in capturing physical and semantic properties of environments.

Original: Augmented Target Description Embedding.
Revised: Augmented Target Description Embedding.

Original: To better master cross-task knowledge, V IENNA augments its episodic targets with all the four learnable task embeddings
Revised: To better master cross-task knowledge, V IENNA augments its episodic targets with all the four learnable task embeddings

Original: τ
Revised: τ

Original: I,A,T,L ( cf.
Revised: I,A,T,L ( cf.

Original: Eq. 7).
Revised: Eq. 7).

Original: We compare this design against two variants in Table 4b, and find such a strategy is conducive to the performance.
Revised: We compare this design against two variants in Table 4b, and find such a strategy is conducive to the performance.

Original: This is because, through end-to-end training, τ I,A,T,L carry task-specific knowledge, e.g .,
Revised: This is because, through end-to-end training, τ I,A,T,L carry task-specific knowledge, e.g .,

Original: τ A is associated with some discriminative audio clips; τ I focuses on essential visual landmarks.
Revised: τ A is associated with some discriminative audio clips; τ I focuses on essential visual landmarks.

Original: By taking τ I,A,T,L together, V IENNA use key knowledge of different tasks in single task episodes.
Revised: By taking τ I,A,T,L together, V IENNA use key knowledge of different tasks in single task episodes.

Original: Diversified Target Parsing.
Revised: Diversified Target Parsing.

Original: We online parse the augmented target description embedding G into N target embeddings Q t =[ q 1 t , · · · , q Nt ] ( cf.
Revised: We online parse the augmented target description embedding G into N target embeddings Q t =[ q 1 t , · · · , q Nt ] ( cf.

Original: Eq.
Revised: Eq.

Original: 8), to achieve vivid and diversified interpretations of G .
Revised: 8), to achieve vivid and diversified interpretations of G .

Original: In Table 4c, we present evaluation scores with different numbers of generated target embeddings, i.e .,
Revised: In Table 4c, we present evaluation scores with different numbers of generated target embeddings, i.e .,

Original: N = 1 , 3 , 5 , 7 .
Revised: N = 1 , 3 , 5 , 7 .

Original: As can be seen, diversified target parsing indeed boots navigation performance.
Revised: As can be seen, diversified target parsing indeed boots navigation performance.

Original: Multitask Planner.
Revised: Multitask Planner.

Original: Several variants of multitask planner f M TP ( cf .
Revised: Several variants of multitask planner f M TP ( cf .

Original: Eq. 9) are compared in Table 4d.
Revised: Eq. 9) are compared in Table 4d.

Original: The two-layer shared trunk design is adopted, due to its relatively better performance.
Revised: The two-layer shared trunk design is adopted, due to its relatively better performance.

Original: Reward Function.
Revised: Reward Function.

Original: Next we examine the design of our reward function (§4.4).
Revised: Next we examine the design of our reward function (§4.4).

Original: As seen in Table 4e, each reward term is indeed useful and combining all the four terms leads to the best performance.
Revised: As seen in Table 4e, each reward term is indeed useful and combining all the four terms leads to the best performance.

Original: Multitask Learning.
Revised: Multitask Learning.

Original: Table 4f reveals the value of training V IENNA on multiple tasks: training with more navigation tasks improves both performance and generalizability.
Revised: Table 4f reveals the value of training V IENNA on multiple tasks: training with more navigation tasks improves both performance and generalizability.

Original: Compared to a composition of four single-task models, multi-task V IENNA also greatly reduces the model size: 101M → 31M.
Revised: Compared to a composition of four single-task models, multi-task V IENNA also greatly reduces the model size: 101M → 31M.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we present VXN , a large-scale 3D indoor dataset for multimodal, multitask navigation in continuous and audiovisual complex environments.
Revised: In this work, we present VXN , a large-scale 3D indoor dataset for multimodal, multitask navigation in continuous and audiovisual complex environments.

Original: Further, we devise V IENNA , a powerful agent that simultaneously learns four famous navigation tasks within a single unified parsing-and-query scheme.
Revised: Further, we devise V IENNA , a powerful agent that simultaneously learns four famous navigation tasks within a single unified parsing-and-query scheme.

Original: We empirically show that, through a fully attentive architecture, V IENNA is able to mine and utilize cross-task knowledge to enhance the performance on all the tasks.
Revised: We empirically show that, through a fully attentive architecture, V IENNA is able to mine and utilize cross-task knowledge to enhance the performance on all the tasks.

Original: These efforts move us closer to a community goal of general-purpose robots capable of fulfilling a multitude of tasks.
Revised: These efforts move us closer to a community goal of general-purpose robots capable of fulfilling a multitude of tasks.

Original: Acknowledgement.
Revised: Acknowledgement.

Original: This research is partially supported by China National Key R&DChecklist
Revised: This research is partially supported by China National Key R&D

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] Our main claims accurately reflect the paper’s contributions and scope.

Original: (b) Did you describe the limitations of your work?

Original: [Yes] The limitations are discussed in the supplemental material.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] The potential negative societal impacts are discussed in the supplemental material.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] We have already read the ethics review guidelines and ensured that our paper conforms to them.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]

Original: Our work does not involve any theoretical result.

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]

Original: Our work does not involve any theoretical result.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] The code and full dataset is available on the project page: https://github.com/hanqingwangai/ VXN .

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] We specify all the training details in §4.5 and §I of supplementary.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We report the error bars in Fig.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We report our compute hardware in §4.5. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We have cited all the used code and data.

Original: (b) Did you mention the license of the assets?

Original: [Yes] Habitat Simulator [15] is obtained ( https://github

Original: .com/facebookresearch/habitat lab ) with MIT li cense.

Original: The algorithm of Chen et al .

Original: [6] is implemented based on the publicly released code ( https://github.com/facebookresearch/sound-spaces ) is with CC-BY-4.0 license.

Original: The algorithm of Chaplot et al .

Original: [2] is implemented based on the publicly released code ( https://github.com/devendrachaplot/

Original: Object-Goal-Navigation ) with MIT license.

Original: The algorithm of Krantz et al .

Original: [25] is implemented based on the publicly released code ( https://github.com/ jacobkrantz/VLN-CE ) with MIT license.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] We do not use any new assets excepting the list above.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] We obtain Matterport3D [19] data by signing the terms of use ( http://kaldir.vc.in.tum.de/matterport/MP_TOS.pdf ).

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes] The data used in our paper neither contain any personally identifiable information nor offensive content.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]

Original: Our work does not involve any crowdsourcing or conducted research with human subjects.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A]

Original: Our work does not involve any crowdsourcing or conducted research with human subjects.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]

Original: Our work does not involve any crowdsourcing or conducted research with human subjects.