AVAST : Attentive Variational State Tracker in a Reinforced Navigator (2022)

Abstract: Recently, emerging approaches have been proposed to deal with robotic navigation problems, especially vision-and-language navigation task which is one of the most realistic indoor navigation challenge tasks. This task can be modelled as a sequential decision-making problem, which is suitable to be solved by deep reinforcement learning. Unfortunately, the observations provided from the simulator in this task are not fully observable states, which exacerbate the difficulty of implementing reinforcement learning. To deal with this challenge, this paper presents a novel method, called as attentive variational state tracker (AVAST), a variational approach to approximate belief state distribution for the construction of a reinforced navigator. The variational approach is introduced to improve generalization to the unseen environment which barely achieved by traditional deterministic state tracker. In order to stabilize the learning procedure, a fine-tuning process using policy optimization is proposed. From the experimental results, the proposed AVAST does improve the generalization relative to previous works in vision-and-language navigation task. A significant performance is achieved without requiring any additional exploration in the unseen environment.

## introduction reinforcement learning (rl) has become a crucial and successful solution in many sequential decision-making problems, such as video game playing ai @xcite and robotic control @xcite . in theory, rl algorithms are designed for solving problems under the assumption of markov decision process (mdp), which means that the observation provided from the environment needs to exactly represent the complete state information of the environment @xcite . however, most of the real-world problems, such as bridge-playing ai, dialogue systems @xcite @xcite , autonomous driving, and first-person navigation @xcite , can not be directly modeled as markov decision processes, because of the incomplete state information. for example, in dialogue task, system does not have an access to the user goal @xcite . in order to improve the generalization, partially observable markov decision process (pomdp) @xcite was designed to model the process in which the agent does not have access to observe complete state information. in case of vision-and-language navigation (vln) task, the problem formulation is considered as pomdp problem, as the agent does not receive full information about the state. it only receive the information about the images of surroundings and the texts which describe the navigation task and agent pose information. there is no information which explicitly tells about agent and goal location coordinates. furthermore, as each observation is unique and complex in the vln task, the common methods which turn pomdp problem into mdp problem by aggregating the observations and estimating the belief states do not work very well. aggregation methods usually use either a framestacking trick @xcite or a recurrent neural network @xcite to aggregate the history observation or the belief state information. these methods mostly work only for either computer vision or natural language processing tasks by considering sufficient information process @xcite ) assumption as well as bayes theory @xcite . meanwhile, the vln task requires agent to consider both domains to solve the problem. motivated by the aforementioned issues, this work formulates vln task as a pomdp problem and solves it by using rl algorithms. we propose a new method named as attentive variational state tracker (avast) to estimate the belief state distribution of the complex observations in the vln task. avast follows sufficient information process assumption to reduce vln task into an mdp problem. by using variational inference approach, the generalization property of the belief state sampled from avast is accordingly held. based on the experiment result, the proposed method can achieve better performance compared to the baselines due to its generalization property. the organization of this work is arranged as follows. in sections 2 and 3, the recent approaches to deal with pomdp state tracking and vln task are discussed, respectively. the proposed method, avast, is explained in section 4. the experimental setup and result are described in section 5. finally, section 6 shows the conclusions. ## partially observable markov decision process state tracking real-world problems usually cannot directly be modelled as mdp problems, because of the information limitation. accordingly, the partially observable markov decision process (pomdp) @xcite is fitted to implement an agent decision process in presence of incomplete state information. in general, a pomdp problem can be described by a 6-tuple set {s, a, p, r, γ, o}. identical to mdp problem, s, a, p, r, γ denote the state, action, transition probability, reward, and discount factor, respectively. the main difference is that the agent can not observe the complete state @xmath0. it only receives an observation @xmath1. according to the probability distribution o(s), the observation o is generated from the underlying system state as o ∼ o(s). generally, estimating a policy distribution from an observation can be arbitrary due to π(a|o; ϕ) ̸ = π(a|s; ϕ). following the sufficient information process @xcite , pomdp state distribution can be approximated by using a state tracker to produce the belief state distribution p(s|i c t ). i c t denotes the complete information state at time t which represents the history information from the beginning to time t. i c t is defined as, i c @xmath2, o 1 , a 1 , . . . , a t-1 , o t ⟩, where ρ(s 0 ) is a distribution over initial stated. once the welltrained state tracker is obtained, a belief state s t can be sampled from the distribution p(s|i c t ), and rl agent will consider it as the system state to generate the action a t . traditionally, common sequential learning using recurrent neural network (rnn) was applied to encode the observations history to produce an appropriate belief state as the input to agent @xcite . such method was likely to summarize history by remembering features from the past trajectories rather than actually estimating belief states. furthermore, naively applying rnn would output suboptimal belief states due to the deterministic computation without any distribution constraint. other approaches @xcite estimated the belief states by introducing bayesian theory. compared to the purely rnn-based methods, introducing stochastic estimation can improve generalization to complex environments. however, dealing with unseen environment is still a major stumbling block in designing a state tracker. therefore, different from the previous works, in this paper, an attentive variational state tracker is proposed to improve the state tracking generalization for vision-language navigation. ## vision-and-language navigation in general, the reinforcement learning agent which is designed for vln task @xcite , will not receive complete state information. instead, the observation @xmath3, generated from the underlying system state according to the probability distribution o ∼ o(s), will be obtained by the agent in vln. the observations o can be separated into three parts which are instructions, visions, and pose information. instructions are provided in natural language @xcite to guide the agent about how to reach the target position ρ goal from the initial position ρ 1 . at different positions ρ t , agent will receive different panoramic visions and pose information. given such a process, vln agent must understand the current situation using the provided instructions, panoramic visions and pose information, and navigate to the target position. formally, an agent will receive one instruction @xmath4, and at the same time receive an initial panoramic vision v 1 ∈ ω v and an initial pose information p 1 ∈ ω p , generated from the initial position ρ 1 . then, it will receive a current panoramic vision v @xmath5, current pose information p @xmath6, and reward r @xmath7, generated from the current position ρ t at each time step t after acting an action a t-1 . due to the difficulty of vln task, the most intuitive way to deal with this task is to apply imitation learning by utilizing expert trajectories through behaviour cloning @xcite . however, behaviour cloning was prone to the out-of-distribution trajectory once it was applied into the environment. previous approach used the adversarial inverse reinforcement learning (airl) @xcite which defined the reward function based on the expert trajectories @xcite and used the learned reward function to train the agent through interactions with the environment. other works developed the crossmodality matching @xcite and modelbased rl @xcite to improve rl agent performance. although previous methods have shown promising results, all of them required the exploration to the unseen environment to obtain additional training data when being evaluated in the unseen validation set. this scenario clearly did not represent real-world implementation where robot needed to provide appropriate actions without requiring any explorations. therefore, in this work, the variational state tracking is proposed to improve generalization. therefore, the agent can perform properly in unseen environments without requiring any environment exploration. ## conclusions this paper has presented attentive variational state tracker to deal with the generalization issue in vision-and-language navigation task. this method developed a variational approach to fulfill the partially observable markov decision process where the belief states were sampled to implement the stochastic machine to improve the generalization to unseen environments. the experimental results demonstrated that the policy optimization using reinforce in combination of the proposed avast outperformed the previous methods in terms of navigation errors and success rates. the generalization was assured by the evaluation in the unseen environments.