SVQN: S EQUENTIAL V ARIATIONAL S OFT QL EARNING N ETWORKS
A BSTRACT
Partially Observable Markov Decision Processes (POMDPs) are popular and ﬂexible models for real-world decision-making applications that demand the information from past observations to make optimal decisions.
Standard reinforcement learning algorithms for solving Markov Decision Processes (MDP) tasks are not applicable, as they cannot infer the unobserved states.
In this paper, we propose a novel algorithm for POMDPs, named sequential variational soft Q-learning networks (SVQNs), which formalizes the inference of hidden states and maximum entropy reinforcement learning (MERL) under a uniﬁed graphical model and optimizes the two modules jointly.
We further design a deep recurrent neural network to reduce the computational complexity of the algorithm.
Experimental results show that SVQNs can utilize past information to help decision making for efﬁcient inference, and outperforms other baselines on several challenging tasks.
Our ablation study shows that SVQNs have the generalization ability over time and are robust to the disturbance of the observation.
I NTRODUCTION
In recent years, substantial progress has been made in deep reinforcement learning for solving various challenging tasks, including the computer Go game (Silver et al., 2016), Atari games (Mnih et al., 2015), StarCraft (Zambaldi et al., 2018; Pang et al., 2018) and the ﬁrst-person shooting (FPS) games (Lample & Chaplot, 2017; Wu & Tian, 2016; Huang et al., 2019).
However, in many real-world applications, decision-making problems are partially observable (Astr ¨ om, 1965), preventing such problems from being solved by standard reinforcement learning algorithms.
Formally, these kinds of problems are often deﬁned as Partially Observable Markov Decision Processes (POMDPs) (Kaelbling et al., 1998), which demand information from past observations to help in the decision-making process (McCallum, 1993).
Although numerous efforts (Hausknecht & Stone, 2015; Foerster et al., 2016; Igl et al., 2018; Zhu et al., 2018) have been paid to tackle this problem, there still exist various challenges.
For example,
Egorov (2015) tries to solve POMDPs by using the belief of the agent as the input of DQN (Mnih et al., 2015), but this algorithm needs access to the environment model.
However, in many reinforcement learning tasks, it is not possible for the agent to acquire the underlying transition function, making such algorithms inapplicable.
Some recent work (Karkus et al., 2017; McAllester & Singh, 2013; Babayan et al., 2018) tries to solve POMDPs under the model-free setting, i.e., the agent does not need to know and learn the transition function of the environment.
For instance, Karkus et al.
(2017) trained an agent to navigate in a partially observable grid world under the model-free setting, i.e., the agent can only observe a part of the grid world and does not learn the transition function.
The agent uses its local observations to update its beliefs (McAllester & Singh, 2013; Babayan et al., 2018).
In their experiments, the ground truth of the state is the full map plus the location of the agent, which means the representation of the state is explicit.
However, in some complex tasks, it is impossible to acquire or design the state or beliefs.
To solve the unknown representation problem of the state, Hausknecht & Stone (2015) and Zhu et al.
(2018) try to represent the state as latent variables of neural networks.
However, they only use a deep recurrent neural network to capture the historical information and fail to utilize the Markov property of the state in POMDPs.
Igl et al.
(2018) apply sequential Monte Carlo (SMC)
(Le et al., 2017) to introduce inductive bias to the neural network, which can embody the Markov properties of the state.
They can infer the state from the past observations online.
However, they separate the planning algorithm from the inference of the state.
To infer the hidden states and optimize the planning module jointly, we represent POMDPs as a uniﬁed probabilistic graphical model (PGM) and derive a single evidence lower bound (ELBO).
We apply structured variational inference to optimize the ELBO.
In our implementation, we design generative models to infer the hidden variables, however, the distribution of the latent variables is conditioned on previous hidden states.
This is different from standard VAEs (Kingma & Welling, 2013), whose prior of the latent variables can be a standard Gaussian distribution.
So, we apply an additional approximate function to tackle the conditional prior problem.
The planning can also be solved under the PGM framework.
Fortunately, maximum entropy reinforcement learning (MERL)
(Levine, 2018) provide a tool to formalize the planning as a probabilistic inference task.
In this paper, we propose a novel end-to-end neural network called the sequential variational soft
Q-learning network (SVQN), which integrates the learning of hidden states and the optimization of the planning within the same framework.
A deep recurrent neural network (RNN) (Cho et al., 2014; Hochreiter & Schmidhuber, 1997) in SVQNs is used to reduce the computational complexity, because the feature extraction can share the same weights in the RNN.
Experimental results show that the SVQN can utilize past information to help in decision making for efﬁcient inference, and outperforms other baselines on several challenging tasks.
Our ablation study shows that SVQNs have the generalization ability over time and are robust to the disturbance of the observation.
Contributions: (1) We derive the variational lower bound for POMDPs, which allows us to integrate the optimization of the control problem and learning of the hidden state under a uniﬁed graphical model.
(2) We propose to tackle the difﬁculty of the inference of the hidden state and solve the problem of a conditional prior using the generative models.
(3) We design an end-to-end deep recurrent neural network, which can reduce the computational complexity and be trained efﬁciently.
R ELATED W ORK
We summarize some related work in POMDPs and inference methods for sequential data.
Model-Based and Model-Free methods for POMDPs: When the environment model is accessible,
POMDPs can be solved by model-based method.
Egorov (2015) used model-based methods to solve POMDPs, but their agents need to know the belief-update function and the transition function.
When the environment model is unknown, model-free methods should be applied.
Recently, some researchers (Hausknecht & Stone, 2015; Zhu et al., 2018) used recurrent neural networks to capture the historical information, but they failed to utilize the Markov property of the state in POMDPs.
Our work proposes generative models for the algorithm learning, which tackles the difﬁculty of the inference of hidden states and introduces inductive bias to the network structure.
Igl et al.
(2018) applied sequential Monte Carlo (SMC) to the POMDPs.
They can infer the hidden state from the past observations online.
However, they separate the planning algorithm and the inference of the hidden state.
Our algorithm is derived from a uniﬁed graphical model, which can train the inference model and the planning algorithm jointly.
Explorations in POMDPs: In contrast to MDP methods, POMDP methods should do exploration to gather information with no immediate reward which can then be used in the future to gain higher rewards.
Pathak et al.
(2017) and Choi et al.
(2018) designed deep reinforcement learning algorithms to help the exploration in their tasks.
Actually, their tasks are POMDP tasks.
However, they just treat their tasks as MDP tasks and intuitively add exploration tricks to standard reinforcement learning algorithms.
Their exploration tricks include the reconstruction of observations and actions, which comes from intuitive concepts such as curiosity and attention.
The generative models in our method also need to reconstruct the observations and actions, but our algorithm is derived from solid theoretical foundations.
Inference for Sequential Data: The observation in POMDP tasks is sequential data, and we need to do inference for the hidden state from observations.
Coquelin et al.
(2009) used a particle ﬁlter to estimate the belief state given past observations.
However, their method needs to access to the
[Figure]
environment model.
Chung et al.
(2015) derived the evidence lower bound of latent variables for sequential data and designed a novel deep neural network to infer recurrent latent variables.
Our inference method has some superﬁcial resemblances to their method, but we derived the evidence lower bound from a different graphical model and tackle the difﬁculty of how to integrate the RL planning algorithm instead of just inferring the hidden variables.
P RELIMINARIES
We start by brieﬂy reviewing the backgrounds and notations related to our method, including partially observable Markov decision processes and maximum entropy reinforcement learning which solves optimal control problems using a probabilistic framework.
3. P ARTIALLY O BSERVABLE M ARKOV D ECISION P ROCESSES
In many control tasks, the complete state of the environment is unknown to the agent.
Instead, the agent can only receive local observations, which are typically conditioned on the current state of the system.
The agent needs to make decisions based on the historical information.
This kind of problems can be formalized as partially observable Markov decision processes (POMDPs) (Smith & Simmons, 2004).
Formally, a POMDP is represented as a tuple ( S, A, O, T, Z, r ) , where S , A and O are state space, action space and observation space, respectively.
The reward function r ( s, a ) is the received reward when taking action a in state s .
T ( s, a, s (cid:48) ) is the state-transition function, which deﬁnes the probability of the succeeding state s (cid:48) after taking action a in state s .
Z ( s, a, o ) is the observation function, which deﬁnes the probability of emitted observation o after taking action a in state s .
POMDPs use the belief b t ( s ) to maintain the distribution of the unknown state at time t , and the agent updates its belief with a Bayesian ﬁlter when receiving a new observation as
[Equation]
where η is a normalizing constant and a t is the action at time t .
POMDP planning needs to ﬁnd a policy π to maximize the accumulative reward as
[Equation]
where s t is the state at time t , and γ ∈ (0 , 1) is a discount factor.
3. M AXIMUM E NTROPY R EINFORCEMENT L EARNING
POMDPs require solving two problems: the inference of state and optimal control.
To integrate both of them under a uniﬁed framework, we represent POMDPs as a probabilistic graphical model.
So the optimal control problem needs to be solved as a probabilistic inference task.
Fortunately, the maximum entropy reinforcement learning (MERL) algorithm provides an effective tool to solve optimal control problem under PGM framework.
The graphical model of Markov decision processes is shown in Fig.
1(a).
We borrow the notation from Levine (2018) to illustrate the algorithm.
Levine (2018) introduces a binary random variable O t to the graphical model, where O t = 1 denotes that the action is optimal at time t , and O t = 0 denotes that the action is not optimal.
The probability distribution of O is p ( O t = 1 | s t , a t ) =
exp( r ( s t , a t )) , and the variational lower bound is given by:
[Equation]
where π ( a | s ) is the policy function.
Standard reinforcement learning only needs to maximize the cumulative reward.
However, MERL will maximize an extra term, which is the policy entropy at each visited state.
To get the optimal solution for MERL, two messages are introduced, i.e., β t ( s t , a t ) =
p ( O t : T | s t , a t ) and β t ( s t )
= p ( O t : T | s t ) , and their relations are given by:
[Equation]
and the optimal policy is given by:
[Equation]
We can deﬁne the Q-value function and V-value function as below:
[Equation]
and the update functions are:
[Equation]
To maximize the variational lower bound in Eq.
(3), we can use a parameterized Q-function
Q θ ( s t , a t ) , where θ is the function parameter, and it can be learned via gradient descent:
[Equation]
where α is the learning rate.
This algorithm is called soft Q-learning, because the update function for Q-value can be considered a soft-update version of the standard Bellman backup.
For a better understanding of MERL, we recommend readers reference this tutorial (Levine, 2018).
S EQUENTIAL V ARIATIONAL S OFT Q-L EARNING N ETWORKS
We now present our algorithm in detail.
We ﬁrst derive the variational lower bound for POMDPs, and then illustrate how to deal with the conditional prior.
4. V ARIATIONAL L OWER B OUND FOR POMDP S
Different from MDPs, the state s of POMDPs in the PGM (shown in Fig. 1(b)) is unobservable, which need to be inferred from the action a and the observation o .
We need to derive different variational lower bound for POMDPs, which can be used to infer the hidden state and do planning jointly.
Unlike the observation function Z ( s, a, o ) in Section 2.1, we assume that the observation is emitted from the hidden state s , which means the probability distribution of observations are only conditioned on states, i.e., o t ∼ p ( o t | s t ) .
This assumption can hold in many tasks.
For example, the observation in a partially observed maze is only determined by the full map and the agent’s location.
We apply the structured variational inference to optimize the evidence lower bound of POMDPs.
In structured variational inference, different parts of the proposal distributions can be optimized separately, which means we can ﬁx some approximate functions and optimize other approximate functions.
In POMDPs, we will use two approximate functions q π ( a t
| s t ) and q θ ( s t | s t − 1 , a t − 1 , o t ) .
q π ( · ) approximates the optimal policy and q θ ( · ) approximates the function of inferring hidden states, where θ is the parameter of the approximate function.
When q θ ( · ) is ﬁxed, the learning procedure is same as MERL, so that q π ( · ) can be learned via the soft Q-learning algorithm.
Conversely, when q π ( · ) is ﬁxed as the optimal policy, we can learn the inference function q θ ( · ) for hidden states.
We can derive the evidence lower bound(ELBO) as bellow: log
[Equation]
and the ELBO can be written as:
[Equation]
To get the optimal action in POMDPs, we need to maximize the ELBO above.
The ﬁrst term in the ELBO is the cumulative reward (cid:80)
Tt =1 r ( s t , a t ) , which can be optimized via maximum entropy reinforcement learning algorithm.
We apply generative models to optimize the rest terms in the ELBO.
The term p ( o t | s t ) means that the hidden state s t needs to have the ability to generate current observation o t .
The Kullback-Leibler divergence term D KL [ q θ ( s t | s t − 1 , a t − 1 , o t )
|| p ( s t | s t − 1 , a t − 1 )] indicates that we should minimize the gap between approximate function q θ ( s t | s t − 1 , a t − 1 , o t ) and the prior p ( s t | s t − 1 , a t − 1 ) .
In standard
VAEs, the prior p ( · ) is generally ﬁxed and can be set as a standard Gaussian.
However, in this generative model, the prior of the hidden state s t is conditioned on previous state s t − 1 and action a t − 1 .
We will show how we deal with the conditional prior in next sub-section.
More details about the derivation process of the ELBO can be found in Appendix A.
4. V ARIATIONAL A UTOENCODERS FOR THE C ONDITIONAL P RIOR
The KL-divergence term in Eq.
(11) introduces a conditional prior for the hidden state s t , but the true conditional distribution p ( s t | s t − 1 , a t − 1 ) is unknown, making it intractable to do inference.
Hence we propose a new parameterized function p priorθ ( s t | s t − 1 , a t − 1 ) to approximate the true distribution p ( s t | s t − 1 , a t − 1 ) .
We use a standard VAE to learn the p priorθ ( · ) .
Hence ( s t − 1 , a t − 1 ) can be treated as the inner observed data in this VAE model and
the inner hidden state
ˆ s t is inferred from ( s t − 1 , a t − 1 ) .
We can write down the ELBO for the conditional prior directly by using the ELBO of standard VAEs (see in Appendix B):
[Equation]
where p ( s t ) can be set as a standard Gaussian.
The p priorθ ( · ) can be learned via a standard VAE paradigm.
The outputs of p priorθ ( · ) are µ prior and σ prior , so that the KL-divergence term in Eq.
(11) can be written as:
[Equation]
Unlike standard VAEs, the second distribution of the KL-divergence function in Eq.
(13) is not a standard normal distribution.
But we can still get the ﬁnal loss function if we expand the KLdivergence function.
We present the ﬁnal formula of the loss function for Eq.
(13) in Appendix C.
S EQUENTIAL V ARIATIONAL S OFT Q-L EARNING N ETWORKS
We design a deep recurrent neural network to improve the ELBO derived in Section 4.1 and Section 4.2.
Fig.
2 shows the overall structure of our algorithm.
In our implementation, there are two generative models, i.e., one of the generate model learns the conditional prior q priorθ ( · ) and reconstructs the inner observed data ( s t − 1 , a t − 1 ) , and another generate model learns the q θ ( · ) in Eq.
(13) and reconstructs the current observation o t .
For the ﬁrst generative model, there are two losses, i.e., L innerKL and L innerMSE .
L innerKL is the negated
KL-divergence between p priorθ ( s t | s t − 1 , a t − 1 ) and a standard Gaussian:
[Equation]
[Equation]
And L innerMSE is the reconstruction loss of ( s t − 1 , a t − 1 ) :
[Equation]
15) where MSE ( · ) is the mean-square error function, the inner hidden state ˆ s t is sampled from p priorθ ( s t | s t − 1 , a t − 1 ) , and ϕ innerθ ( · ) is a reconstruction function.
For the second generative model, there are also two losses, i.e., L elboKL and L elboMSE .
L elboKL is deﬁned as:
[Equation]
And L elboMSE is the reconstruction loss of o t :
[Equation]
) where MSE ( · ) is the mean-square error function, the hidden state s t is sampled from q θ ( s t | s t − 1 , a t − 1 , o t ) , and ϕ elboθ ( · ) is a reconstruction function.
Finally, we get two kinds of loss functions for the two generative models, i.e., the reconstruction loss L MES =
L innerMSE + L elboMSE and the KL-divergence loss L KL =
L innerKL + L elboKL .
And for the planning algorithm, we use the soft Q-learning algorithm (Levine, 2018).
Its loss function is the temporal difference error L TD , which is updated via Eq.
All these losses can jointly be optimized via stochastic gradient descent algorithms.
To reduce the computation complexity, we use a deep recurrent neural network to capture the historical information.
We ﬁrst use the f θ ( o t , a t − 1 , r t − 1 ) to extract a low-dim hidden feature w t from current input, and then feed this features to a recurrent unit rnn θ ( · ) to get the recurrent output h t .
Because h t contains the information of past observations, the loss function in Eq.
(16) can be rewritten as:
[Equation]
In the training, we tried both LSTM cell (Hochreiter & Schmidhuber, 1997) and GRU cell (Cho et al., 2014) as basic recurrent units.
Because of the generalization ability of the recurrent neural network (Lample & Chaplot, 2017), we can just sample a ﬁxed length H of sequential data for training instead of using the full-length data.
In the experiment, we also studied how the training length
H inﬂuences the ﬁnal performance.
We use a parallel training strategy, i.e., the program hosts multiple games in parallel and they all send batched data to a central data memory.
This is quite similar to the data collection method in ELF (Tian et al., 2017).
More details about the training strategy can be found in Appendix E.
E XPERIMENTS
We evaluate our algorithm on ﬂickering Atari (Hausknecht & Stone, 2015) and ViZDoom platform (Kempka et al., 2016).
Flickering Atari was previously used as the test environment in
DRQN (Hausknecht & Stone, 2015), ADRQN (Zhu et al., 2018) and DVRL (Igl et al., 2018).
The
ViZDoom platform is a 3D FPS game for AI research.
The agent in ViZDoom needs to navigate in the 3D environment to accomplish various tasks, such as gathering resources, shooting enemies and looking for the exit.
Because the agent can only see in just one direction each time step, this naturally introduces partial observability to the task.
5. E XPERIMENT S ETUP
We used some recent algorithms as baselines.
Because the true state and transition function are unknown, only the methods which can be trained under model-free setting will be used for comparison.
Baselines are listed as below:
Deep Q-Networks (DQN) (Mnih et al., 2015): A method which uses standard Bellman backup and uses the temporal difference error as the objective function.
Deep Soft Q-Networks (DSQN) (Levine, 2018): A method which uses soft Bellman backup and uses the temporal difference error as the objective function.
Deep Recurrent Q-Networks (DRQN) (Hausknecht & Stone, 2015): A method which applies LSTM on the top of the DQN.
Action
-Speciﬁc
Deep
Recurrent
Q-
Networks
(ADRQN) (Zhu et al
): A method which extends the DRQN, i.e., it uses both the observation and action as inputs.
Deep Variational Reinforcement Learning (DVRL) (Igl et al., 2018)
A method which combines sequential Monte Carlo and A2C (Dhariwal et al., 2017) to solve POMDPs.
5. E VALUATION ON F LICKERING A TARI
Atari environments (Bellemare et al., 2013) are widely used as the benchmark for deep reinforcement learning algorithms due to its high dimensional observation spaces and numerous challenging tasks.
Flickering Atari was introduced by (Hausknecht & Stone, 2015).
In each running step, the observation may be obscured with a certain probability, i.e., the raw screen will be either fully observable or fully obscured with black pixels.
The settings of experiments keep in line with DVRL (Igl et al., 2018), i.e., only one frame is used, the frameskip is set to four and each frame is obscured with a probability of 0.5.
We choose four Atrai games (i.e., Pong, ChopperCommand, DoubleDunk and Asteroids) for evaluation.
Fig.
3 shows the screen shots of these games.
When only given one frame as the input, these tasks are POMDPs because we cannot obtain the velocity of the moving object from a single observation.
These tasks have high dimensional and continuous observation spaces, while discrete action spaces.
The basic network architecture and hyper-parameters are similar to DQN (Mnih et al., 2015).
For the recurrent neural networks, we use a sequence length of 5 for training.
All the algorithms train for 10000,000 steps and run for 100 episodes during evaluation.
The training details can be found in Appendix E.
Table 1 shows the performance results of different algorithms on these Atari games.
We can see that our algorithms signiﬁcantly outperform other baselines on three of the games and get close score to DVRL on ChopperCommand.
Compared with DRQN and ADRQN, our method introduces inductive bias to the network, which helps state estimate and RL planning for POMDPs.
Compared with DVRL, our method can achieve competitive performance with lower sampling complexity.
[Equation]
[Table]
5. E VALUATION ON V I ZD OOM T ASKS
We designed three tasks in ViZDoom as the evaluation tasks for our algorithm.
Health Gather: The agent needs to gather health supplies in a ﬂat map.
The agent will lose health every time step.
Accordingly, if it can’t gather enough health supplies, it will die and the game is over.
At each time step, the agent will receive a reward of 0.001, which encourages the agent to live a longer life.
When it collects one health supply, it will receive a reward of 1.
The observation for the agent is the grayscale image of its vision and its health value.
The agent has three actions to choose, i.e.,
LEFT
RIGHT
Health Gather v2: This is a more difﬁcult task than previous task with a more complex map.
There is some poison in the map.
When the agent encounters poison, it will lose health.
We use the same reward scenario, observations and actions in Health Gather .
Defend Center: In this task, the agent stands in the center of a ﬂat map.
Monsters will walk toward the agent from the edge of the map.
When the monsters touch the agent, the agent will lose health.
The agent holds a gun with limited ammo, and it can kill the monsters by pressing down the shooting button.
The agent gets reward of 4 by killing monsters, reward of -0.2 by using ammo and reward of
[Figure]
[Table]
Evaluations use Mann-Whitney rank test and bold numbers indicate statistical signiﬁcance at the 5% level.
The SVQN models achieve the best performance on these three tasks.
-1 when losing health.
The observation for the agent is the grayscale image of its vision, its health value and the ammo remained.
The agent has three actions to choose, i.e., TURN LEFT , TURN
RIGHT and ATTACK .
Fig.
3 shows the screen shots of these three tasks.
All the algorithms are trained with the same basic network architecture and use the same hyperparameters.
All the models take only one observation as input at each time step and the vision inputs are resized to the resolution of 84 × 84 .
For the recurrent neural networks, we use a sequence length of 5 for training.
All the algorithms train for 300, steps and run for 20 episodes during evaluation.
The discount factor γ is set to 0 .
95 , learning rate is 0 . 0001 and the Adam optimizer (Kingma & Ba, 2014) is used for training.
The network architecture for these tasks and training details are shown in Appendix D.
Fig. 4(a) shows the training curves of different methods on the Health Gather.
Both SVQN models outperform other baselines.
Table 2 reports the ﬁnal scores of different models on these three tasks.
The SVQN models achieve the best performance on these three tasks.
Experiments on ViZDoom indicate that generative models of SVQNs can improve agent’s exploration ability in complicated unknown environment
5. A BLATION S TUDY
Training sequence length: We study how the training sequence length H impacts the algorithm’s performance.
We use the SVQN(LSTM) model and the environment of Health Gather for this study.
Fig. 4(b) shows training processes of models with different training sequence lengths( H = 2 , 5 , 10 , 15 ).
We can see that, when the sequence length is too short( H = 2 ), the model can’t gather enough historical information, which causes performance degradation.
When the sequence length is too long( H = 15 ), it is arduous for the network to do optimization, which will also lead to performance degradation.
This experiment also shows that with a limited training length, the algorithm can generalize to the time of arbitrary lengths during testing.
Thanks to this generalization ability, we can train the agent with a ﬁxed length of data instead of the full-length data, which can reduce the computation complexity.
Observation Probability: We also study whether different models are robust to the noise of the observation.
We add a modiﬁcation to the game Health Gather, i.e., at each time step, the observation of the screen is either fully revealed or fully obscured with a ﬁxed observation probability p .
Fig.
4(c) show the evaluation results of different models under different observation probabilities( p = 0 .
All the models are trained under the standard environment with full observations.
The results show that SVQN models are relatively robust to the disturbance of the observation compared to other algorithms.
C ONCLUSIONS
We propose a novel algorithm named Sequential Variational Soft Q-Learning Networks (SVQN) to solve POMDPs with the discrete action space.
SVQN is model-free and does not need to know the true state’s representation.
We apply generative models to deal with the conditional prior of hidden states and use a recurrent neural network to reduce the computational complexity, i.e., with a small length of training data, it can generalize to the test data with an arbitrary length.
Our designed deep neural network can be trained end-to-end, which optimizes the planning and inference of hidden states jointly.
Experimental results show that SVQN outperforms previous methods on challenging tasks and has the robustness to the disturbance of the observation.
SVQN is also ﬂexible and can be integrated with other maximum entropy reinforcement learning algorithms, such as soft actorcritic (Haarnoja et al., 2018).
In the future, we will try to develop algorithms for POMDPs problems with the continuous action space.
A CKNOWLEDGMENTS
This work was supported by the National Key Research and Development Program of China (No.
2017YFA0700904), NSFC Projects (Nos. 61620106010, U19B2034, U1811461), Beijing NSF
Project (No. L172037), Beijing Academy of Artiﬁcial Intelligence (BAAI), Tsinghua-Huawei Joint
Research Program, a grant from Tsinghua Institute for Guo Qiang, Tiangong Institute for Intelligent Computing, the JP Morgan Faculty Research Program and the NVIDIA NVAIL Program with
GPU/DGX Acceleration.
A D ERIVATION OF THE V ARIATIONAL L OWER B OUND
[Equation]
( o t | s t )] − D KL [ q ( s t | s t − 1 , a t
[Table]
1 , o t )
|| p ( s t | s t − 1 , a t − 1 )] (cid:111) , where s 1: T ∼ q ( s 1: T |O 1: T , a 0: T , o 1: T ) (19)
B V ARIATIONAL A UTOENCODERS
Variational auto-encoders (VAEs) (Kingma & Welling, 2013) are effective generative models that can recover complex distributions over the data space.
In the VAE, there are observed data x and the underlying causal factor z .
Usually, it’s intractable to do inference for the posterior p ( z | x ) .
The
VAE uses a variational function q ( z | x ) to approximate the true posterior.
The lower bound of the marginal distribution of the observed data is given by:
[Equation]
and the lower bound can be equivalently written as :
[Equation] [Equation]
where D KL ( ·||· ) is Kullback-Leibler divergence between two distributions.
The approximate posterior q ( z | x ) is often set as Gaussian N ( µ, diag( σ 2 )) , where µ and σ are the outputs of a non-linear function of x .
The generative model p ( x | z ) and inference model q ( z | x ) can be trained jointly via standard backpropagation techniques.
C O PTIMIZE THE KL S IN E Q . (15)
We give out the ﬁnal form of the loss function as below:
[Equation]
, where p and q are two Gaussian distributions.
σ p , σ q , µ q and µ p are their standard deviations and means respectively.
D N ETWORK A RCHITECTURE
D.
A
TARI
The network architecture for Atari is shown as below:
D. V I ZD OOM
The network architecture for ViZDoom is shown as below:
E T RAINING D ETAILS Training architechture:
The data produce module, training module and evaluation module are run in parallel.
The framework is implemented by Python and Tensorﬂow.
E. A TARI
All the models take only one observation as input at each timestep and the vision inputs are resized to the resolution of 84 × 84 .
For the recurrent neural networks, we use sequence length of 5 for training.
All the algorithms train for 1,0000,000 steps and run for 100 episodes during evaluation.
E. V I ZD OOM
All the algorithms are trained with the same basic network architectures and use the same hyperparameters.
All the models take only one observation as input at each timestep and the vision inputs are resized to the resolution of 84 × 84 .
For the recurrent neural networks, we use sequence length of 5 for training.
All the algorithms train for 300,000 steps and run for 20 episodes during evaluation.
The discount factor γ is set to 0 .
95 , learning rate is 0 . 0001 and the Adam optimizer is used for training.