D ISCRIMINATIVE P ARTICLE F ILTER R EINFORCEMENT L EARNING FOR C OMPLEX P ARTIAL OBSERVATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep reinforcement learning has succeeded in sophisticated games such as Atari,
Go, etc.
Real-world decision making, however, often requires reasoning with partial information extracted from complex visual observations.
This paper presents Discriminative Particle Filter Reinforcement Learning (DPFRL), a new reinforcement learning framework for complex partial observations.
DPFRL encodes a differentiable particle ﬁlter with learned transition and observation models in a neural network, which enables reasoning with partial observations over multiple time steps.
While a standard particle ﬁlter relies on a generative observation model, DPFRL learns a discriminatively parameterized model trained speciﬁcally for decision making.
We show that the discriminative parameterization results in signiﬁcantly improved performance, especially for tasks with complex visual observations, as it circumvents the difﬁculty of modeling complex observations that are irrelevant to decision making.
Experiments show that DPFRL outperforms state-of-the-art
POMDP RL models in Flickering Atari Games, an existing POMDP RL benchmark and in Natural Flickering Atari Games, a new, more challenging POMDP RL benchmark introduced in this work.
Further, DPFRL performs well for visual navigation with real-world data in the Habitat environment.
I NTRODUCTION
Deep Reinforcement Learning (DRL) has attracted signiﬁcant interest with applications ranging from game playing (Mnih et al., 2013; Silver et al., 2017) to robot control and visual navigation (Levine et al., 2016;
Kahn et al., 2018; Savva et al., 2019).
However, more natural or real-world environments pose signiﬁcant challenges for current DRL methods (Arulkumaran et al., 2017), in part because they require (1) reasoning in a partially observable environment (2) reasoning with complex observations, e.g. visually rich images.
For example, a robot navigating in a new environment has to (1) localize and plan a path having only partial information of the environment (2) extract the traversable space from image pixels, where the relevant geometric features are tightly coupled with irrelevant visual features, such as wall textures and lighting.
Decision making under partial observability can be formulated as a partially observable Markov decision process (POMDP).
Solving POMDPs requires tracking the posterior distribution of the states, called the belief .
Most POMDP RL methods track the belief, represented as a vector, using a recurrent neural network (RNN) (Hausknecht & Stone, 2015; Zhu et al., 2018).
RNNs are model-free generic function approximators, and without appropriate structural priors they may need large amounts of data to learn to track a complex belief.
Model-based DRL methods aim to reduce the sample complexity by learning an environment model simultaneously with the policy.
In particular, to deal with partial observability, Igl et al.
(2018) recently proposed DVRL that learns a generative observation model incorporated into the policy through a Bayes ﬁlter.
Because the Bayes ﬁlter tracks the belief explicitly, DVRL performs much better than generic RNNs under partial observability.
However, a Bayes ﬁlter normally assumes a generative observation model, that deﬁnes the probability p ( o | h t ) of receiving an observation o =
o t given the history h t of past observations and actions (Fig. 1b).
Learning this model can be very challenging since the strong generative assumption requires modeling the whole observation space, including features irrelevant for RL.
When o t is an image, p ( o | h t ) is a distribution over all possible images, e.g., parameterized by independent pixel-wise Gaussians with learned mean and variance.
This means, e.g., to navigate in a previously unseen environment, we need to learn the
[Table]
[Figure]
distribution of all possible environments with their visual appearance, lighting condition, etc. — a much harder task than learning to extract features relevant to navigation, e.g. the traversable space.
We introduce the Discriminative Particle Filter Reinforcement Learning (DPFRL), a POMDP RL method that learns to explicitly track a latent belief, while circumventing the difﬁculty of generative observation modeling, and learns to make decisions based on features of the latent belief (Fig. 1a).
DPFRL approximates the belief by a set of weighted learnable latent particles { ( h i t , w it ) } K i =1 , and it tracks the particle belief by a non-parametric Bayes ﬁlter algorithm, a particle ﬁlter , encoded as a differentiable computational graph in the neural network architecture.
Transition and observation models for the particle ﬁlter are neural networks learned jointly end-to-end, optimized for the overall policy.
Importantly, we use a discriminatively parameterized observation model, f obs ( o t , h t ) , a neural network that takes in o t and h t and outputs a single value, a direct estimate of the log-likelihood as shown in Fig. 1c.
The discriminative parameterization relaxes the generative assumption and avoids explicitly modeling the entire complex observation space when computing observation likelihood.
The intuition is similar to that of, e.g., energy-based models (LeCun et al., 2006) and contrastive predictive coding (Oord et al., 2018), but here the learning signal comes directly from the RL objective, backpropagating through the differentiable particle ﬁlter, thus f obs ( o t , h t ) only needs to model the observation features relevant to decision making.
In addition, to summarize the particle belief, we introduce novel learnable features based on Moment-Generating Functions (MGFs) (Bulmer, 1979).
MGF features are computationally efﬁcient and permutation invariant, and they can be directly optimized to provide useful higher-order moment information for learning the policy.
MGF features could be also used as learned features of any empirical distribution in application beyond RL.
We evaluate DPFRL on a range of POMDP RL domains: a continuous control task from Igl et al.
(2018), Flickering Atari Games (Hausknecht & Stone, 2015), Natural Flickering Atari Games, a new domain with more complex observations that we introduce, and the Habitat visual navigation domain using real-world data (Savva et al., 2019).
DPFRL outperforms state-of-the-art POMDP RL methods in most cases.
Results show that the particle ﬁlter structure is effective for handling partial observations, and the discriminative parameterization allows for complex observations.
We summarize our contributions as follows: 1) a differentiable particle ﬁlter based method with a discriminatively parameterized observation model for RL with partial and complex observations.
2) effective MGF features for empirical distributions, e.g., particle distributions 3) a new RL benchmark, Natural Flickering Atari Games, that introduces both partial observability and complex visual observations to the popular Atari domain.
We will open source the code to enable future work.
R ELATED W ORK
Real-world decision-making problems are often formulated as POMDPs given the partial observations.
POMDPs are notoriously hard to solve; in the worst case, they are computationally intractable (Papadimitriou & Tsitsiklis, 1987).
Approximate POMDP solvers have made dramatic progress in solving large-scale POMDPs (Kurniawati et al., 2008).
Particle ﬁlters have been widely adopted as belief tracker for POMDP solvers (Silver & Veness, 2010; Somani et al., 2013) with the ﬂexibility to model complex and multi-modal distributions, compared to Gaussian and Kalman ﬁlters.
However, a predeﬁned model and state representations are required for these methods (see e.g. Bai et al.
Given the advance in generative neural network models, various neural models (Chung et al., 2015;
Maddison et al., 2017; Le et al., 2018; Naesseth et al., 2018) have been proposed for belief tracking.
DVRL (Igl et al., 2018) uses Variational Sequential Monte-Carlo method (Naesseth et al., 2018),
[Figure]
network networkValue
Policy
MGF feature extractor ℎ ¯
RL Loss which is similar to the particle ﬁlters that we use, for belief tracking in RL.
This gives better belief tracking capabilities, but as we demonstrate in our experiments, generative modeling is not robust in complex observation spaces with high-dimensional irrelevant features.
More powerful generative models, e.g., DRAW (Gregor et al., 2015), could be considered to improve generative observation modeling; however, evaluating a complex generative model for each particle would signiﬁcantly increase the computational cost and optimization difﬁculty.
Learning a robust latent representation and avoiding reconstructing observations are of great interest for RL (Oord et al., 2018; Guo et al., 2018; Hung et al., 2018; Gregor et al., 2019; Gelada et al., 2019).
Discriminative RNNs have also been widely used for belief approximation in partially observable domains (Bakker, 2002;
Wierstra et al., 2007;
Foerster et al., 2016).
The latent representation is directly optimized for the policy p ( a | h t ) that skips observation modeling.
Hausknecht & Stone (2015); Zhu et al.
(2018) tackle the partially observable Flickering Atari Games by extending DQN (Mnih et al., 2013) with an LSTM memory.
Our experiments demonstrate that the additional structure provided by using a particle ﬁlter to track beliefs can give improved performance in RL.
Embedding algorithms into neural networks to allow end-to-end discriminative training has gained attention recently.
For belief tracking, the idea has been used in the differentiable histogram ﬁlter (Jonschkowski & Brock, 2016), Kalman ﬁlter (Haarnoja et al., 2016) and particle ﬁlter (Karkus et al., 2018).
Karkus et al.
(2017) combined a learnable histogram ﬁlter with the Value Iteration
Network (Tamar et al., 2016) and introduced the learnable POMDP planner, QMDP-net.
However, these methods require a predeﬁned state representation and are limited to relatively small state spaces.
Ma et al.
(2019) integrated the particle ﬁlter with standard RNNs, e.g., LSTM, and introduced a discriminative PF-RNNs for sequence prediction.
We build on the work in Ma et al. (2019) demonstrating its advantages for RL with complex partial observations and introducing MGF features to the method for improved decision making from particle beliefs.
The proposed discriminative reinforcement framework and algorithms can be applied to most of the discriminative particle ﬁlters.
D ISCRIMINATIVE P ARTICLE F ILTER R EINFORCEMENT L EARNING
We introduce DPFRL, a framework for reinforcement learning under partial and complex observations.
The DPFRL architecture is shown in Fig.
It has two main components, a discriminative particle ﬁlter that tracks a latent belief b t , and an actor network that learns a policy p ( a | b t ) given belief b t .
A pseudocode for the DPFRL algorithm is given by Alg. 1 in the Appendix.
3. D ISCRIMINATIVE P ARTICLE F ILTER FOR T RACKING THE L ATENT B ELIEF
State Representation.
In DPFRL, we use a fully differentiable particle ﬁlter algorithm to maintain a belief state b t .
More speciﬁcally, we approximate the belief state with a set of weighted latent particles b t ≈ { ( h it , w it ) }
Ki =1 , where { h it } Ki =1 are K latent states learned by policy-oriented training, and { w it } Ki =1 represents the corresponding weights.
Each latent state h
it stands for a hypothesis in the belief; the set of latent particles approximates the statistics of the belief.
Belief Update.
We update the latent particles according to particle ﬁlter algorithm
[Equation]
Transition update.
Eqn.
1 implements the transition update step.
We sample the next state h
it from f trans ( h it − 1 , a t , u t ( o t ))
=
p ( h | h
it − 1 , a t , u t ( o t )) , where a t is the agent action and u t ( o t )
parameter- ized function that can learn the environment dynamics.
This formulation assumes a fully controlled system where a learned latent state that captures the dynamics of both the agent and the environment.
Similar formulation has been used for sequence prediction (Ma et al., 2019).
In our experiments, f trans is implemented by a gated function following PF-GRU (Ma et al., 2019), where features are extracted from o t with a neural network.
Further details are in the Appendix.
Measurement update.
Eqn.
2 implements the measurement update.
f obs ( h it , o t ) is the discriminatively parameterized observation function, that is, a learned function that takes h it and o t as inputs, and outputs a scalar value, an estimate of the unnormalized likelihood proportionate to p ( o t | h it ) . We evaluate f obs ( h it , o t ) for all i = 1 : K and normalize using η .
Gradients for training f obs are obtained by backpropagating the standard RL loss through the belief-conditional policy p ( a | b t ) , the update steps of the particle ﬁlter algorithm, and the particle weight update in Eqn. 2.
We expect the discriminatively parameterized f obs to be more effective than a standard generative model for the following reasons. A generative model aims to approximate p ( o | h ) by learning a function that takes h as input and outputs parameterized distribution over o , for example, pixel-wise Gaussians with learned mean and variance. However, the generative assumption forces the model to consider the entire observation space equally, including features irrelevant for RL.
In contrast, f obs has a much simpler functional form that takes o and h as inputs.
It is also more ﬂexible and does not need to form a parametric distribution over o , which relaxes the strong generative assumption. When trained end-to-end for RL, f obs learns to model only the features relevant to RL. Since f obs is only used for particle ﬁltering, it only needs to learn an unnormalized likelihood value that is proportionate to p ( o t | h it ) when evaluated for different h it , i = 1 : K . This can be a much simpler function to learn than the full generative distribution, especially for complex observations, as we avoid modeling features that are irrelevant for belief tracking and decision making.
In implementation we compute weight updates in log-space for better numerical stability. We use a simple fully connected layer for f obs that takes in the concatenated o t and h it , and outputs a single value interpreted as the observation log-likelihood.
Note that more complex network architectures could be considered to further improve the capability of f obs , which we leave to future work.
Differentiable particle resampling.
To avoid particle degeneracy, i.e., most of the particles having near-zero weight, we adopt the differentiable soft-resampling strategy (Karkus et al., 2018; Ma et al., 2019).
Instead of sampling from p t ( i )
=
w it , we sample particles { h (cid:48) it } Ki =1 from a softened proposal distribution q ( i ) =
αw it + (1 − α )1 /K , where α is an trade-off parameter.
The new weights are derived using importance sampling: w (cid:48) it =
w it αw it +(1 − α )1 /K .
We can have the ﬁnal particle belief distribution as { ( h (cid:48) it , w (cid:48) it ) }
Ki =1 =
Soft-Resampling ( { ( h it , w it ) }
Ki =1 ) .
As a result, f obs can be optimized with global belief information and model shared useful features across multiple time steps.
Another concern is that the particle distribution may collapse to particles with the same latent state.
This can be avoided by ensuring that the stochastic transition function f trans has a non-zero variance.
3. B ELIEF -C ONDITIONAL A CTOR N ETWORK
Conditioning a policy directly onto a particle belief distribution is non-trivial.
To feed it to the networks, we need to summarize it into a single vector.
We introduce a novel feature extraction method for empirical distributions based on MomentGenerating Functions (MGFs).
The MGF of an n -dimensional random variable X is given by
M X ( v ) =
E [ e v T X ] , v ∈ R n .
In statistics, MGF is an alternative speciﬁcation of its probability distribution (Bulmer, 1979).
Since particle belief distribution b t is an empirical distribution, the moment generating function of b t can be denoted as M b t ( v ) =
K (cid:80) i =
1 w it e v T h it .
A more detailed background on
MGFs is in Appendix A.2. In DPFRL, we treat v as learnable parameters, and deﬁne m MGF features, determined by v 1: m .
The j -th MGF feature is given by M jb t ( v j ) .
For a clean notation, we use M jt in place of M jb t ( v j ) .
We summarize the belief distribution with b t ≈ (cid:2) ¯ h t , M 1: m t (cid:3) , where ¯ h t = K (cid:80) i =1 w i t
h
i t is the mean particle.
The mean particle ¯ h t , as the ﬁrst-order moment, and m additional MGF features, give a summary of the belief distribution characteristics.
The number of MGF features, m , controls how much additional information we extract from the belief.
We empirically study the inﬂuence of MGF features in ablation studies.
Compared to Ma et al.
(2019) that uses the mean as the belief estimate, MGF features provide additional features from the empirical distribution.
Compared to DVRL (Igl et al., 2018) that treats the Monte-Carlo samples as a sequence and merges them by an RNN, MGF features are permutationinvariant, computationally efﬁcient and easy to optimize, especially when the particle set is large.
Given b t ≈ (cid:2) ¯ h t , M 1: m t (cid:3) , we compute the policy p ( a | b t ) with a policy network π ( b t ) .
In actor-critic setup, a value network V ( b t ) is introduced to assist learning.
In our experiment, we evaluated on the A2C algorithm (Mnih et al., 2016).
E XPERIMENTS
We evaluate DPFRL in a range of POMDP RL domains with increasing belief tracking and observation modeling complexity.
We ﬁrst use benchmark domains from the literature, Mountain Hike, anddifferent Flickering Atari Games.
We then introduce a new, more challenging domain, Natural
Flickering Atari Games, that uses a random video stream as the background.
Finally we apply DPFRL to a challenging visual navigation domain with RGB-D observations rendered from real-world data.
We compare DPFRL with a GRU network, a state-of-the-art POMDP RL method, DVRL, and ablations of the DPFRL architecture.
As a brief conclusion, we show that: 1) DPFRL signiﬁcantly outperforms GRU in most cases because of its explicit structure for belief tracking; 2) DPFRL outperforms the state-of-the-art DVRL in most cases even with simple observations, and its beneﬁt increases dramatically with more complex observations because of DPFRL’s discriminative observation model; 3) MGF features are more effective for summarizing the latent particle belief than alternatives.
4. E XPERIMENTAL S ETUP
We plot the accumulated rewards and all reported results are averaged over 3 different random seeds.
The curves are smoothed over time and averaged over parallel environment executions.
To be comparable with the GRU and DVRL baselines we train
DPFRL with the same A2C algorithm and use a similar network architecture and hyperparameters as the original DVRL implementation.
DPFRL and DVRL differ in the particle belief update structure, but they use the same latent particle size dim ( h ) and the same number of particles K as in the DVRL paper (dim ( h ) = 128 and K =for Mountain Hike, dim ( h )
= 256 and K = 15 for Atari games and visual navigation).
The effect of the number of particles is discussed in Sect.
We train all models for the same number of iterations using the RMSProp optimizer (Tieleman & Hinton, 2012).
Learning rates and gradient clipping values are chosen based on a search in the BeamRider Atari game independently for each model.
Further details are in the Appendix.
We have not performed additional searches for the network architecture and other hyper-parameters, nor tried other RL algorithm, such as PPO (Schulman et al., 2017), which may all improve our results.
4. M OUNTAIN H IKE .
Mountain Hike has been introduced by Igl et al.
(2018) to demonstrate the beneﬁt of belief tracking for POMDP RL.
It is a continuous control problem where an agent navigates on a ﬁxed 20 × 20 map.
In the original observation, partial observability is introduced by disturbing the agent observation with an additive Gaussian noise.
To illustrate the effect of observation complexity in natural environments, we concatenate the original observation vector with a random noise vector.
The complexity of the optimal policy remains unchanged, but the relevant information is now coupled with irrelevant observation features.
More speciﬁcally, the state space and action space in Mountain Hike are deﬁned
[Figure]
R e t u r n a t
I t e r a t i o n 5 e DPFRLDVRLGRU     5 Frames 1e [Figure] R e t u r n
DPFRLDVRLGRU     5 Frames 1e
[Figure]
R e t u r n
DPFRLDVRLGRU
[Figure]
Frames 1e
[Figure]
R e t u r n
DPFRLDVRLGRU
Effect of varying l l =
[Figure]
[Figure]
as S = A = R 2 , where s t =
[ x t , y t ] and a t = [ δx t , δy t ] .
Transitions of the agent are stochastic with an additive Gaussian noise: s t +
=
s t + a t + (cid:15) a , where (cid:15) a ∼ N (0 , 0 . 25) .
The observation space is
O = R 2+ l , where l is a predeﬁned constant and l = 0 corresponds to the original setting.
Observations are o t =
[ o st , o nt ] , where o st = s t + (cid:15) s , (cid:15) s ∼ N (0 , 1) , and o nt ∈ R l is sampled from a uniform distribution U ( − 10 , 10) .
The reward for each step is given by r t = r ( x t , y t ) − 0 .
|| a t || where r ( x t , y t ) is shown in Fig.
Episodes end after 75 steps.
We train models for different settings of the noise vector length l , from l = 0 to l = 100 .
Results are shown in Fig.
We observe that DPFRL learns faster than the DVRL and GRU in all cases, including the original setting l = 0 .
Importantly, as the noise vector length increases, the performance of DVRL and GRU degrades, while DPFRL is unaffected.
This demonstrates the ability of DPFRL to track a latent belief without having to explicitly model complex observations.
4. A TARI G AMES WITH P ARTIAL O BSERVABILITY .
Atari games are one of the most popular benchmark domains for RL methods (Mnih et al., 2013).
Their partially observable variants, Flickering Atari Games, have been used to benchmark POMDP
RL methods (Hausknecht & Stone, 2015; Zhu et al., 2018; Igl et al., 2018).
Here image observations are single frames randomly replaced by a blank frame with a probability of 0 . 5 .
The ﬂickering observations introduce a simple form of partial observability.
Another variant, Natural Atari Games (Zhang et al., 2018), replaces the simple black background of the frames of an Atari game with a randomly sampled video stream.
This modiﬁcation brings the Atari domain one step closer to the visually rich real-world, in that the relevant information is now encoded in complex observations.
As shown by Zhang et al.
(2018), this poses a signiﬁcant challenge for RL.
We propose a new RL domain, Natural Flickering Atari Games , that involves both challenges: partial observability simulated by ﬂickering frames, and complex observations simulated by random background videos. The background videos increase observation complexity without affecting the decision making complexity, making this a suitable domain for evaluating RL methods with complex observations.
We sample the background video from the ILSVRC dataset (Russakovsky et al., 2015).
Examples for the BeamRider game are shown in Fig.
Details are in the Appendix.
We evaluate DPFRL for both Flickering Atari Games and Natural Flickering Atari Games.
We use the same set of games as Igl et al.
To ensure a fair comparison, we take the GRU and DVRL results from the paper for Flickering Atari Games, use the same training iterations as in Igl et al.
and we use the ofﬁcial DVRL open source code to train for Natural Flickering Atari Games.
Results are summarized in Table 1.
We highlight the best performance in bold where the difference is statistically signiﬁcant ( p = 0 . 05 ).
Detailed training curves are in the Appendix.
[Table]
We observe that DPFRL signiﬁcantly outperforms GRU in almost all games, which indicates the importance of explicit belief tracking, and shows that DPFRL can learn a useful latent belief representation.
Despite the simpler observations, DPFRL signiﬁcantly outperforms DVRL and achieves state-of-the-art results on 5 out of 10 standard Flickering Atari Games (ChopperCommand, MsPacman, BeamRider, Bowling, Asteroids), and it performs comparably in 3 other games (Centipede,
Frostbite, IceHockey).
The strength of DFPRL shows even more clearly in the Natural Flickering
Atari Games, where it signiﬁcantly outperforms DVRL on 7 out of 10 games and performs similarly in the rest.
In some games, e.g. in Pong, DPFRL performs similarly with and without videos in the background (15.65 vs. 15.40), while the DVRL performance degrades substantially (-19.
vs. 18.17).
These results show that while the architecture of DPFRL and DVRL are similar, the policy-oriented discriminative observation model of DPFRL is much more effective for handling complex observations, and the MGF features provide a more powerful summary of the particle belief for decision making. However, on some games, e.g. ChooperCommand, even DPFRL performance drops signiﬁcantly when adding background videos.
This shows that irrelevant features can make the task much harder, even for a discriminative approach, as also observed in in Zhang et al.
4. V ISUAL N AVIGATION
[Table]
Visual navigation poses a great challenge for deep RL Mirowski et al. ; Zhu et al. ; Lample & Chaplot (2017). We evaluate DPFRL for visual navigation in the Habitat Environment (Savva et al., 2019), using the real-world Gibson dataset (Xia et al., 2018).
In this domain, a robot needs to navigate to goals in previously unseen environments.
In each time step, it receives a ﬁrst-person RGB-D camera image, and its distance and relative orientation to the goal.
The main challenge lies in the partial and complex observations: ﬁrst-person view images only provide partial information about the unknown environment; and the relevant information for navigation, traversability, is encoded in rich RGB-D observations along with many irrelevant features, e.g., the texture of the wall.
We use the Gibson dataset with the training and validation split provided by the Habitat challenge.
We train models with the same architecture as for the Atari games, except for the observation encoder that accounts for the different observation formats.
We evaluate models in unseen environments from the validation split and compute the same set of metrics as in the literature, SPL and Success Rate, as well as average rewards.
Results are shown in Table 2.
Further details and results are in the Appendix.
DPFRL signiﬁcantly outperforms both DVRL and GRU in this challenging domain.
DVRL performs especially poorly, demonstrating the difﬁculty of learning a generative observation model in realistic, visually rich domains.
DPFRL also outperforms the PPO baseline from Savva et al.
We note that submissions to the recently organized Habitat Challenge 2019 (Savva et al., 2019), such as (Chaplot et al., 2019), have demonstrated better performance than the PPO baseline (while our results are not directly comparable because of the closed test set of the competition).
However, these [Table] approaches rely on highly specialized structures, such as 2D mapping and 2D path planning, while we use the same generic network as for Atari games.
Future work may further improve our results by adding a task-speciﬁc structure to DPFRL or training with PPO instead of A2C.
4. A BLATION S TUDY
We conduct an extensive ablation study on the Natural Flickering Atari Games to understand the inﬂuence of each DPFRL component.
The results are presented in Table 3.
Discriminative parameterization is more effective than generative parameterization.
DPFRLgenerative replaces the discriminative observation function of DPFRL with a generative observation function, where grayscale image observations are modeled by pixel-wise Gaussian distributions with learned mean and variance.
Unlike DVRL, DPFRL-generative only differs from DPFRL in the parameterization of the observation function, the rest of the architecture and training loss remains the same.
In most cases, the performance for DPFRL-generative degrades signiﬁcantly compared to
DPFRL.
These results are aligned with our earlier observations and indicate that the discriminative parameterization is indeed important to extract the relevant information from complex observations without having to learn a more complex generative model.
More particles perform better.
DPFRL with 1 particle performs poorly on most of the tasks (DPFRLLP1).
This indicates that a single latent state is insufﬁcient to represent a complex latent distribution that is required for the task, and that more particles can be expected to improve performance.
MGF features are useful.
We compare DPFRL using MGF features with DPFRL-mean that only uses the mean particle, and with DPFRL-GRUmerge that uses a separate RNN to summarize the belief similar to DVRL.
Results show that DPFRL-mean does not work as well as the standard DPFRL, especially for tasks that may need complex belief tracking, e.g., Pong.
This can be attributed to the more rich belief statistics provided by MGF features, and that they do not constrain the learned belief representation to be always meaningful when averaged.
Comparing to DPFRL-GRUmerge shows that MGF features generally perform better.
While an RNN may learn to extract useful features from the latent belief, optimizing the RNN parameters is harder, because they are not permutation invariant to the set of particles and they result in a long backpropagation chain.
C ONCLUSION
We have introduced DPFRL, a principled framework for POMDP RL in natural environments.
DPFRL combines the strength of Bayesian ﬁltering and end-to-end RL: it performs explicit belief tracking with discriminative learnable particle ﬁlters optimized directly for the RL policy.
DPFRL achieved state-of-the-art results on POMDP RL benchmarks from prior work, Mountain Hike and a number of Flickering Atari Games, and it signiﬁcantly outperformed alternative methods in a new, more challenging domain, Natural Flickering Atari Games, as well as for visual navigation using real-world data.
We have proposed a novel MGF feature for extracting statistics from an empirical distribution.
MGF feature extraction could be applied beyond RL, e.g., for general sequence prediction.
DPFRL does not perform well in some particular cases, e.g., DoubleDunk.
While our discriminatively parameterized observation function is less susceptible to observation noise, it does not allow for additional learning signals that improve sample efﬁciency, e.g., through a reconstruction loss.
Future work may combine generative and discriminative modeling with the principled DPFRL framework. A B ACKGROUND A. P ARTICLE F ILTER A LGORITHM Particle ﬁlter is an approximate Bayes ﬁlter algorithm for belief tracking.
Bayes ﬁlters estimate the belief b t , i.e., a posterior distribution of the state s t , given the history of actions a 1: t and observations o 1: t . Instead of explicitly modeling the posterior distribution, particle ﬁlter approximates the posterior with a set of weighted particles, b t ≈ { ( s it , w it ) } Ki =1 , and update the particles in a Bayesian manner. Importantly, the particle set could approximate arbitrary distributions, e.g., Gaussians, continuous multi-modal distributions, etc.
The mean state can be estimated as the mean particle ¯ s t
= K (cid:80) i
=1 w it s it .
The particle updates include three steps: transition update, measurement update, and resampling.
Transition update.
We ﬁrst update the particles by a given motion model.
More speciﬁcally, we sample the next state s it +1 from a generative transition function
[Equation]
where p ( s | s it , a t ) is the transition function.
Measurement update.
The particle weights are updated again using the observation likelihoods
[Equation]
where η is a normalization factor and p ( o t | s it +1 )
is the observation likelihood computed by evaluating observation o t in a generative observation function p ( o | s it +1 ) .
Resampling.
The particle ﬁlter algorithm can suffer from particle degeneracy, where after some update steps only a few particles have non-zero weights.
This would prevent particle ﬁlter to approximate the posterior distribution effectively.
Particle degeneracy is typically addressed by performing resampling, where new particles are sampled with repetition proportional to its weight.
Speciﬁcally, we sample particles from a categorical distribution p parameterized by the particle weights { w it } Ki =
p
[Equation]
where p ( i ) is the probability for the i -th category, i.e., the i -th particle.
The new particles approximate the same distribution, but they assign more representation capacity to the relevant regions of the state space.
A. M OMENT -G ENERATING F UNCTIONS
In probability theory, the moment-generating function (MGF) is an alternative speciﬁcation of the probability distribution of a real-valued random variable (Bulmer, 1979).
As its name suggests, MGF of a random variable could be used to generate any order of its moments, which characterize its probability distribution.
Mathematically, the MGF of a random variable X with dimension m is deﬁned by
[Equation]
7) where v ∈ R m
and we could consider the MGF of random variable X is the expectation of the random variable e vX .
Consider the series expansion of e vX
[Equation]
Thus we have
[Equation]
To compute the j -th order moment M j , we can compute the j -th order derivative and set v to 0.
[Equation]
[Equation]
In DPFRL, we use MGFs as additional features to provide moment information of the particle distribution.
DPFRL learns to extract useful moment features for decision making by directly optimizing for policy p ( a | b t ) .
Observation Encoders: For the observation encoders, we used the same structure with DVRL (Igl et al., 2018) for a fair comparison.
For Mountain Hike, we use two fully connected layers with batch normalization and ReLU activation as the encoder.
The dimension for both layers is 64.
For the rest of the domains, we ﬁrst down-sample the image size to 84 × 84, then we process images with2D-convolution layers with channel number (32, 64, 32), kernel sizes (8, 4, 3) and stride (4, 2, 1), without padding.
The compass and goal information are a vector of length 2; they are appended after the image encoding as the input.
Observation Decoders: Both DVRL and PFGRU-generative need observation decoders.
For the
Mountain Hike, we use the same structure as the encoder with a reversed order.
The transposed 2D-convolutional network of the decoder has a reversed structure.
The decoder is processed by an additional fully connected layer which outputs the required dimension (1568 for Atari and Habitat
Navigation, both of which have 84 × 84 observations).
Stochastic Transition Function: We directly use the transition function in PF-GRU (Ma et al., 2019) for p ( h | h
it − 1 , a t , o t ) , which is a stochastic function with GRU gated structure.
Action a t is ﬁrst encoded by a fully connected layer with batch normalization and ReLU activation.
The encoding dimension for Mountain Hike is 64 and 128 for all the rest tasks.
The mean and variance of the normal distribution are learned again by two additional fully connected layers; for the variance, we use Softplus as the activation function.
Observation Function: f obs is implemented by a single fully connected layer without activation.
In DVRL, the observation function is parameterized over the full observation space o and p ( o | h
it − 1 , a it ) is assumed as a multivariate independent Bernoulli distribution whose parameters are again determined by a neural network (Igl et al., 2018).
For numerical stability, all the probabilities are stored and computed in the log space and the particle weights are always normalized after each weight update.
Soft-resampling:
The soft-resampling hyperparameter α is set to be 0.9 for Mountain Hike and 0.
for the rest of the domains.
Note that the soft-resampling is used only for DPFRL, not including DVRL.
DVRL averages the particle weights to 1 /K after each resampling step, which makes the resampling step cannot be trained by the RL.
Belief Summary:
The GRU used in DVRL and DPFRL-GRUmerge is a single layer GRU with input dimension equals the dimension of the latent vector plus 1, which is the corresponding particle weight.
The dimension of this GRU is exactly the dimension of the latent vector.
For the MGF features, we use fully connected layers with feature dimensions as the number of MGF features.
The activation function used is the exponential function.
We could potentially explore the other activation functions to test the generalized-MGF features, e.g., ReLU.
Model Learning: For RL, we use an A2C algorithm with 16 parallel environments for both Mountain
Hike and Atari games; for Habitat Navigation, we only use 6 parallel environments due to the GPU memory constraints.
The loss function for DPFRL and GRU-based policy is just the standard A2C loss,
L A2C t =
L
At + λ V L Vt + λ H L Ht , where L At is the policy loss, L Vt is the value loss, L Ht is the entropy loss for encouraging exploration, and λ V and λ H are two hyperparameters.
For all experiments, we use λ V = 0 .
5 and λ H = 0 .
For DVRL, an additional encoding loss L Et is used to train the sequential
VAE, which gives a loss function L DVRL t =
L A2C t + λ E L Et .
We follow the default setting provided by Igl et al.
and set λ E = 0 .
The rest of the hyperparameters, including learning rate, gradient clipping value and α in soft-resampling are tuned according to the BeamRider and directly applied to all domains due to the highly expensive experiment setups.
The learning rate for all the networks are searched among the following values: (3 × 10 − 5 , 5 × 10 − 5 , 1 × 10 − 4 , 2 × 10 − 4 , 3 × 10 − 4 ) ; the gradient clipping value are searched among { 0 . 5 , 1 . 0 } ; the soft-resampling α is searched among { 0 . 5 , 0 . 9 } .
The best performing learning rates were 1 − 4 for DPFRL and GRU, and 2 − 4 for DVRL; the gradient clipping value for all models was 0 .
5 ; the soft-resampling α is set to be 0.9 for Mountain
Hike and 0.5 for Atari games.
B. E XPERIMENT S ETUPS
Natural Flickering Atari games We follow the setting of the prior works (Zhu et al., 2018; Igl et al., 2018): 1) 50% of the frames are randomly dropped 2) a frameskip of 4 is used 3) there is a 0.
chance of repeating an action twice.
In our experiments, we sample background videos from the
ILSVRC dataset (Russakovsky et al., 2015).
Only the videos with the length longer than 500 frames are sampled to make sure the video length is long enough to introduce variability.
For each new episode, we ﬁrst sample a new video from the dataset, and a random starting pointer is sampled in this video.
Once the video ﬁnishes, the pointer is reset to the ﬁrst frame (not the starting pointer we sampled) and continues from there.
Experiment platform: We implement all the models using PyTorch (Paszke et al., 2017) with CUDA 9.2 and CuDNN 7.1.2.
Flickering Atari environments are modiﬁed based on OpenAI Gym (Brockman et al., 2016) and we directly use Habitat APIs for visual navigation.
For Mountain Hike and Atari games, we run our experiments on servers with 4 NVidia GTX1080Ti GPUs on each server.
For
Habitat visual navigation, we run the experiment on servers with 4 NVidia RTX2080Ti GPUs on each server.
C PF-GRU N ETWORK A RCHITECTURE
We implement DPFRL with gated transition and observation functions for particle ﬁltering similar to
PF-GRU (Ma et al., 2019).
In standard GRU, the memory update is implemented by a gated function:
[Equation]
(11) where W n and b n are the corresponding weights and biases, and z t r t are the learned gates.
PF-GRU introduces stochastic cell update by assuming the update to the memory, n it , follows a parameterized Gaussian distribution
[Equation]
With x t = [ f o enc ( o t ) , f a enc ( a t )] , we implement the transition function h
it +1 ∼ f trans ( h it , o t , a t ) , where f o enc is the encoding network for observation and f a enc is the encoding network for the actions.
For the observation function, we directly use a fully connected layer f obs ( h it , o t )
=
W o (cid:2) h it , o t (cid:3) + b o , where W o and b o are the corresponding weights and biases.
D DPFRL A LGORITHM
Algorithm 1: DPFRL
Input: Previous belief b t − 1 ≈ { ( h it − 1 , w it − 1 ) } Ki =1 , observation o t , action a t x ot ← Encoder( o t ) (encode the raw observation)
h it ∼ f trans ( h it − 1 , a t , u t ( x ot )) (transition update) w it ← ηf obs ( x ot , h it )
w it − 1 , η = 1 / K (cid:80) i = 1 w it (observation update) { ( h (cid:48) it , w (cid:48) it ) }
Ki =1 ← Soft-Resampling ( { ( h it , w it ) } Ki =1 ) (soft-resampling) ¯ h t ← K (cid:80) i =1 w (cid:48) it h (cid:48)
it (compute the mean) for j = 1 : m do M jt ← K (cid:80) i =1 w (cid:48) it exp( v j h (cid:48) it ) (compute MGF features) end p ( a | b t ) ← π ( ¯ h t , M 1: m t ) (compute the policy)
V ( b t ) ← V ( ¯ h t , M 1: m t ) (compute the value) Output: Updated belief b t ≈ { ( h (cid:48) it , w (cid:48) it ) } K i =1 , policy p ( a | b t ) and value V ( b t )
E A DDITIONAL R ESULTS E. F LICKEIRNG A TARI G AMES P LOTS
We provide the accumulated reward curves for Atari experiments in this section
. E.1. S TANDARD F LICKERING A TARI G AMES
For the standard Flickering Atari Games
[Figure]
no validation environment is provided.
We directly provide the training curves below.
Results for DVRL and GRU are directly taken from Igl et al.
[Figure]
[Figure] [Figure]
(h) Flickering IceHockey (i) Flickering DoubleDunk
[Figure]
R
[Figure]
t u r n
DPFRLDVRLGRU (j) Flickering Asteroids
E.1. N ATURAL F LICKERING A TARI G AMES
For Natural Flickering Atari Games, we have a separate validation set that contains videos different from the training set.
The validation environment steps once after every 100 training iterations
[Figure] [Figure]
(i) Natural
DoubleDunk
[Figure]
E. V ISUAL N AVIGATION
We present the reward curve for the Habitat visual navigation task below.
DPFRL outperforms both GRU-based policy and DVRL given the same training time.
DVRL struggles with training the observation model and fails during the ﬁrst half of the training time.
GRU based policy learns fast; given only the model-free belief tracker, it struggles to achieve higher reward after a certain point.
We only provide the reward curve here as SPL and success rate are only evaluated after the training is ﬁnished.
[Figure]
E. P ARTICLE V ISUALIZATION WITH PCA
We further visualize the latent particles by principal component analysis (PCA) and choose the ﬁrstcomponents.
We choose a trajectory in the Habitat Visual Navigation experiment, where 15 particles are used.
We observe that particles initially spread across the space ( t = 0 ).
As the robot only receive partial information in the visual navigation task, particles gradually form a distribution with two clusters ( t = 56 ), which represent two major hypotheses of its current state.
After more information is incorporated into the belief, they begin to converge and ﬁnally become a single cluster ( t = 81 ).
We did not observe particle depletion and posterior collapse in our experiment.
This could be better avoided by adding an entropy loss to the learned variance of f trans and we will leave it for future study.
{+2        2        2    (a) t =(b) t =(c) t =    2        2        
+}