D ISCRIMINATIVE P ARTICLE F ILTER R EINFORCEMENT L EARNING FOR C OMPLEX P ARTIAL O BSERVATIONS
A BSTRACT
Deep reinforcement learning is successful in decision making for sophisticated games, such as Atari, Go, etc.
However, real-world decision making often requires reasoning with partial information extracted from complex visual observations.
This paper presents Discriminative Particle Filter Reinforcement Learning (DPFRL), a new reinforcement learning framework for complex partial observations.
DPFRL encodes a differentiable particle ﬁlter in the neural network policy for explicit reasoning with partial observations over time.
The particle ﬁlter maintains a belief using learned discriminative update, which is trained end-to-end for decision making.
We show that using the discriminative update instead of standard generative models results in signiﬁcantly improved performance, especially for tasks with complex visual observations, because they circumvent the difﬁculty of modeling complex observations that are irrelevant to decision making.
In addition, to extract features from the particle belief, we propose a new type of belief feature based on the moment generating function.
DPFRL outperforms state-of-the-art POMDP
RL models in Flickering Atari Games, an existing POMDP RL benchmark, and in Natural Flickering Atari Games, a new, more challenging POMDP RL benchmark introduced in this paper.
Further, DPFRL performs well for visual navigation with real-world data in the Habitat environment.
The code is available online 1 .
I NTRODUCTION
Deep Reinforcement Learning (DRL) has attracted signiﬁcant interest, with applications ranging from game playing (Mnih et al., 2013; Silver et al., 2017) to robot control and visual navigation (Levine et al., 2016;
Kahn et al., 2018; Savva et al., 2019).
However, natural real-world environments remain challenging for current DRL methods (Arulkumaran et al., 2017), in part because they require (i) reasoning in a partially observable environment and (ii) reasoning with complex observations, such as visually rich images.
Consider, for example, a robot, navigating in an indoor environment, with a camera for visual perception. To determine its own location and a traversable path, the robot must extract from image pixels relevant geometric features, which often coexist with irrelevant visual features, such as wall textures, shadows, etc. Further, the task is partially observable: a single image at the current time does not provide sufﬁcient features for localization, and the robot must integrate information from the history of visual inputs received.
The partially observable Markov decision process (POMDP) provides a principled general framework for decision making under partial observability.
Solving POMDPs requires tracking a sufﬁcient statistic of the action-observation history, e.g., the posterior distribution of the states, called the belief .
Most POMDP reinforcement learning (RL) methods summarize the history into a vector using a recurrent neural network (RNN) (Hausknecht & Stone, 2015; Zhu et al., 2018).
RNNs are model-free generic function approximators. Without appropriate structural priors, they need large amounts of training data to learn to track a complex belief well.
Model-based DRL methods aim to reduce the sample complexity by learning a model together with a policy.
In particular, to deal with partial observability, Igl et al.
(2018) recently proposed DVRL, which learns a generative observation model embedded into the policy through a Bayes ﬁlter. Since [Table]
/yuMookN0hCrIRReojm5QAzURRQ/oCb2gV+vRerberPd5a8HKZ/bRL1gf3w/glp8=</latexit>
[Figure] the Bayes ﬁlter tracks the belief explicitly, DVRL performs much better than generic RNNs under partial observability.
However, a Bayes ﬁlter normally assumes a generative observation model, that deﬁnes the probability p ( o | h t ) of receiving an observation o =
o t given the latent state h t Fig. 1b).
Learning this model can be very challenging since it requires modeling all observation features, including features irrelevant for RL.
When o is an image, p ( o | h t ) is a distribution over all possible images.
This means, e.g., to navigate in a previously unseen environment, we need to learn the distribution of all possible environments with their visual appearance, lighting condition, etc. — a much harder task than learning to extract features relevant to navigation, e.g., the traversable space.
We introduce the Discriminative Particle Filter Reinforcement Learning (DPFRL), a POMDP RL method that learns to explicitly track a belief over the latent state without a generative observation model, and make decisions based on features of the belief (Fig. 1a).
DPFRL approximates the belief by a set of weighted learnable latent particles { ( h it , w it ) } Ki =1 , and it tracks this particle belief by a nonparametric Bayes ﬁlter algorithm, an importance weighted particle ﬁlter , encoded as a differentiable computational graph in the neural network architecture.
The importance weighted particle ﬁlter applies discriminative update to the belief with an observation-conditioned transition model and a discriminative state-observation compatibility function (serving as the importance weights), both of which are learnable neural networks trained end-to-end.
By using these update functions instead of the transition and observation models of the standard particle ﬁlter, DPFRL sidesteps the difﬁculty of learning a generative observation model (Fig. 1b).
The model is discriminative in the sense that the compatibility function, f obs ( o t , h t ) , as shown in Fig. 1c, while playing an analogue role as p ( o t | h t ) , is not required to directly represent a normalized distribution over observations; and through end-toend training it only needs to model observation features relevant for the RL task.
Finally, to summarize the particle belief for the policy, we introduce novel learnable features based on Moment-Generating Functions (MGFs) (Bulmer, 1979).
MGF features are computationally efﬁcient and permutation invariant, and they can be directly optimized to provide useful higher-order moment information for learning a policy.
MGF features could be also used as learned features of any empirical distribution in applications beyond RL.
We evaluate DPFRL on a range of POMDP RL domains: a continuous control task from Igl et al.
(2018), Flickering Atari Games (Hausknecht & Stone, 2015), Natural Flickering Atari Games, a new domain with more complex observations that we introduce, and the Habitat visual navigation domain using real-world data (Savva et al., 2019).
DPFRL outperforms state-of-the-art POMDP RL methods in most cases.
Results show that belief tracking with a particle ﬁlter is effective for handling partial observability, and the discriminative update and MGF-based belief features allow for complex observations.
R ELATED W ORK
Real-world decision-making problems are often formulated as POMDPs.
POMDPs are notoriously hard to solve; in the worst case, they are computationally intractable (Papadimitriou & Tsitsiklis, 1987).
Approximate POMDP solvers have made dramatic progress in solving large-scale POMDPs (Kurniawati et al., 2008).
Particle ﬁlters have been widely adopted as a belief tracker for POMDP solvers (Silver & Veness, 2010; Somani et al., 2013) having the ﬂexibility to model complex and multi-modal distributions, unlike Gaussian and Kalman ﬁlters.
However, predeﬁned model and state representations are required for these methods (see e.g. Bai et al. [Figure]
Given the advances in generative neural network models, various neural models have been proposed for belief tracking (Chung et al., 2015; Maddison et al., 2017; Le et al., 2018; Naesseth et al., 2018).
DVRL
(Igl et al., 2018) uses a Variational Sequential Monte-Carlo method (Naesseth et al., 2018), similar to the particle ﬁlter we use, for belief tracking in RL.
This gives better belief tracking capabilities, but as we demonstrate in our experiments, generative modeling is not robust in complex observation spaces with high-dimensional irrelevant observation.
More powerful generative models, e.g., DRAW (Gregor et al., 2015), could be considered to improve generative observation modeling; however, evaluating a complex generative model for each particle would signiﬁcantly increase the computational cost and optimization difﬁculty.
Learning a robust latent representation and avoiding reconstructing observations are of great interest for RL (Oord et al., 2018;
Guo et al., 2018; Hung et al., 2018; Gregor et al., 2019; Gelada et al., 2019).
Discriminative RNNs have also been widely used for belief approximation in partially observable domains (Bakker, 2002;
Wierstra et al., 2007;
Foerster et al., 2016).
The latent representation is directly optimized for the policy p ( a | h t ) that skips observation modeling.
For example, Hausknecht & Stone (2015) and Zhu et al.
(2018) tackle partially observable Flickering Atari Games by extending DQN (Mnih et al., 2013) with an LSTM memory.
Our experiments demonstrate that the additional structure for belief tracking provided by a particle ﬁlter can give improved performance in RL.
Embedding algorithms into neural networks to allow end-to-end discriminative training has gained attention recently.
For belief tracking, the idea has been used in the differentiable histogram ﬁlter (Jonschkowski & Brock, 2016), Kalman ﬁlter (Haarnoja et al., 2016) and particle ﬁlter (Karkus et al., 2018; Jonschkowski et al., 2018).
Further, Karkus et al.
(2017) combined a learnable histogram ﬁlter with the Value Iteration Network (Tamar et al., 2016) and introduced a learnable POMDP planner, QMDP-net.
However, these methods require a predeﬁned state representation and are limited to relatively small state spaces.
Ma et al.
(2019) integrated the particle ﬁlter with standard RNNs, e.g., the LSTM, and introduced PF-RNNs for sequence prediction.
We build on the work of Ma et al.
(2019) and demonstrate its advantages for RL with complex partial observations, and extend it with MGF features for improved decision making from particle beliefs.
Note that our framework is not speciﬁc to PF-RNNs, and could be applied to other differentiable particle ﬁlters as well.
D ISCRIMINATIVE P ARTICLE F ILTER R EINFORCEMENT L EARNING
We introduce DPFRL for reinforcement learning under partial and complex observations.
The DPFRL architecture is shown in Fig.
It has two main components, a discriminatively trained particle ﬁlter that tracks a latent belief b t , and an actor network that learns a policy p ( a | b t ) given the belief b t .
3. P ARTICLE F ILTER FOR L ATENT B ELIEF T RACKING
Latent State Representation.
In POMDPs the semantics of states s is typically deﬁned explicitly. State variables may correspond to the position of a robot, conﬁguration of obstacles, etc. In DPFRL, we do not require explicit speciﬁcation of the state variables, but implicitly represent the state as a vector h of latent variables, that is, the semantics of the state variables are learned instead of being pre-speciﬁed. We use a fully differentiable particle ﬁlter algorithm to maintain a belief over h .
More speciﬁcally, we approximate the belief with a set of weighted latent particles b t ≈ { ( h it , w it ) }
Ki =1 , where { h it } K i =1 are K latent states learned by policy-oriented training, and { w it } K i =1 represents the
Belief Update.
In a basic particle ﬁlter, there are two key steps to update a particle belief { h it − 1 , w it − 1 } Ki =1 to a new particle belief { h it , w it } Ki =1 upon receiving an observation o t after ex- ecuting action a t .
[Equation]
The ﬁrst step, Eq. 1, takes the transition dynamics into account to update each particle. The second step, Eq. 2, takes the observation into account to reweigh the new particles. Our belief update has a similar structure as the standard particle ﬁlter, but we replace the transition model and the observation model with richer functions to make the update more suitable for learning a policy in a partially observable domain.
Speciﬁcally, the update equations are as follows. [Equation] Below, we ﬁrst explain the intuition behind the above updates and the roles of f trans and f obs as compared to the standard transition and observation models. We then derive that above rules from an importance weighed particle ﬁlter in Sect. 3.2.
Observation-conditioned transition update.
Eq. 3 takes a form more general than that in Eq. 2: instead of using the transition dynamics p ( h
| h
it − 1 , a t ) to evolve a particle, we use a more general observation-conditioned transition f trans ( h | h it − 1 , a t , o t ) .
Incorporating the observation allows alleviating the problem of sampling unlikely particles.
In fact, if we take f trans to be p ( h | h it − 1 , a t , o t ) , then this allows us to skip Eq. 2, and completely avoids sampling particles that are likely considering a t only, but unlikely considering both a t and o t . Of course, in RL we do not have access to p ( h | h
it − 1 , a t , o t ) , and instead f trans is learned. In our implementation, a network ﬁrst extracts features from o t , they are fed to a gated function following the PF-GRU of Ma et al. (2019), which outputs the mean and variance of a normal distribution.
Details are in the Appendix.
Importance weighting via a compatibility function.
Eq.
4 is a relaxed version of Eq. 2: instead of using the observation model p ( o t | h it ) to adjust the particle weights based on their compatibility with the observation, we use a general non-negative compatibility function f obs ( h it , o t ) .
If the compatibility function is required to satisfy the normalization constraint that (cid:80) o f obs ( h, o ) is a constant for all h , then it is equivalent to a conditional distribution of o given h .
We do not require this, and thus the update loses the probabilistic interpretation in Eq. 2.
However, eliminating the need for the normalization constraint allows the compatibility function to be efﬁciently trained, as we can avoid computing the normalization constant. In addition, since the observation has already been incorporated in Eq.
3, we actually expect that the weights need to be adjusted in a way different from the standard particle ﬁlter.
In our implementation, f obs ( h it , o t ) is a neural network with a single fully connected layer that takes in a concatenation of h it and features extracted from o t . The output of the network is interpreted as the log of f obs ; and for numerical stability we perform the weight updates of Eq.
4 in the log-space as well. Note that more complex network architectures could improve the capability of f obs , which we leave to future work.
Soft-resampling.
To avoid particle degeneracy, i.e., most of the particles having a near-zero weight, particle ﬁlters typically resample particles. We adopt the soft-resampling strategy of Karkus et al. ; Ma et al. (2019), that provides approximate gradients for the non-differentiable resampling step.
Instead of sampling from p t ( i )
=
w it , we sample particles { h (cid:48) it } Ki =1 from a softened proposal distribution q ( i ) =
αw it + (1 − α )1 /K , where α is an trade-off parameter.
The new weights are derived using importance sampling: w (cid:48) i t =
w it αw it +(1 − α )1 /K .
We can have the ﬁnal particle belief as { ( h (cid:48) it , w (cid:48) it ) }
Ki =1 =
Soft-Resampling ( { ( h it , w it ) }
Ki =1 ) .
As a result, f obs can be optimized with global belief information and model shared useful features across multiple time steps.
Another related concern is that the particle distribution may collapse to particles with the same latent state.
This can be avoided by ensuring that the stochastic transition function f trans has a non-zero variance, e.g., by adding a small constant to the learned variance.
End-to-end training.
In DPFRL the observation-conditioned transition function f trans and the compatibility function f obs are learned.
Instead of training for a modeling objective, they are trained end-to-end for the ﬁnal RL objective, backpropagating gradients through the belief-conditional policy p ( a | b t ) and the update steps of the particle ﬁlter algorithm, Eq. 3-5.
3. C ONNECTION TO I MPORTANCE W EIGHTED P ARTICLE F ILTER
Our belief update can be motivated from the following importance weighted particle ﬁlter.
Learning directly p ( h (cid:48) | h, a, o ) is generally difﬁcult, but if we have a distribution q ( h (cid:48) | h, a, o ) that is easy to learn, then we can use importance sampling to update a particle belief.
[Equation]
where f = p/q is the importance weight.
Consider the case that q ( h (cid:48) | h, a, o ) is the conditional distribution of a joint distribution q ( h (cid:48) , h, a, o ) of the form p ( h (cid:48) | h, a ) q ( o | h (cid:48) ) .
That is, p and q share the same transition dynamics p ( h (cid:48)
| h, a ) .
Then the importance weight f is a function of h (cid:48) and o only, because
[Equation]
This simpler form is exactly the form that we used for f obs in our belief update.
3. D ISCRIMINATIVE VS . G ENERATIVE M ODELING
We expect the discriminative compatibility function to be more effective than a generative model for the following reasons.
A generative model aims to approximate p ( o | h ) by learning a function that takes h as input and outputs a parameterized distribution over o .
When o is, e.g., an image, this requires approximations, e.g., using pixel-wise Gaussians with learned mean and variance.
This model is also agnostic to the RL task and considers all observation features equally, including features irrelevant for ﬁltering and decision making.
In contrast, f obs takes o and h as inputs, and estimates the compatibility of o and h for particle ﬁltering directly.
This function avoids forming a parametric distribution over o , and the function can be easier to learn.
The same functional form is used for the energy-function of energy-based models (LeCun et al., 2006) and in contrastive predictive coding (Oord et al., 2018), with similar beneﬁts.
For example, f obs may learn unnormalized likelihoods that are only proportionate to p ( o | h ) up to a o -dependent value, because after the normalization in Eq. 4, they would give the same belief update as the normalized p ( o | h ) .
Further, because f obs is trained for the ﬁnal RL objective instead of a modeling objective, it may learn a compatibility function that is useful for decision making, but that does not model all observation features and has no proper probabilistic interpretation.
While the task-oriented training of discriminative models may improve policy performance for the reasons above, it cannot take advantage of an auxiliary learning signal like the reconstruction objective of a generative model.
An interesting line of future work may combine generative models with a compatibility function to simultaneously beneﬁt from both formulations.
3. B ELIEF -C ONDITIONAL A CTOR N ETWORK
Conditioning a policy directly onto a particle belief is non-trivial.
To feed it to the networks, we need to summarize it into a single vector.
We introduce a novel feature extraction method for empirical distributions based on MomentGenerating Functions (MGFs).
The MGF of an n -dimensional random variable X is given by
M X ( v ) =
E [ e v (cid:62) X ] , v ∈ R n .
In statistics, MGF is an alternative speciﬁcation of its probability distribution (Bulmer, 1979).
Since particle belief b t is an empirical distribution, the moment generating function of b t can be denoted as M b t ( v ) =
(cid:80) {+Ki =
+}
w it e v (cid:62) h it .
A more detailed background on MGFs is in Appendix A.2.
In DPFRL, we use the values of the MGF at m learned locations v 1: m as the feature vector of the
MGF. The j -th MGF feature is given by M jb t ( v j ) .
For a clean notation, we use M jt in place of M jb t ( v j ) .
We use (cid:2) ¯ h t , M 1: m t (cid:3) as features for belief b t , where ¯ h t = (cid:80) {+Ki =
+}
w it h
it is the mean particle.
The mean particle ¯ h t , as the ﬁrst-order moment, and m additional MGF features, give a summary of the belief characteristics.
The number of MGF features, m , controls how much additional information we extract from the belief.
We empirically study the inﬂuence of MGF features in ablation studies.
Compared to Ma et al.
(2019) that uses the mean as the belief estimate, MGF features provide additional features from the empirical distribution.
Compared to DVRL (Igl et al., 2018) that treats the Monte-Carlo samples as a sequence and merges them by an RNN, MGF features are permutationinvariant, computationally efﬁcient and easy to optimize, especially when the particle set is large.
Given the features (cid:2) ¯ h t , M 1: m t (cid:3) for b t , we compute the policy p ( a | b t ) with a policy network π ( b t ) .
We trained with an actor-critic RL algorithm, A2C
(Mnih et al., 2016), where a value network V ( b t ) is introduced to assist learning.
We use small fully-connected networks for π ( b t ) and V ( b t ) that share the same input b t .
E XPERIMENTS
We evaluate DPFRL in a range of POMDP RL domains with increasing belief tracking and observation modeling complexity.
We ﬁrst use benchmark domains from the literature, Mountain Hike, anddifferent Flickering Atari Games.
We then introduce a new, more challenging domain, Natural
Flickering Atari Games, that uses a random video stream as the background.
Finally we apply DPFRL to a challenging visual navigation domain with RGB-D observations rendered from real-world data.
We compare DPFRL with a GRU network, a state-of-the-art POMDP RL method, DVRL, and ablations of the DPFRL architecture.
As a brief conclusion, we show that: 1) DPFRL signiﬁcantly outperforms GRU in most cases because of its explicit structure for belief tracking; 2) DPFRL outperforms the state-of-the-art DVRL in most cases even with simple observations, and its beneﬁt increases dramatically with more complex observations because of DPFRL’s discriminative update; 3) MGF features are more effective for summarizing the latent particle belief than alternatives.
4. E XPERIMENTAL S ETUP
[Figure]
We train DPFRL and baselines with the same A2C algorithm, and use a similar network architecture and hyperparameters as the original DVRL implementation.
DPFRL and DVRL differ in the particle belief update structure, but they use the same latent particle size dim ( h ) and the same number of particles K as in the DVRL paper (dim ( h ) = 128 and K = 30 for Mountain Hike, dim ( h ) = 256 and
K = 15 for Atari games and visual navigation).
The effect of the number of particles is discussed in Sect.
We train all models for the same number of iterations using the RMSProp optimizer (Tieleman & Hinton, 2012).
Learning rates and gradient clipping values are chosen based on a search in the BeamRider Atari game independently for each model.
Further details are in the Appendix.
We have not performed additional searches for the network architecture and other hyper-parameters, nor tried other RL algorithm, such as PPO (Schulman et al., 2017), which may all improve our results. All reported results are averages over 3 different random seeds.
We plot rewards accumulated in an episode, same as DVRL
(Igl et al., 2018).
The curves are smoothed over time and averaged over parallel environment executions.
4. M OUNTAIN H IKE
Mountain Hike was introduced by Igl et al.
(2018) to demonstrate the beneﬁt of belief tracking for POMDP RL.
It is a continuous control problem where an agent navigates on a ﬁxed 20 × 20 map.
[Figure]
R e t u r n a t
I t e r a t i o n 5 e
DPFRLDVRLGRU
[Figure]
R e t u r n
DPFRLDVRLGRU [Figure] 1e
[Figure]
R e t u r n
DPFRLDVRLGRU
[Figure]
R e t u r n
DPFRLDVRLGRU
Effect of varying l
[Figure] [Figure]
In the original task, partial observability is introduced by disturbing the agent observation with an additive Gaussian noise. To illustrate the effect of observation complexity in natural environments, we concatenate the original observation vector with a random noise vector. The complexity of the optimal policy remains unchanged, but the relevant information is now coupled with irrelevant observation features. More speciﬁcally, the state space and action space in Mountain Hike are deﬁned as S =
A = R 2 , where s t =
[ x t , y t ] and a t = [ δx t , δy t ] .
Transitions of the agent are stochastic with an additive Gaussian noise: s t +
=
s t + a t + (cid:15) a , where (cid:15) a ∼ N (0 , 0 . 25) .
The observation space is
O = R 2+ l , where l is a predeﬁned constant and l = 0 corresponds to the original setting.
Observations are o t =
[ o st , o nt ] , where o st = s t + (cid:15) s , (cid:15) s ∼ N (0 , 1) , and o nt ∈ R l is sampled from a uniform distribution U ( − 10 , 10) .
The reward for each step is given by r t = r ( x t , y t ) − 0 .
|| a t || where r ( x t , y t ) is shown in Fig.
Episodes end after 75 steps.
We train models for different settings of the noise vector length l , from l = 0 to l = 100 .
Results are shown in Fig.
We observe that
DPFRL learns faster than the DVRL and GRU in all cases, including the original setting l = 0 .
Importantly, as the noise vector length increases, the performance of DVRL and GRU degrades, while
DPFRL is unaffected.
This demonstrates the ability of DPFRL to track a latent belief without having to explicitly model complex observations.
4. A TARI G AMES WITH P ARTIAL O BSERVABILITY
Atari games are one of the most popular benchmark domains for RL methods (Mnih et al., 2013).
Their partially observable variants, Flickering Atari Games, have been used to benchmark POMDP
RL methods (Hausknecht & Stone, 2015; Zhu et al., 2018; Igl et al., 2018).
Here image observations are single frames randomly replaced by a blank frame with a probability of 0 . 5 .
The ﬂickering observations introduce a simple form of partial observability.
Another variant, Natural Atari Games (Zhang et al., 2018), replaces the simple black background of the frames of an Atari game with a randomly sampled video stream.
This modiﬁcation brings the Atari domain one step closer to the visually rich real-world, in that the relevant information is now encoded in complex observations.
As shown by
Zhang et al.
(2018), this poses a signiﬁcant challenge for RL.
We propose a new RL domain, Natural Flickering Atari Games , that involves both challenges: partial observability simulated by ﬂickering frames, and complex observations simulated by random background videos.
The background videos increase observation complexity without affecting the decision making complexity, making this a suitable domain for evaluating RL methods with complex observations.
We sample the background video from the ILSVRC dataset (Russakovsky et al., 2015).
Examples for the BeamRider game are shown in Fig.
Details are in Appendix B.
[Table] We evaluate DPFRL for both Flickering Atari Games and Natural Flickering Atari Games.
We use the same set of games as Igl et al.
To ensure a fair comparison, we take the GRU and DVRL results from the paper for Flickering Atari Games, use the same training iterations as in Igl et al.
and we use the ofﬁcial DVRL open source code to train for Natural Flickering Atari Games.
Results are summarized in Table 1.
We highlight the best performance in bold where the difference is statistically signiﬁcant ( p = 0 . 05 ).
Detailed training curves are in Appendix E.
We observe that DPFRL signiﬁcantly outperforms GRU in almost all games, which indicates the importance of explicit belief tracking, and shows that DPFRL can learn a useful latent belief representation.
Despite the simpler observations, DPFRL signiﬁcantly outperforms DVRL and achieves state-of-the-art results on 5 out of 10 standard Flickering Atari Games (ChopperCommand, MsPacman, BeamRider, Bowling, Asteroids), and it performs comparably in 3 other games (Centipede,
Frostbite, IceHockey).
The strength of DFPRL shows even more clearly in the Natural Flickering
Atari Games, where it signiﬁcantly outperforms DVRL on 7 out of 10 games and performs similarly in the rest.
In some games, e.g. in Pong, DPFRL performs similarly with and without videos in the background (15.65 vs. 15.40), while the DVRL performance degrades substantially (-19.78 vs. 18.17).
These results show that while the architecture of DPFRL and DVRL are similar, the policy-oriented discriminative update of DPFRL is much more effective for handling complex observations, and the MGF features provide a more powerful summary of the particle belief for decision making.
However, on some games, e.g. on ChopperCommand, even DPFRL performance drops signiﬁcantly when adding background videos.
This shows that irrelevant features can make a task much harder, even for a discriminative approach, as also observed by Zhang et al.
4. V ISUAL N AVIGATION
[Table]
Visual navigation poses a great challenge for deep RL (Mirowski et al., 2016; Zhu et al., 2017; Lample & Chaplot, 2017).
We evaluate DPFRL for visual navigation in the Habitat Environment (Savva et al., 2019), using the real-world Gibson dataset (Xia et al., 2018).
In this domain, a robot needs to navigate to goals in previously unseen environments.
In each time step, it receives a ﬁrst-person
RGB-D camera image and its distance and relative orientation to the goal.
The main challenge lies in the partial and complex observations: ﬁrst-person view images only provide partial information about the unknown environment; and the relevant information for navigation, traversability, is encoded in rich RGB-D observations along with many irrelevant features, e.g., the texture of the wall.
We use the Gibson dataset with the training and validation split provided by the Habitat challenge.
We train models with the same architecture as for the Atari games, except for the observation encoder that accounts for the different observation format.
We evaluate models in unseen environments from [Table] the validation split and compute the same metrics as in the literature: SPL, success rate, and average rewards.
Results are shown in Table 2.
Further details and results are in Appendix B and E.
DPFRL signiﬁcantly outperforms both DVRL and GRU in this challenging domain.
DVRL performs especially poorly, demonstrating the difﬁculty of learning a generative observation model in realistic, visually rich domains.
DPFRL also outperforms the PPO baseline from Savva et al.
We note that submissions to the recently organized Habitat Challenge 2019 (Savva et al., 2019), such as (Chaplot et al., 2019), have demonstrated better performance than the PPO baseline (while our results are not directly comparable because of the closed test set of the competition).
However, these approaches rely on highly specialized structures, such as 2D mapping and 2D path planning, while we use the same generic network as for Atari games.
Future work may further improve our results by adding a task-speciﬁc structure to DPFRL or training with PPO instead of A2C.
4. A BLATION S TUDY
We conduct an extensive ablation study on the Natural Flickering Atari Games to understand the inﬂuence of each DPFRL component.
The results are presented in Table 3.
The discriminative compatibility function is more effective than a generative observation function.
DPFRL-generative replaces the discriminative compatibility function of DPFRL with a generative observation function, where grayscale image observations are modeled by pixel-wise Gaussian distributions with learned mean and variance.
Unlike DVRL, DPFRL-generative only differs from DPFRL in the parameterization of the observation function, the rest of the architecture and training loss remains the same.
In most cases, the performance for DPFRL-generative degrades signiﬁcantly compared to DPFRL.
These results are aligned with our earlier observations, and indicate that the compatibility function is capable of extracting the relevant information from complex observations without having to learn a more complex generative model.
More particles perform better.
DPFRL with 1 particle performs poorly on most of the tasks (DPFRLLP1).
This indicates that a single latent state is insufﬁcient to represent a complex latent distribution that is required for the task, and that more particles may improve performance.
MGF features are useful.
We compare DPFRL using MGF features with DPFRL-mean, that only uses the mean particle; and with DPFRL-GRUmerge, that uses a separate RNN to summarize the belief, similar to DVRL.
Results show that DPFRL-mean does not work as well as the standard DPFRL, especially for tasks that may need complex belief tracking, e.g., Pong.
This can be attributed to the more rich belief statistics provided by MGF features, and that they do not constrain the learned belief representation to be always meaningful when averaged.
Comparing to DPFRL-GRUmerge shows that MGF features generally perform better.
While an RNN may learn to extract useful features from the latent belief, optimizing the RNN parameters is harder, because they are not permutation invariant to the set of particles and they result in a long backpropagation chain.
C ONCLUSION
We have introduced DPFRL, a framework for POMDP RL in natural environments.
DPFRL combines the strength of Bayesian ﬁltering and end-to-end RL: it performs explicit belief tracking with learnable particle ﬁlters optimized directly for the RL policy.
DPFRL achieved state-of-the-art results on POMDP RL benchmarks from prior work, Mountain Hike and a number of Flickering Atari Games. Further, it signiﬁcantly outperformed alternative methods in a new, more challenging domain, Natural Flickering Atari Games, as well as for visual navigation using real-world data.
We have proposed a novel MGF feature for extracting statistics from an empirical distribution.
MGF feature extraction could be applied beyond RL, e.g., for general sequence prediction.
DPFRL does not perform well in some particular cases, e.g., DoubleDunk.
While our task-oriented discriminative update are less susceptible to complex and noisy observations than a generative model, they do not beneﬁt from an additional learning signal that could improve sample efﬁciency, e g., through a reconstruction loss.
Future work may combine a generative observation model with the discriminative update in the DPFRL framework.
A CKNOWLEDGEMENT
This research is partially supported by ONR Global and AFRL grant N62909-18-1-2023.
We want to thank Maximilian Igl for suggesting to add videos to the background of Atari games
A B ACKGROUND A. P ARTICLE F ILTER A LGORITHM
Particle ﬁlter is an approximate Bayes ﬁlter algorithm for belief tracking.
Bayes ﬁlters estimate the belief b t , i.e., a posterior distribution of the state s t , given the history of actions a 1: t and observations o 1: t .
Instead of explicitly modeling the posterior distribution, particle ﬁlter approximates the posterior with a set of weighted particles, b t ≈ { ( s it , w it ) }
Ki =1 , and update the particles in a Bayesian manner.
Importantly, the particle set could approximate arbitrary distributions, e.g., Gaussians, continuous multi-modal distributions, etc.
The mean state can be estimated as the mean particle ¯ s t = (cid:80) Ki
{+=
+} w it s it .
The particle updates include three steps: transition update, measurement update, and resampling.
Transition update.
We ﬁrst update the particles by a given motion model.
More speciﬁcally, we sample the next state s it +1 from a generative transition function
[Equation]
where p ( s | s it , a t ) is the transition function.
Measurement update.
The particle weights are updated again using the observation likelihoods
[Equation]
where η is a normalization factor and p ( o t | s it +1 )
is the observation likelihood computed by evaluating observation o t in a generative observation function p ( o | s it +1 ) .
Resampling.
The particle ﬁlter algorithm can suffer from particle degeneracy, where after some update steps only a few particles have non-zero weights.
This would prevent particle ﬁlter to approximate the posterior distribution effectively.
Particle degeneracy is typically addressed by performing resampling, where new particles are sampled with repetition proportional to its weight.
Speciﬁcally, we sample particles from a categorical distribution p parameterized by the particle weights { w it } Ki =
p
[Equation]
10) where p ( i ) is the probability for the i -th category, i.e., the i -th particle.
The new particles approximate the same distribution, but they assign more representation capacity to the relevant regions of the state space.
A. M OMENT -G ENERATING F UNCTIONS
In probability theory, the moment-generating function (MGF) is an alternative speciﬁcation of the probability distribution of a real-valued random variable (Bulmer, 1979).
As its name suggests, MGF of a random variable could be used to generate any order of its moments, which characterize its probability distribution.
Mathematically, the MGF of a random variable X with dimension m is deﬁned by
[Equation]
(11) where v ∈ R m
and we could consider the MGF of random variable X is the expectation of the random variable e v (cid:62) X .
Consider the series expansion of e v (cid:62) X
[Equation]
This leads to the well-known fact that the j -th order moment M j ( j -way tensor) is the j -th order derivative of the MGF at v = 0 .
[Equation]
In DPFRL, we use MGFs as additional features to provide moment information of the particle distribution.
DPFRL learns to extract useful moment features for decision making by directly optimizing for policy p ( a | b t ) .
B E XPERIMENT D ETAILS B. I MPLEMENTATION D ETAILS
Observation Encoders: For the observation encoders, we used the same structure with DVRL (Igl et al., 2018) for a fair comparison.
For Mountain Hike, we use two fully connected layers with batch normalization and ReLU activation as the encoder.
The dimension for both layers is 64.
For the rest of the domains, we ﬁrst down-sample the image size to 84 × 84, then we process images with2D-convolution layers with channel number (32, 64, 32), kernel sizes (8, 4, 3) and stride (4, 2, 1), without padding.
The compass and goal information are a vector of length 2; they are appended after the image encoding as the input.
Observation Decoders: Both DVRL and PFGRU-generative need observation decoders.
For the
Mountain Hike, we use the same structure as the encoder with a reversed order.
The transposed 2D-convolutional network of the decoder has a reversed structure.
The decoder is processed by an additional fully connected layer which outputs the required dimension (1568 for Atari and Habitat
Navigation, both of which have 84 × 84 observations).
Observation-conditioned transition network: We directly use the transition function in PFGRU (Ma et al., 2019) for f trans ( h it − 1 , a t , o t ) , which is a stochastic function with GRU gated structure.
Action a t is ﬁrst encoded by a fully connected layer with batch normalization and ReLU activation.
The encoding dimension for Mountain Hike is 64 and 128 for all the rest tasks.
The mean and variance of the normal distribution are learned again by two additional fully connected layers; for the variance, we use Softplus as the activation function.
State-observation compatibility network: f obs is implemented by a single fully connected layer without activation.
In DVRL, the observation function is parameterized over the full observation space o and p ( o | h
it − 1 , a it ) is assumed as a multivariate independent Bernoulli distribution whose parameters are again determined by a neural network (Igl et al., 2018).
For numerical stability, all the probabilities are stored and computed in the log space and the particle weights are always normalized after each weight update.
Soft-resampling:
The soft-resampling hyperparameter α is set to be 0.9 for Mountain Hike and 0.
for the rest of the domains.
Note that the soft-resampling is used only for DPFRL, not including
DVRL.
DVRL averages the particle weights to 1 /K after each resampling step, which makes the resampling step cannot be trained by the RL.
Belief Summary:
The GRU used in DVRL and DPFRL-GRUmerge is a single layer GRU with input dimension equals the dimension of the latent vector plus 1, which is the corresponding particle weight.
The dimension of this GRU is exactly the dimension of the latent vector.
For the MGF features, we use fully connected layers with feature dimensions as the number of MGF features.
The activation function used is the exponential function.
We could potentially explore the other activation functions to test the generalized-MGF features, e.g., ReLU.
Actor Network and Policy Network: The actor network and policy network are two fully connected layers, which take in the belief summary b t = (cid:2)
¯ h t , M 1: m t (cid:3) as input.
The output dimension of these two networks are chosen according to the RL tasks.
Model Learning: For RL, we use an A2C algorithm with 16 parallel environments for both Mountain
Hike and Atari games; for Habitat Navigation, we only use 6 parallel environments due to the GPU memory constraints.
The loss function for DPFRL and GRU-based policy is just the standard A2C loss,
L A2C t =
L
At + λ V L Vt + λ H L Ht , where L At is the policy loss, L Vt is the value loss, L Ht is the entropy loss for encouraging exploration, and λ V and λ H are two hyperparameters.
For all experiments, we use λ V = 0 .
5 and λ H = 0 .
For DVRL, an additional encoding loss L Et is used to train the sequential
VAE, which gives a loss function L DVRL t =
L A2C t + λ E L Et .
We follow the default setting provided by Igl et al.
and set λ E = 0 .
The rest of the hyperparameters, including learning rate, gradient clipping value and α in soft-resampling are tuned according to the BeamRider and directly applied to all domains due to the highly expensive experiment setups.
The learning rate for all the networks are searched among the following values: (3 × 10 − 5 , 5 × 10 − 5 , 1 × 10 − 4 , 2 × 10 − 4 , 3 × 10 − 4 ) ; the gradient clipping value are searched among { 0 . 5 , 1 . 0 } ; the soft-resampling α is searched among { 0 . 5 , 0 . 9 } .
The best performing learning rates were 1 − 4 for DPFRL and GRU, and 2 − 4 for DVRL; the clipping value models was
5 ; the resampling is to be Mountain
Hike and 0.5 for Atari games.
B. E XPERIMENTAL S ETUP
Natural Flickering Atari games We follow the setting of the prior works (Zhu et al., 2018; Igl et al., 2018): 1) 50% of the frames are randomly dropped 2) a frameskip of 4 is used 3) there is a 0.25 chance of repeating an action twice.
In our experiments, we sample background videos from the ILSVRC dataset (Russakovsky et al., 2015).
Only the videos with the length longer thanframes are sampled to make sure the video length is long enough to introduce variability.
For each new episode, we ﬁrst sample a new video from the dataset, and a random starting pointer is sampled in this video.
Once the video ﬁnishes, the pointer is reset to the ﬁrst frame (not the starting pointer we sampled) and continues from there.
Experiment platform: We implement all the models using PyTorch (Paszke et al., 2017) with CUDA 9.2 and CuDNN 7.1.2.
Flickering Atari environments are modiﬁed based on OpenAI Gym (Brockman et al., 2016) and we directly use Habitat APIs for visual navigation.
Collecting experience and performing gradient updates are done on a single computation node with access to one GPU. For Mountain Hike and Atari games we use NVidia GTX1080Ti GPUs.
For Habitat visual navigation we use NVidia RTX2080Ti GPUs.
C PF-GRU N ETWORK A RCHITECTURE
We implement DPFRL with gated transition and observation functions for particle ﬁltering similar to
PF-GRU (Ma et al., 2019).
In standard GRU, the memory update is implemented by a gated function:
[Equation]
where W n and b n are the corresponding weights and biases, and z t r t are the learned gates.
PF-GRU introduces stochastic cell update by assuming the update to the memory, n it , follows a parameterized Gaussian distribution
[Equation]
With x t = [ f o enc ( o t ) , f a enc ( a t )] , we implement the transition function h
it +1 ∼ f trans ( h it , o t , a t ) , where f o enc is the encoding network for observation and f a enc is the encoding network for the actions.
For the observation function, we directly use a fully connected layer f obs ( h it , o t )
=
W o (cid:2) h it , o t (cid:3) + b o , where W o and b o are the corresponding weights and biases.
D DPFRL A LGORITHM
Algorithm 1: DPFRL
Input: Previous belief b t − 1 ≈ { ( h it − 1 , w it − 1 ) }
Ki =1 , observation o t , action a t x ot ← Encoder( o t ) (encode the raw observation)
h it ∼ f trans ( h it − 1 , a t , u t ( x ot ))
(transition update) w it ← ηf obs ( x ot , h it )
w it − 1 , η = 1 / K (cid:80) i =
1 w it
(observation update) { ( h (cid:48)
it , w (cid:48) it ) }
Ki =1 ← Soft-Resampling ( { ( h it , w it ) }
Ki =1 ) (soft-resampling) ¯ h t ←
K (cid:80) i =1 w (cid:48) it h (cid:48)
it (compute the mean) for j = 1 : m do
M jt ←
K (cid:80) i =1 w (cid:48) it exp( v j h (cid:48) it ) (compute MGF features) end p ( a | b t ) ← π ( ¯ h t , M 1: m t ) (compute the policy)
V ( b t ) ← V ( ¯ h t , M 1: m t ) (compute the value)
Output: Updated belief b t ≈ { ( h (cid:48)
it , w (cid:48) it ) } K i =1 , policy p ( a | b t ) and value V ( b t )
E A DDITIONAL R ESULTS E. F LICKEIRNG A TARI G AMES P LOTS
We provide the accumulated reward curves for Atari experiments in this section.
Standard Flickering Atari Games.
For the standard Flickering Atari Games we provide the training curves below.
Results for DVRL and GRU are directly taken from Igl et al.
[Figure]
Flickering
[Figure]
(h) Flickering IceHockey (i)
Flickering DoubleDunk
[Figure]
R
[Figure]
t u r n
DPFRLDVRLGRU (j) Flickering Asteroids
Natural Flickering Atari Games. For Natural Flickering Atari Games we report results for a separate validation set, where the background videos are different from the training set.
The validation environment steps once after every 100 training iterations
[Figure] [Figure]
i [Figure]
DoubleDunk
[Figure]
E. V ISUAL N AVIGATION
We present the reward curve for the Habitat visual navigation task below.
DPFRL outperforms both GRU-based policy and DVRL given the same training time.
DVRL struggles with training the observation model and fails during the ﬁrst half of the training time.
GRU based policy learns fast; given only the model-free belief tracker, it struggles to achieve higher reward after a certain point.
We only provide the reward curve here as SPL and success rate are only evaluated after the training is ﬁnished.
[Figure]
E. P ARTICLE V ISUALIZATION WITH PCA
We further visualize the latent particles by principal component analysis (PCA) and choose the ﬁrstcomponents.
We choose a trajectory in the Habitat Visual Navigation experiment, where 15 particles are used.
We observe that particles initially spread across the space ( t = 0 ).
As the robot only receive partial information in the visual navigation task, particles gradually form a distribution with two clusters ( t = 56 ), which represent two major hypotheses of its current state.
After more information is incorporated into the belief, they begin to converge and ﬁnally become a single cluster ( t = 81 ).
We did not observe particle depletion and posterior collapse in our experiment.
This could be better avoided by adding an entropy loss to the learned variance of f trans and we will leave it for future study.
[-2        2        2    (a) t =(b) t =(c) t =    2        2        
-][Equation] [Equation]