P ROVABLE RL WITH E XOGENOUS D ISTRACTORS VIA M ULTISTEP I NVERSE D YNAMICS
A BSTRACT
Many real-world applications of reinforcement learning (RL) require the agent to deal with high-dimensional observations such as those generated from a megapixel camera.
Prior work has addressed such problems with representation learning, through which the agent can provably extract endogenous, latent state information from raw observations and subsequently plan efﬁciently.
However, such approaches can fail in the presence of temporally correlated noise in the observations, a phenomenon that is common in practice.
We initiate the formal study of latent state discovery in the presence of such exogenous noise sources by proposing a new model, the Exogenous Block MDP (EX-BMDP), for rich observation RL.
We start by establishing several negative results, by highlighting failure cases of prior representation learning based approaches.
Then, we introduce the Predictive
Path Elimination ( PPE ) algorithm, that learns a generalization of inverse dynamics and is provably sample and computationally efﬁcient in EX-BMDPs when the endogenous state dynamics are near deterministic.
The sample complexity of PPE depends polynomially on the size of the latent endogenous state space while not directly depending on the size of the observation space, nor the exogenous state space.
We provide experiments on challenging exploration problems which show that our approach works empirically.
I NTRODUCTION
In many real-world applications such as robotics there can be large disparities in the size of agent’s observation space (for example, the image generated by agent’s camera), and a much smaller latent state space (for example, the agent’s location and orientation) governing the rewards and dynamics.
This size disparity offers an opportunity: how can we construct reinforcement learning (RL) algorithms which can learn an optimal policy using samples that scale with the size of the latent state space rather than the size of the observation space?
Several families of approaches have been proposed based on solving various ancillary prediction problems including autoencoding (Tang et al., 2017; Hafner et al., 2019), inverse modeling (Pathak et al., 2017; Burda et al., 2018), and contrastive learning (Laskin et al., 2020) based approaches.
These works have generated some signiﬁcant empirical successes, but are there provable (and hence more reliable) foundations for their success?
More generally, what are the right principles for learning with latent state spaces?
In real-world applications, a key issue is robustness to noise in the observation space.
When noise comes from the observation process itself, such as due to measurement error, several approaches have been recently developed to either explicitly identify (Du et al., 2019;
Misra et al., 2020; Agarwal et al., 2020a) or implicitly leverage (Jiang et al., 2017)
the presence of latent state structure for provably sample-efﬁcient RL.
However, in many real-world scenarios, the observations consist of many elements (e.g. weather, lighting conditions, etc.) with temporally correlated dynamics (see [Table] [Table] /qdx5ksUjAotXCuG7Ice6WwqKSGcYUXDnIhb8UAup4akYLrldPnxvSLV2KaZNaXQTpV/5woRercKI18Zypw6P71JuJ7XrfApNUrlckLBCPfFiWFppjRSVI0VhYk6pEnQlrlb6VyKKyQ6POs+BB+f0r/T66ajXCv0bzYrx2fzOJYJtvkM6mTkByRY3JGzkmHSPKVPJJn8hI8BE/Ba/DtrXUumM1skb8Q/PgFxpGhKA==</latexit> e.g. Figure 1 and the example below) that are entirely independent of the agent’s actions and rewards.
The temporal dynamics of these elements precludes us from treating them as uncorrelated noise, and as such, most previous approaches resort to modeling their dynamics.
However, this is clearly wasteful as these elements have no bearing on the RL problem being solved.
As an example, consider the setting in Figure 1.
An agent is walking in a park on a lonely sidewalk next to a pond.
The agent’s observation space is the image generated by its camera, the latent endogenous state is its position on the sidewalk, and the exogenous noise is provided by motion of ducks or people in the background, swaying of trees and changes in lighting conditions, typically unaffected by the agent’s actions.
While there is a line of recent empirical work that aims to remove causally irrelevant aspects of the observation (Gelada et al., 2019; Zhang et al., 2020), theoretical treatment is quite limited (Dietterich et al., 2018) and no prior works address sample-efﬁcient learning with provable guarantees.
Given this, the key question here is:
How can we learn using an amount of data scaling with just the size of the endogenous latent state while ignoring the temporally correlated exogenous observation noise?
We initiate a formal treatment of RL settings where the learner’s observations are jointly generated by a latent endogenous state and an uncontrolled exogenous state, which is unaffected by the agent’s actions and does not affect the agent’s task.
We study a subset of such problems called
Exogenous Block MDPs (EX-BMDPs), where the endogenous state is discrete and decodable from the observations.
We ﬁrst highlight the challenges in solving EX-BMDPs by illustrating the failures of many prior representation learning approaches (Pathak et al., 2017; Misra et al., 2020; Jiang et al., 2017; Agarwal et al., 2020a; Zhang et al., 2020).
These failure happen either due to creating too many latent states, such as one for each combination of ducks and passers-by in the example above leading to sample inefﬁciency in exploration, or due to lack of exhaustive exploration.
We identify one recent approach developed by Du et al.
with favorable properties for a class of EX-BMDPs with near-deterministic latent state dynamics.
In Section 4 and Section 5, we develop a variation of their algorithm and analyze its performance for near-deterministic EX-BMDPs.
The algorithm, called Path Prediction and Elimination ( PPE ), learns a form of multi-step inverse dynamics by predicting the identity of the path that generates an observation.
For near-deterministic
EX-BMDPs, we prove that PPE successfully explores the environment using O (( SA ) 2 H log( |F| /δ ))
samples where S is the size of the latent endogenous state space, A is the number of actions, H is the horizon and F is a function class employed to solve a maximum likelihood problem of inferring the path which generated an observation.
Several prior works (Gregor et al., 2016; Paster et al., 2020) have also considered a multi-step inverse dynamics approach to learn a near optimal policy.
However, these works do not consider the EX-BMDP model.
Further, it is unknown whether these algorithms have guarantees for EX-BMDP similar to what we present for PPE .
Theoretical analysis of the performance of these algorithms in the presence of exogenous noise is an interesting future work direction.
Empirically, in Section 6, we demonstrate the performance of PPE and various prior baselines in a challenging exploration problem with exogenous noise.
We show that baselines fail to decode the endogenous state and either end up over-abstracting (creating an abstract state which aliases essentially-different states) or under-abstracting (creating essentially the same clones of states).
We further, show that PPE is able to recover the latent endogenous model in a visually complex navigation problem, in accordance with the theory.
E XOGENOUS B LOCK MDP S ETTING
We introduce a novel Exogenous Block Markov Decision Process (EX-BMDP) setting to model environments with exogenous noise.
We brieﬂy describe notations before presenting a formal treatment of EX-BMDP.
Notations.
For a given set U , we use ∆( U ) to denote the set of all probability distributions over U .
For a given natural number N ∈ N , we use the notation [ N ] to denote the set { 1 , 2 , · · · , N } .
Lastly, for a probability distribution p ∈ ∆( U ) , we deﬁne its support as supp p = { u | p ( u ) > 0 , u ∈ U} .
We start with describing the Block Markov Decision Process (BMDP) Du et al.
This process consists of a ﬁnite set of observations X , a set of latent states Z with cardinality Z , a ﬁnite set of actions A with cardinality A , a transition function T : Z × A → ∆( Z ) , an emission function q :
Z × A → ∆( X ) , a reward function R :
X ×
A → [0 , 1] , a horizon H ∈ N , and a start state distribution µ ∈ ∆( Z ) .
The agent interacts with the environment by repeatedly generating H -step trajectories ( z 1 , x 1 , a 1 , r 1 , · · · , z H , x H , a H , r H )
where z 1 ∼ µ ( · ) and for every h ∈ [ H ] we have x h ∼ q ( · | z h ) , r h =
R ( x h , a h ) , and if h < H , then z
h +1 ∼ T ( · | z h , a h ) .
The agent does not observe the states ( z 1 , · · · , z H ) , instead receiving only the observations ( x 1 , · · · , x H ) and rewards ( r 1 , · · · , r H ) .
We assume that the emission distributions of any two latent states are disjoint, usually referred as the block assumption : supp ( q ( · | z 1 ))
∩ supp ( q ( ·| z 2 ))
= ∅ when z 1 (cid:54) =
z 2 .
The agent chooses actions using a policy π : X → ∆( A ) .
We also deﬁne the set of non-stationary policies Π NS =
Π H as a H -length tuple, with ( π 1 , · · · ,
π H ) ∈
Π NS denoting that the action at time step h is taken as a h ∼ π h ( · | x h ) .
The value V ( π ) of a policy π is the expected episodic sum of rewards V ( π ) :
= E π [ (cid:80)
Hh =1 R ( x h , a h )] .
The optimal policy is given by π (cid:63) = arg max π ∈ Π NS V ( π ) .
We denote by P h ( x | π ) the probability distribution over observations x at time step h when following a policy π .
Lastly, we refer to an open loop policy as an element in all A H sequences of actions.
An open loop policy follows a pre-determined sequence of actions { a 1 , .., a H } for H time steps, unaffected by state information.
Given the aforementioned deﬁnitions, we deﬁne an EX-BMDP as follows:
Deﬁnition 1 (Exogenous Block Markov Decision Processes) .
An EX-BMDP is a BMDP such that the latent state can be decoupled into two parts
z = ( s, ξ ) where s ∈ S is the endogenous state and ξ ∈ Ξ is the exogenous state.
For z ∈ Z the initial distribution and transition functions are decoupled, that is: µ ( z ) =
µ ( s ) µ ξ ( ξ ) , and T ( z (cid:48)
[Equation]
The observation space X can be arbitrarily large to model which could be a high-dimensional real vector denoting an image, sound, or haptic data in an EX-BMDP.
The endogenous state s captures the information that can be manipulated by the agent.
Figure 1, center, visualizes the transition dynamics factorization.
We assume that the set of all endogenous states S is ﬁnite with cardinality S .
The exogenous state ξ captures all the other information that the agent cannot control and does not affect the information it can manipulate.
Again, we make no assumptions on the exogenous dynamics nor on its cardinality | Ξ | which may be arbitrarily large.
We note that the block assumption of the
EX-BMDP implies the existence of two inverse mappings: φ (cid:63) : X → S to map an observation to its endogenous state, and φ (cid:63)ξ : X → Ξ to map it to its exogenous state.
Justiﬁcation of assumptions.
The block assumption has been made by prior work (e.g., Du et al.
(2019), Zhang et al.
to model many real-world settings where the observation is rich , i.e., it contains enough information to decode the latent state.
The decoupled dynamics assumption made in the EX-BMDP setting is a natural way to characterize exogenous noise; the type of noise that is not affected by our actions nor affects the endogenous state but may have non-trivial dynamic.
This decoupling captures the movement of ducks, captured in the visual ﬁeld of the agent in Figure 1, and many additional exogenous processes (e.g., movement of clouds in a navigation task).
Goal.
Our formal objective is reward-free learning.
We wish to ﬁnd a set of policies, we call a policy cover , that can be used to explore the entire state space.
Given a policy cover, and for any reward function, we can ﬁnd a near optimal policy by applying dynamic programming (e.g., Bagnell et al.
(2004)), policy optimization (e.g., Kakade and Langford (2002); Agarwal et al. (2020b); Shani et al.
or value (e.g., Antos et al.
based methods.
Deﬁnition 2 ( α -policy cover) .
Let Ψ h be a ﬁnite set of non-stationary policies.
We say Ψ h is an α -policy cover for the h th time step if for all z ∈
Z it holds that
[Equation]
we call Ψ
[Equation]
a policy cover.
For standard BMDPs the policy cover is simply the set of policies that reaches each latent state of the
BMDP (Du et al., 2019; Misra et al., 2020; Agarwal et al., 2020a).
Thus, for a BMDP, the cardinality of the policy cover scales with |Z| .
The structure of EX-BMDPs allows to reduce the size of the policy cover signiﬁcantly to
[Equation]
| when the size of the exogenous state space is large.
Speciﬁcally, we show that the set of policies that reach each endogenous state, and do not depend on the exogenous part of the state is also a policy cover (see Appendix B, Proposition 4).
Further, the proposed algorithm PPE , learns such a policy cover and requires number of samples that depends polynomially on |S| instead of |Z| or |X| .
F AILURES OF P RIOR A PPROACHES
We now describe the limitation of prior RL approaches in the presence of exogenous noise.
We provide an intuitive analysis over here, and defer a formal statement and proof to Appendix A.
Limitation of Noise-Contrastive learning.
Noise-contrastive learning has been used in RL to learn a state abstraction by exploiting temporal information.
Speciﬁcally, the H OMER algorithm
(Misra et al., 2020) trains a model to distinguish between real and imposter transitions.
This is done by collecting a dataset of quads ( x, a, x (cid:48) , y ) where y = 1 means the transition was ( x, a, x (cid:48) ) was observed and y =
means that ( x, a, x (cid:48) ) was not observed.
H OMER then trains a model p θ ( y | x, a, φ θ ( x (cid:48) )) with parameters θ , on the dataset, by predicting whether a given pair of transition was observed or not.
This provides a state abstraction φ θ : X → N for exploring the environment.
H OMER can provably solve Block MDPs.
Unfortunately, in the presence of exogenous noise, H OMER distinguishes between two transitions that represent transition between the same latent endogenous states but different exogenous states.
In our walk in the park example, even if the agent moves between same points in two transitions, the model maybe able to tell these transitions apart by looking at the position of ducks which may have different behaviour in the two transitions.
This results in the H OMER creating O ( |Z| ) many abstract states.
We call this the under-abstraction problem.
Limitation of Inverse Dynamics.
Another common approach in empirical works is based on modeling the inverse dynamics of the system, such as the ICM module of Pathak et al.
In such approaches, we learn a representation by using consecutive observations to predict the action that was taken between them.
Such a representation can ignore all information that is not relevant for action prediction, which includes all exogenous/uncontrollable information.
However, it can also ignore controllable information. This may result in a failure to sufﬁciently explore the environment.
In this sense, inverse dynamics approaches result in an over-abstraction problem where observations from different endogenous states can be mapped to the same abstract state.
The over-abstraction problem was described at Misra et al.
(2020), when the starting state is random.
In Appendix A.3 we show inverse dynamics may over-abstract when the initial starting state is deterministic.
Limitation of Bisimulation.
Zhang et al.
(2020) proposed learning a bisimulation metric to learn a representation which is invariant to exogenous noise.
Unfortunately, it is known that bisimulation metric cannot be learned in a sample-efﬁcient manner (Modi et al.
(2020), Proposition B.1).
Intuitively, when the reward is same everywhere, then bisimulation merges all states into a single abstract state.
This creates exploration issue in sparse reward settings, since the agent can merge all states into a single abstract state until it receives a non-trivial reward.
Hence, the learned abstract states provide no help in exploring the environment.
Bellman rank might depend on | Ξ | .
The Bellman rank was introduced in Jiang et al.
(2017) as a complexity measure for the learnability of an RL problem with function approximations.
To date,
1: Set Ψ 1 = {⊥} , stochasticity level η ≤ 1 4 SH // ⊥ denotes an empty path 2: for h = 2 , . . .
, H do 3:
Set N = 16 ( | Ψ h − 1 ◦ A | ) 2 log (cid:16) |F|| Ψ h − 1 | AH δ (cid:17) 4:
Collect a dataset D of N i.i.d.
tuples ( x, υ ) where υ ∼ Unf (Ψ h − 1 ◦ A ) and x ∼ P ( x h | υ ) .
Solve multi-class classiﬁcation problem: ˆ f h = arg max f ∈F (cid:80) ( x,υ ) ∈D ln f ( idx ( υ ) | x ) .
6: for 1 ≤ i < j ≤ | Ψ h − 1 ◦ A| do 7:
Calculate the path prediction gap: (cid:98) ∆( i, j )
=
1 N (cid:80) ( x,υ ) ∈D (cid:12)(cid:12)(cid:12) ˆ f h
( i | x ) − ˆ f h ( j | x ) (cid:12)(cid:12)(cid:12) .
If (cid:98) ∆( i, j ) ≤ 5 /| Ψ h − 1 ◦A| , then eliminate path υ with idx( υ ) =
j . //
υ i and υ j visit the same state 9: Ψ h is deﬁned as the set of all paths in Ψ h − 1 ◦
A that have not been eliminated in line 8.
most of the learnable RL problems have a small Bellman rank.
However, we show in Appendix A that Bellman rank for EX-BMDP can scale as O ( | Ξ | ) .
This shows that EX-BMDP is a highly non-trivial setting as we don’t even have sample-efﬁcient algorithms that can solve it without being computationally-efﬁcient.
In Appendix A we also describe the failures of FLAMBE (Agarwal et al., 2020a)) and autoencoding based approaches (Tang et al., 2017).
R EINFORCEMENT L EARNING FOR EX-BMDP S
In this section, we present an algorithm Predictive Path Elimination ( PPE ) that we later show can provably solve any EX-BMDP with nearly deterministic dynamics and start state distribution of the endogenous state, while making no assumptions on the dynamics or start state distribution of the exogenous state (Algorithm 1).
Before describing PPE , we highlight that PPE can be thought of as a computationally-efﬁcient and simpler alternative to Algorithm 4 of Du et al.
(2019) who studied rich-observation setting without exogenous noise.
1 Due to the apparent difference of the algorithms we keep the presentation of our algorithm fully self-contained.
PPE performs iterations over the time steps h ∈ { 2 , · · · , H } .
In the h th iteration, it learns a policy cover Ψ h for time step h containing open-loop policies.
This is done by ﬁrst augmenting the policy cover for previous time step by one step.
Formally, we deﬁne Υ h
= Ψ
h − 1 ◦
A
=
{ π ◦ a | π ∈ Ψ h − 1 , a ∈ A} where π ◦ a is an open-loop policy that follows π till time step h − 1 and then takes action a .
Since we assume the transition dynamics to be near-deterministic, therefore, we know that there exists a policy cover for time step h that is a subset of Υ h and whose size is equal to the number of reachable states at time step h .
Further, as the transitions are near-deterministic, we refer to an open-loop policy as a path, as we can view the policy as tracing a path in the latent transition model.
PPE works by eliminating paths in Υ h so that we are left with just a single path for each reachable state.
This is done by collecting a dataset D of tuples ( x, υ ) where υ is a uniformly sampled from Υ h and x ∼ P h ( x | υ ) (line 4).
We train a classiﬁer ˆ f h using D by predicting the index idx ( υ ) of the path υ from the observation x (line 5).
Index of paths in Υ h are computed with respect to Υ h and remain ﬁxed throughout training.
Intuitively, if ˆ f h ( i | x ) is sufﬁciently large, then we can hope that the path υ
i visits the state φ (cid:63) ( x ) .
Further, we can view this prediction problem as learning a multistep inverse dynamics model since the open-loop policy contains information about all previous actions and not just the last action.
For every pair of paths in Υ h , we ﬁrst compute a path prediction gap (cid:98) ∆ (line 7).
If the gap is too small, we show it implies that these paths reach the same endogenous state, hence we can eliminate a single redundant path from this pair (line 8).
Finally, Ψ h is deﬁned as the set of all paths in Υ h which were not eliminated.
PPE reduces RL to performing H standard classiﬁcation problems.
Further, the algorithm is very simple and in practice requires just a single hyperparameter ( N ).
We believe these properties will make it well-suited for many problems.
Recovering an endogenous state decoder.
We can recover a endogenous state decoder ˆ φ h for each time step h ∈ { 2 , 3 , · · · , H } directly from ˆ f h as shown below:
[Equation]
Intuitively, this assigns the observation to the path with smallest index that has the highest chance of visiting x , and therefore, φ (cid:63) ( x ) .
We are implicitly using the decoder for exploring, since we rely on using ˆ f h for making planning decisions.
We will evaluate the accuracy of this decoder in Section 6.
Recovering the latent transition dynamics.
PPE can also be used to recover a latent endogenous transition dynamics.
The direct way is to use the learned decoder ˆ φ h along with episodes collected by PPE during the course of training and do count-based estimation.
However, for most problems, recovering an approximate deterministic transition dynamics sufﬁces, which can be directly read from the path elimination data.
We accomplish this by recovering a partition of paths in Ψ h − 1 × A where two paths in the same partition set are said to be merged with each other.
In the beginning, each path is only merged with itself.
When we eliminate a path υ
j on comparison with υ i in line 8, then all paths currently merged with υ j get merged with υ i .
We then deﬁne an abstract state space (cid:98) S h for time step h that contains an abstract state j for each path υ j ∈ Ψ h .
Further, we recover a latent deterministic transition dynamics for time step h − 1 as ˆ T h − 1 : (cid:98) S h − 1 ×
A → (cid:98) S h where we set ˆ
T h − 1 ( i, a ) =
j if the path υ
j ∈ Ψ h gets merged with path υ (cid:48)
i ◦ a ∈ Ψ h
where υ (cid:48) i ∈ Ψ
h − 1 .
Learning a near optimal policy given a policy cover.
PPE runs in a reward-free setting.
However, the recovered policy cover and dynamics can be directly used to optimize any given reward function with existing methods.
If the reward function depends on the exogenous state then we can use the
PSDP algorithm (Bagnell et al., 2004) to learn a near-optimal policy.
PSDP is a model-free dynamic programming method that only requires policy cover as input (see Appendix D.1 for details).
However, if the reward function only depends on the endogenous state, then we can use a computationally cheaper value-iteration algorithm VI that uses the policy cover and recovered transition dynamics.
VI is a model-based algorithm that estimates the reward for each state and action, and performs dynamic programming on the model (see Appendix D.2 for details).
In each case, the sample complexity of learning a near-optimal policy, given the output of the PPE algorithm, scales with the size of endogenous and not the exogenous state space or the size of observation space.
T HEORETICAL A NALYSIS AND D ISCUSSION
We provide the main sample complexity guarantee for PPE as well as additional intuition for why it works.
We analyze the algorithm in near-deterministic MDPs deﬁned as follows:
Two transition functions T 1 and T 2 are η -close if for all h ∈
[ H ] , a ∈ A , s ∈ S h
it holds that || T 1 ( · | s, a ) − T 2 ( · | s, a ) || 1 ≤ η .
Analogously, two starting distribution µ 1 and µ 2 are η -close if || µ 1 ( · ) − µ 2 ( · ) || 1 ≤ η .
We emphasize that near-deterministic dynamics are common in real-world applications like robotics.
Assumption 1 (Near deterministic endogenous dynamics) .
We assume the endogenous dynamics is η -close to a deterministic model ( µ D,η , T D,η ) where η ≤ 1 / (4 SH ) .
We make a realizability assumption for the regression problem solved by PPE (line 5).
We assume that F is expressive enough to represent the Bayes optimal classiﬁer of the regression problems created by PPE .
Assumption 2 (Realizability) .
For any h ∈ [ H ] , and any set of paths Υ ⊆
A h with | Υ | ≤ SA and where
A h denotes the set of all paths of length h , there exists f (cid:63) Υ ,h ∈ F such that: f (cid:63) Υ ,h ( idx ( υ ) | x ) =
[Equation]
In this assumption, the function f (cid:63) Υ ,h ( υ | x ) denotes the Bayes optimal classiﬁer for the regression problem in the h th iteration when Υ h
= Υ .
Realizability assumptions are common in theoretical analysis (e.g., Misra et al.
(2020), Agarwal et al.
(2020a)).
In practice, we use expressive neural networks to solve the regression problem, so we expect the realizability assumption to hold.
Note that there are at most A S ( H +1)
Bayes classiﬁers for different prediction problems.
However, this is acceptable since our guarantees will scale as ln |F| and, therefore, the function class F can be exponentially large to accommodate all of them.
We now state the formal sample complexity guarantees for PPE below.
Theorem 1 (Sample Complexity) .
Fix δ ∈ (0 , 1) .
Then, with probability greater than 1 − δ , PPE returns a policy cover { Ψ h } Hh =1 such that for any h ∈
[ H ] , Ψ h is a ηH -policy cover for time step h and | Ψ h | ≤ S , which gives the total number of episodes used by PPE as O (cid:16) S 2 A 2 H ln |F| SAHδ (cid:17) .
We defer the proof to Appendix C.
Our sample complexity guarantees do not depend directly on the size of observation space or the exogenous space.
Further, since our analysis only uses standard uniform convergence arguments, it extends straightforwardly to inﬁnitely large function classes by replacing ln |F| with other suitable complexity measures such as Rademacher complexity.
Why does PPE work?
We provide an asymptotic analysis to explain why PPE works.
Consider a deterministic setting and the h th iteration of PPE .
Assume by induction that Ψ h − 1 is an exact policy cover for time step h − 1 . Let P h ( ξ ) denote the distribution over exogenous states at time step h which is independent of agent’s policy.
The Bayes optimal classiﬁer ( f (cid:63)h := f Υ h ,h ) of the prediction problem can be derived as:
[Equation]
(cid:48) ) P ( υ (cid:48) ) ( a ) = P h ( x | υ ) (cid:80) υ (cid:48) ∈
Υ h P h ( x | υ (cid:48) ) ( b ) =
P h ( φ (cid:63) ( x )) | υ ) (cid:80) υ (cid:48) ∈ Υ h P h ( φ (cid:63) ( x )) | υ (cid:48) ) , where ( a ) holds since all paths in Υ h are chosen uniformly, and ( b ) critically uses the fact that for any open-loop policy υ we have: [Equation] Factorization
[Equation]
As Ψ h − 1 is a policy cover for time step h − 1 , therefore, Υ h is also a policy cover for time step h . This implies that we can hope to converge to f (cid:63)h ( · | x ) for every observation reachable at time step h . It also implies that the denominator ω ( φ (cid:63) ( x )) {+: = (cid:80) υ (cid:
+}
) ∈ Υ h P h ( φ (cid:63) ( x ) | υ (cid:48) ) will be non-zero for every observation x reachable at time step h , and, therefore, f (cid:63) ( · | x ) is well-deﬁned. The Bayes optimal classiﬁer depends only on the endogenous state φ (cid:63) ( x ) of the observation x .
Therefore, there is no signal available to separate observations that map to the same endogenous state, and we don’t have an under-abstraction issue.
Conversely, let x and x (cid:48) be two observations reachable at time step h that map to two different endogenous state s and s (cid:48) respectively. Let υ ∈ Υ
h be a path that visits s and has an index i . Then υ cannot visit s (cid:48) due to deterministic dynamics. Using the structure of f (cid:63)h derived above, we get f (cid:63)h ( i | x ) = 1 / ω ( s ) > 1 / | Υ h | and f (cid:63)h ( i | x (cid:48) ) = 0 .
This provides the signal to separate observations from different endogenous states.
Therefore, there is no over-abstraction issue. PPE uses a similar reasoning to ﬁlter out redundant paths that reach the same endogenous state. This is necessary to keep the size of policy cover small and get polynomial sample complexity. Let υ 1 , υ 2 ∈
Υ h be two paths with indices i and j respectively.
We deﬁne their exact path prediction gap as ∆( i, j ) : = E x h [ | f (cid:63)h ( i | x h ) − f (cid:63)h ( j | x h ) | ] .
Let υ 1 visit an endogenous state s at time step h .
Then f (cid:63)h ( i | x h ) = 1 / ω ( s ) if φ (cid:63) ( x h ) =
s , and 0 otherwise.
If υ 2 also visits s at time step h , then f (cid:63)h ( i | x h ) =
f (cid:63)h ( j | x h ) for all x h .
This implies ∆( i, j ) = 0 and PPE will ﬁlter out the path with higher index.
Conversely, let υ 2 visit a different state at time step h .
If x is an observation that maps to s , then f (cid:63)h ( i | x ) = 1 / ω ( s ) and f (cid:63)h ( j | x ) = 0 .
This gives | f (cid:63)h ( i | x ) − f (cid:63)h ( j | x ) | = 1 / ω ( s ) ≥ 1 / | Υ h | and, consequently, ∆( i, j ) > 0 .
In fact, we can show ∆( i, j ) ≥ O ( 1 / | Υ h | ) .
Thus, PPE will not eliminate these paths upon comparison.
Our complete analysis in the Appendix generalizes the above reasoning to ﬁnite sample setting where we can only approximate f (cid:63)h and ∆ , as well as to EX-BMDPs with near-deterministic dynamics.
We stress that our analysis critically relies on Equation 1 that holds for open-loop policies but not for an arbitrary policy class.
This is the main reason why we build a policy cover with open-loop policies. [Table] =</latexit> [Figure]
c <latexit sha1_base64="1/uDFG36QoEDGFebRyJRvDmloow=">AAAB7XicdVBNSwMxEJ2tX7V+VT16CRbB05JtK+ux6MVjBVsL7VKyadrGZrNLkhXK0v/gxYMiXv0/3vw3pu0KKvpg4PHeDDPzwkRwbTD+cAorq2vrG8XN0tb2zu5eef+
[Table] + [Table] +
[Table]
[Table]
+etowBWjRkwtIVRxeyuiY6IINTagkg3h61P0P2lXXa/
[Table]
rlcZFHkcRjuAYTsEDHxpwBU1oAYU7eIAneHZi59F5cV6XrQUnnzmEH3DePgHMrI9I</latexit>
[Figure] [Figure]  E XPERIMENTS We evaluate PPE on two domains: a challenging exploration problem called combination lock to test whether PPE can learn an optimal policy and an accurate state decoder, and a visual-grid world with complex visual representations to test whether PPE is able to recover the latent dynamics.
Combination Lock Experiments.
The combination lock problem is deﬁned for a given horizon
H by an endogenous state space S
= { s 1 ,a } ∪ { s h,a , s h,b , s h,c } Hh =2 , an exogenous state space Ξ = { 0 , 1 } H , an action space
A with 10 actions, and a deterministic endogenous start state of s 1 ,a .
For any state s h,g we call g as its type which can be a, b or c .
States with type a and b are considered good states and those with type c are considered bad states.
Each instance of this problem is deﬁned by two good action sequences ( a h )
Hh =2 , ( a (cid:48) h )
Hh =2 with a h (cid:54) = a (cid:48) h , which are chosen uniformly randomly and kept ﬁxed throughout training.
At h = 1 , the agent is in s 1 ,a and action a 1 leads to s 2 ,a , a (cid:48)
h leads to s 2 ,b , and all other actions lead to s 2 ,c .
For h > 2 , taking action a h in s h,a leads to s h +1 ,a and taking action a (cid:48)
h in s h,b leads to s h +1 ,b .
In all other cases involving taking an action in a state s h,g , we transition to the next bad state s h +1 ,c .
We visualize the latent endogenous dynamics in Figure 2a.
The exogenous state evolves as follows: we sample ξ 1 as a vector in { 0 , 1 } H by selecting the value of each dimension independently uniformly in { 0 , 1 } .
At time step h , ξ h is generated from ξ h − 1 by uniformly ﬂipping each bit in ξ h − 1 independently with probability 0.1.
There is a reward of 1.0 on taking the good action a H,a in s H,a and a reward of 0.1 on taking action a H,b in s H,b , and in all other cases the agent gets a reward of 0.
This gives a V ( π (cid:63) )
= 1 , and the probability that a random open loop policy gets this optimal return is 10 − H .
An observation x is generated stochastically from a latent state z
= ( s, ξ ) .
We map s to a vector w encoding the identity of the state.
We concatenate ( w, ξ ) , add Gaussian noise to each dimension, and multiply the result with a Hadamard matrix to generate x .
See Appendix F for full details.
Our construction is inspired by similar combination lock problems in prior work (Du et al., 2019; Misra et al., 2020) (which did not consider exogenous distractors).
Baseline.
We compare PPE with ﬁve baselines on the combination lock problem.
These include
PPO (Schulman et al., 2017) which is an actor-critic algorithm, PPO + RND (Burda et al., 2019) which adds an exploration bonus to PPO using prediction errors, Homer that uses contrastive learning (Misra et al., 2020), and another algorithm ID which is similar to Homer but instead of contrastive learning it learns an inverse dynamics model to recover the state abstraction.
Lastly, we also compare with
Bisim that learns a bisimulation metric along with an actor-critic agent (Zhang et al.
We use existing publicly available codebases for these baselines.
Our implementation of PPE very closely follows the pseudo-code in Algorithm 1.
We model F using a two-layer feed-forward network with ReLU non-linearity.
We train F with Adam optimization and use a validation set to do model selection.
We refer readers to Appendix F for additional experimental details.
Results.
Figure 2b shows results for values of H in { 5 , 10 , 20 , 40 } .
For each value of H , we plot the minimal number of episodes n needed to achieve a mean regret of at most V ( π (cid:63) ) / 2 = 0 .
We run each algorithm 5 times with different seeds and report the median performance.
If an algorithm is unable to achieve the desired regret in 5 × 10 5 episodes then we set n = ∞ .
We observe that
PPO is unable to solve the problem at H = 5 .
PPO + RND is able to solve the problem at H = 5 and
H = 10 , showing the exploration bonus induced by random network distillation helps.
However, it is unable to solve the problem for larger values of H .
We observe that Homer and ID are also unable to solve the problem for any value of H .
Bisim also fails to solve the problem for any H ≥ 5 .
This agrees with the theoretical prediction that Bisim provides no learning signal when running in sparse-reward settings.
In such settings, there can be many episodes before any reward is received.
In the absence of any reward, the bisimulation objective incentivizes mapping all observations to the same representation which is not helpful for further exploration.
Lastly, PPE is able to solve the problem for all values of H and is signiﬁcantly more sample efﬁcient than baselines.
Since the reward function of the combination-lock problem depends only on the endogenous state, we run PPE and then a value-iteration like algorithm (see Appendix D.2) to learn a near optimal policy.
In order to understand the failure of Homer and ID , we investigate the accuracy of the state abstraction learned by these methods and compare that with PPE .
We focus on the combination lock setting with H = 2 and evaluate the learned decoder for the last time step.
As the state abstraction models are invariant to label permutation we use the following evaluation metric: given a learned abstraction for the endogenous state ˆ φ : X → [ N ] we compute 1 / m (cid:80) mi
=1 1 { ˆ φ ( x i, 1 ) = ˆ φ ( x i, 2 ) ⇔ φ (cid:63) ( x i, 1 ) =
φ (cid:63) ( x i, 2 ) } , where { x i, 1 , x i, 2 } m i =1 are drawn independently from a ﬁxed distribution D with good support over all states.
We report the percentage accuracy in Figure 2c.
When there is no exogenous noise, Homer is able to learn a good state decoder with enough samples while ID fails to learn, in accordance with the theory.
On inspection, we found that ID suffers from the under-abstraction issue highlighted earlier as it has difﬁculty separating observations from s 3 a and s 3 b .
On adding exogenous noise, the accuracy of Homer plummets signiﬁcantly.
The accuracy of ID also drops but this drop is mild since unlike Homer , the ID objective is able to ﬁlter exogenous noise.
Lastly, we observe that PPE is always able to learn a good decoder and is more sample efﬁcient than baselines.
Visual Grid World Experiments.
We test the ability of PPE to recover the latent endogenous transition dynamics in visual grid-world problem.
The agent navigates in a N × N grid world where each grid can contain a stationary object, the goal, or the agent.
The agent’s endogenous state is given by its position in the grid and its direction amongst four possible canonical directions.
The agent can take ﬁve different actions for navigation.
The world is visible to the agent as a 8 N × 8 N sized
RGB image.
We add exogenous noise as follows: at the beginning of each episode, we independently sample position, size and color of 5 ellipses.
The position and size of these ellipses is perturbed after each time step independent of the action.
We project these ellipses on top of the world’s image.
Figure 3 shows sampled observations from the 7 × 7 gridworld that we experiment on.
The exogenous state is given by the position, size and color of ellipses and is much larger than |S| ≤ 4 N 2 .
We model
F using a two-layer convolutional neural network and train it using Adam optimization.
We defer the full details of setup to Appendix F.
Since the problem has deterministic dynamics, we can evaluate the accuracy of the learned transition model by measuring it in terms of accuracy of the elimination step (Algorithm 1, line 8), since this step induces our algorithm’s mapping from observations to endogenous latent states.
For a ﬁxed h ∈ { 2 , · · · , H } , let ν i and
ν
j be two paths in Ψ h − 1 ◦ A .
We compute two type of errors.
Type 1 error computes whether PPE merged these paths, i.e., predicted them as mapping to the same abstract state, when they go to different endogenous states.
Type 2 error computes whether PPE predicted the paths as mapping to different abstract states, when they map to the same endogenous state.
We report the total number of errors of both types by summing over all values of h and all pairs of different paths in Ψ h − 1 ◦ A .
Type 1 errors are more harmful, since they can lead to exploration failure.
Speciﬁcally, merging paths going to different states may result in the algorithm avoiding one of the two states when exploring at the next time step.
Type 2 errors are less serious but lead to inefﬁciency due to using redundant paths for exploration.
When both errors are 0, then the model recovers the exact latent model up to relabeling.
Results.
We report results on learning the model in in Figure 3c.
We see that PPE is able to reduce the number of type 1 errors down to 0 using 2 × 10 5 episodes per time step.
This is important since even a single type 1 error can cause exploration failures.
Similarly, PPE is able to reduce typeerrors and is able to get them down to 56 with 5 × 10 5 episodes.
This is acceptable since type [Figure] errors do not cause exploration failures but only cause redundancy.
Therefore, at 2 × {+
 ,+} the algorithm makes 0 type 1 errors and just a handful type 2 errors.
This is remarkable considering that PPE compares roughly 2 × 10 5 pairs of paths in the entire run.
Hence, it makes only ≤ 0 .
03% type 2 errors.
Further, the agent is able to plan using the learned transition model and receive the optimal return.
We also evaluate the accuracy of state decoding on this problem.
We compare the state decoding accuracy of PPE and Homer at H = 2 using an identical evaluation setup to the one we used for combination lock.
Figure 3d shows the results.
As expected, PPE rapidly learns a highly accurate decoder while Homer performs only as well as a random uniform decoder.
C ONCLUSION
In this work, we introduce the EX-BMDP setting, an RL setting that models exogenous noise, ubiquitous in many real-world systems.
We show that many existing RL algorithms fail in the presence of exogenous noise.
We present PPE that learns a multi-step inverse dynamics to ﬁlter exogenous noise and successfully explores.
We derive theoretical guarantees for PPE in near-deterministic setting and provide encouraging experimental evidence in support of our arguments.
To our knowledge, this is the ﬁrst such algorithm with guarantees for settings with exogenous noise.
Our work also raises interesting future questions such as how to address the general setting with stochastic transitions, or handle more complex endogenous state representations.
Another interesting line of future work direction is the analysis of other approaches that learn multi-step inverse dynamics (Gregor et al., 2016; Paster et al., 2020) and understanding whether these approaches can also provably solve EX-BMDPs. A CKNOWLEGMENTS
We would like to thank the reviewers for their suggestions and comments.
We acknowledge the help of Microsoft’s GCR team for helping with the compute.
YE is partially supported by the Viterbi scholarship, Technion