Provably sample-efficient RL with side information about latent dynamics
Abstract
We study reinforcement learning (RL) in settings where observations are highdimensional, but where an RL agent has access to abstract knowledge about the structure of the state space, as is the case, for example, when a robot is tasked to go to a specific room in a building using observations from its camera, while having access to the floor plan.
We formalize this setting as transfer reinforcement learning from an abstract simulator , which we assume is deterministic (such as a simple model of moving around the floor plan), but which is only required to capture the target domain’s latent-state dynamics approximately up to unknown (bounded) perturbations (to account for environment stochasticity).
Crucially, we assume no prior knowledge about the structure of observations in the target domain except that they can be used to identify the latent states (but the decoding map is unknown).
Under these assumptions, we present an algorithm, called TASID , that learns a robust policy in the target domain, with sample complexity that is polynomial in the horizon, and independent of the number of states, which is not possible without access to some prior knowledge.
In synthetic experiments, we verify various properties of our algorithm and compare it with several transfer RL algorithms that require access to “full simulators” (i.e., those that also simulate observations).
Introduction
When learning from scratch, reinforcement learning (RL) in the real world can be very expensive.
For example, a robot learning to navigate in a building might need to explore every possible state or location, which can be painstakingly costly and time-consuming.
Sometimes, however, it is not necessary to begin such a learning process from scratch.
For instance, in the robot example, we might have access to a general floor map of the building.
How can this kind of high-level but imprecise information be used to learn how to operate in the environment more quickly and more effectively?
In this paper, we study how to effectively leverage prior information in the form of such “abstract” descriptions of the environment.
We formalize this abstract description as an “abstract simulator,” which, like a map, can be used as an imperfect model.
Importantly, our abstract simulators differ from more standard simulators in that they only focus on the “latent structure” of the environment dynamics, not on the observations that might be experienced by an agent in the environment.
In general, fully faithful simulators of even the latent dynamics might be hard to build, for instance, due to the difficulty of exactly modeling all probabilistic outcomes, as when the robot’s actions do robust dynamic programming perturbed transitions deterministic abstract simulator ⇡ <latexit sha1_base64="MMfoOZUbGzRkaB76umvTEWj+CN8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz00Et4v1xxq+4cZJV4OalAjka//NUbxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bH7qlJxZZUDCWNmShszV3xMZjbSeRIHtjKgZ6WVvJv7ndVMTXvsZl0lqULLFojAVxMRk9jcZcIXMiIkllClubyVsRBVlxqZTsiF4yy+vklat6l1Ua/eXlfpNHkcRTuAUzsGDK6jDHTSgCQyG8Ayv8OYI58V5dz4WrQUnnzmGP3A+fwBRC43R</latexit> target environment policy inverse dynamic models
[Equation]
<
[Equation]
FHMqDae
[Equation]
/MHSxieWg==</latexit> robust abstract policy ⇢ <latexit sha1_base64="94wfdLRRBjCkyTC3Y5p5yTs2egQ
[Table]
>AAAB73icbVDLSgNBEOyNrxhfUY9eBoPoxbAbBT0GvXiMYB6QLGF2MkmGzM6uM71CWPITXjwo4tXf8ebfOEn2oIkFDUVVN91dQSyFQdf9dnIrq2vrG/nNwtb2zu5ecf+gYaJEM15nkYx0K6CGS6F4HQVK3oo1p2EgeTMY3U795hPXRkTqAccx90M6UKIvGEUrtehpNx2ee5NuseSW3RnIMvEyUoIMtW7xq9OLWBJyhUxSY9qeG6OfUo2CST4pdBLDY8pGdMDblioacuOns3sn5MQqPdKPtC2FZKb+nkhpaMw4DGxnSHFoFr2p+J/XTrB/7adCxQlyxeaL+okkGJHp86QnNGcox5ZQpoW9lbAh1ZShjahgQ/AWX14mjUrZuyhX7i9L1ZssjjwcwTGcgQdXUIU7qEEdGEh4hld4cx6dF+fd+Zi35pxs5hD+wPn8AT+Hj3M=</latexit> not have exactly their intended effect.
More complex simulators might be more faithful, but simpler simulators might be easier to build and also more computationally tractable.
In this paper, we address this trade-off with a particular design choice: First, we assume that the abstract simulator is deterministic.
Indeed, an ordinary map, which implicitly represents what new position will be reached by a particular action, is such a deterministic model.
Compared to fully probabilistic models, deterministic ones are especially simple to build and work with.
On the other hand, a deterministic model will not in general capture the “noise” of real-world dynamics.
Therefore, we seek algorithms that will learn policies that are robust to errors or “perturbations” in the model represented by the abstract simulator.
To this end, in Section 4, we present a new algorithm called TASID that provably finds a policy achieving a certain level of performance for any target environment whose dynamics can be reasonably approximated by a given abstract simulator, up to some quantifiable perturbation level, and with arbitrarily different observations.
(Figurevisualizes our setup and approach.)
In the robot example, although a map is intuitively helpful for reducing the need for exploration, there is still much to be inferred, for instance, how locations on the map correspond to locations in the building, and, even more challenging, how they correspond to rich, high-dimensional observations experienced via cameras or other sensors.
Our algorithm solves these challenges, building on prior work on learning from such observations [Krishnamurthy et al., 2016, Jiang et al., 2017].
Furthermore, our theoretical guarantees show that we do indeed save dramatically on sample complexity (measured by the required number of interactions with the target environment) by not having to explore the entire environment, and instead focusing the task of learning observations just to the states that lead towards accomplishing the task.
Indeed, our algorithm’s sample complexity is entirely independent of the size of the state space, assuring efficiency even when the state space is extremely large, as is often the case.
We are able to achieve this result because of specific assumptions and criteria for success, as outlined above, namely, near-deterministic latent dynamics, knowledge of the abstract simulator, and the benchmark of a robust policy rather than an optimal policy.
In Section 5, we empirically evaluate TASID in two domains.
The first represents a challenging problem requiring strategic exploration.
We show that TASID is able to efficiently achieve the robust policy value while the PPO algorithm [Schulman et al., 2017] augmented with exploration bonus [Burda et al., 2019b] and domain randomization [Tobin et al., 2017] does not solve the problem.
The second domain is a visual grid world that tests the robustness of TASID to dynamics perturbation and the change of state-space size.
We show that TASID succeeds empirically in agreement with our theory.
Related Work
Transfer RL.
The goal of transfer RL is to speed up learning in a target environment by leveraging a model (or some other form of knowledge) learnt in a source environment [Taylor and Stone, 2009, Lazaric, 2012].
For our setting, the most relevant are the methods that can transfer between environments with different observation spaces.
The dominant line of research requires access to a hand-coded inter-task mapping, which is used to introduce constraints, shape rewards, or warm-start learning in the target environment [Taylor and Stone, 2005, Taylor et al., 2007a, Fernández et al., 2010, Mann and Choe, 2013, Brys et al., 2015].
Instead of hand-coding, several works seek to learn inter-task mappings using structural information such as factorization [Liu and Stone, 2006, Taylor et al., 2007b, Soni and Singh, 2006].
Van Driessel and Francois-Lavet [2021] and Sun et al.
[2022] do not require such structural information, and only posit assumptions similar to our block MDP assumption, but without requiring near-deterministic latent dynamics.
Latent dynamics learnt in the source domain is used to create auxiliary tasks that regularize learning in the target domain.
These methods show empirical promise of transferring information about latent dynamics.
Their focus is mainly on modeling and experiments, whereas our work also seeks to derive theoretical guarantees that showcase the benefits of transfer.
We derive sample complexity that is independent of the size of the state space and observation space, which would be impossible without transfer.
Among the cited works, Mann and Choe [2013] also show benefits of transfer, but their sample complexity generally scales linearly with the size of the target observation space.
The strong bound that we derive requires a near-deterministic latent dynamics, and so it is not as broadly applicable.
Provably efficient RL with rich observations.
Our modeling setup and theoretical analysis build on the prior work that considers large-dimensional “rich” observations, but simpler (tabular) latent dynamics [Krishnamurthy et al., 2016, Jiang et al., 2017, Du et al., 2019].
To circumvent the hardness of learning in general partially-observed MDPs (POMDPs), these methods impose structural assumptions, such as the block MDP assumption used here.
These methods, like ours, learn with sample complexity that is independent of the size of the observation space, but, unlike ours, that is polynomial in the size of the latent state space.
Using existing MDP lower bounds [Kakade, 2003], this dependence is unavoidable for general latent dynamics, even when the decoding function of observations to latent states is known.
We overcome this fundamental obstacle by assuming that the latent dynamics is near-deterministic, and the deterministic approximation is known.
Robust RL.
Our solution concept and the underlying algorithm build on robust dynamic programming methods [Iyengar, 2005, Bagnell et al., 2001].
However, we are using this concept in a fundamentally new way to achieve sample complexity improvements in transfer learning.
This is one of the key conceptual innovations of our work.
Problem Setting
We next formalize our modeling setup and assumptions.
As a running example, we consider a navigation problem on a floor of a building, where the goal is to reach a certain location by a robot that can turn left and right and move forward.
The robot senses the environment via lidar readings and/or a camera, but it does not have access to its position or orientation.
We use the notation ∆( S ) for the set of probability distributions over a set S , and [ n ] to denote the set { 1 , 2 , . . . , n } .
3.1 Target environment
We assume that the target environment is a block
MDP (see, e.g., Du et al., 2019, Misra et al., 2020,
Zhang et al., 2020).
In a block MDP, observations are high-dimensional (e.g., lidar and camera readings), but emitted by a finite number of latent states (e.g., location and orientation of a robot); latent states are not directly observed, but they can be determined from the observations they emit.
Formally, a block MDP is a triple M = ⟨ M, X , q ⟩ .
The first component of the triple is a standard episodic MDP M = ⟨S , A , s init , H, T, R ⟩ , referred to as the latent MDP , with a finite state space S , referred to as the latent state space , a finite action space
A , initial latent state s init , horizon H , transition function T :
[ H ] × S ×
A → ∆( S ) , and reward function R :
[ H ] × S ×
A → [0 , 1] ; transition probabilities and reward function values are written as T h ( s ′ | s, a ) and R h ( s, a ) .
The second component of the block MDP triple is an observation space X , which is typically large and possibly infinite.
The final component is an emission function q : S → ∆( X ) , which describes the conditional distribution over observations given any latent state; its values are written as q ( s | x ) .
We assume that q satisfies the block MDP assumption, meaning that there exist disjoint sets {X s } s ∈S such that if x ∼ q ( · | s ) then x ∈ X s with probability 1.
This means that there exists a perfect decoder ϕ
M : X → S that maps an observation x to the unique state ϕ M ( x ) that emits it.
An agent interacts with a block MDP in a sequence of episodes, each generated as follows:
Initially, s
= s init .
At each step h = 1 , . . .
, H , the agent observes x h ∼ q ( · | s h ) , then takes action a h , accrues reward r h
=
R h
( s h , a h ) , after which the MDP transitions to state s h +1 ∼ T h ( ·| s h , a h ) .
The agent does not observe the latent states s h , only the observations x h and rewards r h .
Observations and actions up to step h are denoted x 1: h and a 1: h .
We denote the block MDP describing the target environment by M ⋆ = ⟨ M ⋆ , X , q ⋆ ⟩ where
M ⋆ = ⟨S , A , s init , H, T ⋆ , R ⋆ ⟩ is the target latent MDP.
The perfect decoder for M ⋆ is denoted as ϕ ⋆ .
Practicable policy.
Behavior of an agent in a target environment is formalized as a (non-Markovian) practicable policy .
A practicable policy prescribes which action to take given any sequence of observations, i.e., it is a mapping π : X ≤ H
→ A , where X ≤ H = ∪ Hh =1 X h .
The action taken by π on x 1: h is written π h ( x 1: h ) .
The expected sum of rewards in M , when actions a h are chosen according to a practicable policy π , i.e., when a h = π h ( x 1: h ) , is referred to as the value of π in M and denoted
V π M = E M ,π [ r 1 + r 2 + · · · + r H ] ; the subscript in the expectation signifies the probability distribution over episode realizations when the environment follows M , and actions are chosen according to π .
We also allow practicable policies to be (Markovian) mappings π : [ H ] × X
→ A which choose every action according to the last observation, so that a h = π h ( x h ) for all h .
Whether a particular practicable policy is Markovian or not will generally be clear from context.
3.2 Abstract simulator
Our learning algorithm can interact with the target environment, but it additionally has access to an abstract simulator , which provides an idealized and abstracted version of the target environment.
Formally, an abstract simulator is an episodic MDP denoted M ◦ = ⟨S , A , s init , H, T ◦ , R ◦ ⟩ , with the same state space, action space, start state and horizon as the latent MDP M ⋆ , but not necessarily the same transition and reward functions.
Furthermore, we assume that abstract simulator is deterministic:
Assumption 1.
The abstract simulator M ◦ is deterministic, i.e., T ◦ h
( s h
+
| s h , a h ) ∈ { 0 , 1 } .
In order for the abstract simulator to be useful, it must approximate target environment.
In this paper we assume that the target environment can be viewed as a “perturbed” version of the abstract simulator, using a notion of perturbation inspired by the concept of trembling-hand equilibria from extensive-form games [Selten, 1975].
Specifically, we say that an MDP M ′ is an η -perturbation of another MDP M if its dynamics can be realized by following M ’s dynamics while distorting agent actions according to some (unknown) “noise” distribution, referred to as ξ in the definition below, which keeps actions unchanged with probability at least 1 − η :
Definition 1 ( η -perturbation) .
We say that an MDP M ′ =
⟨S , A , s init , H, T ′ , R ′ ⟩ is an η -perturbation of an MDP M = ⟨S , A , s init , H, T, R ⟩ if there exists a function ξ : [ H ] × S ×
A → ∆( A ) that satisfies ξ h ( a | s, a ) ≥ 1 − η for all h , s , a , and such that
[Equation]
for all h , s , a , s ′ ; thus MDP M ′ can be viewed as following the dynamics of M in which each action a is stochastically replaced (“perturbed”) according to ξ .
The set of all η -perturbations of M is denoted C ( M, η ) .
We assume that the target environment is an η -perturbation of the abstract simulator for a value of η < 0 .
Thus, most of the time the target environment transitions after each action “as intended” (i.e., following known dynamics of the abstract simulator), but with a probability at most η it may depart from the intended action due to an inherent, but unknown stochasticity:
Assumption 2.
M ⋆ is an η -perturbation of the abstract simulator M ◦ for some η < 0 .
Abstract policy.
Behavior of an idealized agent that can directly access latent state is formalized as a (Markovian) abstract policy .
An abstract policy prescribes what action to take in each state s at a given step h , i.e., it is a mapping ψ : [ H ] × S → A ; we write ψ h ( s ) for the action taken by ψ in step h and state s .
The expected sum of rewards in an episodic MDP M = ⟨S , A , s init , H, T, R ⟩ , when following ψ is called the value of ψ in M and denoted V ψM =
E M,ψ [ r 1 + r 2 + · · · + r H ] .
Robust abstract policy.
Since the latent MDP in the target domain is a perturbation of the abstract simulator, we will seek to obtain policies that are robust to any allowed perturbation.
For a given abstract simulator M ◦ and the perturbation level η , we define a robust abstract policy ρ to be a policy that achieves the largest possible value under the worst-case choice of perturbation:
[Equation]
where Ψ is the set of all mappings from [ H ] × S to A .
In a natural way, a robust abstract policy ρ can be composed with the perfect decoder ϕ ⋆ to obtain a (Markovian) practicable policy ρ ◦ ϕ
⋆ : [ H ] ×X
→ A mapping observations in the target environment to actions while still maximizing the worst-case reward among perturbations of M ◦
We aim for algorithms that find practicable policies that perform almost as well as this robust practicable policy .
3.3 The learning setting
We can now formally define our learning setting.
A learning algorithm Alg in this setting receives as input a deterministic abstract simulator (episodic MDP) M ◦ = ⟨S , A , s init , H, T ◦ , R ◦ ⟩ , meaning it receives the entire MDP represented in tabular form (or some other computationally convenient form).
The algorithm is also provided with oracle access to a target environment (block MDP),
M ⋆ = ⟨ M ⋆ , X , q ⋆ ⟩ .
This means that the algorithm cannot directly access M ⋆ itself, but can interact with it as an agent would, executing actions a h , and receiving back observations x h and rewards r h in a sequence of episodes, as described above.
Finally, Alg is given parameters ϵ > 0 , δ > 0 , and η < 0 .
It is assumed that M ⋆ is an η -perturbation of M ◦ .
The assumption that we know the value of η is not essential and we discuss in Section 4.4 how it can be removed.
After interacting with M ⋆ , the algorithm outputs a practicable policy π .
The goal of learning is for π to have value almost as good as the robust practicable policy with high probability, that is, for V π M ⋆ ≥ V ρ ◦ ϕ ⋆ M ⋆ − ϵ with probability at least 1 − δ (where probability is over the algorithm’s randomization as well as randomness in the interactions with the target environment).
Furthermore, we require the number of episodes executed by the algorithm before outputting a policy π to be bounded by a polynomial in the number of actions |A| , the horizon H , 1 /ϵ , 1 /δ , and 1 / (1 − 2 η ) .
Note importantly that this polynomial must have no explicit dependence on the number of states |S| or observations |X| in the target environment.
An algorithm that satisfies these criteria (given the stated assumptions) is said to achieve an efficient transfer from abstract simulator .
Main Algorithm
Our main contribution is an algorithm TASID , which achieves efficient transfer from an abstract simulator.
The algorithm operates in two stages.
In the first stage, it determines the robust abstract policy ρ for the provided abstract simulator via robust dynamic programming (Algorithm 1).
In the second stage, it interacts with the target environment (via oracle access) in order to learn to predict the current latent state based on the history of observations and actions.
The learnt decoding map is then composed with the robust abstract policy to obtain the practicable policy that is returned by the algorithm (see Algorithm 2).
4.1 Robust dynamic programming for abstract simulator
We obtain the robust abstract policy by instatiating the robust dynamic programming algorithm of Bagnell et al.
[2001] and Iyengar [2005] to our specific notion of perturbation.
The algorithm proceeds by filling out values of the robust value function ˜ V , which quantifies the largest sum of rewards achievable starting at any step h and state s , when assuming the worst-case perturbation of the input MDP M ◦ .
Specifically,
[Equation]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] Follow π 1: for h − 1 steps to observe x
[Table]
Take action a h uniformly at random and observe
[Table] [Table] [Equation] (cid:80)
[Equation]
[Equation] [Equation] [Equation] // Learn “inverse dynamics” 11:
Define α h : X 2 → A as // Define “shadow action” decoder α h ( x h , x h +1 ) = argmax [Equation] f
[Equation] + [Equation]
Define ϕ h
+1 :
X h +1 → S such that //
Define state decoder ϕ h {++
+}
( x 1: h +1 ) is the state s h +1 reached in M ◦ , when starting in s init and executing a[-(cid:48) 
 = α 1 ( x 1 , x 2 ) , . .-]
[Equation]
, x
[Equation] [Equation]
Define π h +1 : X h +
→ A as // Define practicable policy
[-h +
 ( x 1: h +1 ) = ρ h-][Equation]
14: return π = ( π 1 , . .
. , π H )
Similar to standard dynamic programming, the value function in robust dynamic programming can be filled out beginning with h = H + 1 , where we have ˜ V h ( s ) = 0 , and proceeding backward.
In our case, the values ˜
V h ( s ) can be obtained from
˜ V
h +1 ( s ) using a closed-form expression (line 4), which leverages intermediate values ˜ Q
h ( s, a ) .
Note that ˜ Q is not quite the robust state-action value function, because it assumes that the action at step h is left unperturbed (and only considers the worst-case perturbation in the following steps).
The function ˜ Q is used to derive the robust policy (line 5).
Theorem 1.
For any abstract simulator M ◦ and any perturbation level η , Algorithm 1 returns the robust policy ρ = argmax ψ ∈
Ψ min M ∈C ( M ◦ ,η ) V ψM .
(The proof of this theorem and all other proofs in this paper are deferred to the appendix.)
4.2 Learning a decoder in the target environment
We cannot directly apply the robust abstract policy in the target environment, because the latent states are not observable.
Therefore, we construct a “state decoder” in the target environment, which uses the history of observation in an episode to predict the current latent state; the state decoder is combined with the robust abstract policy to obtain the practicable policy.
Formally, a (non-Markovian) state decoder is a mapping ϕ : X ≤ H → S .
In step h of an episode, the history of previous observations x 1: h is used as an input, and the decoder predicts the latent state s h .
To construct such a state decoder, we crucially leverage the abstract simulator M ◦ .
The abstract simulator M ◦ is deterministic, so a specific sequence of actions a 1 , . .
. , a h always leads to the same state.
The latent MDP in the target environment is a perturbation of the abstract simulator.
This means that when an agent takes action a h in step h , most of the time the environment transitions according to T ◦ ( · | s h , a h ) , but sometimes (with probability at most η ) it transitions according to some other action.
The action a h gets replaced with some “shadow” action a ′ h according to an unknown noise distribution ξ , and the latent state then transitions according to T ◦ ( · | s h , a ′ h ) .
If we knew shadow actions a ′ 1 , . .
a ′ h , we could then recover the current latent state by simulating that same sequence of actions in the abstract simulator.
To obtain shadow actions, we learn an “inverse dynamics” model, which predicts a ′ h from the observations x h and x h +1 (an approach also used in previous work on block MDPs).
We learn a separate inverse dynamics model for each step of an episode.
In step h , we sample triplets of the form ( x h , a h , x ′ h ) across multiple episodes in the target environment, and then fit a model f h for the conditional probability of a h given x h and x ′ h .
The model f h ( a h | x h , x ′ h ) is referred to as an inverse dynamics model, because it “inverts” the dynamics represented by the transition function.
We show that if we were able to obtain an exact model f ⋆h of the conditional probability, then the action a with the largest probability f ⋆h ( a | x h , x ′ h ) would be the correct shadow action.
As is standard in the block MDP literature, in order to fit an inverse dynamics model, we assume access to an optimization algorithm capable of fitting functions from some class F ⊆ {X 2 → ∆( A ) } to data; we call this algorithm an optimization oracle for
F .
The class F should be sufficiently expressive to approximate the required conditional probability distribution.
We now have all the pieces required to describe our algorithm.
The algorithm first constructs the robust abstract policy (line 1), and defines the practicable policy π 1 at the initial step h = 1 , where the latent state is known to be s init (line 3).
The algorithm then proceeds iteratively to fill in π 2 , . .
. , π H .
In iteration h , the algorithm first learns the inverse dynamics model f h with the help of the optimization oracle (line 5–10).
The inverse dynamics model is then used to obtain the shadow action decoder α h (line 11), which predicts which action caused the transition from x h to x h +1 .
Using the shadow action decoders up to step h , we can construct a state decoder ϕ h +1 (line 12), which for a given history of observations x 1: h +1 , first predicts their corresponding shadow actions a ′ 1 , . .
. , a ′ h
and then uses the abstract simulator to determine the state s h +1 that they lead to.
Finally, using the state decoder ϕ h +1 , we define the practicable policy at the step h
+ 1 to return the same action as the abstract robust policy would return on the decoded state (line 13).
4.3 Sample Complexity of TASID
We next provide the sample complexity analysis of TASID , showing that it indeed achieves efficient transfer from an abstract simulator.
In addition to Assumptions 1 and 2, we also need to ensure that the function class F is expressive enough to approximate the conditional probability distribution being fitted by the inverse dynamics model.
It turns out that this target probability distribution can be expressed in terms of the transition function of the block MDP M ⋆ , which is the function
T ⋆ :
[ H ] × X × A → ∆( X ) equal to
[Equation]
Using T ⋆ , we can write the exact inverse dynamics model as f ⋆h ( a | x, x ′ ) =
[Equation]
To state our assumption, let P h denote the distribution over triples ( x h , a h , x h +1 ) sampled in line 7-8, and let (cid:101)
f h be the maximizer (in F ) of the expected log likelihood under P h :
[Equation]
Assumption 3 (Approximate realizability) .
There exists ϵ F ≥ 0 such that for every h ∈ [ H ] , and for all triples ( x, a, x ′ ) in the support of P h
[Equation]
Exact realizability assumptions (with ϵ F = 0 ) are standard in the block MDP literature
[Du et al., 2019, Agarwal et al., 2020].
Here we only assume approximate realizability, which can be assured for any ϵ F > 0 by choosing a sufficiently expressive class such as deep neural networks.
In practice, we choose classes that express some inductive biases (such as feature invariance, factorization, etc.).
We are now ready to state our main theoretical result—sample complexity of TASID .
Theorem 2.
Let M ◦ be a deterministic abstract simulator.
Let M ⋆ be a target environment for which M ⋆ is an η -perturbation of M ◦ for some η < 0 .
Let F be a class of functions that satisfies Assumption 3 with ϵ F ≤ (1 − 2 η ) 2 8 H 4 |A| 3 .
Then for any ϵ > 0 and δ ∈ (0 , 1) , Algorithm 2 with oracle access to M ⋆ , optimization-oracle access to F , and inputs M ◦ , η , ϵ/ 2 , and δ , executes n = O (cid:0) H 3 |A| 3 ln( |F| /δ ) ϵ (1 − 2 η )(cid:1) episodes and returns a practicable policy π that with probability at least 1 − δ satisfies
[Equation]
Thus, Algorithm 2 achieves efficient transfer from an abstract simulator, meaning that its sample complexity is independent of the sizes of |S| and |X| .
We also achieve a fast rate, O (1 /ϵ ) , with respect to the sub-optimality ϵ .
The assumption on ϵ
F is not restrictive, because we can always choose a sufficiently expressive class.
Note that previous works simply assume ϵ F =
[e.g., Du et al., 2019,
Agarwal et al., 2020, Misra et al., 2020].
The dependence on ln |F| is standard; for neural networks it is proportional to the number of bits to store the network weights.
For example, if the neural network has 1000 weights represented by doubles (i.e., with 64 bits) then ln |F| =
(ln 2) · 64000 .
There may be room to improve dependence on the horizon H and action space size |A| , but these were not our focus here.
4.4 Extensions
In the appendix, we show how TASID , under additional assumptions, can be extended to settings where the abstract simulator has a stochastic start state (but deterministic transitions).
The assumption that η is known is not necessary.
If η is not known, it is possible to carry out a form of binary search as follows:
Run the algorithm with η = 1 / 4 and verify whether it achieves the performance guarantee.
If it does, continue with η = 1 / 8 (splitting the interval (0 , 1 / 4] ), otherwise continue with η = 3 / 8 (splitting the interval (1 / 4 , 1 / 2) ), and run the algorithm again.
If the goal is to recover η up to the precision ϵ η , then we just need to repeat this log 2 (1 / 2 ϵ η ) times.
Although our focus here is on transfer from an abstract simulator, and we emphasize the relative ease of constructing abstract simulators, our approach could also be used for transfer learning between two block MDP environments that share latent space structure.
In the source domain, we could use an existing block MDP approach [e.g., Misra et al., 2020] to infer latent structure, which could then be used as an abstract simulator to speed up learning in the target domain using TASID .
Experiments
We evaluate TASID in two simulation environments, focusing on four of its properties: sample complexity independent of state-space size; scalability to complex visual observations; robustness of the policy it learns; and scalability to large state spaces.
We summarize the results here and defer the details, including hyperparameter selection, detailed environment descriptions, algorithm and baselines implementation and a full description of results under various setups to the appendix.
Can TASID solve problems that require strategic exploration?
We theoretically showed that
TASID can solve problems that require performing strategic exploration using a small number of episodes.
We test this empirically on a challenging environment called combination lock .
Combination lock.
We first describe the abstract simulator M ◦ visualized in Figure 2a.
It has an action space A , horizon H , and a state space { ( h, i ) | h ∈ { 0 } ∪ [ H ] , i ∈
[3] } , with the initial state (0 , 1) .
As we will see, states { ( h, 1) , ( h, 2) | h ∈ [ H ] } are good states from which optimal return is possible, while states { ( h, 3) | h ∈
[ H ] } are bad states.
In state ( h, 1) , one good action leads to state ( h + 1 , 1) , and all other |A| − 1 actions lead to state ( h + 1 , 2) .
In state ( h, 2) , one good action leads to state ( h + 1 , 2) , another good action leads to state ( h + 1 , 1) , and all other |A| − 2 actions lead to ( h + 1 , 3) .
All actions in state ( h, 3) lead to state ( h + 1 , 3) .
The identities of good actions are unknown and are different for different states.
Any transition to state ( H, 1) gives a reward of 9 . 5 , whereas transition to state ( H, 2) gives a reward of 10 , and transition to state ( H, 3) gives a reward of 0 .
To “mislead” the agent, transitions from ( h, 2) to ( h + 1 , 3) give a reward of 1 and transitions from ( h, 1) to ( h + 1 , 1) give a reward of − 1 /H .
All other transitions give a reward of 0.
H,
H, H,
R =
R =  1 /H
R =9 .
R =
R =  1 /H
R =
R =  1 /H (0 , 1) <latexit sha1_base64="Yrx1AM5ors3cfUcNx3CEgNN4SOs=">AAAB7XicdVDLSgMxFM3UV62vqks3wSJUkCHTsbbdFd24rGAf0A4lk2ba2MxkSDJCGfoPblwo4tb/ceffmD4EFT1w4XDOvdx7jx9zpjRCH1ZmZXVtfSO7mdva3tndy+8ftJRIJKFNIriQHR8ryllEm5ppTjuxpDj0OW3746uZ376nUjER3epJTL0QDyMWMIK1kVpFdAad036+
[Table]
/HtvIEgS0kgTjpXqOijWXoqlZoTTaa6XKBpjMsZD2jU0wiFVXjq/dgpPjDKAgZCmIg3n6veJFIdKTULfdIZYj9Rvbyb+5XUTHVS9lEVxomlEFouChEMt4Ox1OGCSEs0nhmAimbkVkhGWmGgTUM6E8PUp/J+0Srbj2qWb80L9chlHFhyBY1AEDqiAOrgGDdAEBNyBB/AEni1hPVov1uuiNWMtZw7BD1hvn8XGjfU=</latexit> (1 , 1) <latexit sha1_base64="0pjEpEIVhIIq0jWMWhZrX73zns0=">AAAB7XicdVDLSgMxFM3UV62vqks3wSJUkCHTsbbdFd24rGAf0A4lk2ba2MxkSDJCGfoPblwo4tb/ceffmD4EFT1w4XDOvdx7jx9zpjRCH1ZmZXVtfSO7mdva3tndy
[Table]
[Figure]
fduX5AkpJEmHCvVcVCs
[Figure]
vZn4l9dJ9KDipyyKE00jslg0SDjUAs5eh30mKdF8YggmkplbIRlhiYk2AeVMCF
[Figure] fwv9J07Udz3Zvzgu1y2UcWXAEjkEROKAMauAa1EEDEHAHHsATeLaE9Wi9WK [Figure] L1oy1nDkEP2C9fQLL4I35</
[Figure] The latent MDP M ⋆ of the target environment is constructed by taking a random η -perturbation of M ◦ for η = 0 .
Exploration in both M ⋆ and M ◦ can be difficult due to misleading rewards and challenging dynamics, where most actions lead to bad states.
The optimal policy in M ◦ finishes in the state ( H, 2) and achieves the value of 10.
However, the robust policy will attempt to visit the state ( H, 1) which lies on a more stable trajectory.
Meanwhile, the optimal policy in M ◦ will fail in M ⋆ , because perturbations are likely to move the agent into a bad state.
Observations in the target environment are real-valued vectors of dimension 2 ⌈ log 2 ( H +4) ⌉ .
For a latent state ( h, i ) , the observation is generated by first creating an ( H + 4) -dimensional
vector by concatenating one-hot encodings of h and i , element-wise adding Gaussian noise, applying a fixed coordinate permutation, and finally multiplying the vector with a Hadamard matrix.
We compare TASID with four baselines.
The first is PPO [Schulman et al., 2017], a policy-gradientbased algorithm.
The second is PPO-RND, which augments PPO with an exploration bonus based on prediction error [Burda et al., 2019b].
These two baselines do not use abstract simulator and run on the target environment from scratch.
The other two baselines, PPO+DR and PPO+RND+DR, enhance PPO and PPO+RND with domain randomization (DR), where the policy is pre-trained on a set of randomized block MDPs [Tobin et al., 2017].
PPO+DR and PPO+RND+DR are designed for transfer RL, but unlike our work, they rely on observation-based similarity.
During the pre-training phase of domain randomization, we generate block MDPs by following a similar process as the one used to generate the target environment.
However, when generating the emission function, we only generate a random permutation matrix and the other components are kept the same as in the target environment.
This randomization set includes the true emission function and transition dynamics.
For baseline algorithms, we run grid search over hyperparameters listed in Table 3 in Appendix D, separately for each environment specification (each value of H ), and report the best results of
PPO(+RND)(+DR).
For TASID , we consider only one hyperparameter, the number of training episodes per time step n D , and search over three possible values: 1000 , 2500 , 10000 .
Figure 2b plots the size of the problem (expressed as horizon H ) that each algorithm can solve (defined as reaching at least 95% of the value of the robust policy) within a given number of episodes; we considered H = 5 , 10 , 15 , 20 , 25 , 40 .
The plot shows that PPO and PPO+DR fail to solve the problem beyond the smallest size ( H = 5 ), which is not surprising as they are not designed to perform strategic exploration.
PPO+RND can solve the problem until H = 20 , but fails for
H = 25 , showing that RND bonus helps to an extent but cannot solve harder problems.
Interestingly,
PPO+RND+DR underperforms compared to PPO+RND even though it has access to pre-training.
We believe this is because RND reward bonus does not work well with domain randomization which can mislead the RND bonus by randomizing the observation.
Finally, TASID can solve the problem for all values of H and is more sample efficient than the baselines.
Can TASID scale to complex observation spaces?
We evaluate TASID in the visual MiniGrid environment [Chevalier-Boisvert et al., 2018] with noisy observations.
We test on the grid world
[Figure]
shown in Figure 3b.
This type of random noise has been studied in the literature and has been shown to pose challenge for RL algorithms
[Burda et al., 2019a].
In this grid world, the agent can either use the top route to reach the goal, or a bottom route.
The top route is shorter but the agent is more likely to visit the lava due to perturbations.
Therefore, the robust policy prefers the bottom route which is longer but more robust.
How robust is TASID to different η ?
Our theoretical analysis suggests that the suboptimality scales as O (1 / (1 − 2 η ) 2 ) .
We evaluate this error empirically in the grid world environment for various values of η in [0 . 1 , 0 . 5] and show the results in Figure 3c.
The performance matches the theoretical prediction of V ρ ◦ ϕ
⋆ − O (1 / (1 − 2 η ) 2 ) until η = 0 . 3 , and becomes even better for larger η .
Interestingly, even though the theoretical guarantee is vacuous at η = 0 . 5 , the algorithm still learns a non-trivial policy with the value greater than that of a random policy.
Note that we tune the value of n D rather than use the value suggested by our theory.
How does TASID ’s performance scale with |S| ?
We showed theoretically that TASID ’s sample complexity to learn a near-optimal robust policy is independent of the size of the state space.
We empirically test this by varying the height of the grid world environment while keeping the width and the horizon H , and overall layout the same.
Results are presented in Figure 3d.
As height changes from 10 to 100 , the size of the state space increases ten times; however, we do not see a significant drop in the performance of TASID when using a fixed number of episodes in the target environment.
Conclusion and Future Work
We presented a new algorithm TASID for transfer RL in block MDPs that quickly learns a robust policy in the target environment by leveraging an abstract simulator.
We have also proved a sample complexity bound, which does not scale with the size of the state space or the observation space.
Finally, we demonstrated theoretical properties by empirical evaluation in two domains.
Our work raises many important questions.
Our theoretical analysis depends on the assumption that the abstract simulator is deterministic and the the target environment is its perturbation.
Is it possible to relax these requirements and still obtain sample bounds that do not scale with the state space size?
One direction might be to leverage “full simulators” (including observations), and incorporate observation similarity between the simulator and the real environment, along with similarity in the latent dynamics.
Another important question is how to extend the presented approach to continuous control problems in robotics, which are natural domains for sim-to-real transfer.
A Proofs for Algorithm
We introduce the definition of the (single-step) perturbed transition and reward set, following the definition of C ( M ◦ , η ) .
Definition 2 ( η -perturbation) .
We say that a transition and reward function pair ( T ′ , R ′ ) at step h is an η -perturbation of the pair ( T h , R h ) if there exists a function ξ : S × A → ∆( A ) that for all s, s ′ and a satisfies ξ ( a | s, a ) ≥ 1 − η and
[Equation]
With slight abusing of the notation, the set of all η -perturbations of T h , R h is denoted C (( R h , T h ) , η ) .
It is straightforward to see that ( R ′ , T ′ )
∈ C
(( R h , T h ) , η ) if and only if there exists an MDP
M ′ ∈ C ( M, η ) such that ( R ′ , T ′ ) is the h -th step reward and transition functions in M ′ and ( R h , T h ) is the h -th step reward and transition functions in M .
Now we prove a lemma about the function ˜ Q h used in Algorithm 1.
Lemma 1.
For any h ∈ [ H ] , and s ∈ S h , the learned function ˜ Q h in Algorithm 1 satisfies:
[Equation]
Proof.
By the definition of C (( R ◦ h , T ◦ h ) , η ) , we know that
[Equation]
where min over ξ ( ·| s
[Equation]
a ) is taken so that it still satisfies the requirement of ξ ( a | s, a ) ≥ 1 − η .
This is then used in the last equality [Equation] where the minimum is achieved by placing the maximum allowed mass, which is η , on the action with smallest value of ˜ Q h ( s, a ′ ) and remaining on ˜ Q
h ( s, a ) .
Thus we proved that
[Equation]
The proof is completed by taking maximum over actions on both side.
Completing the Proof of Theorem
Now we prove the Theorem 1 as the main result for Algorithm 1.
Proof.
By Theorem 2.2 in Iyengar [2005], since Q h and V h satisfy Lemma 1, we have that ˜ V 1 ( s ) = sup ψ ∈ Ψ min M ∈C ( M ◦ ,η ) V ψM, 1 ( s ) .
The perturbation we defined consists of independent perturbation at different time step h , therefore it satisfy the “rectangularity” condition defined in Iyengar [2005].
From Iyengar [2005], the robust policy is given by
[Equation]
This is how Algorithm 1 computes the policy it returns.
Therefore, Algorithm 1 returns the robust policy ρ .
A.1 An alternative Proof for Algorithm 1 from the First Principles
The previous proof relies on a more general results from Iyengar [2005] in a more complicated form.
However the proof can be simplified since our perturbation set takes a particular linear structure.
So for completeness and readability, we also include another complete proof derived from the first principle.
We acknowledge that the proof is heavily inspired by previous work in robust dynamics programming [Bagnell et al., 2001, Iyengar, 2005].
Let Ψ be the set of all mappings from [ H ] × S to ∆( A ) .
The following proposition follows directly from application of max-min inequality.
Proposition 1.
∀ h ∈
[ H ] , s ∈ S
[Equation]
Given this, we are going to explain the high level structure of the proof.
For ˜ V h , we are going to show that
[Equation]
[Equation]
(12) for some ˜ ψ .
Then from Proposition 1, we will have that
[Equation]
Then we will have that all the inequalities must be equality and the ˜ ψ must be the robust policy ρ .
Now we will show that Equation 11 and Equation 12 holds for the ˜ ψ which is the output from
Algorithm 1, in the following two lemmas.
Lemma 2.
For any s ∈ S and h ∈ [ H ] , ˜ V h ( s ) ≥ min M ∈C ( M ◦ ,η ) V ⋆M,h ( s )
Proof.
For any h ∈ [ H ] , we will construct a M ∈ C ( M ◦ , η ) such that ∀ s, ˜ V h
( s ) =
V ⋆M,h ( s ) .
Then for any s ∈ S , ˜ V h ( s ) =
V ⋆M,h ( s ) ≥ min M ∈C ( M ◦ ,η ) V ⋆M,h ( s ) and we prove this lemma.
Now we prove the following statement by induction on h .
For any h ∈ [ H + 1] , ∃ M ∈ C ( M ◦ , η ) such that [Equation]
s [Equation]
[Equation]
For the basement of induction, ˜ V H +1 ( s ) =
[-+
-]V ⋆M,h +1 ( s ) = [-
-]0 for any M .
Thus the induction statement holds for h = H + 1 .
Second, let us assume that the induction statement holds for h + 1 , where h ∈ [ H ] : ∀ s, ˜
V h
+1 ( s ) =
V ⋆M h
+1 ,h +1 ( s ) for some M h +1 ∈ C ( M ◦ , η ) .
Notice that this statement is for the value function at h + 1 step and holds for M h +1 , thus for any M ∈ C ( M ◦ , η ) that shares the reward and transition functions on and after step h +1 , the statement also holds.
This is because in our definition of episodic
MDP the perturbation class the reward and transition functions at different steps are independent.
Now we are going to construct M h such that ˜ V h ( s )
=
V ⋆M h ,h ( s ) .
More specifically, we will only construct the reward and transition functions at h -th step: R h and T h , and concatenate it with other component in M h
+1 .
For any s , let a be argmin a ∈A ˜ Q h ( s, a ) .
Given the s and a , construct T h ( s ′ | s, a ) and R h ( s, a ) for any s, a, s ′ as the following.
[Equation]
Then we finish the proof of statement for h by
[Equation]
By induction, we proved that there exist a M ∈ C ( M ◦ , η )
such that
˜
V
h
( s ) =
V ⋆M,h ( s ) for any s ∈ S , h ∈
[ H + 1] .
Thus we finish the proof of ˜ V h ( s ) ≥ min M ∈C ( M ◦ ,η ) V ⋆M,h ( s ) .
Lemma 3.
For the policy ˜ ψ computed from Algorithm 1,
[Equation]
[Equation] ∈ [Equation]
Proof.
We prove it by induction on h from H + 1 to 1 .
In the base case ( h = H + 1 ) we have,
˜ V
H +1 ( s ) =
= V ˜ ψM,H +
( s ) for all s ∈ S and any M .
This implies: ∀ s, ˜ V
H
+
( s ) ≤ min M ∈C ( M ◦ ,η ) V ˜ ψM,H +1 ( s ) .
We now consider the general case.
We assume that ∀ s ∈ S , ˜ V h
+1 ( s ) ≤ min M ∈C ( M ◦ ,η ) V ˜ ψM,h +1 ( s ) .
Fix any s ∈ S and any M ∈ C ( M ◦ , η ) , we are going to show that ˜ V h ( s ) ≤ V ˜ ψM,h ( s ) .
Let R h and
T h be its reward and transition function at h -th step, and ξ h be the perturbation variables.
We have 1 − ξ h ( a | s, a ) ≤ η by the definition of M ∈ C ( M ◦ , η ) .
So 1 − ξ h ( ˜ ψ ( s ) | s, ˜ ψ (
[Equation] [Equation]
≤ η .
[Equation]
[Equation]
Since this is for any M ∈ C ( M ◦ , η ) , we proved that ˜ V h ( s ) ≤ min M ∈C ( M ◦ ,η ) V πM,h ( s ) .
Now we prove the Theorem 1 as the main result for Algorithm 1.
Proof.
We have proved Equation 11 and Equation 12 holds for the ˜ ψ which is the output from
Algorithm 1, in previous two lemmas.
Now combine this with the inequality of min max values, we have that
[Equation]
So we will have that all the inequalities must be equality and the ˜ ψ output by Algorithm 1 must be the robust policy ρ defined in Equation 1.
B Proofs for Algorithm
In this section, we are going to prove a less restrictive and more general version of Theorem 2 stated below.
We allow the ϵ F to take any values and the initial state in the true environment has a small probability {+ϵ 
+} when it is not equals to the s init .
We show the how the error bounds scales with these approximation errors.
Theorem 3.
Let M ◦ be a deterministic abstract simulator.
Let M ⋆ be a target environment for which M ⋆ is an η -perturbation of M ◦ for some η < 0 .
Let {+ϵ 
+} be the probability that s 0 ̸ = s init in
M .
Let F be a class of functions realizable with respect to M ⋆ .
Then for any ϵ > 0 and δ ∈ (0 , 1) ,
Algorithm 2 with oracle access to M ⋆ , optimization-oracle access to F , and inputs M ◦ , η , ϵ , and δ , executes n =
[Equation]
(cid:1) 3 episodes and returns a practicable policy π that with probability at least 1 − δ satisfies
[Equation]
The proof to Theorem 3 has three steps.
First we show the accuracy of learned ERM classifier of the classification problem in line 10.
Second we show the accuracy of the learned action decoder
α h
+
( x h , x h +1 ) under the roll-in distribution of x h .
Last the error bound in the theorem can be proved by bounding the union probability of failing to predict the action in each steps.
Now we give some lemmas and proofs in these three steps in the following three subsections, followed by the proof of the final sample complexity results.
To prove Theorem 2 in the main paper, we set the ϵ in Algorithm 2 to be ϵ/ 2 , let ϵ F satisfy the assumptions in Theorem 2, and let [-(cid:15) 
-]ϵ 0 = 0 .
Thus the right hand side in Theorem 3 can be becomes ϵ and we proved Theorem 2.
B.1 Accuracy of action classification
First we define some notation that will be used in the proofs for TASID (Algorithm 2).
Let the uniform distribution over action space be denoted Unf ( A ) .
We will use Unf ( a ) = 1 / |A| to denote the probability over action a given by uniform distribution.
We define P h ( x ) to be the distribution from which x h is sampled from in Line 9 in the h th iteration of Algorithm 2.
Formally, P h ( x h ) is the probability of receiving observation x h at time step h when sampling the first h − 1 actions according to the learned practicable policy π 1: h − 1 in the target environment (when h
= 1 , then x 1 is directly emitted from the start state, i.e., x 1 ∼ q ( · | s init ) ).
Finally, we denote the joint distribution
P h ( x, a, x ′ ) :
=
P h ( x ) × Unf ( A ) × T ⋆h ( x ′ | x, a ) from which ( x h , a h , x h +1 ) are sampled from in Line 9.
Now we show the form of conditional distribution of action given two observations, f ⋆h ( a | x, x ′ ) , under the joint distribution P h ( x, a, x ′ ) .
Moreover, we prove the form of f ⋆h ( a | x, x ′ ) for any roll-in distribution of the first observation x is the same.
Lemma 4.
Given any joint distribution P ( x, a, x ′ ) =
P ( x ) Unf ( a ) T ⋆h ( x ′ | x, a ) , where P ( x ) is any prior distribution over x and Unf ( a ) denotes the uniform distribution over actions, the posterior distribution over action a given pair of observation x, x ′ is:
[Equation]
Proof.
We start by applying the Bayes’ rule.
[Equation]
where the last equality uses T ⋆h ( x ′ | x, a ′ )
=
q ( x ′ | ϕ
⋆ ( x ′ ))
T ⋆h ( ϕ ⋆ ( x ′ )
| ϕ ⋆ ( x ) , a ′ ) for all a ′ ∈ A , which follows from the block MDP structure of M ⋆ .
Note that, in particular, Lemma 4 holds for the joint distribution P h ( x, a, x ′ ) that we described earlier.
Now we can show the error bound of empirical log-likelihood maximizer.
Before that, we introduce some useful lemmas from Agarwal et al.
[2020] for the completeness of this paper.
Lemma 5 (Lemma 24 in Agarwal et al.
Let D be a dataset of n ( x, a, x ′ ) samples, and let
D ′ be another independent dataset from the same distribution.
Let L ( f, D ) =
(cid:80) ni
=1 ℓ ( f, ( x i , y i )) be any function that decomposes additively across examples where ℓ is any function, and let ˆ f ( D ) be any estimator taking as input random variable D and with range F .
Then, with probability 1- δ ,
[Equation]
Lemma 6 (Lemma 25 in Agarwal et al.
For any two conditional probability densities f 1 , fand any distribution D ∈ ∆( X × X ) we have
[Equation]
The proof of the empirical log-likelihood maximizer’s error bound follows the proof in Agarwal et al.
[2020] as well.
However we modified the proof to only assume approximate realizability assumption instead of perfect realizaibility.
Theorem 4.
Let D h be a data set with n D transitions (x,a,x’) sampled i.i.d.
from P h ( x, a, x ′ ) .
Let f h ( a | x, x ′ ) and (cid:101)
f h ( a | x, x ′ ) denote the maximizers of empirical log-likelihood and expected log-likelihood respectively:
[Equation]
Then for any δ ∈ (0 , 1) , with probability at least 1 − δ we have that:
[Equation]
Proof.
We set L ( f, D ) in Lemma 5 to be (cid:80)
[Equation]
x,a,x ′ ) ∈D − 12 log
[Equation]
| x
[Equation]
x ′ )
f h ( a | x,x ′ ) where D is D h and D ′ to be another I.I.D. dataset with n D samples drawn independently with D . With this choice, the right hand side is
[Equation]
[Equation]
since f h is the empirical maximum likelihood estimator and f ⋆h ∈ F .
The left hand side is
[Equation]
By Assumption 3, we have that
[Equation]
f
[Equation]
a |
[Equation]
x ′ ) is upper bounded by √ 1 + ϵ F uniformly overall x, a, x ′ .
So we have that
[Equation]
The last steps follows from Lemma 6.
After combine this with the right hand side and rearrange the terms, we have
[Equation]
B.2 One-step accuracy of action decoder
In the last subsection, we have showed the learned function f h approaches the posterior distribution of action at a rate of 1 / n D .
In Algorithm 2, the learned function f h is used to build the state decoder α h .
This state decoding process relies on identified the correct “shadow” actions.
In this section we first show that there is a separation between the correct shadow action’s probability and random actions’ probabilities, in the posterior distribution of action.
This relies on the transition dynamics in target environment lies in C ( T ◦ , η ) .
Then we show the action decoding accuracy.
The idea to bound the 1-step action decoding accuracy is to apply the Markov’s inequality on the event that the argmax action of f h is wrong.
First, we define the set of shadow actions given two successive states.
Definition 3 (Shadow action set) .
For any states pair ( s, s ′ ) , we define a set of actions
A h ( s, s ′ )
= { a ∈ A : T ◦ h ( s ′ | s, a ) = 1 } and A ch ( s, s ′ ) =
A\ A h ( s, s ′ ) .
Since the observation emission function q ⋆ maps different s to disjoint observation subspace, the shadow action sets can also be defined on the corresponding observation pairs x, x ′ .
We define
A h ( x, x ′ )
= { a ∈ A : T ◦ h ( ϕ ⋆ ( x ′ )
| ϕ ⋆ ( x ) , a ) = 1 } , and similarly for A ch ( x, x ′ )
By definition, we have that A h ( x, x ′ )
=
A h ( s, s ′ ) if x ∼ q ⋆ ( ·| s ) and x ′ ∼ q ⋆ ( ·| s ′ ) .
So later we may use these two notations interchangeably for convenience.
Next we prove the accuracy result of decoder α under the joint distribution with any practicable policy, but learned from the dataset from uniform action distribution.
We use 1 I as an indicator function of random events.
Lemma 7 (Accuracy of decoder) .
For any practicable policy π , let P h,π ( x, a, x ′ ) :
=
P h ( x ) π ( a | x ) T ⋆ ( x ′ | x, a ) be the joint distribution of P h ( x ) , policy π and transition function T ⋆ .
Let D h be a data set with n D transitions (x,a,x’) sampled i.i.d.
from P h ( x, a, x ′ ) .
For any h ∈ [ H ] and δ ∈ (0 , 1) , and any practicable policy π , with probability at least 1 − δ , we have that
[Equation]
Proof.
For any a 1 ∈ A h ( x, x ′ ) , and a 2 ∈ A ch ( x, x ′ ) ,
[Equation]
Given that the gap is (1 − 2 η ) / |A| , we have that for any fixed x, x ′ , if ∥ f h ( · , x, x ′ ) − f ⋆h ( · , x, x ′ )
∥ TV < ( − 2 η ) / 2 |A| , then argmax f h ( · , x, x ′ ) ∈
A h ( x, x ′ ) .
Recall that α h ( x, x ′ )
= argmax a ∈A f h ( a | x, x ′ ) .
We have
E P h [1 I ( α h ( x, x ′ ) /
∈
A h ( x, x ′ ))]
=
Pr ( α h
[Equation]
The second to last step follows from Markov’s inequality, and the last step follows from Theorem 4.
Notice that this is the error under distribution of uniform action P h ( x, a, x ′ ) :
= P ( x ) ◦ Unf ◦ T ⋆h .
For any practicable policy π , since P h,π ( x,a,x ′ ) P h ( x,a,x ′ )
=
π ( a | x ) 1 / |A| ≤ |A| ,
[Equation]
Then taking the complement of the event in the indicator function finished the proof.
B.3 Analysis of the sample complexity
Now we are going prove that the learned policy recover the input latent policy with high probability.
Lemma 8.
For any h ∈ [ H ] , let x 1: h +1 :
= x 1 , x 2 , . .
, x h +1 and a 1: h :=
a 1 , a 2 , . .
. , a h be the state and action sequence generated from ( π 1 , . . .
, π h ) output by Algorithm 2.
Given the high probability event in Theorem 4, we have that for any h ∈
[ H ]
[Equation]
Proof.
If the action decoder α k is correct, i.e. α k ( x k , x k +1 )
∈
A k ( x k , x k +1 ) for any k ∈ [ h ] .
Then if the initial state is s init , we have that for any k ∈ [ h ] , ϕ k ( x 1: k ) =
ϕ
⋆ ( x k +1 ) due to the determinism of the abstract simulator.
Then for any k ∈ [
[Equation]
That means if we have a k ̸
= ρ ( ϕ ⋆ ( x k )) [Equation] we must have α k ( x k , x k +1 )
/ ∈
A k ( x k , x k +1 ) for some j ≤ k , or the initial state is not s init .
So under the condition that the initial state is s init ,
[Equation]
Thus we have
[Equation]
This immediately gives the following theorem by letting h = H and bounding the value gap for non-optimal actions by H .
The proof of Theorem 2 follows from this.
Proof.
We first bound the value gap using preivous lemma and the Performance Difference Lemma [Kakade, 2003].
[Equation]
Finally
[Equation]
we can solve the sample complexity by denote the value gap ϵ , and finish the proof
[Equation]
C Analysis with Stochastic Initial State
We can extend TASID to a more general setting where initial state can be stochastically chosen, instead of being deterministic.
This setting captures problems such as navigation in a set of house simulators, where dynamics of each house simulator can be deterministic but the choice of initial state, i.e., choice of current house and position of the agent inside the house, can be stochastically chosen.
As Algorithm 1 does not rely on the deterministic initial state, therefore, we only need to show that we can extend Algorithm 2 to the stochastic initial state setting, and find the robust policy learned by Algorithm 1.
First, we introduce the difference in problem settings and our main results under this setting formally.
We assume that in both abstract simulator M ◦ and the target environment M ⋆ , the initial states are sampled from the same distribution µ over a finite set of initial states S 1 of size |S 1 | = N .
We make an assumption that each initial state occurs with a reasonable probability and has a different transition dynamics.
Assumption 4.
(Conditions on initial state)
For all initial states s ∈ S 1 , we assume µ ( s ) ≥ µ min for some µ min ∈ (0 , 1] .
Further, there exists a margin Γ > 0 such that for any two initial states s, ˜ s ∈ S 1 we have:
[Equation]
Informally, this assumption is required so that we can visit each initial state sufficiently, and use the margin assumption to learn an accurate decoder to cluster initial states.
However, note that we cannot directly cluster in the observation space since we do not want to make any additional structural assumptions on it.
In contrast, we will use a function-approximation approach where we interact with the observation via a function class.
We present a variation of TASID in Algorithm 3 that can address stochastic initial state.
The algorithm assumes access to an approximate decoder ˆ ϕ : X → N that can cluster observations from the same initial state together.
This decoder can be thought of partitioning the observation space into decoder states which recover the true initial states upto relabeling.
In Section C.1 we discuss how to learn this decoder using the Homer algorithm Misra et al.
Algorithm 3 learns a mapping from these decoder states to initial states in S 1 by performing a hypothesis testing algorithm that uses Algorithm 2 in the main paper as a sub-routine.
In Section C.2 we discuss this hypothesis test.
We will prove that under a realizability assumption (stated later), this algorithm has the following guarantee.
state s an episodes counter cnt ( · ) and a value logger v ( · the state cluster i calls return17: else 18: s ← s + 1 19: if s = N + 1 then 20:
Run TASID for one episode on initial state argmax s v ( s ) 21: return argmax s v ( s ) 22: else 23: returnTheorem 5.
With the assumption about ϵ F in Theorem 2, Algorithm 3 will execute a policy that is close to robust policy by ϵ in all but poly (cid:110) N, H, A, 1 ϵ ,
[Equation]
episodes with probability at least 1 − δ .
Note that when there is a deterministic initial state, i.e., N = 1 and µ min = 1 , we recover dependence on the same set of parameters as our main result.
For some problems, N maybe significantly smaller than the set of all states.
For example, a robot may start an episode from its charging station and there maybe a small number of charging stations in the environment.
For these problems, the dependence on N may be acceptable.
In the second part, we propose a hypothesis testing based algorithm, that use Algorithm 2 in the main paper as a sub-routine, and prove the new sample complexity.
C.1 Learning decoder initial states
We use the Homer algorithm
[Misra et al., 2020] to learn the decoder ˆ ϕ .
We briefly describe the application of this algorithm for time step h = 1 .
Homer collects a dataset D of n quads as follows: we sample y uniformly in { 0 , 1 } and collect two independent transitions ( x (1) , a (1) , x ′ (1) ) , ( x (2) , a (2) , x ′ (2) ) at the first time step by taking actions a (1) and (2) uniformly.
If y = 1 then we add ( x (1) , a (1) , x ′ (1) , y ) to D , otherwise, we add ( x (1) , a (1) , x ′ (2) , y ) .
Note that ( x (1) , a (1) , x ′ (2) ) is an unobserved transition, therefore, we call it an imposter transition, whereas, ( x (1) , a (1) , x ′ (1) ) is a real transition.
We know that there are exactly N initial states since we have access to the simulator.
Given a bottleneck function class Φ : { ϕ : X → [ N ] } and another regressor class G : { f : [ N ] × A × X → [0 , 1] } , we train a model to differentiate between real and imposter transition as follows:
[Equation]
Difference from Misra et al.
While our approach and analysis in this subsection closely follows Misra et al.
[2020], we differ from them in two crucial ways.
Firstly, we apply bottleneck on x instead of x ′ , since we want to recover a decoder for initial states.
Secondly, Misra et al.
[2020] do not assume a margin assumption Γ since their approach does not concern with recovering an exact decoder, but only in learning a good set of policies for exploration.
We will denote the function class G ◦ Φ = { g ◦ ϕ : ( x, a, x ′ )
(cid:55)→ g ( ϕ ( x ) , a, x ′ )
| g ∈ G , ϕ ∈ Φ } .
Let D ( x, a, x ′ ) be the marginal distribution over real and imposter transitions, and let ρ ( x ′ ) =
E x ∼ µ,a ∼ unf ( A )
[ T ( x ′ | x, a )] be the marginal probability over x ′ for real transitions where µ is the initial state distribution.
It can be shown that:
[Equation]
where T ( x ′ | x,a ) µ ( x ) / |A| is the probability of observing a real transition ( x, a, x ′ ) and ρ ( x ′ ) µ ( x ) / |A| is the probability of observing an imposter transition ( x, a, x ′ ) and the factor of 1 / 2 comes due to uniform selection over real and imposter transition.
Misra et al.
[2020] showed that the Bayes optimal classifier of the prediction problem is given by:
Lemma 9 (Bayes Optimal Classifier) .
For any ( x, a, x ′ ) in support of D , we have:
[Equation]
Proof.
See Lemma 9 of Misra et al.
Similar to Misra et al.
[2020], we make a realizability assumption stated below that allows us to solve the classification problem well.
Assumption 5 (Realizability) .
We assume that g ⋆ ∈ G ◦ Φ .
We can use the realizability assumption to get the following generalization bound guarantee: for any δ ∈ (0 , 1) we have:
[Equation]
with probability at least 1 − δ , where C ( G ◦ Φ) is a complexity measure for class G ◦ Φ such as ln( |G|| Φ | ) or Rademacher complexity.
For proof see Proposition 11 and Corollary 6 in Misra et al.
Note that even though their proof uses a bottleneck model ϕ on x ′ instead of x , essentially the same argument holds by symmetry.
Coupling Distribution.
We define a coupling distribution as
[Equation]
where D ( x 1 | a, x ′ ) is the conditional distribution derived from the joint distribution D ( x 1 , a, x ′ ) defined earlier.
We also define the marginal distribution D ( a, x ′ ) which gives us D ( x 1 | a, x ′ )
=
D ( x 1 ,a,x ′ ) / D ( a,x ′ ) .
We present some result related to the distributions defined above that will be useful later for proving important results later.
[Equation]
Using Equation 69 we can prove:
[Equation]
Similarly
[Equation]
we can prove:
[Equation]
Further [Equation] we have:
[Equation]
which gives us:
[Equation]
We now state a useful lemma
[Equation]
Lemma 10.
Fix δ ∈ (0 , 1) .
Then with probability at least 1 − δ we have
[Equation]
Proof.
We use triangle inequality to decompose the left hand side as:
[Equation]
The first term is bounded as shown below:
[Equation]
where the second inequality uses Equation 73 and Equation 67.
The second term is bounded as:
[Equation]
where the inequality results from following similar steps used for bounding the first term.
Adding the two upper bounds we get 4∆
[Equation] Using Lemma 9, we have for every x 1 , x 2 , x ′ ∈ X , a ∈ A :
[Equation]
We use this to prove the following result:
Lemma 11.
With probability at least 1 − δ we have:
[Equation]
Proof.
Let’s define a shorthand E = 1 { ˆ ϕ ( x 1 )
=
ˆ ϕ ( x 2 ) } .
Starting with left hand side of Lemmawe get:
[Equation]
using Equation
[Equation]
where the last inequality uses
[Equation]
∥ TV which is either zero when ϕ ⋆ ( x 1 ) =
ϕ ⋆ ( x 2 ) or at least Γ .
Combining these two conditions we get a lower bound of 1 { ϕ ⋆ ( x 1 ) ̸ = ϕ
⋆ ( x 2 ) } Γ on TV distance.
The proof is then completed with application of Lemma
[Equation]
Theorem 6.
(Initial State Clustering Result).
Let N > 1 and let ∆ <
[Equation]
Then there exists a bijection mapping σ : S 1 → [ N ] such that with probability at least 1 − δ :
[Equation]
Proof.
We will use i ∈
[ N ] to denote a decoder state defined by { x | ˆ ϕ ( x ) =
i, x ∈ X 1 } and s ∈ S 1 to denote a real state defined by { x | ϕ ⋆ ( x ) =
s, x ∈ X 1 } .
For any i, s we can bound the left hand side of Lemma 11 as:
[Equation]
where the second last step follows from observing that x 1 and x 2 are sampled independently.
We define a few shorthands: Pr( i ) =
Pr( ˆ ϕ ( x ) = i ) , Pr( s ) =
Pr( ϕ ⋆ ( x ) = s )
=
µ ( s ) and Pr
( i, s ) =
Pr( ϕ ⋆ ( x ) =
s, ˆ ϕ ( x ) = i ) .
This combined with above and Lemma 11 gives us:
[Equation]
We define a mapping σ : S 1 → [ N ] as follows:
[Equation]
This gives us:
[Equation]
Since ∆ ′ can be brought arbitrarily small
[Equation]
we will assume ∆ ′ <
Pr( σ ( s )) 2 / 4 which allows us to write:
[Equation]
By definition of σ ( s ) we have:
[Equation]
where the first inequality uses the fact that maximum of a set of values is greater than its average
[Equation]
and the last inequality uses Assumption 4.
We now place another condition on ∆ ′ , namely,
[Equation]
which implies that:
[Equation]
This eliminates Equation 86.
Hence the Pr( σ ( s ) , s ) must satisfy Equation 85 which can be simplified as:
[Equation]
where the third step uses √ 1 − y ≥ 1 − y for y ∈ [0 , 1] and that 4∆ ′ / Pr( σ ( s ))
2 < 1 from constraints on ∆ ′ , and the last step uses Pr( σ ( s )) ≥ Pr( σ ( s ) , s ) ≥ µ min / N (Equation 87).
We can finally prove our main result as:
[Equation]
What is left is to show that σ ( s ) is a bijection mapping and collect all constraints on ∆ ′ .
Let s and s ′ be two initial states such that σ ( s )
=
σ ( s ′ )
=
k .
We then get:
[Equation]
where the last equality follows from Equation 84.
Further, we have Pr( k, s ) ≥ µ min / N and Pr( k, s ′ ) ≥ µ min / N from Equation 87.
This gives us µ 2 min N 2 ≤ ∆ ′ .
Hence, if ∆ ′ < µ 2 min N 2 , then, we cannot have two different initial states mapping to the same decoder state.
Further, as |S 1 | = N , hence the map σ : S 1 → [ N ] is a bijection.
Finally, we made three constraints on ∆ ′ .
The first is ∆ ′ <
Pr( σ ( s )) 2 / 4 , second is Equation 88, and third is ∆ ′ <
µ 2 min N 2 .
Note that Equation 88 already implies that ∆ ′ <
Pr( σ ( s )) 2 / 4 .
Hence, the overall constraint on ∆ ′ is:
[Equation]
We can simplify this constraint by making it tighter using Pr( σ
[Equation]
s ))
[Equation]
[Equation]
Note that we are assuming here that N ≥ 2 and, therefore, µ min < 1 , which implies 2 µ min / N ≤ µ min < 1 .
When N = 1 , we can trivially align the single initial state.
This completes the proof.
This allows us to separate all initial states at time step h
= 1 with high probability.
C.2 Aligning the learned decoder states
The only thing left is aligning learned decoder states in the target domain with simulator initial states, which we do as follows.
At a high level Algorithm 3 first clusters the initial observation into clusters, then each time it start with a cluster, it run a subroutine, InitialStateTest, to test the latent state index of that cluster.
The testing algorithm Algorithm 4, run our main algorithm TASID the hypothesis of the state index from 0 to N .
By the analysis of TASID , we know that if the hypothesis is correct, it will learn a policy with nearly robust value.
Thus for each cluster, with in Nn episodes the algorithm will find the nearly robust policy.
Theorem 7.
If ϵ F ≤ (1 − 2 η ) 2 ϵ 4 H 4 |A| {+
+} we learn the initial state decoder ˆ ϕ with n 0 samples such that
[Equation]
Algorithm 3 will execute a policy that is close to robust policy by 5 ϵ in all but
O
[Equation]
episodes with probability at least 1 − 3 δ .
Proof.
We prove this theorem by two steps.
First, we show that for each cluster
ˆ ϕ ( x ) , after we run
Algorithm 4 with N ( n l + n t ) steps we can find the correct state of that cluster.
Second, we show that once we find the correct state, we run TASID and learn a policy at most 4 ϵ worse than the robust policy.
First, by the definition of n 0 , we have that ∆ :
[Equation]
(cid:1) 2 (cid:17) for N ≥ 2 and η < 1 .
By Theorem 6, we have for each i ∈
[ N ] , there must exist a state σ ( i ) ∈
[ N ] such that with probability at least 1 − δ ,
[Equation]
Thus if we run Algorithm 4 InitialStateTest for s = σ ( i ) .
Then by Theorem 3, we have that the value of learned policy π is at least
[Equation]
with probability at least 1 − δ/N 2 .
By Hoeffding’s inequality, we know that the Monte-Carlo estimates v ( s ) of the learned policy value with N t samples is at least
[Equation]
with probability at least 1 − δ/N 2 .
Thus let ˆ s =
argmax
s v ( s ) and π be the corresponding learned policy from TASID given hypothetical initial state ˆ s .
The policy value v π is at least
[Equation]
with probability 1 − 2 δ/N by taking the union bound on all N possible hypothetical initial states.
Thus, for a given state cluster, after run Algorithm 4 InitialStateTest for N ( n l + n t ) episodes, the policy value is at least V ρ ◦ ϕ ⋆ M − 5 ϵ .
That means for each state cluster, we make at most N ( n l + n t ) mistakes.
Thus we make at most N 2 ( n l + n t ) mistakes in total during running Algorithm 4.
Notice that we also need n 0 samples to learn the decoder ˆ ϕ .
The total number of episodes we may make mistake on is therefore
[Equation]
Taking the union bound over all initial state clusters ˆ ϕ ( x ) and the high probability statement in
Theorem 6, we have that with probability 1 − 3 δ
the statement holds.
We finish the proof by plugging in n l , n t , n 0 .
D Experiment Details D.1 Experiment Details in Combination Lock
Domain details We describe the details of the combination lock domain here.
The deterministic
MDP is described in section 5.
The transition dynamics and reward functions in the target domain is defined by the the coefficients ξ ( a ′ | s, a ) and Definition 1.
For each s, a , we set ξ ( a ′ | s, a ) :
= ηp ( a ′ | s ) + (1 − η ) {+
+}
I ( a ′ = a ) where p ( ·| s ) is a random probability mass distribution and each probability is drawn uniformly from [0 , 1] and then normalized.
We set η = 0 .
1 in the experiment.
The only exception is that all transitions from state ( H, 2) are not perturbed in the true target domain.
This settings makes the robust policy without this knowledge not optimal in this specific instance, but our goal is still learning the robust policy here.
The observation mapping is defined as below.
Let v s be a 2-sparse encoding in R H +4 of the state ( h, i ) where the first 3 bits is a 1-sparse encoding of i and last H + 1 bits is a 1-sparse encoding of h .
The the observation o is computed by
[Equation]
where perm is an arbitrary permutation of [ H + 4] , and H is the 2 ⌈ log [-
-]2 ( H +4)
⌉ by 2 ⌈ log 2 ( H +4)
Hadamard matrix, consists of 2 ⌈ log 2 ( H +4) ⌉ mutually orthogonal columns.
The permutation perm ( v ) operator shuffles the dimensions of the vector v .
E.g., if perm denotes the permutation (3 , 1 , 2) , then perm ( v ) =
( v 3 , v 1 , v 2 ) .
We create a Hadamard matrix H n of size n × n for n = 2 l for some l ∈ N ∪ { 0 } using Sylvester’s construction.
4 If H +4 < 2 ⌈ log 2 ( H +4) ⌉ , then the vector perm ( v s + N (0 , 0 . 01)) will be padded with zeros to ensure it is of size 2 ⌈ log 2 ( H +4) ⌉ .
Our constructions are significantly based on previous hard combination locks studied in Du et al.
[2019], Misra et al.
Implementation details of TASID
We describe the details of TASID and hyper-parameters below.
We model the action predictor class F by a two-layer MultiLayer Perceptron (MLP) with Leaky
ReLU activations in this domain.
The input to the MLP is the concatenation of observations x and x ′ .
This is processed through two non-linear layers, and finally a softmax operation is performed to generate probabilities over actions.
We implement the model in PyTorch and train it using Adam optimization.
The h + 1 -th action predictor, i.e., ˆ f h +1 is initialized with the parameter from h -th action predictor ( f h ).
We initialize the first action predictor ( f 1 ) using PyTorch’s default initialization.
For action predictor we remove 20% of the n D datapoints for use as a held-out validation set and use the remaining for training parameters of the MLP.
We perform 100 epochs of training and evaluate the current model on the validation set after each epoch.
We stop training if the validation loss does not increase for 10 consecutive epochs.
We save the model after each epoch, and use the model with the smallest validation loss.
The only hyperparameter we search for TASID is n D whose values we search over the grid { 1000 , 2500 , 10000 } .
For all algorithms we report results with best hyperparameter for every value of H .
For every value of H and hyperparameter setting, we run the experiment 5 times with different seeds.
We select the best hyperparameter as one that takes the least median number of episodes to achieve a moving average return of 0 .
95 v ρ , where ρ is the latent robust policy and v ρ is its value in the target domain.
We compute median across 5 seeds.
In case of a tie, we look at the average return over the entire course of training and across all seeds.
The hyper-parameters of training neural networks is the same across our methods and all baseline methods, and are specified in Table 2 on page 31.
[Table]
and RND are listed in
Table 3.
We perform a grid search over the entropy coefficient in PPO and the coefficient of RND bonus.
The grid search we use for PPO+RND experiments contains 15 different hyperparameter choices (5 for
RND bonus and 3 for entropy coefficient).
This results in 15 × 5 = 75 experiments for every value of H .
In contrast, the grid search for TASID only contains 3 different hyperparameter choices.
To avoid further increasing the search space, we only use the best PPO+RND hyperparameters for every value of H when running the domain randomization experiments.
The best hyperparameter values are chosen using the procedure described earlier with sole exception we use compute episodes needed to achieve a moving average return of 0 .
5 v ρ instead of 0 .
95 v ρ .
This weaker metric is more helpful since the latter often gives infinities.
For domain randomization pretraining, we first train in the source domain with 5 × 10 5 episodes.
During this training, every 100 episodes, we randomize the domain uniformly by regenerating a permutation function and the perturbation in transitions ξ ( ·|· , · ) , in the same way as how the target domain is generated.
We assume that the domain randomization algorithm knows η (same as TASID ), and everything about the observation mapping except the specific permutation function and transition perturbation (not known by TASID ).
Rather than uniform domain randomization, EPOpt [Rajeswaran et al., 2017] uses only episodes with a return smaller than lower ϵ -quantile in the batch ( 100 episodes) during domain randomization pretraining.
We searched for values of ϵ in { 0 . 1 , 0 . 2 , 0 . 5 , 1 . 0 } where 1 . 0 gives uniform domain randomization.
Therefore, the hyperparameter grid search for domain randomization has 4 different choices.
Details of the experiment results We report the details of experiments we used to generate Figure 2b in the paper.
We run each algorithm with 5 random seeds for a maximum of 5 × 10 5 number
[Table]
of episodes in the target domain
, saving checkpoints every 1000 episodes.
We decide whether the combination lock is solved or not if the algorithm can achieve a moving average return of 0 .
95 v ρ .
In
Figure 2b we report the median of the number of episodes needed to solve the combination lock with horizon H .
The results of each random seeds is list in the following table.
[Table]
∞
PPO+RND
[Table]
∞ , [Equation] [Equation] [Equation]
[Table]
In the main paper, Figure 2c shows the reward curves for different algorithms with horizon equalsin the target environment.
Here we include the reward curves for all values of horizon in Figure 4.
It shows our algorithm stably learned a robust policy with number of samples that is smaller than baseline algorithms.
Though baseline algorithms aim to learn the near optimal policy, they did not converge to a policy that is significantly better than the robust policy.
Amount of compute We run our experiments on a cluster containing mixture of P40, P100, and
V100 GPUs.
Each experiment runs on a single GPU in a docker container.
We use Python3 and
[Figure]
Pytorch 1.4.
We found that for H=40 and V100 GPUs, on average, POTAS took 2.5 hours, PPO + DR took 14 hours, and PPO + RND + DR took 31 hours.
Performing domain randomization increased the computational time by a factor of 2.
D.2 Experiment Details in MiniGrid
Domain details We use the code of the MiniGrid environment [Chevalier-Boisvert et al., 2018] under the Apache License 2.0.
In MiniGrid, the agent is placed in a discrete grid world and needs to solve different types of tasks.
We customized the lava crossing environment in several ways.
We first create a shorter but more narrow crossing path, as the optimal path in deterministic environment.
Then we construct a longer but more safe path under perturbation.
The map of the mini-grid is shown in Figure 3a.
The height of the map can be changed without changing the problem structure and the optimal and robust policy.
The state consists of the x - y coordinate of the position, a direction that takes four values, and the time step.
The agent only observe a 7 × 7 area in front of it.
(see
Chevalier-Boisvert et al.
[2018] for details.)
We set the observation to be the visual map (RGB image) with a random noise between (50 , 50 , 50) and (150 , 150 , 150) for all background pixels.
The action space is changed to having five actions: moving forward, turning left, turning right, turning right and moving forward, turning left and moving forward.
We set the horizon to three times the number of steps the robust policy needs to reach the goal in the deterministic environment.
Implementation details of TASID We describe details of TASID and hyper-parameters below.
We implement the action predictor model class by a convolutional neural network (CNN).
We take the two images x and x ′ and concatenate them along the channel dimension.
This concatenated image is passed through CNN which applies two convolutional layers with ReLu activations followed by a flattening the feature and applying a linear layer to reduce it to a vector of size the number of actions.
Finally, we apply a softmax layer to generate probabilities over actions.
We train the algorithm with 5 × 10 5 episodes, for all values of η and gridworld height.
We run each experiment 5 times and report averaged results in Figure 3.
Shaded bands in the Figure correspond to the 95% Student’s t confidence intervals.
Other hyperparameters for minigridworld experiments are shown in Table 5.
Amount of compute We run our experiments on a cluster containing P100 GPUs.
Each experiment runs on a single GPU in a docker container.
We use Python3 and Pytorch 1.4.
We found that on P
GPUs, POTAS took 2 to 6 hours (2 hours for η = 0 .
1 and 6 hours for η = 0 . 5 ).
[Table]