M ETA R EINFORCEMENT L EARNING FROM OBSERVATIONAL DATA
Anonymous authors
Paper under double-blind review
A BSTRACT
Pre-training is transformative in supervised learning: a large network trained with large and existing datasets can be used as an initialization when learning a new task.
Such initialization speeds up convergence and leads to higher performance.
In this paper, we seek to understand what the formalization for pre-training from only existing and observational data in Reinforcement Learning (RL) is and whether it is possible.
We formulate the setting as Batch Meta Reinforcement
Learning.
We identify MDP mis-identiﬁcation to be a central challenge and motivate it with theoretical analysis.
Combining ideas from Batch RL and Meta RL, we propose tiMe, which learns dis ti llation of multiple value functions and M DP e mbeddings from only existing data.
In challenging control tasks and without additional exploration on unseen MDPs, tiMe is competitive with state-of-the-art model-free RL method trained with hundreds of thousands of interactions.
I NTRODUCTION
Deep Reinforcement Learning algorithms still require millions of environment interactions to obtain reasonable performance, hindering their applications (Mnih et al., 2015;
Lillicrap et al., 2016; Vuong et al., 2018; Fujimoto et al., 2018b; Jaderberg et al., 2018; Arulkumaran et al., 2019; Huang et al., 2019).
This is due to the lack of good pre-training methods.
In supervised learning, a pre-trained network signiﬁcantly reduces sample complexity when learning new tasks (Zeiler & Fergus, 2013;
Devlin et al., 2018; Yang et al., 2019).
Meta Reinforcement Learning (RL) has been proposed as a framework for pre-training in RL (Wang et al., 2016a;
Duan et al., 2016; Finn et al., 2017).
However, such methods still require the collection of millions of interactions during meta-train, which means that they face the same sample complexity challenge as standard RL algorithms.
In this work, we use the following deﬁnition of pre-training: the ability to use data from a set of tasks to improve performance on unseen, but related tasks.
In supervised learning, a key reason why pre-training is incredibly successful is that the dataset used for pre-training can be collected from naturally occurring large-scale processes.
This removes the need to manually collect data and allows for scalable data collection, resulting in massive datasets.
For example, Mahajan et al.
pre
-trains using existing images and their corresponding hashtags from Instagram to obtain state-of-the-art performance on ImageNet (Russakovsky et al., 2014).
In this paper, we seek to formalize pre-training in RL in a way that allows for scalable data collection.
The data used for pre-training should be purely observational and the policies that are being optimized for should not need to interact with the environment during pre-training.
To this end, we propose Batch Meta Reinforcement Learning (BMRL) as a formalization of pre-training in RL from only existing and observational data.
During training, the learning algorithms only have access to a batch of existing data collected a priori from a family of Markov Decision Process (MDP).
During testing, the trained policies should perform well on unseen MDPs sampled from the family.
A related setting is Batch RL (Antos et al., 2007; Lazaric et al., 2008; Lange et al., 2012), which we emphasize assumes the existing data comes from a single MDP.
To enable scalable data collection, this assumption must be relaxed: the existing data should come from a family of related
MDPs.
Consider smart thermostats, whose goal is to maintain a speciﬁc temperature while minimizing electricity cost.
Assuming Markovian dynamics, the interactions between a thermostat and its environment can be modelled as a MDP.
Data generated by a thermostat operating in a single building can be used to train Batch RL algorithms.
However, if we consider the data generated by the same thermostat operating in different buildings, much more data is available.
While the interactions between the same thermostat model and different buildings correspond to different MDPs, these MDPs share regularities which can support generalization, such as the physics of heat diffusion.
In section 6, we further discuss the relations between BMRL and other existing formulations.
The ﬁrst challenge in BMRL is the accurate inference of the unseen MDP identity.
We show that existing algorithms which sample mini-batches from the existing data to perform Q-learning style updates converge to a degenerate value function, a phenomena we term MDP mis-identiﬁcation .
The second challenge is the interpolation of knowledge about seen MDPs to perform well on unseen
MDPs.
While Meta RL algorithms can explicitly optimize for this objective thanks to the ability to interact with the environment, we must rely on the inherent generalizability of the trained networks.
To mitigate these issues, we propose tiMe, which learns from existing data to dis ti ll multiple value functions and M DP e mbeddings.
tiMe is a ﬂexible and scalable pipeline with inductive biases to encourage accurate MDP identity inference and rich supervision to maximize generalization.
The pipeline consists of two phases.
In the ﬁrst phase, Batch RL algorithm is used to extract MDPspeciﬁc networks from MDP-speciﬁc data.
The second phase distills the MDP-speciﬁc networks.
To summarize, our contributions are three folds: (1) Formulation of Meta RL from observational data as Batch Meta Reinforcement Learning (BMRL) (2) A simple stage-wise approach which works well on standard benchmarks in the BMRL setting (3) Most importantly, demonstration that Meta RL from only observational data is possible.
We hope this work will direct the attention of the meta RL community towards this research direction.
P RELIMINARIES 2. B ATCH R EINFORCEMENT L EARNING
We model the environment as a Markov Decision Process (MDP), uniquely deﬁned as a 5 element tuple M i =
( S, A, T i , R i , γ ) with state space S , action space A , transition function T i , reward function R i and discount factor γ (Puterman, 1994; Sutton & Barto, 1998).
At each discrete timestep, the agent is in a state s , pick an action a , and arrives at the next state s (cid:48) and receives a reward r .
The goal of the agent π is to maximize the expected sum of discounted rewards
J ( π ) =
E τ ∼ π,M i [ (cid:80) ∞ t =
γ t R i ( s t,i , a t,i , s (cid:48) t,i )]
where τ = ( s 0 ,i , a 0 ,i , r 0 ,i , s 1 ,i , a 1 ,i , r 1 ,i , . . . ) is a tra- jectory generated by using π to interact with M i .
We will consider a family of MDPs, deﬁned formally in subsection 2.3.
We thus index each MDP in this family with i .
In Batch RL, policies are trained from scratch to solve a single MDP M i using existing batch of N transition tuples B i =
{ ( s t,i , a t,i , r t,i , s (cid:48) t,i ) | t = 1 , . .
. , N } without any further interaction with M i .
At test time, we use the trained policies to interact with M i to obtain an empirical estimate of its performance J .
Batch RL optimizes for the same objective as standard RL algorithms.
However, during training, the learning algorithm only has access to B i and are not allowed to interact with M i .
2. B ATCH -C ONSTRAINED Q-L EARNING
Fujimoto et al.
(2018a)
identiﬁes extrapolation error and value function divergence as the modes of failure when modern Q-learning algorithms are applied to the Batch RL setting.
Concretely, deep Qlearning algorithms approximate the expected sum of discounted reward starting from a state-action pair E [ (cid:80) ∞ t =
γ t R ( s t , a t , s (cid:48) t ) | s 0 =
s, a 0 = a ] with a value estimate Q ( s, a ) .
The estimate can be learned by sampling transition tuples from the batch and applying the temporal difference update:
[Equation]
The value function diverges if Q fails to accurately estimate the value of π ( s (cid:48) ) .
Fujimoto et al.
(2018a) introduces Batch-Constrained Q-Learning, constraining π to select actions that are similar to actions in the batch to prevent inaccurate values estimation.
Concretely, given s (cid:48) , a generator
G outputs multiple candidate actions { a m } m .
A perturbation model ξ takes each state-candidate action pair as input and generates small correction term ξ ( s (cid:48) , a m ) for each candidate.
The corrected candidate action a m + ξ ( s (cid:48) , a m ) with the highest value as estimated by a learnt Q is π ( s (cid:48) ) :
[Equation]
Estimation error has also been previously studied and mitigated in model-free RL algorithms (Hasselt, 2010; van Hasselt et al., 2015; Fujimoto et al., 2018b).
2. M ETA R EINFORCEMENT L EARNING
Meta RL optimizes for average return on a family of MDPs and usually assume that the MDPs in this family share S, A, γ .
Each MDP is uniquely deﬁned by a tuple ( T i , R i ) .
A distribution p ( T i , R i ) deﬁnes a distribution over MDPs.
During meta-train, we train a policy by sampling MDPs from this distribution and sampling trajectories from each sampled MDP, referred to as the meta-train MDPs.
During meta-test, unseen MDPs are sampled from p ( T i , R i ) , referred to as the meta-test MDPs.
The trained policy is used to interact with the meta-test MDPs to obtain estimate of its performance.
The choice of whether to update parameters (Finn et al., 2017) or to keep them ﬁxed during meta-test (Hochreiter et al., 2001) is left to the learning algorithms, both having demonstrated prior successes.
2. MDP I DENTITY I NFERENCE WITH S ET N EURAL N ETWORK
A Meta RL policy needs to infer the meta-test MDP identity to pick actions with high return.
Rakelly et al.
(2019) introduces PEARL, which uses a set neural network (Qi et al., 2016;
Zaheer et al., 2017) f
as the MDP identity inference function.
f takes as input a context set c
= { ( s k , a k , r k , s (cid:48) k ) } k
and infers the identity of a MDP in the form of distributed representation in continuous space.
The parameters of f is trained to minimize the error of the critic Q :
[Equation]
where ¯ V is a learnt state value function.
PEARL also adopts an amortized variational approach (Kingma & Welling, 2013; Rezende et al., 2014; Alemi et al., 2016; Kingma & Welling, 2019) to train a probabilistic f , which is interpreted as an approximation to the true posterior over the set of possible MDP identities given the context set.
B ATCH M ETA R EINFORCEMENT L EARNING
Let K be the number of meta-train MDPs, N be the number of transition tuples available from each meta-train MDP, θ be the parameter of the policy, we can formulate Batch Meta Reinforcement
Learning (BMRL) as an optimization problem:
[Equation]
where the learning algorithms only have access to the batch B during meta-train:
[Equation]
We assume we know which MDP each transition in the batch was collected from.
This assumption simpliﬁes our setting and is used to devise the algorithms.
To maintain the ﬂexibility of the formalization, we do not impose restrictions on the controller that generates the batch.
However, the performance of learning algorithms generally increases as the training data becomes more diverse.
MDP identity inference challenge To obtain high return on the unseen meta-test MDPs, the trained policies need to accurately infer their identities (Ritter et al., 2018; Gupta et al., 2018; Humplik et al., 2019).
In BMRL, previously proposed solutions based on Q-learning style updates, where mini-batches are sampled from the batch to minimize TD error, converge to a degenerate solution.
subsection 5.1 provides experimental result that demonstrates the phenomena.
In ﬁnite MDP, this degenerate solution is the optimal value function of the MDP constructed by the relative frequencies of transitions contained in the batch.
We can formalize this statement with the following proposition.
Proposition 1.
Let N ( s, a, s (cid:48) ) be the number of times the triple ( s, a, s (cid:48) ) appears in B (with any reward).
Performing Q-learning on ﬁnite S and A with all the Q -values Q ( s, a ) initialized to 0 and update rule (1) where ( s, a, s (cid:48) , r ) is sampled uniformly at random from B at each step t , will lead the Q -values to converge to the optimal Q -value of the MDP ( S, A, ˆ T, ˆ R, γ ) almost surely as long
[Table]
/Gi/FufMxbC0Y+cwj+wPj8ARkdniQ=</latexit> as α t ≥ 0 , (cid:80) ∞ t =
α t = ∞ , (cid:80) ∞
t =
α 2 t < ∞ , where ˆ
[Equation]
otherwise.
Thus, performing Q-learning style update directly on data sampled from the batch B fails to ﬁnd a good policy because the value function converges to the optimal value function of the wrong MDP.
We refer this phenomena as MDP mis-identiﬁcation .
The proof is shown in subsection A.1.
Interpolation of seen MDPs to unseen MDPs challenge The trained policies need to generalize from the meta-train MDPs to unseen meta-test MDPs.
Meta RL tackles this challenge by formulating an optimization problem that explicitly optimizes for the average return of the meta-trained policy after additional gradient steps in unseen MDPs (Finn et al., 2017; Rothfuss et al., 2018; Nichol et al., 2018).
This is possible thanks to the ability to interact with the environment during metatrain.
However, in the meta-train phase of BMRL, the learning algorithms do not have access to the environment.
We must rely on the inherent generalizability of the trained networks to perform well on the unseen meta-test MDPs.
The key challenge is therefore ﬁnding the right inductive biases in the architecture and training procedure to encourage such generalization.
The need to ﬁnd the right inductive biases in RL was highlighted by Botvinick et al.
; Zambaldi et al.
; Hessel et al.
We note that previous works phrase the need to ﬁnd inductive biases as a means to forgo generality for efﬁcient learning.
In our setting, these two goals need not be mutually exclusive.
L EARNING D ISTILLATION OF VALUE FUNCTIONS AND MDP E MBEDDINGS 4. D ESCRIPTION OF ARCHITECTURE AND TRAINING PROCEDURE
We propose a ﬂexible and scalable pipeline for BMRL.
Figure 1 (left) provides an overview of the pipeline in the simplest setting.
Meta-train comprises of two separate phases.
The ﬁrst phase consists of independently training a value function Q ∗ i for each MDP-speciﬁc batch B i using Batch
RL algorithms.
In the second phase, we distill the set of batch-speciﬁc value functions { Q ∗ i } i into a super value function Q S through supervised learning (Hinton et al., 2015).
Compared to the normal value function, a super value function takes not only a state-action pair as input, but also an inferred MDP identity, and outputs different values depending on the inferred MDP identity.
The pipeline is ﬂexible in that any Batch RL algorithms are applicable in the ﬁrst phase.
Figure(right) illustrates the architecture for the second phase given that the Batch RL algorithm used in the ﬁrst phase is Batch Constrained Q (BCQ) Learning.
As described in subsection 2.2, BCQ maintains three separate components, a learnt value function Q , a candidate action generator G and a perturbation model ξ .
Therefore, the output of the ﬁrst phase consists of 3 sets { Q ∗ i } i , { G ∗ i } i , { ξ ∗ i } i .
[Table]
jointly by θ
[Table]
The second phase distills { Q ∗ i } i , { G ∗ i } i , { ξ ∗ i } i to Q S , G S , ξ S respectively.
The distillation of G and ξ is necessary to pick actions that lead to high return because each learnt value function Q ∗ i only provides reliable estimates for actions generated by G ∗ i and ξ ∗ i , a consequence of the training procedure of BCQ.
In addition to Q S , G S , ξ S , the architecture consists of 3 other networks, f, P, E .
f takes as input a context { ( s k , a k , r k , s (cid:48) k ) } k and outputs a distributed representation of the MDP identity in a ﬁxed-dimension continuous space.
The output of f is an input to Q S , G S , ξ S , P .
E and P predicts s (cid:48)
j , r j given a state-action pair ( s j , a j ) .
P has low capacity while the other networks are relatively large.
During the second phase, all networks are jointly trained end-to-end by the regression losses of predicting s (cid:48) , r and distilling { Q ∗ i } i , { G ∗ i } i , { ξ ∗ i } i .
This is illustrated in details in Algorithm 1.
During meta-test, f is used to infer the identity of the meta-test MDP as a ﬁxed-dimension continuous vector.
The super functions Q S , G S , ξ S are used to pick actions in the meta-test MDPs, using the same procedure as BCQ (subsection 2.2).
The super functions also take as input the inferred
MDP identity.
The key idea behind the approach is a simple stage-wise approach for the Meta RL from observational data problem.
In the second phase, we distill many policies into one for related tasks by jointly learning dis ti llation of value functions and M DP e mbeddings.
We therefore name the approach tiMe.
MDP embeddings refer to the ability to infer the identity of a MDP in the form of distributed representation in continuous space given a context.
4. B ENEFITS OF THE PROPOSED PIPELINE
Inductive biases to encourage accurate MDP identiﬁcation
The ﬁrst inductive bias is the relationship between f and Q S .
They collectively receive as input state-action pair ( s j , a j ) and context { ( s k , a k , r k , s (cid:48) k ) } k and regress to target value Q ∗ j ( s j , a j ) .
The target for each state-action pair can take on the values within the set { Q ∗ i ( s j , a j ) } i .
Similar state-action pairs can have very different regression targets if they correspond to different meta-train MDPs.
The context is the only information in the input to f and Q S that correlates with which Q ∗ ( s j , a j ) out of the set { Q ∗ i ( s j , a j ) } i f and
Q S should regress to.
Thus, f and Q S must learn to interpret the context to predict the correct value for ( s j , a j ) .
The second inductive bias is the auxiliary task of predicting s (cid:48)
j , r j .
A key design choice is that the network P which takes as input E ( s j , a j ) and f ( { ( s k , a k , r k , s (cid:48) k ) } k ) and predicts s (cid:48) j , r j has low capacity.
As such, the output of f must contain meaningful semantic information such that a small network can use it to reconstruct the MDP.
This is to prevent the degenerate scenario where f learns to copy its input as its output.
To summarize, these two explicit biases in the architecture and training procedure encourage f to accurately infer the MDP identity given the context.
Richness and stability of supervision Previous approaches update f to minimize the critic loss (subsection 2.4).
It is well-known that RL provides sparse training signal.
This signal can also cause instability since the target values in the critic loss change over time.
In contrast, our pipeline provides training signal for f that is both rich and stable.
It is rich because f is trained to infer a representation of the MDP identity that can be used for multiple downstream tasks, such as predicting s (cid:48) , r .
This encourages general-purpose learnt representation and supports generalization.
The training signal is also stable since the regression targets are ﬁxed during the second phase of tiMe.
Scalability The pipeline is scalable in that an arbitrary amount of purely observational data can be used in the ﬁrst phase so long as computational constraints permit.
The data can also be heterogeneous in the sense that they do not need to contain only trajectories with high return.
In the experimental section, we demonstrate the beneﬁt of the approach when the data contains trajectories of varying qualities, some of which were generated by random policies.
The extraction of the batchspeciﬁc networks, such as the batch-speciﬁc value functions { Q ∗ i } i , from the MDP-speciﬁc batches can be trivially parallelized and scales gracefully as the number of meta-train MDPs increases.
E XPERIMENTAL R ESULTS
Our experiments have two main goals: (1) Demonstration of the MDP mis-identiﬁcation phenomena and tiMe’s ability to effectively mitigate it.
(2) Demonstration of the scalability of the tiMe pipeline to challenging continuous control tasks and generalization to unseen MDPs.
In all experiments, the MDP-speciﬁc batch B i is the replay buffer when training Soft Actor Critic (SAC) (Haarnoja et al.
(2018a;b)) in the MDP M i for a ﬁxed number of environment interactions.
Thus, the batch B i contains transitions with varying reward magnitude, some of which were generated by random and poorly performing policies.
While our problem formulation BMRL and the pipeline tiMe allow for varying both the transition and reward functions within the family of MDPs, we consider the case of changing reward function in the experiments and leave changing transition function to future work.
Thus, for the auxiliary prediction task in the second phase of the pipeline,
P only predicts r and not s (cid:48) .
5. T OY E XPERIMENTS
This section illustrates MDP mis-identiﬁcation as the failure mode of existing Batch RL algorithms in BMRL.
The toy setting allows for easy interpretability of the trained agents’ behavior.
We also show that in the standard Batch RL setting, the Batch RL algorithm tested ﬁnds a near-optimal policy.
This means the failure of existing Batch RL algorithm in BMRL is not because of the previously identiﬁed extrapolation issue when learning from existing data (Fujimoto et al., 2018a).
Environment Description In this environment, the agent needs to navigate on a 2d-plane to a goal location.
The agent is a point mass whose starting location is at the origin (0 , 0) .
Each goal location is a point on a semi-circle centered at the origin with radius of 10 units.
At each discrete timestep, the agent receives as input its current location ( x, y ) , takes an action indicating the change in its position (∆ x, ∆ y ) , transitions to a new position ( x + ∆ x, y + ∆ y ) and receives a reward.
The reward is the negative distance between the agent’s current location and the goal location.
The agent does not receive the goal location as input and | ∆ x | ≤ 1 , | ∆ y | ≤ 1 .
Since the MDP transition function is ﬁxed, each goal location uniquely deﬁnes a MDP.
The distribution over MDPs is deﬁned by the distribution over goal locations, which corresponds to a distribution over reward functions.
Batch SAC We modify SAC to learn from the batch by initializing the replay buffer with existing transitions.
Otherwise, training stays the same.
We test Batch SAC on a simple setting where there is only one meta-train MDP and one meta-test MDP which share the same goal location.
This is the standard Batch RL setting and is a special case of BMRL.
Batch SAC ﬁnds a near-optimal policy.
Three meta-train and meta-test MDPs
This experiment has 3 meta-train MDPs with different goal locations.
The goals divide the semi-circle into two segments of equal length.
There are three metatest MDPs whose goal locations coincides with the goal locations of the meta-train MDPs.
This setting only tests the ability of the trained policies to correctly identify the meta-test MDPs and do not pose the challenge of generalization to unseen MDPs.
Batch SAC was trained by combining the transitions from the 3 meta-train MDPs into a single replay buffer and sampling transitions from this buffer to perform gradient updates.
Otherwise, training stays the same as SAC.
Figure(left, middle) illustrates that Batch SAC fails to learn a reasonable policy because of the MDP misidentiﬁcation phenomena.
Batch SAC with task inference function We also tried adding to the architecture of Batch SAC the probabilistic MDP identity inference function as described in subsection 2.4.
This is the equivalent of adapting PEARL (Rakelly et al., 2019) to work in the BMRL setting.
This approach fails to train a policy that performs well on all 3 meta-test MDPs.
We note that the off-policy meta RL setting that the PEARL paper considers and the BMRL setting we consider are solving different problems.
We do not argue that one is easier than the other.
Performance of tiMe Since Batch SAC can extract the optimal value function out of the batch in the single meta-train MDP case, we use it as the Batch RL algorithm in the ﬁrst phase of the tiMe pipeline.
The architecture in the second phase thus consists of E, P, f and Q S .
To pick an action, we randomly sample multiple actions and choose the action with the highest value as estimated by
Q S .
This method is termed random shooting (Chua et al., 2018).
As illustrated in Figure 2 (left, right), tiMe can identify the identities of the three meta-test MDPs and pick near-optimal actions.
5. M UJOCO EXPERIMENTS
Environment Description
This section illustrates the test of tiMe in challenging continuous control robotic locomotion tasks.
Each task requires the application of control action to a simulated robot so that it moves with a particular velocity in the direction of its initial heading.
Formally, the MDP within each MDP family share S, A, T, γ and only differs in R where R is deﬁned to be:
[Equation]
where α and β are positive constant.
A one-to-one correspondence exists between a MDP within the family and a target velocity.
Deﬁning a family of MDP is equivalent to picking an interval of possible target velocity.
This setting is instantiated on two types of simulated robots, hopper and halfcheetah, illustrated in Figure 3.
Experiments are performed inside the Mujoco simulator (Todorov et al., 2012).
The setting was ﬁrst proposed by Finn et al.
Zero-shot meta-test During testing, in contrast to prior works, we do not update the parameters of the trained networks, as is done in gradient-based meta RL, or allow for an initial exploratory phase where episode returns do not count towards the ﬁnal meta-test performance, as is done in off-policy meta RL (Rakelly et al., 2019).
This allows for testing the inherent generalizability of the trained networks without confounding factors.
The meta-test MDPs are chosen such that they are unseen during meta-train, i.e. none of the transitions used during meta-train was sampled from any of the meta-test MDPs.
At the beginning of each meta-test episode, the inferred MDP identity is initialized to a zero vector.
Subsequent transitions collected during the episode is used as the context.
The meta-test MDPs are also chosen to provide wide coverage over the support of the MDP distribution.
This is to test whether our approach generalizes to a variety of meta-test MDPs, or simply overﬁt to a small region inside the support.
Meta-train conditions The target velocities of the meta-train MDPs divide the target velocity interval into equal segments.
This removes the bias of sampling meta-train MDPs when evaluating performance.
The target velocity intervals, episode length, and number of meta-train MDPs for hopper and halfcheetah are [0 . 0 , 2 . 35] and [0 . 0 , 1 . 5] , 1000 and 200, 16 and 29 respectively.
For hopper and halfcheetah, each meta-train MDP has one million and sixty transitions respectively.
Performance analysis Figure 3 illustrates tiMe’s performance on unseen meta-test MDPs.
tiMe is competitive with the state-of-the-art model-free RL methods trained from scratch for one million and sixty thousands
environment interactions in hopper and halfcheetah respectively.
We perform experiments on halfcheetah with an episode length 200 because of computational constraints.
Previous Meta RL works also use an episode length of 200 (Rakelly et al., 2019).
The same network trained with tiMe also performs well in a variety of different meta-test MDPs, demonstrating that it does not over-ﬁt to one particular meta-train MDP.
We compare with SAC to demonstrate BMRL is a promising research direction.
We do not include other Meta RL algorithms as baseline because they would require interacting with the environment during meta-train and thus, is not solving for the problem that BMRL poses.
We tried removing G S , ξ S from the architecture in Figure 1 and picked action with Cross Entropy Method (Rubinstein & Kroese, 2004), but that lead to poor performance because Q S over-estimates the values of actions not generated by G S , ξ S .
Limitations Our approach assumes that the transitions in the batch contain good enough transitions to learn a good policy in the batch RL setting.
However, we note that the data in the batch contain data of varying qualities, some of which were generated by poorly performing policies.
Also, our approach has only been demonstrated to work on tasks where reset are not crucial for exploration needed for task inference, e.g. sparse reward setting.
We leave this venue for future work.
R ELATED W ORKS
Supervised Learning and Imitation Learning The main differences between Batch (Meta) RL and supervised learning are: actions have long-term consequences and the actions in the batch are not assumed to be optimal.
If they are optimal in the sense that they were collected from an expert,
Batch RL reduces to Imitation Learning (Abbeel & Ng, 2004; Ho & Ermon, 2016).
In fact, Fujimoto et al.
(2018a) demonstrates that Batch RL generalizes Imitation Learning in discrete MDPs.
Meta RL Equation 3 is the same objective that existing Meta RL algorithms optimize for (Wang et al.
(2016b);
Finn et al.
We could have formulated our experimental setting as a Partially
Observable MDP, but we chose to formulate it as Batch Meta Reinforcement Learning to ensure consistency with literature that inspires this paper.
The main difference between Meta RL and our formulation is access to the environment during training.
Meta RL algorithms sample transitions from the environment during meta-train.
We only have access to existing data during meta-train.
Context Inference Zintgraf et al.
(2019) and Rakelly et al.
(2019) propose learning inference modules that infer the MDP identity.
Their procedures sample transitions from the MDP during metatrain, which differs from our motivation of learning from only existing data.
Killian et al.
(2017) infers the MDP’s “hidden parameters”, inputs the parameters to a learnt transition function to generates synthetic data and train a policy from the synthetic data.
Such model-based approaches are still outperformed by the best model-free methods (Wang et al.
(2019)), which our method is based on.
Batch RL Fujimoto et al.
(2018a) and Agarwal et al.
(2019) demonstrate that good policies can be learnt entirely from existing data in modern RL benchmarks.
Our work extends their approaches to train policies from data generated by a family of MDPs.
Li et al.
(2004) selects transitions from the batch based on an importance measure.
They assume that for state-action pair in the batch, their value under the optimal value function can be easily computed.
We do not make such assumption.
Factored MDPs In discrete MDP, the number of possible states increases exponentialy in the number of dimension.
Kearns & Koller (2000) tackles this problem by assuming each dimension in the next state is conditionally dependent on only a subset of the dimensions in the current state.
In contrast, our method makes no such assumption and applies to both discrete and continuous settings.
Joint MDP
The family of MDPs can be seen as a joint MDP with additional information in the state which differentiates states between the different MDPs (Parisotto et al., 2015).
Sampling an initial state from the joint MDP is equivalent to sampling a MDP from the family of MDPs.
However, without prior knowledge, it is unclear how to set the value of the additional information to supports generalization from the meta-train MDPs to the meta-test MDPs.
In fact, the additional information in our approach is the transitions from the MDP and the network learns to infer MDP identity.
C ONCLUSION
We propose a new formalization of pre-training in RL as Batch Meta Reinforcement Learning (BMRL).
BMRL differs from Batch RL in that the existing data comes from a family of related
MDPs and thus enables scalable data collection.
BMRL also differs from Meta RL in that no environment interaction happens during meta-train.
We identiﬁed two main challenges in BMRL: MDP identity inference and generalization to unseen MDPs.
To tackle these challenges, we propose tiMe, a ﬂexible and scalable training pipeline which jointly learn distillation of value functions and MDP embeddings.
Experimentally, we demonstrate that tiMe obtains performance competitive with those obtained by state-of-the-art model-free RL methods on unseen MDPs.
A A PPENDIX A. MDP MIS - IDENTIFICATION CONVERGENCE PROOF
Statement : Performing Q-learning on ﬁnite S and A with all the Q -values Q ( s, a ) initialized toand update rule (1) where ( s, a, s (cid:48) , r ) is sampled uniformly at random from B at each step t , will lead the Q -values to converge to the optimal Q -value of the MDP ( S, A, ˆ T, ˆ R, γ ) almost surely as long as α t ≥ 0 ,
(cid:80) ∞ t =
α t = ∞ , (cid:80) ∞
t =
α 2 t < ∞ , where
[Equation]
if
[Equation]
>
[Equation]
1 s
[Equation]
, otherwise.
ˆ R ( s, a, s (cid:48) )
=
  [Equation] s [Equation]
a
[Equation]
s
[Equation]
r
[Equation]
r
N
[Equation]
s
[Equation]
a
[Equation]
s (cid:48) ) , if N ( s, a, s (cid:48) ) >0 , otherwise.
Proof.
First note that for any ( s, a ) ∈ S ×
A such (cid:80) s
(cid:48)(cid:48) ∈
S N ( s, a, s (cid:48)(cid:48) )
= 0 , the initial Q ( s, a ) is already optimal and will never be updated; for all other ( s, a ) ∈ S × A and any s (cid:48) ∈ S , we have
[Equation]
and with probability 1 ,
[Equation]
Then convergence follows from the same argument for the convergence of Q -learning
(Watkins &
Dayan, 1992)
B H YPER - PARAMETERS
The small, medium and large target velocity in hopper corresponds to 0 . 2 , 1 . 1 , 2 . 0 .
The small, medium and large velocity in halfcheetah corresponds to 0 .
The learning rate is 3 e 4 and the Adam optimizer is used in all experiment.
All neural networks used are feed-forward network.
All experiments are performed on machines with up to 48 CPU cores and 4 Nvidia GPU.
All experiments are performed in Python 3 .
7 , mujoco-py 2 . 0 .
5 running on top of mujoco 200 .
All neural network operations are in Pytorch 1 . 2 .
In the toy experiment, Q S consists of 2 hidden layers, each of size 256 .
The inferred MDP size is 32 .
The context size is 1 .
f consists of 1 hidden layers of size 256 .
E consists of 1 hidden layers of size 256 and outputs 256 values.
Same goes for P .
Random shooting was performed withrandom actions at each iterations.
In hopper, the size of the inferred MDP is 8 .
The context size is 1 .
f consists of 3 hidden layers, each of size 256 .
E consists of 4 hidden layers, each of size 16 , and outputs a vector of size 8 .
B consists of 2 hidden layers, each of size 4 .
Q S consists of 8 hidden layers, each of size 128 .
The training mini-batch size is 32 .
When BCQ is ran to extract the value function out of the batch, the same hyper-parameters as found in the ofﬁcial implementation are used https: //github.com/sfujim/BCQ , except the learning rate is lowered from 0 .
003 to 0 .
The alpha and beta in the reward function deﬁnition are 1 . 0 and 0 .
The alive bonus is 1 . 0 .
In halfcheetah, the size of the inferred MDP is 64 .
The context size is 1 .
f consists of 7 hidden layers, each of size 512 .
E consists of 7 hidden layers, each of size 512 , and outputs a vector of size 64 .
B consists of 1 hidden layers, each of size 64 .
Q S consists of 8 hidden layers, each of size 512 .
The training mini-batch size is 64 .
G S consists of 7 hidden layers, each of size 750 .
ξ S consists of 7 hidden layers, each of size 400 .
When BCQ is ran to extract the value function out of the batch, unless otherwise mentioned, the same hyper-parameters as found in the ofﬁcial implementation are used https://github.com/sfujim/BCQ .
The learning rate is lowered from 0 .
003 to 0 .
The perturbation model has 2 hidden layers, of size 400 , 300 .
The critic also has 2 hidden layers, of size 400 , 300 .
The alpha and beta in the reward function deﬁnition are 1 .and 0 .
The alive bonus is 0 . 0 .
In both hopper and halfcheetah, except for the super Q function loss, the terms in the loss L in
Algorithm 1 are scaled so that they have the same magnitude as the super Q function loss.
Graphs for the mujoco experiments are generated by smoothing over the last 100 evaluation datapoints.
The performance on Mujoco was averaged over 5 seeds 0 − 4 .
The hyper-parameters for SAC are the same as those found in the Pytorch public implementation https://github.com/vitchyr/ rlkit .
The standard deviations are averaged over 5000 timesteps during evaluation.
This corresponds to 5 episodes in halfcheetah because there is no terminal state termination in halfcheetah and variable number of episodes in hopper because there is terminal state termination.