R ECURRENT
M ODEL -F REE
RL
IS A S TRONG B ASELINE FOR M ANY POMDP S
Anonymous authors
Paper under double-blind review
A BSTRACT
Many problems in RL, such as meta RL, robust RL, and generalization in RL, can be cast as POMDPs.
In theory, simply augmenting model-free RL with memory, such as recurrent neural networks, provides a general approach to solving all types of POMDPs.
However, prior work has found that such recurrent model-free
RL methods tend to perform worse than more specialized algorithms that are designed for speciﬁc types of POMDPs.
This paper revisits this claim.
We ﬁnd that a careful architecture and hyperparameter decisions yield a recurrent model-free implementation that performs on par with (and occasionally substantially better than) more sophisticated recent techniques in their respective domains.
We also release a simple and efﬁcient implementation of recurrent model-free RL for future work to use as a baseline for POMDPs.
I NTRODUCTION
While reinforcement learning (RL) is often cast as the problem of learning a single fully observable task, also known as MDP, training and testing on that same task, most real-world applications of RL demand some degree of transfer and partial observability.
For example, visual navigation (Zhu et al., 2017) requires adaptation to unseen scenes with occlusion in observations, and human-robot collaboration requires that robots infer the intentions of human collaborators.
(Chen et al., 2018).
Many subareas in RL study problems that are special cases of POMDPs, and we summarize them in Table 1.
For example, meta RL (Duan et al., 2016; Schmidhuber, 1987; Thrun & Pratt, 2012; Wang et al., 2017) is a POMDP where certain aspects of the reward function or (less commonly) dynamics function are unobserved but held constant through one episode.
The robust RL problem (Bagnell et al., 2001; Pattanaik et al., 2018; Pinto et al., 2017; Rajeswaran et al., 2017a) assumes that certain aspects of the dynamics or reward function are unknown, aiming at ﬁnding optimal policies that perform against adversarially-chosen perturbations.
Generalization in RL (Cobbe et al., 2019; Packer et al., 2018; Whiteson et al., 2011; Zhang et al., 2018a) focuses on unobserved aspects of the dynamics or reward function that are novel during testing, using an average-case objective instead of a worst-case objective like robust RL.
Recent work has proposed efﬁcient and performant algorithms for solving these specialized problem settings.
However, these algorithms often make assumptions that preclude their application to other classes of POMDPs.
For example, methods for robust RL are rarely used for the meta RL setting
Nonetheless, many prior works have used a simple baseline that is applicable to all POMDPs: model-free RL equipped with a recurrent policy and (sometimes) value function (Duan et al., 2016; Fakoor et al., 2020; Igl et al., 2018; Packer et al., 2018; Rakelly et al., 2019; Wang et al., 2017; Yu et al., 2019).
We will refer to this approach as recurrent model-free RL .
This baseline is simultaneously simple (requiring changing only a few lines of code from a model-free RL algorithm) and general.
However, prior work has consistently found that recurrent model-free RL performs poorly across a wide range of problem settings, including meta RL (Rakelly et al., 2019; Zintgraf et al., 2020), general POMDPs (Han et al., 2020; Igl et al., 2018), robust RL (Zhang et al., 2021), and generalization in RL (Packer et al., 2018).
One common explanation is that specialized algorithms that are tailored to speciﬁc types of POMDPs are very likely to outperform recurrent model-free RL because they (implicitly) encode inductive biases for solving these speciﬁc tasks.
For example, algorithms for meta RL may leverage the assumption that the underlying dynamics (while unknown) are ﬁxed, and the underlying goals are ﬁxed within one episode (Rakelly et al., 2019; Zintgraf et al., 2020); algorithms for robust RL may assume that the dynamics parameters are known (Rajeswaran et al., 2017a) and dynamics is Lipschitz continuous (Jiang et al., 2021).
This paper challenges this explanation.
We argue that, contrary to popular belief, recurrent modelfree RL is competitive with recent state-of-the-art algorithms across a range of different POMDP settings.
Similar to prior work in Markovian on-policy RL methods (Andrychowicz et al., 2021;
Engstrom et al., 2020), our experiments show that implementation in recurrent model-free RL matters.
Fig.
1 shows a typical scenario in PyBullet occlusion environments (Coumans & Bai, 2016) to support this argument.
Through extensive experiments, we show that the careful design and implementation of recurrent model-free RL is critical to its performance.
Design decisions, such as the actor-critic architecture, conditioning on previous actions and/or rewards, the underlying model-free
RL algorithms, and context length in RNNs, are especially crucial.
The main contribution of this paper is a performant implementation of recurrent-model free RL.
We demonstrate that simple yet important design decisions, such as the underlying RL algorithm and the context length, yield a recurrent model-free RL algorithm that performs on par with prior specialized POMDP algorithms on the environments those algorithms were designed to solve.
Ablation experiments identify the importance of these design decisions.
We also open-sourced our code that is easy to use and memory-efﬁcient.
B ACKGROUND
MDP.
A Markov decision process (MDP) (Bellman, 1957) is a tuple ( S , A , T, T 0 , R, H, γ ) , where
S is the set of states, A is the set of actions, T : S ×
A × S → [0 , 1] is the transition function (dynamics), T 0 : S → [0 , 1] is the initial state distribution, R : S
×
A × S → R is the reward function, H ∈ N is the time horizon, and γ ∈ [0 , 1) is the discount factor.
Solving an MDP requires learning a memoryless policy π : S ×
A → [0 , 1] that maximizes the expected discounted return: π ∗ = arg max π E s t ,a t ,r t ∼ T,π (cid:104)(cid:80)
H − 1 t =
γ t r t +1 | s 0 (cid:105) .
For any MDP, there exists an optimal policy that is both memoryless and deterministic (Puterman, 2014).
MaxEnt RL algorithms (Ziebart, 2010), such as SAC (Haarnoja et al., 2018a), add an entropy bonus to the RL objective.
POMDP.
A partially observable Markov decision process (POMDP) ( ˚Astr¨om, 1965) is a tuple ( S , A , O , T, T 0 , O, O 0 , R, H, γ ) , where the underlying process is an MDP ( S , A , T, T 0 , R, H, γ ) .
Let O be the set of observations and let O : S ×
A × O → [0 , 1] be the emission function.
Let the observable trajectory up to time-step t be τ 0:
t =
( o 0 , a 0 , o 1 , r 1 , . .
. , a t − 1 , o t , r t ) , the memory-based policy in the most general form is deﬁned as π ( a t | τ 0: t ) , conditioning on the whole history.
At the ﬁrst time step t = 0 , an initial state s 0 ∼ T 0 ( · ) and initial observation o 0 ∼ O 0 ( · | s 0 ) are sampled.
At any time-step t ∈ { 0 , . . .
, H − 1 } , the policy emits the action a t ∈ A to the system, the system updates the state following the dynamics, s t +1 ∼ T ( · | s t , a t ) , then the next observation is sampled o t +1 ∼ O ( · | s t +1 , a t ) and the reward is computed as r t
+1 =
R ( s t , a t , s t +1 ) .
We refer to the part of the state s t at current time-step t that can be directly unveiled from current observation o t as the observable state s ot , and the rest part of the state as the hidden state s ht . We call the hidden state s ht stationary if it does not change
[Table]
In this scenario
[Table]
the policy objective can be rewritten as π ∗ =
[Equation]
| s h (cid:105)(cid:105) for the average-case POMDP objective, or π ∗ = arg max π min s h ∈ supp
( T 0 )
{+E s t ,a t ,r t ∼ T,O,O 0 ,π (cid:104)(cid:80) H − 1 t =
+}
γ t r t +1 | s h (cid:105) for the worst-case
POMDP objective.
R ELATED W ORK
In this section, we discuss several subareas of RL that both explicitly and implicitly solve POMDPs, as well as algorithms proposed for these specialized settings.
Table 1 summarizes these subareas.
RL for “Standard” POMDPs.
We use the term “standard” to refer to prior work that explicitly labels the problems studied as POMDPs.
Common tasks include scenarios where the states are partially occluded (Heess et al., 2015), different states correspond to the same observation (perceptual aliasing (Whitehead & Ballard, 1990)), random frames are dropped (Hausknecht & Stone, 2015), observations use egocentric images (Zhu et al., 2017), or the observations are perturbed with random noise (Meng et al., 2021).
These POMDPs often have hidden states that are non-stationary and affect both the rewards and the dynamics.
POMDPs are hard to solve (Littman, 1996; Papadimitriou & Tsitsiklis, 1987) because of the curse of dimensionality: the size of the history grows linearly with the horizon length.
Many prior POMDP algorithms (Cassandra et al., 1994; Kaelbling et al., 1998) attempt to infer the state from the past sequence of observations, and then apply standard RL techniques to that inferred state.
However, the exact inference requires the knowledge of the dynamics, emission, and reward functions, and is intractable in all except the most simple settings.
A common strategy for solving these general POMDPs is to use recurrent policies, which take the entire history of past observations as inputs (Bakker, 2001; Schmidhuber, 1991; Wierstra et al., 2007).
This strategy is very simple and general, and can be applied to arbitrary tasks without knowledge of the task structure (e.g., whether the hidden states change within an episode) across long time horizons (Duan et al., 2016).
These recurrent strategies can be further subdivided into model-free methods (Hausknecht & Stone, 2015; Heess et al., 2015; Meng et al., 2021) , where the single objective is to maximize the return, and model-based methods (Freeman et al., 2019; Han et al., 2020; Igl et al., 2018; Watter et al., 2015) that have explicit objectives on modeling the belief states and use them as the inputs of memoryless policies.
The recurrent model-free RL that we focus on belongs to the class of model-free off-policy memory-based algorithms.
Meta RL.
Meta RL, also called “learning to learn” (Schmidhuber, 1987; Thrun & Pratt, 2012), focuses on POMDPs where some parameters in the rewards or (less commonly) dynamics are varied from episode to episode, but remain ﬁxed within a single episode, which represent different tasks with different values (Humplik et al., 2019).
The meta RL setting is almost the same as multitask RL (Wilson et al., 2007; Yu et al., 2019), but differs in that multi-task RL can observe the task parameters, making it an MDP instead of a POMDP.
Algorithms for meta RL can be roughly categorized based on how the adaptation step is performed.
Gradient-based algorithms (Fakoor et al., 2020; Finn et al., 2017; Hochreiter et al., 2001) run a few gradient steps on the pre-trained models to adapt.
Memory or context-based algorithms use RNNs to implicitly adapt, which can be further subdivided into implicit and explicit task inference methods.
Implicit task inference methods (Duan et al., 2016; Wang et al., 2017) use RL objective only to learn recurrent policies.
Explicit task inference methods (Rakelly et al., 2019; Zintgraf et al., 2020) train an extra inference model to explicitly estimate task embeddings (i.e., a representation of the unobserved parameters) by variational inference.
Task embeddings are then used as additional inputs to memoryless policies.
Robust RL.
The goal of robust RL is to ﬁnd a policy that maximizes returns in the worst-case environments.
Early work in the control and operations research community (Khalil et al., 1996;
Nilim & Ghaoui, 2005) and RL community (Bagnell et al., 2001; Morimoto & Doya, 2005) focused on linear or ﬁnite systems.
Prior work designs deep RL algorithms that are robust against a variety of adversarial attacks, including attacks on the dynamics (Jiang et al., 2021; Rajeswaran et al., 2017a), observations (Huang et al., 2017; Pattanaik et al., 2018; Zhang et al., 2021), and actions (Gleave et al., 2020; Pinto et al., 2017; Tessler et al., 2019).
Treating the robust RL problem as a POMDP, rather than an MDP (as done in most prior work), unlocks a key capability for RL agents, because agents can use their memory to identify the hidden states of the current adversarial environment, although previous work (Jiang et al., 2021; Rajeswaran et al., 2017a) only train Markovian policies on POMDPs.
While some work ﬁnd memory-based policies are more robust to the adversarial attacks than Markovian policies (Russo & Prouti`ere, 2021; Zhang et al., 2021), they train these baselines in a single MDP without adversaries, which differs from our training setting where the recurrent model-free RL can have access to a set of MDPs.
Generalization in RL.
The goal of generalization in RL is to make RL algorithms perform well in test domains that are unseen during training, which emphasizes the average case on the novel test domains instead of the worse case in the possibly seen test domains as in robust RL.
Prior work have studied generalization to initial states in the same MDP (Rajeswaran et al., 2017b; Whiteson et al., 2011; Zhang et al., 2018b), random disturbance in dynamics (Rajeswaran et al., 2017b), states (Stulp et al., 2011), observations (Song et al., 2020; Zhang et al., 2018a), and actions (Srouji et al., 2018), and different modes in procedurally generated games (Cobbe et al., 2019; Farebrother et al., 2018; Justesen et al., 2018).
Among them, Packer et al.
(2018) provides a benchmark on both in-distribution (ID) and out-of-distribution (OOD) generalization to different dynamics parameters, and Zhao et al.
(2019) extends the benchmark by introducing random noise in states, observations, and actions.
Algorithms for improving generalization in RL can be roughly divided into classic regularization methods such as weight decay, dropout, batch normalization, and entropy regularization (Cobbe et al., 2020; Farebrother et al., 2018;
Igl et al., 2019), model architectures (Raileanu & Fergus, 2021; Srouji et al., 2018), data augmentation through randomization (Lee et al., 2020;
Tobin et al., 2017), Although introducing observational noise and the change in dynamics parameters will transform MDPs to POMDPs, few work study memory-based policies such as model-free recurrent RL with mixed results.
Same algorithm RL2 (Duan et al., 2016) was found to perform badly in Packer et al.
but relatively well in Yu et al.
D ESIGN C ONSIDERATIONS FOR R ECURRENT M ODEL -F REE RL
Implementing a recurrent model-free RL algorithm requires making a number of design decisions.
This section describes the decisions that we found most important to make recurrent model-free RL competitive with more complex, recent algorithms.
We will focus on continuous control problems with state-based inputs (i.e., not image-based inputs).
Importantly, we assume that the policy can observe the reward and done signals (the end of one episode during one trial (Duan et al., 2016)) from the environment during evaluation .
This assumption is common in prior work (Han et al., 2020; Zintgraf et al., 2020), but many recurrent model-free implementations do not provide the agent with information.
In the following paragraphs, we will describe the important decision factors in recurrent model-free RL.
Table 2 summarizes how prior work and our method makes these design decisions when implementing recurrent model-free RL.
Recurrent Off-Policy Actor-Critic Architecture.
The ﬁrst important design decision is whether the recurrent policy (actor) and the recurrent Q-value function (critic) use shared RNN encoder (and embedders) or use separate ones.
In the experiment section (Sec. 5.2) we will show that a shared encoder would cause large gradient norm in the (off-policy) recurrent actor-critic and thus hinder learning, while separate encoders can greatly mitigate this issue and learn efﬁciently.
This echoes prior work (Fakoor et al., 2020;
Meng et al., 2021; Sun et al., 2021; Wang et al., 2020) that also use separate encoders in their (off-policy) recurrent actor-critic.
To avoid running an inordinate number of experiments, we will use the separate architecture in the rest of the paper. [Table]
Policy Input Space.
The next consideration is the input space of the model-free policy.
The maximal input space of policy to emit an action a t at time t , should be the history of all quantities that the policy has observed, namely the past observations o 0: t , the past actions a 0: t − 1 , the past rewards r 0: t , and the past done signals d 0: t , which was already employed in the early work (Duan et al., 2016).
Generally, the input space of optimal policy should only depend on the quantities that have connections with hidden states (deﬁned in Sec. 2) (Izadi & Precup, 2005; Poupart & Boutilier, 2002).
We show the policy input spaces that are connected with the hidden states for the discussed subareas in the “Inputs” column of Table 1.
While prior work often only conditions the recurrent RL baseline on previous observations (and actions) (Han et al., 2020; Igl et al., 2018; Kostrikov, 2018; Meng et al., 2021; Wang et al., 2020; Yang & Nguyen, 2021), our experiments in Sec.
5.2 ﬁnd that additionally conditioning on other previous information, such as previous rewards, can increase reward by up to 30% .
Model-free RL Algorithms.
Recurrent model-free RL can be understood as applying an off-theshelf model-free RL algorithm with an actor and a Q function parametrized to take sequences of inputs.
As such, the choice of the underlying model-free RL algorithm is paramount.
Most prior work on continuous control POMDP problems used on-policy algorithms, such as A2C (Mnih et al., 2016), TRPO (Schulman et al., 2015) or PPO (Schulman et al., 2017).
While off-policy algorithms such as TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018a;b) greatly improve the performance in continuous control MDP problems in terms of sample efﬁciency and asymptotic performance, these methods are rarely used in recurrent model-free RL baselines (Rakelly et al., 2019;
Zhang et al., 2020; Zintgraf et al., 2020).
In the experiment section (Sec. 5.1), we will show that using these off-policy algorithms for recurrent model-free RL provides results that are better than using on-policy algorithms and are comparable to their specialized methods in POMDP.
This echoes the ﬁnding that model-free off-policy TD3-Context (Fakoor et al., 2020) can be better than the specialized method PEARL (Rakelly et al., 2019) in meta RL.
RNN Variants and Context Length.
RNN training is known to be unstable, especially with long sequences input (Bengio et al., 1994).
The RNN variants like LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Chung et al., 2014) mitigate the training issues, but still may fail to learn long-term dependencies (Trinh et al., 2018).
In POMDP problems, these dependencies reﬂect the memory that an agent must have to solve a task.
For example, a POMDP that hides velocities from observations theoretically requires a short memory length to infer velocities through consecutive positions (Meng et al., 2021).
Prior work in POMDPs choose a variety set of context lengths for RNNs from 1 to 2048 (see the “Len” column of Table 2), and we select three representatives of short (5), medium (64), and long length (larger than 100) in the experiments (Sec. 5) for comparison.
We also try both
LSTM and GRU as RNN variants to compare their performance.
We ﬁnd that the optimal context length and RNN variant are task-speciﬁc (see Sec. 5.2).
E XPERIMENTS
Our experiments aim to answer two questions.
First, how does a well-tuned implementation of recurrent model-free RL compare to specialized POMDP methods, such as purpose-designed meta
RL and robust RL algorithms?
To give these prior methods the strongest possible footing, we will compare prior methods on the speciﬁc problem types for which they were developed (i.e., meta RL algorithms were tested on meta RL tasks).
Our second question studies which design decisions are essential for recurrent model-free RL.
We put the environment details in Appendix D.
Code Implementation.
We release a modular and highly-conﬁgurable implementation of recurrent (off-policy) model-free RL: url.
Our implementation is efﬁcient in terms of computer memory compared to previous off-policy RL methods for POMDPs (200x less RAM than Han et al.
and
9x less GPU memory than Dorfman et al.
Please see the appendix A for details, including an explanation of why our implementation is more memory-efﬁcient than prior work.
5. R ECURRENT M ODEL -F REE RL IS C OMPARABLE WITH P RIOR S PECIALIZED M ETHODS
While prior work has studied a range of different POMDP settings (e.g., meta RL, occluded observations), recurrent model-free RL is a ubiquitous baseline (Han et al., 2020; Humplik et al., 2019; Igl et al., 2018; Rakelly et al., 2019; Zintgraf et al., 2020).
However, prior work consistently report that this baseline is reported to be unperformed to more specialized methods.
This section casts doubt on that claim, showing that a well-tuned implementation of recurrent model-free RL can perform at least as well as more specialized methods.
We study four subareas of POMDPs: the “standard” POMDP, meta RL, robust RL, and generalization in RL.
We tune a wide range of decision factors shown in Sec.
in our implemented recurrent model-free RL.
Appendix A.3 shows the details of the tuning options.
For each subarea, we show the performance of a single variant that works best across the environments in that subarea, compared with the prior specialized methods in this subsection.
In other words, the following plots of each subarea report the same model-free recurrent RL algorithm with the same hyperparameters.
The exact conﬁgurations of each subarea can be found in the last four rows of Table 2.
Under this restricted setting, we ﬁnd that our implementation can actually outperform prior (specialized) methods by a wide margin across the four subareas.
For each plot of learning curves, we show three approaches as reference.
First, an Oracle policy has access to the POMDP hidden states, turning the POMDP into an MDP; this policy should therefore be treated as an upper bound on the performance that any POMDP method should receive.
Second, as a lower bound, we use a Markovian policy to solve the POMDP.
Both Oracle policy and
Markovian policy are trained with the same hyperparameters as our recurrent model-free RL implementation.
Third, we add a Random policy, which represents a trivial lower bound. We show the full learning curves in Appendix E.1 due to the space limit.
See Appendix B for details about the implementation of these comparisons.
“Standard” POMDP.
Our ﬁrst experiments look at the “standard” POMDPs that typically occlude some part of states in the environment.
We will compare against VRM (Han et al., 2020), a recent state-of-the-art model-based POMDP algorithm.
We directly apply the environment design of VRM paper that occludes either positions&angles or velocities of the simulated robot in PyBullet (Coumans & Bai, 2016).
There are 8 environments { Hopper, Ant, Walker, Cheetah } - { P,V } , where “-P” stands for observing positions&angles only, and “-V” stands for observing velocities only.
Fig.
1 and Fig. 18 in appendix show that the best single variant of our model-free recurrent RL implementation outperform VRM in 6 out of 8 environments, especially in { Cheetah,Hopper } - { P } (over 80% of the Oracles).
Our results suggest that, while the variational dynamics model used by VRM may be useful for some tasks, a simple recurrent model-free RL baseline can outperform VRM if properly tuned.
While we are primarily interested in sample complexity, but not compute, it is worth noting that our recurrent model-free RL implementation is substantially more efﬁcient than the open-source VRM implementation, training 5 × faster and requiring at most 200 × less RAM usage (see Appendix A).
[Figure]
Meta RL.
We next compare the recurrent model-free RL to the meta RL setting, where some indicator of the task is unobserved.
We compare our implementation of recurrent model-free RL to a specialized state-of-the-art method, VariBAD (Zintgraf et al., 2020) that explicitly learns the task embeddings by variational model-based objectives.
We make VariBAD stronger by replacing RL algorithm PPO with SAC as Dorfman et al.
We adopt the three environments used in Dorfman et al.
(2020) for experiments, including Semi-Circle and Cheetah-Vel,
and we also adapt Wind to make it harder to solve.
Figure 2 shows that our best single variant outperforms VariBAD and even reaches Oracles in the two meta RL environments, Cheetah-Vel and Wind, leaving Semi-Circle in the appendix.
Prior work (Rakelly et al., 2019; Zintgraf et al., 2020) show that disentangling task inference and control can stabilize training.
However, our experiments suggest that joint training of task inference and control could also have comparable performance if well implemented.
Additionally, because recurrent model-free RL is trained end-to-end, without using pre-trained task representations saved in the replay buffer like the off-policy version of VariBAD (Dorfman et al., 2020), our implementation does not have non-stationarity issue in task representations.
Robust RL.
Thirdly, we focus on the robust RL that aims to maximize the worst returns over the tasks.
We choose the recent specialized algorithm MRPO
(Jiang et al., 2021) as the compared method, and adopt their used environments based on SunBlaze benchmark (Packer et al., 2018).
These environments have hidden states that are ﬁxed during one episode, including the density and the friction coefﬁcients of the simulated robots, namely { Cheetah, Hopper, Walker } -Robust.
Fig.shows both the average return and worst return of our single best variant and MRPO on the three environments, where the worst return is measured by the average return in the worst 10% testing tasks following the practice in Jiang et al.
The results are quite surprising: although our implementation, using average-case RL objective and without access to the hidden states, is not expected to surpass MRPO and Oracle with access to hidden states in worst return, we found that our best variant vastly outperforms the specialized MRPO and Oracle in both average return and worst return, with over 80% fewer simulation steps.
Our implementation beneﬁts from its memory and off-policy algorithms, while MRPO might suffer from its Markovian on-policy algorithm and a bit ideal Lipschitz assumption in dynamics.
Nevertheless, our implementation is around 17.5x slower than MRPO given the same simulation steps (see Appendix A), so we only run it with 3M steps with a limited time budget.
Generalization in RL.
Finally, we focus on the SunBlaze benchmark from Packer et al.
(2018) for investigating generalization in RL, including { Hopper, Cheetah } -Generalize.
We pick the best specialized method in the tables of ﬁnal performance in Packer et al.
(2018), Markovian on-policy robust RL method EPOpt-PPO-FF (Rajeswaran et al., 2017a).
Fig.
4 show the interpolation and extrapolation success rates in one environment, where in the interpolation the testing tasks have same distribution of hidden states as that of the training tasks, while in the extrapolation the testing distribution is disjoint from that of training.
We can see that our model-free method is on par with the EPOpt-PPO-FF in the interpolation benchmark, while EPOpt-PPO-FF requires access to the dynamics parameters but ours does not.
In the extrapolation benchmark, our method greatly outperforms the previous method, although our objective does not consider extrapolation.
Overall, we can see that with careful tuning on recurrent model-free RL, it can at least perform as well as the specialized or more complicated methods, in various kinds of POMDPs.
5. W HAT M ATTERS IN R ECURRENT M ODEL -F REE RL A LGORITHMS ?
In the previous subsection, we showed that recurrent model-free RL can perform on par with the specialized (state-of-the-art) methods, then a natural question comes: Why our implementation of recurrent model-free RL outperforms the implementation used in prior work?
Our analysis will focus on ablating the ﬁve important design decisions introduced in Sec. 4: the actor-critic architecture ( Arch ), the policy input space ( Inputs ), the underlying model-free RL algorithm ( RL ), the RNN encoder ( Encoder ), and the RNN context length ( Len ).
See Table.
2 for a summary of how prior work made these design decisions.
Due to the space limit, we show the ablation results in some but not all the environments to compare the performance between the best single variant and the other variant that only differs in one decision factor.
We also provide “single factor analysis” plots for each decision factor by averaging the performance over the other factors in
Appendix E.2.
Recurrent Off-Policy Actor-Critic Architecture.
First, we ran some experiments with both shared and separate architectures on two toy POMDP environments.
Fig.
5 show the results in one of them (see Appendix E.3 for the other).
We can see that the shared architecture failed to learn, compared to the separate architecture.
The large RNN gradient norm in the shared architecture suggests that the actor and critic losses may cause conﬂicts in gradient update.
Our results echo prior work (Fakoor et al., 2020;
Meng et al., 2021; Sun et al., 2021) that only consider separate RNN encoders that can achieve high asymptotic rewards, and also echo that (Han et al., 2020) shows poor results in the shared architecture of
SAC-LSTM.
Policy Input Space.
The 1st row of Table 3 shows the effect of policy input space in a POMDP environment
Walker-P.
The reward signals could help reveal the missing information of the velocity of the robot base, which is occluded in Walker-P. Similarly, the single factor analysis on policy input in Fig. 12 shows that oar is among the best in “-P” environments. Therefore, it is reasonable that adding previous rewards into policy inputs can increase the performance.
[Table]
Model-free RL Algorithms.
Table 2 shows that TD3 dominates all the best variants in each subarea, which may be due to dense reward setting in most environments. However, the 2nd row of
Table 3 shows the effect of RL algorithm in a POMDP environment Ant-P. SAC is signiﬁcantly better than TD3 (increase by 6.8 × , surpassing the PPO-GRU (Kostrikov, 2018) in Fig. 1), possibly due to strategic exploration where the action noise conditions on the history instead of being independent.
This is prominent mainly in Ant-P as it might be much harder than the others.
RNN Variants and Context Length.
Generally, there is no signiﬁcant difference between LSTM and GRU (see the single factor analysis in Appendix E.2), which is understandable as both are designed for general purpose.
Howover, the 3rd row of Table 3 shows the effect of RNN encoder in a robust RL environment.
We can see replacing LSTM with GRU can increase the worst-case metric in Walker-Robust.
For the context length in RNNs, a medium length (64) dominates in all the best variants in each subarea (see Table 2), which could be viewed as a trade-off between memory capacity and computation costs.
However, the remaining rows of Table 3 show the mixed effects of context length in RNNs.
Both increasing and decreasing the context length can boost the performance in different environments.
Speciﬁcally, decreasing the length from 64 to 5 makes our method surpass VRM in Walker-V (increase by 2.2 × ).
This might explain why the prior methods adopt a wide range of context lengths from 1 to 2048 (see Table 2).
Therefore, the choice of context length is problem-speciﬁc and can require tuning.
Summary.
We now summarize the main ﬁndings of our experiments:
Using separate weights for the recurrent actor and recurrent critic boosts performance, likely because it avoids gradient explosion (Fig. 5 and Fig. 17 in Appendix).
Using state-of-the-art off-policy RL algorithms as the backbone in recurrent model-free RL can improve asymptotic performance (Fig. 1 and Figures in Appendix E.1).
The context length for the recurrent actor and critic has a large inﬂuence on task performance, but the optimal length is task-speciﬁc.
Reasonable values are 5 to 500, and 64 is a good start (Rows 4–6 in Table 3 and Figures in Appendix E.2).
It is important that the inputs to the recurrent actor and critic, such as past observations and past returns, contain enough information to infer the POMDP hidden states (Row 1 in Tableand Figures in Appendix E.2).
These ﬁndings may provide a useful initialization for researchers studying recurrent model-free RL.
C ONCLUSION
In this paper, we show that a carefully-designed implementation of recurrent model-free RL can perform well across a range of POMDP domains, often on par with (if not signiﬁcantly better than) prior methods that are speciﬁcally designed for speciﬁc types of POMDPs.
Our ablation experiments demonstrate the importance of key design decisions, such as the underlying RL algorithm and RNN context length.
While the best choices for some decisions (such as using separate RNNs for the actor and the critic) are consistent across domains, the best choices for other decisions (such as RNN context length) are problem-dependent.
R EPRODUCIBILITY S TATEMENT
We release our code at the url for reproducibility.
In this code repository, we provide instructions on how to run our method and the compared methods in this paper on all the environments involved in the experiment section.
We provide the default conﬁguration ﬁles for training and evaluating these algorithms that are adopted in our experiments.
We also attach the numeric results of all the bar charts shown in the experiment section
E THICS S TATEMENT
We do not believe that our work has direct ethical or societal implications.
A C ODE - LEVEL D ETAILS
In this section, we ﬁrst introduce the outline of code design, especially the replay buffer for sequences, and then compare the system usage, including computing speed, RAM, and GPU memory with previous POMDP methods.
A. C ODE D ESIGN
Easy to use.
Our code can be either used as an API to call the recurrent model-free RL class or a framework to tune the details in the class.
The recurrent model-free RL class takes the hyperparameters of RNN encoder type, shared or separate actor-critic architecture, and whether include previous observations, and/or actions, and/or rewards into the inputs, to generate different instances.
The details of the hyperparameter tuning set are shown in Sec.
A.3.
Memory-efﬁcient replay buffer for sequences.
Moreover, we design an efﬁcient replay buffer for off-policy RL methods to cope with sequential inputs.
Previous methods (Dorfman et al., 2020;
Han et al., 2020; Yang & Nguyen, 2021; Zintgraf et al., 2020) mainly use three-dimensional replay buffer to store sequential inputs, with the dimensions of (num episodes, max episode length, observation dimension) , taking observation storage as example.
This kind of implementation becomes memory-inefﬁcient if the actual episode length is far smaller than the max episode length (Han et al., 2020; Yang & Nguyen, 2021), which is common in locomotion tasks (Coumans & Bai, 2016; Schulman et al., 2017).
Otherwise, it requires the ideal assumption that the episode has ﬁxed length (Dorfman et al., 2020; Zintgraf et al., 2020).
Instead, we manage to implement a two-dimensional replay buffer of shape (num transitions, observation dimension) for observation storage, which also records the locations where each stored episode ends.
In case of actual episodes that are shorter than provided sampled sequence length, the buffer can also generate masks to indicate if the corresponding transitions are valid.
This enables the policy to receive a batch of previous experiences in a tensor-like data structure when sampling from the replay buffer.
To sum up, our replay buffer can support varying-length sequence inputs and subsequence sampling in a 2-dim manner.
Flexible training speed.
Finally, our code supports ﬂexible training speed by controlling the ratio of the numbers of gradient updates in RL w.r.t.
the environment rollout steps.
The training speed is approximately proportional to the ratio if the simulator speed is much faster than the policy gradient update.
Typically, the ratio is less than or equal to 1.0 to enjoy higher training speed.
A. S YSTEM U SAGE
Table 4 shows the typical system usage of our method and the compared specialized methods on different environments.
The time cost for our method and VariBAD depends on how many processes in parallel are run on a single GPU – our method is run with 8 processes on a single GPU while
VariBAD is run with one process due to large GPU memory usage.
From the results we can see that our method is memory-efﬁcient in both RAM and GPU, and has an acceptable training speed with default hyperparameters.
The computer system we used during the experiments includes a
GeForce RTX 2080 Ti Graphic Card (with 11GB memory) and Intel(R)
Xeon(R) Gold 6148 CPU @
2.40GHz (with 250GB RAM and 80 cores).
A. O UR H YPERPARAMETER T UNING S ET
Our proposed method has the following decision factors (introduced in Sec. 4) to tune in the experiments with the following options (the names in brackets are abbreviated ones):
• Actor-Critic architecture ( Arch ): share the encoder weights between the recurrent actor and recurrent critic or not, namely shared and separate .
• Model-free RL algorithms ( RL ): td3 (Fujimoto et al., 2018) or sac (Haarnoja et al., 2018a)
• Encoder architecture ( Encoder ): lstm (Hochreiter & Schmidhuber, 1997) or gru (Cho et al., 2014)
[Table]
• Policy input space ( Inputs ): o , oa , or , oar , oard (the notation is introduced in Sec. 4; depending on the POMDPs, see “Policy input” row in Table 5) • Context length ( Len ): short (5), medium (64), long (larger than 100, depending on the POMDPs).
For each instance, we label it with the names of all the hyperparameters it used in lowercase as notation.
For example, td3-lstm-64-or-separate in Fig.
5 refers to the instance that uses the separate actor-critic architecture, TD3 RL algorithm, LSTM encoder, the policy input space of previous observations and reward sequences, and RNN context length of 64.
B T RAINING D ETAILS
hidden layer
, each RNN is one-layer LSTM or GRU, each MLP has two hidden layers.
Fig.
6 shows our (separate) recurrent actor-critic architecture.
Table 5 shows the main hyperparameters we adopt for each subarea.
We did not tune these hyperparameters, except that we adjusted the number of gradient steps so that all the experiments could be completed in 36 hours.
For Markovian policies (SAC and TD3), we remove the encoders and RNNs from the actor-critic architecture, and train them with same hyperparameters as those of recurrent policies.
For each task, we report the results of either SAC or TD3, whichever achieves higher returns.
For Oracle policies, we use the well-tuned results from Table 1 (“SAC w/ unstructured row”) in Rafﬁn et al.
{+(2021), which is based on Stable Baseline
+}
(Rafﬁn, 2020), for “standard” POMDPs.
For the other subareas, we have to run the Markovian policies (SAC and TD3) with access to the hidden states, using the same training hyperparameters as those of recurrent policies.
But these Oracle policies might be not
[Table]
[Table]
(cid:55)
Input space oard oar s s
Access to hidden states?
(cid:55) (cid:55) (cid:51) (cid:51) well-tuned given same environment and gradient steps, especially in robust RL and generalization in RL.
We also show the settings of the specialized methods we compared in the main paper in Table 6.
Note that our recurrent model-free RL share the exactly same settings as (off-policy) VariBAD (Dorfman et al., 2020) and VRM (Han et al., 2020).
For MRPO (Jiang et al., 2021) and EPOPT (Rajeswaran et al., 2017a), they adopt totally different settings, i.e. on-policy Markovian approaches to MDPs (with access to the ground-truth state ( s ) of environment).
Thus, in fact, MRPO and EPOPT should be more viewed as oracle policies as upper bounds of recurrent model-free RL.
C E VALUATION D ETAILS
The bar charts in Fig. 1 and 18 and Table 3 adopt the ﬁnal performance of each method.
We run each instance/variant in our method and each compared method with 4 random seeds.
The ﬁnal performance is calculated by the average performance of the last 20% environment steps across the 4 seeds.
In terms of selecting the best variant in our method for each subarea, we ﬁrst compute the ﬁnal performance of each variant, then normalize the ﬁnal performance into [0 , 1] , and ﬁnally select the best variant in the average of the normalized ﬁnal performance across all the environments in each subarea. For the bar charts in Fig.
1 and 18, we show the normalized returns of each method, calculated by R − R min
R max − R min ∈ [0 , 1] , where R is the raw average return of that method and R max and R min are the maximum and minimum of all the methods including Oracle policy and Random policy.
D E NVIRONMENT D ETAILS D. “S TANDARD ” POMDP
Except for the classic Pendulum environment, we use PyBullet (Coumans & Bai, 2016) as the simulator for “standard” POMDP environments.
As the practice in VRM (Han et al., 2020), we remove all the position/angle-related entries in the observation space for “-V” environments and velocityrelated entries for “-P” environments, to transform the original MDP into POMDP.
{ Pendulum,Ant,Cheetah,Hopper,Walker } -P. The “-P” stands for the environments that keep position-related entries by removal of velocity-related entries.
Thus, the observed state s o includes positions p , while the hidden state s h is the velocities v .
{ Pendulum,Ant,Cheetah,Hopper,Walker } -V.
The “-V” stands for the environments that keep velocity-related entries by removal of position-related entries.
Thus, the observed state s o includes positions v , while the hidden state s h is the velocities p .
D.
[Equation]
Semi-Circle.
We directly follow off-policy version of VariBAD (Dorfman et al., 2020).
The observed state s o includes the agent’s 2D position p , and the hidden state s h is referred to the goal state p g .
The goal state only appears in reward function: R ( s ot , s ot +1 , a t , s h ) ≡ R ( p t +1 , p g ) =( (cid:107) p t +1 − p g (cid:107) 2 ≤ r ) .
The dynamic function T is independent of the goal state.
Cheetah-Vel.
We directly follow Dorfman et al.
(2020) using MuJoCo (Todorov et al., 2012) simulator of HalfCheetah-v2 .
The hidden state s h is the target velocity v g and observed state s o includes the velocity v .
Reward function includes both the hidden state and action:
R ( s ot , s ot +1 , a t , s h ) ≡ R ( v t , v g , a t ) = −(cid:107) v t − v g (cid:107) 1 − 0 .
05 (cid:107) a t (cid:107) 2 .
The dynamic function T is independent of the goal state.
Wind.
We modiﬁed the parameters of Wind environment in Dorfman et al.
(2020) to make it harder to solve.
The agent must navigate to a ﬁxed (but unknown) goal p g within a distance of
D = 1 from its ﬁxed initial state.
Similarly to Semi-Circle, the reward function is goal conditioned but without hidden state: R ( s ot , s ot +1 , a t , s h ) ≡ R ( p t +1 , p g )
= 1 ( (cid:107) p t +1 − p g (cid:107) 2 ≤ r ) .
The hidden state s h appear in the deterministic dynamics as a noise term, i.e. s ot +
=
s ot + a t + s h , where s h is sampled from U [ − 0 .
at the initial time-step and then kept ﬁxed.
D. R OBUST RL
{ Hopper,Walker,Cheetah } -Robust.
We directly adopt the environments used in MRPO (Jiang et al., 2021).
In each environment, the hidden state is the dynamics parameters including the density and friction coefﬁcients of the simulated robot in roboschool, adapted from the SunBlaze (Packer et al., 2018).
The exact ranges of the hidden states in each environment can be found in Table 1 of
MRPO (Jiang et al., 2021).
We evaluate the algorithms with 100 tasks in each environment, and use the average of them as average returns, and the average of the worst 10% of them as worst returns, following MRPO paper.
D. G ENERALIZATION IN RL
{ Hopper,Cheetah } -Generalize.
We directly adopt the environments used in SunBlaze (Packer et al., 2018).
In each environment, the hidden state is the dynamics parameters including the density, friction coefﬁcients, and the power of the simulated robot in roboschool.
The exact ranges of both interpolation and extrapolation in the hidden state distribution for each environment can be found in Table 1 of SunBlaze (Packer et al., 2018).
We follow the practice of SunBlaze to evaluate interpolation and extrapolation success rates.
E F ULL E XPERIMENTAL R ESULTS E. L EARNING C URVES OF A LL THE C OMPARED M ETHODS
In this subsection, we show all the learning curves of all the compared methods (including Oracle policy as upper bound, Markovian and Random policies as lower bounds) in each subarea, namely “Standard” POMDPs (Fig. 7 and Fig.
8), meta RL (Fig. 9), robust RL (Fig. 10), and generalization in RL (Fig. 11).
The ﬁnal performance of these learning curves generate the bar charts in Sec. 5.1.
[Figure]
We show the results from the single best variant of our implementation on recurrent model-free
RL, the popular recurrent model-free on-policy implementation (PPO
GRU, A2C GRU) (Kostrikov 2018), and also model-based method VRM (Han et al., 2020). Note that VRM is around 5x slower than ours, so we have to run 0.5M environment steps for it.
E. S INGLE F ACTOR A NALYSIS ON O UR M ETHOD
Our analysis will focus on ablating the important design decisions: the actor-critic architecture ( Arch ), the policy input space ( Inputs ), the underlying model-free RL algorithm ( RL ), the RNN encoder ( Encoder ), and the RNN context length ( Len ).
As there are several decision variables in our method, we could only show the results of each variable in the plots by averaging the performance over the other variables, which we call single factor analysis .
In this kind of analysis, we can only say one value is better than another in one factor in the average sense (not the maximal sense); therefore it can show the robustness of each factor, but cannot tell the best choices (showed in Sec. 5.1).
We show single factor analysis plots in each subarea, namely “Standard” POMDPs (Fig.
12 and Fig. 13), meta RL (Fig. 14), robust RL (Fig. 15), and generalization in RL (Fig. 16).
From these plots, we can see that each decision factor can make a difference in some environments.
For example, the choice of RL algorithm is crucial in Ant-P (Fig. 12), Cheetah-V (Fig. 13), Wind (Fig. 14) and Hopper-Generalize (Fig. 16).
The context length is essential in all the “-P” environments (Fig. 12), Cheetah-Vel (Fig. 14), and both the generalization environments (Fig. 16).
The policy input space can make a difference in most “-P” environments (Fig. 12) possibly because oar contains the information of missing velocities. [Figure]
E. A DDITIONAL R ESULTS ON S EPARATE VS S HARED R ECURRENT A CTOR -C RITIC A RCHITECTURE
In Fig.
5 of Sec.
5.2, we show the importance of selecting a separate recurrent actor-critic architecture in a meta RL environment, Semi-Circle.
Now we show the result in another POMDP environment, Pendulum-V, which occludes the positions and angles, in Fig.
We can see that the shared encoder architecture is also worse than the separate one, possibly due to the different gradient scales in actor and critic losses w.r.t. the encoder. E. A DDITIONAL R ESULTS ON C OMPARISON WITH VRM Both Fig.
1 and Fig.
18 shows the ﬁnal performance of the same single variant of our implementation, but the former shows our results with 1.5M simulation steps while the latter shows our results with 0.5M simulation steps to match with those of VRM due to the time budget.
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
F A BLATION S TUDY ON RNN A RCHITECTURES
Our implementation of recurrent model-free RL uses the popular 1-layer LSTM or GRU. To investigate the effect of RNN architecture, we ablate two RNN variants: one is 2-layer stacked LSTM/GRU, the other is Particle-Filter RNN (PF-RNN (Ma et al., 2020a)). PF-RNN maintains a stochastic belief (posterior distribution) through a set of weighted particles to better capture the uncertainty and multi-modality, compared to classic RNN’s deterministic belief.
DPFRL (Ma et al., 2020b) applies PF-GRU to POMDP tasks, using the mean of particles and MGF features of particles as the belief state.
We follow the implementation of PF-GRU in DPFRL (Ma et al., 2020b) to replace regular GRU, keeping the other model components the same.
We try 2-layer LSTM/GRU, on our best variant across all the environments.
We run PF-GRU on “standard” POMDP environments where the best variant also uses GRU for fair comparison, and do not try PF-LSTM as it is not adopted in DPFRL (Ma et al., 2020b).
Since 2-layer LSTM/GRU doubles the training time of 1-layer LSTM/
GRU given the same gradient updating frequency, we have to decrease the frequency from 1.0 to 0.6 in “standard” POMDPs.
Similarly, PF-GRU costs 15 × than 1-layer GRU, so we also have to decrease the frequency from 1.0 to 0.6. All the other hyperparameters remain the same. Figs. 19 to 23 show all the learning curves of the ablation study on RNN architectures.
Except for rare cases (e.g. Ant-P, Walker-V), the RNN variants perform worse than or are par with 1-layer
LSTM/GRU, although in most cases the RNN variants can still outperform the Markovian policies.
Possible reasons are that the RNN variants need hyperparameter tuning and more training samples to converge.
[Figure] [Figure] [Figure] [Figure]
[Figure]