T HE GUIDE AND THE EXPLORER : SMART AGENTS FOR RESOURCE - LIMITED ITERATED BATCH REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Iterated batch reinforcement learning (RL) is a growing subﬁeld fueled by the demand from systems engineers for intelligent control solutions that they can apply within their technical and organizational constraints.
Model-based RL (MBRL) suits this scenario well for its sample efﬁciency and modularity.
Recent MBRL techniques combine efﬁcient neural system models with classical planning (like model predictive control; MPC).
In this paper we add two components to this classical setup.
The ﬁrst is a Dyna-style policy learned on the system model using model-free techniques.
We call it the guide since it guides the planner.
The second component is the explorer, a strategy to expand the limited knowledge of the guide during planning.
Through a rigorous ablation study we show that exploration is crucial for optimal performance.
We apply this approach with a DQN guide and a heating explorer to improve the state of the art of the resource-limited Acrobot benchmark system by about 10%.
I NTRODUCTION
John is a telecommunication engineer.
His day job is to operate a mobile antenna.
He has about forty knobs to turn, in principle every ﬁve minutes, based on about a hundred external and internal system observables.
His goal is to keep some performance indicators within operational limits while optimizing some others.
In the evenings John dreams about using reinforcement learning (RL) to help him with his job.
He knows that he cannot put an untrusted model-free agent on the antenna control (failures are very costly), but he manages to convince his boss to run live tests a couple of days every month.
John’s case is arguably on the R&D table of a lot of engineering companies today.
AI adoption is slow, partly because these companies have little experience with AI, but partly also because the algorithms we develop fail to address the constraints and operational requirements of these systems.
What are the common attributes of these systems?
• They are physical, not getting faster with time, producing tiny data compared to what model-free RL (MFRL) algorithms require for training.
• System access is limited to a small number of relatively short live tests, each producing logs that can be used to evaluate the current policy and can be fed into the training of the next.
• They are relatively small-dimensional, and system observables were designed to support human control decisions, so there is no need to ﬁlter them or to learn representations (with the exception when the engineer uses complex images, e.g., a driver).
• Rewards are non-sparse, performance indicators come continually.
Delays are possible but usually not long.
• Safe operation even during live tests is crucial.
Simulators and digital twins are more and more available, but some systems are too complex to simulate from ﬁrst principles.
The RL setup that ﬁts this scenario is neither pure batch (ofﬂine; Levine et al.
as interacting with the system is possible during short periods of time nor pure online as the policy can only be updated ofﬂine between two interaction periods.
We refer to it as iterated batch RL (Lange et al.
(2012) call it growing batch).
Furthermore, we are interested in model-based RL (Deisenroth &
Rasmussen, 2011; Chua et al., 2018; Moerland et al., 2021) because i) it is sample efﬁcient (Chua et al., 2018; Wang et al., 2019), ii) it works well on small-dimensional systems with dense rewards, and iii) the system model (data-driven simulator) itself is an object of interest because it can ease the adoption of data-driven algorithms by systems engineers.
Given a robust system model, simple model predictive control (MPC) agents using random shooting (RS; Richards (2005); Rao (2010)) or the cross entropy method (CEM; de Boer et al.
(2004)) have been shown to perform remarkably well on many benchmark systems (Nagabandi et al., 2018; Chua et al., 2018; Wang et al., 2019; Hafner et al., 2019; K´egl et al., 2021) and real-life domains such as robotics (Yang et al., 2020).
On the other hand, implementing successfully the seemingly elegant
Dyna-style approach (Sutton, 1991; Kurutach et al., 2018; Clavera et al., 2018;
Luo et al., 2019), when we learn model-free agents on the system model and apply them on the real system, remains challenging especially on systems that require planning with long horizon.
Our main ﬁnding is that the Dyna-style approach can be an excellent choice for iterated batch RL by adding to it a decisiontime planning guided by the model-free policy and a dynamically self-tuning exploration.
The key is that exploration and bootstrapping with a value function estimate affords us shorter rollouts so we can reduce the accumulation of errors that plagues the Dyna-style approach with long horizons (Janner et al., 2019).
We also innovate on the experimental framework (metrics, statistically rigorous measurements), so we can proﬁt from the modularity of the Dyna-style approach, tuning ingredients (model, the MFRL guide policy, exploration, planning, bootstrapping) independently.
This modular approach makes engineering easier (as opposed to monolithic approaches like AlphaZero (Silver et al., 2017)), which is an important aspect if we want to give the methodology to non-expert systems engineers.
We demonstrate our G UIDE &E XPLORE approach on the small-dimensional but difﬁcult Acrobot system which is known to be especially tough for Dyna-style techniques (Wang et al., 2019).
We show in a rigorous ablation study how each ingredient adds a small but signiﬁcant improvement, achieving a 10% total margin over the current state of the art (Wang et al., 2019; Wang & Ba, 2020; K ´ egl et al., 2021).
D YNA Z ERO (pure Dyna-style with an AlphaZero guide) also matches the asymptotic performance with a larger sample complexity (3x) and computational (5x) price tag
[Table] [Table]
1. S UMMARY OF CONTRIBUTIONS
• A conceptual framework with interchangeable algorithmic bricks for iterative batch reinforcement learning, suitable to bring intelligent control into slow, physical, lowdimensional engineering systems and the organizational constraints surrounding them.
• A rigorous experimental framework to optimize such systems.
• An ablation study that helped us ﬁnd the combination of a neural model, a bootstrapping DQN guide [Table] and a heating explorer, which lead to a 10% jump in the state of the art on the resource-limited Acrobot system.
R ELATED WORK
The MBRL subﬁeld has seen a proliferation of powerful methods, but most of them miss the speciﬁc requirements (solving problems irrelevant in this scenario like representation learning or sparse rewards) and missing others (limited and costly system access; data taking and experimentation through campaigns, live tests; safety) (Hamrick, 2019).
The Dyna framework developed by Sutton (1991) consists in training an agent from both real experience and from simulations from a system model learned from the real data.
Its efﬁcient use of system access makes it a natural candidate for iterated batch RL.
The well-known limitation of this approach is the agent overﬁtting the imperfect system model (Grill et al., 2020).
A ﬁrst solution is to use short rollouts on the model to reduce error accumulation as done in Model-Based Policy
Optimization (MBPO; Janner et al. (2019)).
Another solution is to rely on ensembling techniques for the model.
Kurutach et al.
(2018)’s ME-TRPO is based on an ensemble of models and Trust
Region Policy Optimization (TRPO; Schulman et al.
At each TRPO step a transition is sampled from a randomly picked model of the ensemble, preventing the policy from overﬁtting one model.
An ensemble is also used in Model-Based Meta-Policy Optimization (MP-MPO; Clavera et al.
(2018)) where each model of the ensemble is seen as a different task used to meta-learn a policy.
The meta-policy is then able to quickly adapt to any of the dynamics model and is more robust to model inaccuracies.
Instead of learning the model and then the policy from the model, Stochastic
Lower Bound Optimization (SLBO; Luo et al.
alternates between model and policy updates.
In our I TERATED MBRL skeleton (Fig 1), this strategy would couple the L EARN and MFRL steps, which we do not study in this paper.
We note that according to the results shown in Fig 1.(a) and
Table 1 in Wang et al.
(2019), ME-TRPO, SLBO and MB-MPO alone (pure Dyna, without planning) are clearly suboptimal on the Acrobot system 1 , similar to the performance obtained by our pure Dyna DQN (Section 4).
Finally, Yu et al.
(2020) and Kidambi et al.
(2020) use a Dyna-style approach in the context of pure batch RL where no further data collection and therefore no further model updates are assumed.
The idea of using a guide and a value function when planning is not novel (Silver et al., 2017; Schrittwieser et al., 2020; Wang & Ba, 2020; Argenson & Dulac-Arnold, 2020).
We were greatly inspired by these elements in our objective of building smarter agents as they can make the search more efﬁcient and thus lead to a better performance.
POPLIN-A (Wang & Ba, 2020) relies on behavior cloning (using only real experience, unlike our Dyna-style approach that mainly uses the model), but their guide is similar to our approach.
During the planning, they add random noise to the actions recommended by a deterministic policy network and update the noise distribution using a CEM strategy.
In a similar way our G UIDE &E XPLORE strategy also adds a carefully controled amount of noise to the recommended actions.
Our results highlight the importance of a well-calibrated exploration, which also contributes to the understanding of POPLIN-A. Argenson & Dulac-Arnold (2020) and Lowrey et al.
both found that bootstrapping with a value estimate improves the performance of simple guided MPC strategies.
The popular AlphaZero (Silver et al., 2017) and
MuZero (Schrittwieser et al., 2020) algorithms also rely on a guide and a value function for their
Monte Carlo Tree Search (MCTS).
We implement a Dyna-style version of AlphaZero, which we call
D YNA Z ERO .
The principal issue of MuZero (Schrittwieser et al., 2020) in our micro-data iterated batch RL context is that it does not control the number of system access steps: it needs to simulate a lot from the real environment to establish the targets for the value function.
In these two algorithms the guide is updated from the results obtained during the search that it guided, a procedure known as Dual Policy Iteration (Sun et al., 2018).
We prefer experiencing with Dyna-style approaches ﬁrst to leverage popular MFRL algorithms and defer the study of Dual Policy Iteration to future work.
Our results show that planning is an important ingredient, a claim already made by Hamrick et al.
They use MuZero to run their ablation study while we prefer using an explicit model for practical reasons explained in the introduction.
Besides planning we also study the importance of exploration.
T HE FRAMEWORK FOR RESOURCE - LIMITED ITERATIVE BATCH RL 3. T HE FORMAL SETUP
Let T T =
(cid:0) ( s 1 , a 1 ) , . . .
, ( s T , a T ) (cid:1) be a system trace consisting of T steps of observable-action pairs ( s t , a t ) : given an observable s t of the system state at time t , an action a t was taken, leading to a new system state observed as s t +1 .
The observable vector s t =
( s 1 t , . . .
, s d s t ) contains d s numerical or categorical variables, measured on the system at time t .
The action vector a t contains d a numerical or categorical action variables, typically set by a control function a t =
π ( s t ) of the current observable s t (or by a stochastic policy a t ∼ π ( s t ) ; we will also use the notation π : s t (cid:59) a t ).
The performance of the policy is measured by the reward r t which we assume to be a function of the observables s t .
Given a trace T T and a reward r t obtained at each step t , we deﬁne the mean reward as R ( T T ) = 1 T (cid:80)
Tt =1 r t .
The system model p : ( s t , a t ) (cid:59) s t +1 can be a deterministic point predictor or a probabilistic (generative) model that, besides the point prediction E (cid:8) p (cid:0) s t +1 | ( s t , a t ) (cid:1)(cid:9) , also provides information on the uncertainty of the prediction and/or to model the randomness of the system (Deisenroth & Rasmussen, 2011; Chua et al., 2018).
3. A NOTE ON TERMINOLOGY
By model we will consistently refer to the learned transition or system model p (never to any policy).
Rollout is the procedure of obtaining a trace T from an initial state s 1 by alternating a model or real system p and a policy π (Fig 1).
We decided to rename what Silver et al.
(2017) calls the prior policy to guide since prior clashes with Bayesian terminology (as, e.g., Grill et al.
Hamrick et al.
(2021) also note), and guide expresses well that the role of this policy is to guide the search/planning.
Sometimes the guide is also called the reactive policy (Sun et al., 2018) since it is typically an explicit function or conditional distribution ξ : s (cid:59) a that can be executed or drawn from rapidly.
We will call the (often implicit) policy π : s (cid:59) a resulting from the guided plan/search the actor (sometimes also called the non-reactive policy since it takes time to simulate from the model, before each action).
We will learn the guide ξ using MFRL mainly on the model p , but, when possible, off-policy (since π (cid:54) = ξ ) data (system trace T ) obtained by acting on the real system will also be used.
In that sense, through the data from the real system, planning is part of training the guide.
However, since from the point of view of the MFRL guide, the model p is the world, we will not use the term planning to refer to the rollouts of the model p and the guide ξ when training the guide on p , rather we will reserve planning to the guided search procedure that results in acting on the real system.
3. E XPERIMENTAL SETUP : THE ITERATED BATCH MBRL
For rigorously studying and comparing algorithms and algorithmic ingredients, we need to ﬁx not only the simulation environment but also the experimental setup.
We parameterize the iterated batch
RL loop (Fig 1) by four parameters:
• the number of episodes N , • the number of system access steps T per episode, • the planning horizon L , and • the number of generated rollouts n at each planning step.
N and T are usually set by hard organizational constraints (number N and length T of live tests) that are part of the experimental setup.
Our main goal is to measure the performance of our algorithms at a given (and challengingly small) number of system access steps N × T .
Planning happens in silico , and so n and L (and so the total number of calls N × T × n × L to the simulator/system model p ) are softer constraints determined by the (physical) time between two steps and the computational resources available for the planning.
In benchmark studies, such as this paper, we argue that ﬁxing N , T , n , and L is important for making the problem well deﬁned (taking some of the usual algorithmic choices out of the input of
R OLLOUT ( π, p, s 1 , T ) :
T ← {} for t ← 1 to T : a t (cid:59) π ( s t ) (cid:46) draw action from policy
T ← T ∪ ( s t , a t ) (cid:46) update trace s t +1 (cid:59) p ( s t , a t ) (cid:46) draw next state return T
I TERATED MBRL ( p real , S 0 , π (0) , N, T, L, n ) :
[Figure]
(cid:46) draw initial state
[Figure]
(cid:46) initial random trace for τ ← 1 to N : (cid:46) for N episodes
[Figure]
(cid:46) learn system model
[Figure] [Figure]
(cid:46)
[Figure]
initial state
[Figure]
(cid:46) episode trace
[Figure]
system.
π (0) : s t (cid:59) a t is an initial (typically random) policy and in R OLLOUT π : s t (cid:59) a t is any policy.
N is the number of episodes;
T is the length of the episodes; L is the planning horizon and n is the number of planning trajectories used by the actor policies π ( τ ) .
τ = 1 , . .
, N is the episode index whereas t = 1 , . .
, T is the system (or model) access step index.
L EARN is a supervised learning (probabilistic or deterministic time-series forecasting) algorithm applied to the collected traces and A CTOR is a wrapper of the various techniques that we experiment with in this paper (Fig 2).
An A CTOR typically updates π ( τ − 1) using the model p ( τ ) in an n × L planning loop, but it can also access the initial policy π (0) and the trace ∪ ττ (cid:48)
=1 T ( τ (cid:48) ) collected on p real up to episode τ .
the optimizer), affording meaningful comparison across papers and steady progress of algorithms.
As in all benchmark designs, the goal is to make the problem challenging but not unsolvable.
That said, we are aware that these choices may change the task and the research priorities implicitly but signiﬁcantly (for example, a longer horizon L will be more challenging for the model but may make the planning easier), so if the MBRL community can agree, it would make sense to carefully design several settings (quadruples N − T − n − L ) on the same environment.
Our main operational cost is system access step so we are looking for any-time algorithms that achieve the best possible performance at any episode τ .
Hence, in the MBRL iteration (Fig 1), we use the same traces T ( τ ) , rolled out in each iteration (Line 7), to i) update the model p (Line 4) and the actor policy (Line 5) and ii) to measure the performance of the techniques (Section 3.5).
3. M ODEL - BASED ACTOR POLICIES : GUIDE AND EXPLORE
Our main contribution is a Dyna-style G UIDE &E XPLORE strategy (Fig 2).
The gist is to learn a guide policy ξ using a model-free RL technique on the model p and on the traces collected on the real system T .
It is known that the guide ξ , executed as an actor π = ξ on the real system, does not work (we also conﬁrm it in Section 4), partly because ξ overﬁts the model (Fig 5 in Kurutach et al.
; Grill et al.
(2020)), partly because the goal of π is not only to exploit the traces
T = ∪ Nτ =1 T ( τ ) collected so far and model p =
L EARN ( T ) , but also to collect data to self-improve p and ξ/π in the next episode τ .
This second reason is particular in our iterated batch setup: in pure batch RL, exploration is not an issue.
We explore implicitly because of the randomness of planning (H ETEROGENEOUS RS in Fig 3), but it turns out that it needs to be augmented by explicit exploration strategies.
To show this, we experiment with two strategies, i)
H EATING : modulating the temperature of the guide distribution ξ ( a | s ) , and ii) E PS G REEDY : choosing a random action with probability
ε (Fig 4).
The novelty of our approach is that, instead of constant T and ε , tuned as hyperparameters, we use a set of temperatures [ T i ] ni =1 and probabilities [ ε i ] ni =1 to further diversify the search and to let the planner to dynamically choose the right amount of randomization when selecting the best trace in Line 4 in Fig 3.
Finally, similarly to Lowrey et al.
; Argenson &
Dulac-Arnold (2020), we found that bootstrapping the planning with the learned value function at the end of each rollout trace (B OOTSTRAP in Fig 3) is crucial for optimizing the performance with a short horizon.
The main competitor of our G UIDE &E XPLORE actor is an actor that delegates all planning and exploration to the Monte-Carlo tree search of Silver et al.
(2017)’s
A LPHA Z ERO (Fig 4).
[Equation]
return H ETEROGENEOUS RS (cid:0)
[ π (
[Equation]
] ni =1 , p, L, n (cid:1) (cid:46) planning
[Equation]
[Equation] [Equation]
π
[Equation]
p
[Equation]
T
[Equation]
L
[Equation]
n (cid:1)
[Equation] [Equation]
(cid:46) guide policy (“Dyna-style”)
[Equation]
(cid:46) a set of guided explorer policies return H ETEROGENEOUS RS (cid:0) [ ρ i ] ni =1 , p, L, n (cid:1) (cid:46) planning
[Equation]
, π prev , p, T , L, n (cid:1) :
[Figure]
×
[Figure]
3. M ETRICS
We use two rigorously deﬁned and measured metrics (K´egl et al., 2021) to assess the performance of the different algorithmic combinations.
MAR measures the asymptotic performance after the learning has converged, and MRCP measures the convergence pace.
Both can be averaged over seeds, and MAR is also an average over episodes, so we can detect statistically signiﬁcant differences even when they are tiny, leading to a proper support for experimental development.
M EAN ASYMPTOTIC REWARD (MAR).
Our measure of asymptotic performance, the mean asymptotic reward, is the mean reward MR ( τ ) =
R (cid:16) T ( τ ) T (cid:17) in the second half of the episodes (after convergence; we set N in such a way that the algorithms converge after less than N/ 2 episodes)
MAR = 2 N (cid:80)
Nτ = N/ 2 MR ( τ ) .
M EAN REWARD CONVERGENCE PACE (MRCP (¯ r ) ).
To assess the speed of convergence, we deﬁne the mean reward convergence pace MRCP (¯ r ) as the number of steps needed to achieve mean reward ¯ r , smoothed over a window of size 5: MRCP (¯ r )
= T × arg min τ (cid:16) 15 (cid:80) τ
+2 τ (cid:48)
=
τ − 2 MR ( τ ) > ¯ r (cid:17) .
The unit of MRCP (¯ r ) is system access steps, not episodes, ﬁrst to make it invariant to episode length, and second because in micro-data RL the unit of cost is a system access step.
For Acrobot, we use ¯ r = 1 . 8 in our experiments, which is roughly 70% of the best achievable mean reward.
T OTAL R EWARD (cid:0) T (cid:1) : return T × R ( T ) (cid:46) total reward (a.k.a return) of trace
B OOTSTRAP (cid:0) V, α (cid:1) ( T ) : return T × R (cid:0) T ( i ) (cid:1) + αV (cid:0) T ( i )
[ L, 1] (cid:1) (cid:46) total reward + value of last state
H ETEROGENEOUS RS (cid:0)
[ ρ i ] ni =1 , p, L, n (cid:1)
[ s ] : for i ← 1 to n :
T ( i ) ← R OLLOUT ( ρ i , p, s , L ) (cid:46)
i th roll-out trace
V ( i ) ← V ALUE (cid:0) T ( i ) (cid:1) (cid:46) total reward of T ( i ) or bootstrap i ∗ ← arg max i V ( i ) (cid:46) index of the best trace return T ( i ∗ ) [1 , 2] (cid:46) ﬁrst action of the best trace (cid:0) ( ∗ 1 , a ∗ 1 ) , ∗ T , a ∗ T ) (cid:1) .
Its parameters are the shooting policies [ ρ i ] ni =1 , the model p , and the number n and length L of rollouts, but to properly deﬁne it, we also need the state s which we plan from, so we use a double argument list ()[] . [Equation] for i ← 1 to n : [Equation]
[Equation]
[ ρ i
[Equation]
ni =
[Equation]
E
[Equation]
[Equation]
: for i ← 1 to n :
[Equation]
with probability (1 − ε i ) ,
[Equation]
[Equation]
a
[Equation]
with probability ε i .
[Equation]
E XPERIMENTS 4. T HE A CROBOT BENCHMARK ENVIRONMENT
Acrobot is an underactuated double pendulum with four observables s t =
[ θ 1 , θ 2 , ˙ θ 1 , ˙ θ 2 ] which are usually augmented to six by taking sine and cosine of the angles (Brockman et al., 2016); θ 1 is the angle to the vertical axis of the upper link; θ 2 is the angle of the lower link relative to the upper link, both being clipped to [ − π, π ] ; ˙ θ 1 and ˙ θ 2 are the corresponding angular momenta.
For the starting position s 1 of each episode, all four state variables are sampled uniformly from an approximately hanging and stationary position s j 1 ∈ [ − 0 .
The action is a discrete torque on the lower link a ∈ {− 1 , 0 , 1 } .
The reward is the height of the tip of the lower link over the hanging position r ( s ) = 2 − cos θ 1 − cos( θ 1 + θ 2 ) ∈ [0 , 4]
Acrobot is a small but relatively difﬁcult and fascinating system, so it is an ideal benchmark for continuous-reward engineering systems.
Similarly to K´egl et al.
(2021), we set the number of episodes to N = 100 , the number of steps per episode to T = 200 , the number of planning rollouts to n = 100 , and the horizon to L = 10 .
With these settings, we can identify four distinctively different regimes (see the attached videos): i) the random uniform policy π (0) achieves
MAR ≈ 0 .
1 − 0 .
2 (Acrobot keeps approximately hanging), ii) reasonable models with random shooting or pure Dyna-style controllers achieve MAR ≈ 1 . 4 − 1 .
(Acrobot gains energy but moves its limb quite uncontrollably), iii) random shooting n = 100 , L = 10 with good models such as
PETS (Chua et al., 2018; Wang et al., 2019) or DARMDN (K´egl et al., 2021)
keep the limb up and manage to have its tip above horizon on average MAR ≈ 2 . 0 − 2 .
1 (previous state of the art), and iv) in our experiments we could achieve a quasi perfect policy (Acrobot moves up like a gymnast and stays balanced at the top) MAR ≈ 2 .
7 − 2 .
8 using random shooting with n = 100 K , L = 20 on the real system , giving us a target and a possibly large margin of improvement.
Acrobot is also an ideal benchmark for making our point since it turned out quite challenging for Dyna-style techniques in
Wang et al.
(2019)’s benchmarks (MAR ≈ 1 .
6 − 1 . 7 ).
4. M ODELS , GUIDES , AND ACTORS
Following K ´ egl
et al.
(2021), we tried different system models (Fig 1/Line 4) from the family of
Deep Autoregressive Mixture Density Networks (DARMDN) and decided to use DARMDN(1) det , an autoregressive network trained probabilistically (the output of the net for each observable dimension is a single Gaussian learned by minimizing the negative log-likelihood loss) and sampled deterministically (the mean of the Gaussian).
In K ´ egl et al.
(2021) this was the best model using random shooting; Table 4 in the Appendix conﬁrms this using our best actor.
In principle, any MFRL technique working with discrete action space and providing a value function and a policy can be used as a guide ξ when applying I TERATED MBRL to the Acrobot system (Fig 2/Line 1).
We experimented with Deep Q Networks (DQN; Mnih et al. (2015)) and PPO (Schulman et al., 2017).
For the DQN, following Janner et al.
(2019), short rollouts starting from real observations are performed on the model to sample transitions which are then placed in an experience replay buffer, along with the real transitions observed during the rollouts (Fig 1/Line 7).
The DQN is then updated by sampling batches from this buffer.
For DQN alone (without planning), an ε -greedy strategy is used for its training ( ε decaying as more training samples are used) and at decision time ( ε = 0 . 05 ).
For actors (Fig 1/Line 5, Fig 2), we start from the simple DQN( ε = 0 . 05 ) or PPO guide which is interacting with the system without planning.
We then add different ingredients to these simple agents to improve the performance.
Adding ( n , L ) to the name of the agent means that we use the agent to guide a planning of n rollouts with horizon L .
It is important to note here that planning without exploration using the greedy guide is, in our case, equivalent to no planning since both the model p and the guides ξ are deterministic.
XXX-H EATING E XPLORE and XXX-E PS G REEDY E XPLORE refer to the additional use of the associated exploration strategies (Fig 1/Line 2, Fig 4).
When a ﬁxed ε is used for the exploration strategy, we add it as an explicit parameter, e.g., E PS G REEDY E XPLORE ( ε ).
No parameters means that a different ε or temperature is used for each of the n rollouts.
By default the value estimate is the T OTAL R EWARD (Fig 3).
We add B OOTSTRAP when we use bootstrapping by the value function estimate.
Finally, we also apply AlphaZero (Silver et al., 2017) in a Dyna-style fashion (Fig 2) which we refer to as D YNA Z ERO .
Appendix A contains detailed information on the various algorithmic choices.
4. R ESULTS
Table 2 and Figure 5 present our the results with the main actors.
We see that planning with exploration and bootstrapping gives the best results among all the DQN actors.
The approach beats the previous state of the art: both RSA CTOR ( n = 100 , L = 10 ) from K´egl et al.
and
POPLIN-A (Wang & Ba (2020); Table 1) by 10% in terms of MAR.
PPO is slightly suboptimal compared to
DQN.
D YNA Z ERO matches the best MAR but converges three times slower and takes about ﬁve times more computational time, mainly because, unlike random shooting, MCTS cannot easily be parallelized.
Table 2 and the detailed ablation study in Appendix B show that all ingredients add to the performance.
Although DQN alone (DQN( ε = 0 . 05 )) is far better than a random policy, it is clearly not sufﬁcient to obtain a reasonable actor.
Using it as a planning guide (DQN( n = 100 , L = 10 )-
E PS G REEDY E XPLORE ( ε = 0 . 05 )) signiﬁcantly improves the performance but remains worse than random shooting RSA CTOR ( n = 100 , L = 10 ).
To propel the Dyna-style actor above simple MPC, we need to ﬁne tune exploration.
We found that allowing the planner to choose the right amount is a robust and safe approach.
The ﬁnal improvement was attained by adding value bootstrapping to the heated explorer/planner; bootstrapping does not improve the ε -greedy actor, possibly because the value estimate of the guide ξ is off on random actions, unlike in the heating exploration when the explorer “softly” increases the probability of actions, given by the guide.
As a reference, we also include a simple but costly RSA CTOR ( n = 100 K , L = 10 ).
It would not be ofﬁcially accepted in our benchmark as we restrict n to 100 and L to 10, but it can serve as a target to reach.
[Table]
DQN
[Table] [Table]
C ONCLUSION
In this paper we show that an ofﬂine Dyna-style approach can be successfully applied on the Acrobot system where previously Dyna-style algorithms were failing.
Our empirical results exhibit the importance of using a planning guided by a policy with the correct amount of exploration.
We thus propose a strategy to automatically ﬁne-tune exploration and add bootstrapping with a value function estimate to further improves the performance.
This combination leads to an improvement over the previous state of the art by 10% while respecting the same resource constraints.
Future work includes modelling the uncertainties of the value estimates so as to use them for better exploration.
R EPRODUCIBILITY STATEMENT
All the code to produce our results will be made publicly available after publication.
Details for the implementations of I TERATED MBRL, the different exploration strategies, the system models, and the different MFRL agents are also provided in Section A.
[Figure]
A I MPLEMENTATION DETAILS A. C ODE AND DEPENDENCIES
Our code will be made publicly available after publication to ease the reproducibility of all our results.
We use Pytorch (Paszke et al., 2019) to build and train the neural network system models and policies.
To run the I TERATED MBRL experiments we use the rl_simulator ( https: //github.com/ramp-kits/rl_simulator ) python library developed by K´egl et al.
(2021) which relies on Open AI
Gym (Brockman et al., 2016) for the Acrobot dynamics.
For the PPO agent we use the StableBaselines3 implementation (Rafﬁn et al., 2019).
A. M ODELS AND AGENTS
It is known that carefully tuning hyperparameters of deep reinforcement learning algorithms is crucial for success and fair comparisons (Henderson et al., 2018; Zhang et al., 2021).
To reduce the computational cost and consider a reasonable search space the models and the agents were optimized independently.
For the systems models we use the same hyperparameters as the ones used in
K ´ egl et al.
Please refer to Appendix D in K ´ egl et al.
(2021) for a complete description of the hyperparameter search and the selected hyperparameters.
The DQN is a neural network with one hidden layer of 64 neurons.
The input and hidden layers are both made of a linear layer followed by the ReLU activation function.
The output layer is a linear layer.
At the start of each epoch of I TERATED MBRL, the system model is trained using all the data collected at the previous epochs and the DQN is updated ofﬂine from both transitions generated with the model and the real observed transitions.
Following Janner et al.
and
Holland et al.
(2019), short rollouts starting from real observations are performed on the model to sample transitions which are then placed in an experience replay buffer with total size 25 , 000 .
The real transitions observed during the rollouts on the real system are also placed in this buffer.
We train and act with an ε -greedy strategy with an ε decaying as we see more samples.
The decay schedule is given by max(0 .
6 − 0 .
001375 t, 0 . 05) where t is the current number of steps realized on both the model and the real system.
The DQN is updated by sampling batches of size 64 from this buffer.
To generate transitions from the model we use a total number of 1800 steps.
Each rollout starts from a randomly sampled real observation and performs 50 steps before being reset to a new real observation.
The DQN is updated after each two steps.
We use a target network which is updated after every 400 steps.
Finally the learning rate is set to 0 . 0001 and the discount factor to 1.
To obtain these values for the different hyperparameters, we started from default values and carefully tuned them to achieve the best performance.
The PPO agent is also updated ofﬂine at the start of each epoch with the model.
We rely on the StableBaselines3 (Rafﬁn et al., 2019)
implementation using the default Multi-Layer Perceptron policy.
A total of 15000 steps are generated and each rollout resets from a real observation each 20 steps.
For D YNA Z ERO we modiﬁed the version of AlphaZero available at https://github.com/ tmoer/alphazero_singleplayer .
For the policy-value network we use two hidden layers with 64 neurons each.
The input and hidden layers are both made of a linear layer followed by the
ELU activation function.
The output layer for the policy is a softmax layer and a linear layer for the value.
The constant c in the UCT rule is set to 1 .
5 and the discount factor to 0 .
The policy-value network is updated at the beginning of each epoch by performing rollouts on the model, reset after each 50 steps.
The total number of steps performed on the model at each update is 400 .
Data are put in a buffer of size 10 , 000 .
The batch size is set to 64 and the learning rate to 0 .
Finally the number of MCTS searches at each step is set to 500 .
A. E XPLORATION
For the multi- ε exploration strategy based on E PS G REEDY E XPLORE we use one ε value for each of the n = 100 rollouts: { 0 . 001 , 0 . 01 , 0 . 02 , . . .
For the multi-temperature H EATINGE XPLORE strategy, we ﬁrst normalized the Q values by their maximum value, ˜ Q
( s, a ) =
Q ( s, a ) / max a (cid:48) Q ( s, a (cid:48) ) , before applying a softmax: ρ i ( a | s t ) =
e ˜ Q ( a,s t ) /T
i (cid:80) a (cid:48)
e ˜ Q ( a (cid:48) ,s t ) /T
i where { T i , 1 ≤ i ≤ n } is an increasing sequence of temperatures.
A large temperature gives a uniform distribution, whereas a low temperature corresponds to taking arg max Q ( a, s t ) .
Different shapes of sequences were tried (linear, logarithmic, exponential, polynomial, logistic), and best performance was obtained with a logistic schedule (with a linear end).
The exact values will be provided in the code.
A. B OOTSTRAPPING
For DQN the value estimate is computed by taking the maximum of the Q values over the actions.
The α parameter is set to 0 .
B A BLATION STUDY B. A CTORS
We show in this section that a guided planning and exploration are needed to obtain the best result.
Remove either planning or exploration and you will obtain suboptimal performance.
To support this claim we ran an ablation study with E PS G REEDY E XPLORE .
E PS G REEDY E XPLORE makes it easy to control the degree of exploration through the ε parameter.
Setting ε to 0 corresponds to no exploration and is equivalent to using the guide greedily without planning ( n = 1 and L = 1 ) as our model is used deterministically when sampled from.
Setting ε to 1 corresponds to full exploration and is equivalent to the purely random RSA CTOR ( n = 100 , L = 10 ).
Figure 6 shows the results obtained by DQN( ε = 0 . 05 ), DQN with a forced ε -greedy scheme at decision time with ε = 0 .and
no planning (DQN( ε = 0 . 4 )), and DQN with planning and ﬁxed ε values (DQN( n = 100 ,
L = 10 )-E PS G REEDY E XPLORE ( ε ) for ε ∈ { 0 . 0001 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 4 , 0 . 8 , 0 . 99 , 0 . 9999 } ).
All the results are also put in Table 3.
First, we see that planning deﬁnitely helps when comparing the performance of DQN( ε = 0 . 4 ) which uses no planning but forces the exploration with ε = 0 .
4 and the performance of DQN( n = 100 , L = 10 )-E PS G REEDY E XPLORE ( ε = 0 . 4 ) which uses planning and the same ﬁxed value of ε .
Second, as expected, the closer ε is to 0 the closer the performance is to DQN( ε = 0 . 05 ), and the closer ε is to 1 the closer the performance is to RSA CTOR ( n = 100 , L = 10 ).
With a well-chosen ε between these two extremes, say ε = 0 .
4 , we obtain a better performance than either extremes.
We can thus claim that planning is required and planning with a correct amount of exploration.
Our
E PS G REEDY E XPLORE or H EATED E XPLORE exploration strategies, used with multiple ε or temperatures values allows for the automatic and dynamic selection of the good amount of exploration.
B. M ODELS
In Table 4 we compare the performances of models from K´egl et al.
(2021) when applied with our best agent DQN( n = 100 , L = 10 )-H
EATING E XPLORE -B OOTSTRAP .
Most models belong to the family of Deep (Autoregressive) Mixture Density Networks with D components,
D { AR } MDN ( D ) { det } , trained by minimizing the negative log-likelihood.
The ‘det’ sufﬁx means that the model is sampled from deterministically, returning the mean of the predicted distribution.
DARNNs are Deep Autoregressive Neural Networks point-estimating a mean by minimizing the mean squared error and estimating a constant variance using the residual errors.
These models are homoscedastic with a variance independent of the input.
The reader can also refer to K´egl et al.
(2021) for a complete description of these models.
In the paper we use the second best model
DARMDN (1) det over the non-autoregressive DMDN (1) det because we ﬁnd autoregressive models to be easier to use in practice on real-life projects due to their ability to model dependence between the observable dimensions and to model different feature types.
We also note that, as observed in
K´egl et al.
(2021) with RSA CTOR ( n = 100 , L = 10 ), the deterministic versions perform slightly better than the stochastic ones, in general.
[Table]
C RSA CTOR PERFORMANCE ON THE REAL SYSTEM
We present the results one can obtain on the real system with an RSA CTOR and different values of the planning horizon L and the number of generated rollouts n in Fig 7.
For the considered planning horizons a larger number of generated rollouts lead to a better performance.
We also observed in our simulations that for the Acrobot to stay balanced, it was necessary (although not always sufﬁcient) to have a reward larger than 2 . 6 .
We see from Fig 7 that this can be achieved with a simple agent such as RSA CTOR but at the price of a very large number of generated rollouts.
The goal is therefore to design a smarter agent that can come as close as possible to this performance with a limited budget
[Table] [Table]