G ENERATIVE P LANNING FOR T EMPORALLY C OORDINATED E XPLORATION IN R EINFORCEMENT L EARNING
A BSTRACT
Standard model-free reinforcement learning algorithms optimize a policy that generates the action to be taken in the current time step in order to maximize expected future return.
While ﬂexible, it faces difﬁculties arising from the inefﬁcient exploration due to its single step nature.
In this work, we present Generative Planning method (GPM), which can generate actions not only for the current step, but also for a number of future steps (thus termed as generative planning ).
This brings several beneﬁts to GPM.
Firstly, since GPM is trained by maximizing value, the plans generated from it can be regarded as intentional action sequences for reaching high value regions.
GPM can therefore leverage its generated multi-step plans for temporally coordinated exploration towards high value regions, which is potentially more effective than a sequence of actions generated by perturbing each action at single step level, whose consistent movement decays exponentially with the number of exploration steps.
Secondly, starting from a crude initial plan generator, GPM can reﬁne it to be adaptive to the task, which, in return, beneﬁts future explorations.
This is potentially more effective than commonly used action-repeat strategy, which is non-adaptive in its form of plans.
Additionally, since the multi-step plan can be interpreted as the intent of the agent from now to a span of time period into the future, it offers a more informative and intuitive signal for interpretation.
Experiments are conducted on several benchmark environments and the results demonstrated its effectiveness compared with several baseline methods.
Code is available: https://github.com/Haichao-Zhang/generative_planning .
I NTRODUCTION
Reinforcement learning (RL) has received great success recently (Mnih et al., 2015;
Lillicrap et al., 2016; Gu et al., 2017; Haarnoja et al., 2018), reaching or even surpassing human-level performances on some tasks ( e.g. Mnih et al., 2015; Silver et al., 2016).
However, the number of samples required for training is still quite large, rendering sample efﬁciency improvement a critical problem.
Standard model-free RL methods operate at the granularity of individual time step.
Although ﬂexible, one prominent issue is inefﬁcient exploration, as the consecutive one-step actions are not necessarily temporally consistent, which makes the probability of consistent movement decays exponentially with the exploration steps, rendering many of the exploration efforts wasted (Dabney et al., 2021).
Extensive efforts have been made on improving the temporal consistency of exploration from different perspectives with different approaches, including parameter space perturbation ( e.g. Plappert et al., 2018b; Fortunato et al., 2018; Xie et al., 2019), enhancing correlation between policy steps ( e.g.
Lillicrap et al., 2016; Korenkevych et al., 2019; Amin et al., 2021; Zhang & Van Hoof, 2021) and skill/option-based exploration ( e.g. Agarwal et al., 2018; Tirumala et al., 2020;
Pertsch et al., 2020).
Among them, simple action repeat-based temporal persistency has been shown to be very successful ( e.g. Mnih et al., 2015; Sharma et al., 2017).
Action repeat refers to the strategy of repeating a previous action for a ﬁxed or variable number of time steps.
This simple strategy demonstrates strong performance on a broad range of tasks and has been actively investigated recently (Neunert et al., 2019;
Dabney et al., 2021; Biedenkapp et al., 2021; Yu et al., 2021).
Planning is another effective approach for obtaining a control policy alternative to RL ( e.g. Chua et al., 2018; Srinivas et al., 2018; Hafner et al., 2019;
Eysenbach et al., 2019).
Typically, it is used to [Table] Environment plan
[Table]
/DgCupwBw1oAoMhPMMrvDnSeXHenY9Fa8HJZ47hD5zPH0+AjdA=</latexit> ego
[Figure] position setpoint optimize a sequence of actions by online minimization of a cost function computed together with a model (Chua et al., 2018; Srinivas et al., 2018; Hafner et al., 2019). The control loop is closed with model-predictive control (MPC), i.e. by executing the ﬁrst action only and re-run the optimization again in the next time step. Planning excels at reasoning over long horizons and has a high sample efﬁciency, but is more computationally demanding and requires access to a model (Levine & Koltun, 2013a;b; Lowrey et al., 2019; Srinivas et al., 2018;
Chua et al., 2018; Hafner et al., 2019).
A large body of work in the literature has explored how to leverage the notable complementary strengths of RL and planning from many perspectives, e.g. , guiding policy learning with planning (Levine &
Koltun, 2013a;b), improving planning with a learned policy (Schrittwieser et al., 2019; Wang & Ba, 2020) and generating intrinsic reward for exploration (Sekar et al., 2020).
This work is motivated by recognizing both the apparent complementary strengths and less apparent connections between these two streams of seemingly different approaches, i.e. , simple but successful action repeat strategy and ﬂexible but more demanding planning-based control.
We note that they can be connected from the perspective of planning for exploration .
On the one hand, we can go beyond simply repeating the same actions by using planning to induce more ﬂexible forms of action sequences for temporally coordinated exploration.
On the other hand, the empirical success of action repeat suggests that the form of planning that is useful for exploration could be possibly done in a way that is lightweight and retains the overall model-free nature of the whole algorithm.
Motivated by these observations, we propose a Generative Planing Method (GPM) as an attempt towards the goal of model-free RL with temporally coordinated, planning-like behavior for exploration.
The overall framework is shown in Figure 1.
At the beginning of an episode, a plan τ is generated.
The plan is a temporally extended sequence of actions and the one-step action for the current time step of execution is derived from it as ω ( τ ) .
Upon entering the next time step, the old plan is adjusted by “time-forwarding” one step for alignment as ρ ( τ ) .
The old plan is committed to until certain conditions are met when compared to a new one generated at each time step.
By doing this, we have enabled both ﬂexible intentional action sequences and temporally extended exploration.
We refer this type of temporally extended and intentional exploration as temporally coordinated exploration .
The contributions of this work can be summarized as follows:
• we provide a natural perspective on extending standard model-free RL algorithms to generate multi-step action sequences for intentional exploration via planning by leveraging a connection between planning and temporally extended exploration.
• we propose a generative planning method as a concrete instantiation and initial attempt in this direction; • we verify the effectiveness of the proposed method by comparing it with a number of baseline methods; • we show that the proposed method has additionally improved interpretability on several aspects, including both the evolvement of its exploration behavior and ﬁnal policy behavior.
B ACKGROUND
We will focus on continuous control tasks in this work.
We ﬁrst review some basics on model-free
RL for continuous control, temporally extended exploration and model-based planning.
2. M ODEL - FREE RL FOR C ONTINUOUS C ONTROL
Standard RL methods like SAC (Haarnoja et al., 2018) learn a policy function which maps the current state s to action a as π ( s )
→ a .
The policy is typically modelled with a neural network with learnable parameters θ
and the optimization of the policy parameters is achieved by maximizing the following function w.r.t.
θ :
[Equation]
where D denotes replay buffer and α is a temperature parameter.
Q ( s, a ) denotes the soft state-action value function and is deﬁned as:
[Equation]
where T ( s, a ) denotes the dynamics function and ¯ r ( s, a ) the policy entropy augmented reward function (Haarnoja et al., 2018).
γ ∈ (0 , 1) is a discount factor.
By deﬁnition, Q ( s, a ) represents the accumulated discounted future entropy augmented reward starting from s , taking action a , and then following policy π θ thereafter.
2. T EMPORALLY E XTENDED E XPLORATION M EETS M ODEL - BASED P LANNING
The policy of the form in Eqn.(1) generates a one-step action at a time and the exploration is typically achieved by incorporating noise at single step level (Haarnoja et al., 2018).
While this is simple and ﬂexible in representing the control policy, it might not be effective during exploration as the consecutive one-step actions are not necessarily temporally consistent, limiting its ability to escape local optima thus wasting many of the exploration efforts (Dabney et al., 2021).
Instead, a directed form of exploration can potentially mitigate this issue by generating consistent actions for a certain period of time (Sutton et al., 1999; Mnih et al., 2015; Sharma et al., 2017; Dabney et al., 2021).
Build upon this philosophy, action repeat-based temporally extended exploration, despite of its simplicity, has been proven to be very effective and has been actively explored recently (Sharma et al., 2017; Dabney et al., 2021; Biedenkapp et al., 2021; Yu et al., 2021).
Apart from directly improving the temporal property of the exploration behavior, another set of approaches guide policy optimization with model-based planning ( e.g. Levine & Koltun, 2013a;b;
Lowrey et al., 2019; Hoppe et al., 2019).
In model-based planning, the optimized action for a particular time step is obtained by online minimization of a cost function.
To better leverage the temporal structure of the cost function, planning is typically done for a span of time period by optimizing a sequence of actions τ =
[Equation] 0 , a 1 , · · · ] ( e.g. Williams et al., 2017; Srinivas et al., 2018;
Chua et al., 2018; Hafner et al., 2019; Schrittwieser et al., 2019):
[Equation]
This type of approaches has the potential to generate more intentional and ﬂexible action sequences than repeating actions, as they are optimized at the sequence level.
The generated action sequences is typically used with MPC by executing only the ﬁrst action and discarding the rest.
Also, the prerequisite of an additional model for calculating C and the online optimization of it indicate a large gap between this type of approaches and model-free RL.
This makes the task of how to leverage planning for exploration in model-free RL more challenging.
G ENERATIVE P LANNING M ETHOD FOR M ODEL - FREE RL
Model-based planning can generate ﬂexible plans, but requires online planning with a model which is computationally intensive.
On the other hand, action-repeat is more efﬁcient, but is less adaptive in its form of plans.
This motivates us to present GPM, which supports ﬂexible plans beyond repeating for exploration without online optimization and resembles typical model-free RL algorithms.
We deﬁne a plan τ as a sequence of decisions that spans a number of time steps in the corresponding space A .
For example, if A is the space of primitive actions, τ is a sequence of primitive actions, i.e. τ =
[ a 0 , a 1 , · · · ] .
If A is the spatial coordinate space as common in robotics control and driving, then τ is a sequence of spatial positions and is essentially a spatial-temporal trajectory.
The framework of GPM is illustrated in Figure 1.
At the core of GPM is a plan generator Π θ ( s ) (cid:55)→ τ , which is a generative model that takes state s as input and outputs a plan τ ( c.f. Section 3.1).
In essence, it is a form of policy that outputs multi-step actions.
(therefore termed as generative planning ).
The plan generator is trained by maximizing plan values, which shapes it to generate plans that are more adaptive along the training process ( c.f. Section 3.2).
During rollout, at each time step t , give the observation s t , the plan generator Π generates a plan τ new =
[ a t , a t +1 , a t +2 , · · · ] .
Also, the old plan (if available) from the previous step is “time-forwarded” one step to generate τ old = ρ ( τ (cid:48) actual ) , where τ (cid:48) actual denotes the actual plan from the previous time step.
The replanning mechanism is based on the values of both plans τ old and τ new to determine which one is used as the actual plan τ actual ( c.f. Section 3.3).
[Equation]
[Table]
sha1_base64="WkPyA36AcRsaTjWY1AlAqUdXGEM=">AAAB9XicbVDLSgMxFL1TX7W+qi7dBIsgCGWmCrosunFZwT6gHUsmk7ahmWRIMkoZ+h9uXCji1n9x59+YaWehrQdCDufcS05OEHOmjet+O4WV1bX1jeJmaWt7Z3evvH/Q0jJRhDaJ5FJ1AqwpZ4I2DTOcdmJFcRRw2g7GN5nffqRKMynuzSSmfoSHgg0YwcZKD71A8lBPInulZ9N+ueJW3RnQMvFyUoEcjX75qxdKkkRUGMKx1l3PjY2fYmUY4XRa6iWaxpiM8ZB2LRU4otpPZ6mn6MQqIRpIZY8waKb+3khxpLNodjLCZqQXvUz8z+smZnDlp0zEiaGCzB8aJBwZibIKUMgUJYZPLMFEMZsVkRFWmBhbVMmW4C
[Figure]
M6inAEx3AKHlxCHW6hAU0goOAZXuHNeXJenHfnYz5acPKdQ
[Figure]
Q ( s t , a t ) , Q ( s t , a t , a t +1 ) , Q ( s t , a t , a t +1 , a t +2 ) , · · · ] .
Q ( s t , a t ) uses a separate decoder different from the rest of steps t + 1 , t + 2 , · · · , whose values are computed as the sum of the value from the previous step and the output from the decoder.
3. P LAN G ENERATOR M ODELING
It is natural to model the plan generator Π with an autoregressive structure to capture the inherent temporal relations between consecutive actions.
In practice, we use a recurrent neural network (RNN) to model Π , with its structure shown in Figure 2 (a).
More concretely, the plan generator Π θ ( s ) is implemented as a stochastic recurrent network as follows:
[Equation]
z denotes the feature extracted from s .
h ( z ) denotes a network for initializing the state for RNN.
In the special case when τ only contains a single action, Eqn.(3) reduces to the standard stochastic actor as used in Haarnoja et al.
(2018), with µ mean and σ std the network for predicting mean and standard derivation of the action distribution for the current time step.
f denotes the action decoder for future time steps, which can be implemented with some prior incorporated when available.
For example, we can implement f with a skip-connection to leverage the temporal structure:
[Equation]
where g is implemented with neural networks.
In this case, if we initialize g to generate small values, then we actually incorporated an action repeat prior to start with for exploration and learning.
The degrees of stochastic of Π θ ( s ) can be adjusted by matching a speciﬁed a target entropy in a way similar to SAC (Haarnoja et al., 2018), contributing a term J ( α ) into the total cost ( c.f. Appendix A.1).
3. P LAN V ALUE L EARNING AND P LAN G ENERATOR T RAINING
We propose to optimize the plan generator by solving the following optimization problem:
[Equation]
where τ l denotes sub-plan of length l extracted from τ , starting from its beginning.
U (1 , L ) denotes a discrete uniform distribution and L denotes the maximum plan length.
By taking expectation w.r.t.
l ∼ U (1 , L ) , we essentially train the plan generator by maximizing not only the value of the full plan, but also the value of the sub-plans.
The plan value Q ( s t , τ l ) is deﬁned as:
[Equation]
Here s t denotes the current state, r t +1 =
R ( s t , a t ) , r t +2 =
E s t +1 ∼ P ( s t ,a t ) R ( s t +1 , a t +1 ) , with R the reward function.
P ( s t , a t ) and T ( s, τ ) denote the action-level and plan-level transition functions respectively.
This formulation has a number of potential beneﬁts: i ) it can generate ﬂexible and adaptive form of plans to be used for temporally extended exploration; ii ) the generative approach used for plan generation retains the resemblance to typical model-free RL algorithms while avoids expensive online planning; iii ) it gains additional interpretability because of its multi-step plan.
To train the plan generator, we use the re-parametrization trick as in SAC (Kingma & Welling, 2014;
Schulman et al., 2015; Haarnoja et al., 2018), and also leverage the differentiability of the plan generator for gradient calculation: ∆ θ
[Equation]
(cid:105) .
Q needs to be learned in order to calculate ∆ θ .
In practice, we parametrize it with neural networks as Q φ , whose structure shown in Figure 2(b).
More concretely, the plan value function is implemented as a recurrent network, taking the one-step action as input at each step, and outputting the plan value up to the current step.
To leverage the temporal structure, we compute the plan value as the sum of the plan value from the previous step and the output from the decoder of the current step.
Since there is no previous step value for the ﬁrst step of plan, the decoder of the ﬁrst step is not shared with the rest of plan steps.
To train the plan value function Q φ , we use TD-learning as commonly used for value learning in RL algorithms ( e.g. Haarnoja et al., 2018; Fujimoto et al., 2018)
[Equation]
(6) where T ( s, τ l , r ) =
r
[Equation]
+
[Equation]
denotes the target value computed with a target critic network Q φ (cid:48) , with φ (cid:48) denoting the parameter values periodically copied from φ softly (Haarnoja et al., 2018).
By taking expectation w.r.t.
l ∼ U (1 , L ) , we learn not only the value of the full plan, but also the value of the sub-plans, which is used in replanning.
In addition, an entropy term (computed on the ﬁrst step of the plan generator) is also added to the target value following Haarnoja et al.
This form of learning has the following potential beneﬁts: i ) it can be used to train the plan generator as in Eqn.(5); ii ) it can learn from trajectories sampled from a replay buffer, which is not necessarily aligned with the plans used during rollout ( c.f. Appendix A.6.1); iii ) it supports the evaluation of sub-plans, which is necessary for adaptive replanning, as detailed below.
3. T EMPORAL P LAN U PDATE AND R EPLANNING
To encourage temporally coordinated exploration, we propose to commit to the current plan by default and switch if a new plan is better according to certain criteria.
More concretely, at the current step, we have a plan candidate τ old obtained from the plan τ (cid:48) actual of the previous step as τ old =
ρ ( τ (cid:48) actual ) .
In the typical case where the plan is consisted of a sequence of actions,
The action to be executed at the current step is ω ( τ ) =
τ [0] , and ρ is simply a shifting operator extracting the rest of the plan: ρ ( τ )=
τ [1 :] , serving as a plan candidate for the current time step.
At the same time, we generate another new plan τ new ∼ Π( s t ) .
Now we can make a decision on which of the two candidates we shall use.
If the length of τ old (denoted as l old = | τ old | ) is zero, meaning the previous plan has been fully executed, then we just need to switch to τ new .
Otherwise, we compare their plan values for determining a switch.
Note that since τ old is always shorter than τ new , the plan values need to be computed up to l old .
Denoting the plan values under this setting as Q l old ( s, τ old ) and Q l old ( s, τ new ) , we then construct a categorical distribution for generating replanning signals as replan ∼ Categorical( z ) , where z denotes the logits and is deﬁned as z (cid:44) [ (cid:15), Q l old ( s, τ new ) − Q l old ( s, τ old )] .
If replan=1 , we switch to τ new ; otherwise, we commit to τ old at the current time step.
Intuitively, if the value of the new plan is much higher than that of the old one, a plan switch is preferred.
Otherwise, it prefers to commit to the old plan.
To adjust (cid:15) , we use an objective similar to the entropy-based temperature adjustment in SAC (Haarnoja et al., 2018):
[Equation]
where (cid:15) ≥ 0 , l commit _ target is a target commit length and l commit is the actual commitment length computed as the exponential moving average of commitment length during rollout.
By doing this, we can better leverage the temporally coordinated plans from the plan generator for exploration, while allowing to switch plans if necessary.
The overall algorithm is summarized in Appendix A.1.
R ELATED W ORK
Exploration with Temporally Correlated Action Sequences.
Our work mainly lies in this category.
Methods in this category focus on improving the correlations of consecutive actions to improve over simple but widely used exploration methods like (cid:15) -greedy (Dabney et al., 2021).
Some methods focus on making the stochastic process underlying the exploration policy more temporally correlated ( e.g.
Lillicrap et al., 2016; Korenkevych et al., 2019).
Action repeat strategy leads to a family of successful methods in this category by fully discarding action space randomness for the repeating steps after the ﬁrst one ( e.g. Mnih et al., 2015; Sharma et al., 2017; Neunert et al., 2019; Dabney et al., 2021;
Biedenkapp et al., 2021; Yu et al., 2021).
The proposed method is connected with action repeat in the sense that the source of stochasticity is also introduced in the ﬁrst step of the action sequence, but it supports more ﬂexible form of action sequences beyond repeating, and achieves this through plan generator training, rather than relying on certain type of speciﬁcally designed stochastic process as in Lillicrap et al.
; Korenkevych et al.
This category of methods mostly retain the ﬂat
RL structure, in contrast to the more complicated hierarchical RL structures as reviewed below.
Hierarchical RL.
There is a large body of work on improving exploration and learning efﬁciency by operating at skill level with a higher level meta-controller ( e.g. Dayan & Hinton, 1993;
Sutton et al., 1999; Randlov, 1999; Vezhnevets et al., 2016; Bacon et al., 2017; Florensa et al., 2017; Co-Reyes et al., 2018; Konidaris et al., 2010; Pertsch et al., 2020).
However, most hierarchical RL methods are nontrivial to adapt because of some task-speciﬁc aspects, e.g. the proper design of skill space, its form of representation and its way of acquisition (Dayan & Hinton, 1993;
Sutton et al., 1999;
Florensa et al., 2017; Co-Reyes et al., 2018; Konidaris et al., 2010; Pertsch et al., 2020).
In contrast, the proposed method is a light-weight design that resembles standard model-free RL algorithms to a great extend, thus lies in a space that is mostly complementary to hierarchical RL methods.
Goal-conditional RL.
Goal conditional RL methods learn a goal-conditional policy and address the efﬁciency of policy learning by using techniques such hindsight experience replay.
There are some recent efforts on generating goals beyond simply relabeling the experienced states ( e.g. Ren et al., 2019; Pong et al., 2020; Pitis et al., 2020).
In this setting, it resembles the hierarchical RL structure, where the goal generator plays the role of a high-level meta-controller and the goal-conditional policy functions as a low-level worker.
Differently, we focus on how to improve action sequence based exploration for plain model-free RL algorithms without resorting to goal-oriented designs.
Therefore, we view the proposed approach as orthogonal to goal-conditional RL and they could potentially be integrated together to bring further improvements.
Model-based RL and Plan-to-Explore.
Model-based RL is another large family of methods that has been explored from many perspectives ( e.g. Sutton, 1991; Silver et al., 2016; Srinivas et al., 2018;
Chua et al., 2018; Agarwal et al., 2018; Lowrey et al., 2019; Hoppe et al., 2019).
i)
Learning with latent dynamics.
While some works may require a given model (Silver et al., 2016; Lowrey et al., 2019), many recent methods work with a latent dynamics model ( e.g. Oh et al., 2017; Silver et al., 2017; Schrittwieser et al., 2019; Hafner et al., 2020).
Our work is related to Oh et al.
Silver et al.
; Schrittwieser et al.
in the sense that our plan value function is trained by predicting future values, and related to Hafner et al.
(2020) in the way of policy training by maximizing values.
However, these approaches do not leverage their implicit models for temporally extended exploration, which is the focus of our work, thus are orthogonal to our work.
Also, our value function has a dedicated latent dynamics model that is not shared with the plan generator, which offers more ﬂexibilities in structural design ( e.g. the usage of a structural prior Eqn.(4) in plan generator) and learning (each is responsible for its own objective).
Therefore, the overall algorithmic structure resemble standard model-free RL methods, making the transition between them more natural.
ii) Integrating model into model-free RL.
There are many possible ways to integrate model into model-free RL methods ( e.g. Sutton, 1991; Levine & Koltun, 2013a;b; Racanière et al., 2017;
[Figure]
Srinivas et al., 2018).
Imagination-augmented agent (Racanière et al., 2017) uses the predictions from a model as additional contexts for a model-free policy.
Srinivas et al.
(2018) embeds differentiable planning within a goal-directed policy and optimizes the action plan through gradient descent.
Our work is different in that it uses a plan generator to directly generate action sequences conditioned on the current state, much like a model-free policy, without using additional context from a model-based prediction (Racanière et al., 2017) or requiring expensive gradient based plan generation (Srinivas et al., 2018).
iii) Plan-to-explore.
The general idea of plan-to-explore is broadly applicable and has appeared in many categories including hierarchical ( e.g. Konidaris et al., 2010;
Pertsch et al., 2020), goal-conditional ( e.g. Ren et al., 2019; Pong et al., 2020; Pitis et al., 2020) as well as model-based
RL ( e.g. Pathak et al., 2017; Henaff, 2019; Shyam et al., 2019; Sekar et al., 2020).
In this work, we focus on improving standard model-free RL methods by generating coordinated action sequence for temporally extended exploration, which has a different focus and complementary to these approaches.
E XPERIMENTS
Baselines Details.
We implement and compare the performance of the proposed method with seven baseline methods (Xu et al.). : i) SAC : the Soft Actor Critic (Haarnoja et al., 2018) algorithm which is a representative model-free RL algorithm; ii) EZ : temporally-extended (cid:15) -Greedy exploration ( (cid:15)z -Greedy)
(Dabney et al., 2021), which repeat the sampled action for a random duration during exploration; iii) FAR : Fixed Action Repeat (Mnih et al., 2015), which repeats the policy action for a ﬁxed number of steps; iv) DAR : Dynamic Action Repeat with TempoRL (Biedenkapp et al., 2021), which improves over FAR with a action-conditional policy for predicting a duration to repeat, outperforming non-conditional alternatives (Sharma et al., 2017); v) TAAC :
Temporally Abstract Actor Critic (Yu et al., 2021), which uses a switching policy for determining whether to repeat the previous action, instead of generating a duration to repeat beforehand as in DAR (Biedenkapp et al., 2021); vi) ICM :
Intrinsic Curiosity Module (Pathak et al., 2017), which is a model-based method for reward bonus based exploration.
vii)
Dreamer : a model-based RL method which trains the policy by back-propagation through the learned latent dynamics (Hafner et al., 2020).
SAC is used as the backbone algorithm for ii) ∼ vi) .
More results and resources are available on the project page.
[Figure]
SAC
EZ [Figure] 199 episodes
GPM
SAC
EZ 0.
1.0 Environment Steps 1e
A v e r age
R e t u r n
CARLA 0.0 0.2 0.4 0.
0.8 1.0 Environment Steps 1e
A v e r age
R e t u r n
SACEZFARDARTAACGPM (b)
Benchmark Results. We conduct experiments on 12 benchmark tasks and the results are shown in Figure 3 (image-based CARLA results are deferred to Figure 4). It is observed that vanilla
SAC can achieve reasonable performance on most of the tasks, albeit with a large performance variance and high sample complexity. FAR shows better performance than SAC on some tasks ( e.g.
BipedalWalker and FetchPush ), but no clear improvements ( e.g. InvertedPendulum ) or even worse performance on some other tasks ( e.g. CartpoleSwingUp ).
This is potentially because of the relatively more restrictive policy with a ﬁxed repetition scheme.
EZ outperforms SAC on some tasks ( e.g. MountainCarContinuous and FetchPickAndPlace ) and retain mostly comparable performance to SAC on others, demonstrating the contribution of incorporating temporal consistency to the exploration policy.
ICM and Dreamer improves over SAC on some tasks but not always.
Both
DAR and TAAC show consistent improvements over SAC due to their ﬂexibility on leveraging action repeat to the extend where it is useful.
The proposed GPM demonstrates further improvements in terms of sample efﬁciency on some of the tasks, with ﬁnal performance matching or outperforming those of the baseline methods.
Apart from state-based control, GPM also delivers strong performance in a more challenging image-based control task ( c.f. Figure 4), as detailed in Section 5.2.
State Trajectory Visualization.
To understand the behavior of different algorithms, we further visualize the state visitation distribution and its evolution during training (Figure 12).
It shows that during the initial phase of training, the state coverage of the methods with extended temporal exploration is much higher than that of SAC , which are consistent with the results reported in (Dabney et al., 2021).
Furthermore, as training progresses, GPM is able to ﬁnd the high rewarding state more quickly, visually depicting its improved exploration performance and sample efﬁciency.
5. E ND - TO -E ND A UTONOMOUS D RIVING WITH E MERGED I NTERPRETABLE P LANS
Autonomous Driving (AD) is an important application for learning-based algorithms.
Although the end-to-end approaches have showed encouraging results ( e.g. Bojarski et al., 2016; Toromanoff et al., 2020), one great limitation is their limited interpretability.
Although GPM is also an end-to-end approach, it has improved interpretability, in addition to its better performance.
It is important to note that this ability of generating interpretable plans is emerged from pure reinforcement learning, without direct supervisions.
This is in clear contrast to the methods from the decomposed paradigm, which requires additional labels as direct supervision for training (Phan-Minh et al., 2020; Rhinehart et al., 2020).
We use CARLA (Dosovitskiy et al., 2017) to setup a challenging image-based driving task.
We set the plan length as L = 20 to enable more perceivable results when visualized.
More details including observations, actions, rewards and network structure are presented in Appendix A.5.
Behavior and Performance Comparison .
The exploration behavior comparison of GPM with SAC and EZ are shown in Figure 4 (a).
It can be observed that the effective range of exploration of a
SAC agent increases slowly and is quite limited, demonstrating its inefﬁciency in exploration.
EZ shows enhanced exploration ability over SAC , as depicted by the enlarged range of explored areas.
GPM shows a further improved exploration ability.
More speciﬁcally, GPM shows some directional movement ability during the initial phase of training.
It further shows reasonable turning behavior after about 100 training episodes and more complex behavior with multiple turns after abouttraining episodes.
In terms of performance, it can be observerd from Figure 4 (b) that GPM learns faster than other baseline methods, with ﬁnal performance better than or competitive to other methods. Long i t ud i na l [Figure] [Figure]
Emerged and Evolved Planning .
The evolution of the plans from GPM is visually shown in Figure 5.
It shows the emergence of meaningful plans and the evolution of them along the training process.
This naturally depicts several advantages of GPM over standard end-to-end AD systems: i) it has more interpretable plans, which can serve as a media for communicating the intent of the algorithm; ii) the evolution of the algorithm behavior during training is more perceivable; iii) the plans, with grounded meaning of spatial-temporal trajectories, offer the opportunity of further adjustment according downstream requirements in the AD pipeline, which could potentially narrow down the gap between end-to-end and decomposed approaches and provide an opportunity to integrate.
The plans from
GPM together with the visual context images are provided in Figure 6.
It can be observed that GPM plans are interpretable and meaningful.
Note that this a type of emerged planning ability as there is no direct supervision on it but purely acquired through interactions with the environment.
C ONCLUSION AND F UTURE W ORK
We explore a perspective on using planning for inducing temporally extended, ﬂexible and intentional action sequence for exploration in a generative fashion.
We present a Generative Planning method as an attempt towards this goal.
The proposed approach features an inherent mechanism for generating temporally coordinated plans for exploration, while retaining the major features and advantages of typical model-free RL algorithms.
We hope this work can serve as an initial encouraging step that can inspire more developments in this direction.
For future work, the quality of the plans can potentially be further improved by learning a more reﬁned cost with the help of additional data (Ziebart et al., 2009; Wulfmeier et al., 2016; Choi et al., 2019; Rosbach et al., 2019).
While we have mainly focused on continuous control tasks in this work, it is possible to generalize GPM to handle discrete actions (Tennenholtz & Mannor, 2019).
The network architecture also provides another axis for potential improvements.
We leave the exploration of these interesting directions as our future work.
Acknowledgements.
We would like to thank Jingchu Liu, Lisen Mu, Jerry Bai, Le Zhao, Break
Yang, Lingfeng Sun and anonymous reviewers for discussions and feedback on the project and paper.
We would also like to thank the Horizon AI platform team for infrastructure support.
A A PPENDIX A. GPM A LGORITHM
The overall procedure of the proposed method is presented in Algorithm 1.
Algorithm
Generative
Initialize: θ , φ φ (cid:48) ← φ , D ← ∅ for each iteration do for each environment step do τ old = ∅ if | τ old | is 0 else τ old ← ρ ( τ t − 1 ) (cid:46) time-forward the old plan τ new ∼ Π( s t ) (cid:46) generate a new plan m = 1 if l old =
| τ old | is 0 else m ∼ Categorical([ (cid:15), Q l old ( s, τ new ))
− Q l old ( s, τ old )])
τ t = (1 − m ) · τ old + m · τ new (cid:46) select the actual plan from old and new plans a t ← ω ( τ t ) (cid:46) extract the primitive action for the current step s t +1 ∼ p ( s t +1 | s t , a t )
D ← D ∪ { ( s t , a t , r ( s t , a t ) , s t +1 ) }
(cid:46) save transition to replay buffer end for for each train step do φ ← φ − λ ∇ φ J Q ( φ ) (cid:46) plan value update θ ← θ − λ ∇ θ J Π ( θ ) (cid:46) plan generator update α ← α − λ ∇ α J ( α ) (cid:46) temperature update (cid:15) ← (cid:15) − λ ∇ (cid:15) J ( (cid:15) )
(cid:46) replan threshold update φ (cid:48) ← ηφ + (1 − η ) φ (cid:48) (cid:46) target parameter soft-copy end for end for
J ( α ) =
E a t = ω ( τ ) ,τ ∼ Π (cid:2) − α log π 0 ( a t | s t , α ) − α ¯ H (cid:3) , where π 0 denotes the policy corresponding to the ﬁrst step of the plan, which is similar to the one used in Haarnoja et al.
¯ H is the target entropy set in the same way as SAC (Haarnoja et al., 2018) for controlling the degree of stochasticity of the policy by adjusting the temperature α .
A. D ETAILS ON ρ AND ω O PERATORS
Here we summarize in Table 1 the meaning and implementations of some entities as well as operators that are used in this work to further help understanding.
[Table]
In Table 1, T ego2world is an operator for transforming the plan from ego to world coordinate and
T world2ego the reverse operator.
A. N ETWORK S TRUCTURE AND D ETAILS
Network details for different tasks are summarized in Table 2, where we use [ K ] × L to denote an
MLP with L FC layers and each with the output size of K .
For baseline methods, they use the network structure as shown in Table 2 for both actor and critic networks.
For GPM , we implement the plan generator and plan value network as in Figure 2.
The plan generator ﬁrst use an MLP that has the same structure as baseline methods as shown in Table 2 for encoding the observation s to a feature vector z .
z is used as the input of the Normal Layer, which uses FCs for generating mean and std vectors of a diagonal Gaussian distribution (Haarnoja et al., 2018).
z is passed through a FC layer to the initial state of a GRU network (plan RNN), with its hidden size the same as the single layer size in Table 2.
Action is detached before inputting to the GRU to to enforce the training of
GRU and improve training stability.
A shared FC layer is used for generating the actions after the ﬁrst step, taking the output from plan RNN as input.
The plan value network is implemented with
LSTM (value RNN) with hidden size the same as the single layer size in Table 2.
An MLP decoder with the same structure as those used in other baseline methods (Table 2) are used for plan value estimation, taking the output from value RNN as decoder input.
The ﬁrst step uses one such decoder with parameters not shared with others.
The rest of the steps uses a shared one.
The reason for this is that the conditioning state for the ﬁrst step is different from the rest.
ReLU activation is used for hidden layers.
Following standard practice, the minimum over the outputs of two value networks is used to mitigate value overestimation (Fujimoto et al., 2018; Haarnoja et al., 2018)
[Table]
A. G YM T ASK D ETAILS
The Gym tasks used in this paper and their corresponding Gym environment names are summarized in Table 3.
Fetch tasks are from Plappert et al.
(2018a).
[Table]
A. CARLA T ASK D ETAILS
The CARLA driving task is implemented based on the open source Car Learning to Act (CARLA) simulator
(Dosovitskiy et al., 2017).
More details on sensor/observation/action/reward, network structure etc. are presented in the sequel.
A.5. S ENSORS AND O BSERVATIONS
The following types of sensors are used in CARLA driving task ( c.f. Figure 7):
• Camera : frontal camera capturing RGB image of size 64 × 64 × 3 ; • IMU sensor : 7D vector containing acceleration (3D), gyroscope (3D), compass (1D), providing some measurements on the status of the current vehicle;
• Velocity : 3D vector containing velocity relative to ego-coordinate in m/s; • Navigation : the collection of the positions of 8 waypoints d meters away from the ego vehicle along the route, with d ∈ { 1 , 3 , 10 , 30 , 100 , 300 , 1000 , 3000 } meters.
The positions of the waypoints are represented in 3D ego-coordinate.
The sensory reading is a 24D vector; • Goal : 3D position in ego-coordinate representing the target position the ego car should navigate to and stop at; • Previous action : 2D vector representing the previous action.
More details on the action space are provided in Appendix A.5.3.
The delta time between two time steps is ∆ t = 0 .
05 s,
i
[Table]
e. , the perception and control rate is 20 Hz.
[Table]
[Figure]
A.5. CARLA N ETWORK S TRUCTURE
The network structure used in CARLA task is shown in Figure 7.
This is the common structured shared by all the methods including GPM .
Given the observation containing multi-modality signals, an observation encoder is ﬁrst used to fuse them into a low-dimensional feature vector.
This is achieved by processing each modality of signals with a dedicated encoder with the same dimensionality for the outputs.
More speciﬁcally, the visual encoder is a ResNet-18 based CNN.
Each of other modalities are encoded by a fully connected layer.
The outputs from each of the encoder are fused together with a sum operator followed by a FC layer to produce the feature vector as input to downstream networks.
The observation encoder is shared by both the actor and critic network.
During training, the the gradient from the actor is stopped and the observation encoder is only trained by value learning.
A.5. CARLA C ONTROLLER AND A CTION S PACE
The CARLA environment with its controller interface is illustrated in Figure 8.
It takes as input a setpoint in ego-coordinate, specifying an immediate position to go to for the current step.
For the driving case, the setpoint can be represented in the 2D spatial ego-coordinate.
Therefore there are two dimensions ( x : lateral, y : longitudinal) in the action space, with a = ( x, y ) ∈ [ − 20 , 20] × [ − 20 , 20] .
The actor controls the movement of the car by generating 2D action as the setpoint.
Target direction and speed will be computed based on the setpoint and ego-position (origin), which will be passed to a
PID-controller to produce control signals including throttle, steer, brake and reverse.
These signals will be used by the simulator to generate the next state of the car.
As part of the environment, a shrinkage is applied to the setpoint before computing the control signals:
[Equation]
This is used to balance the portion of the braking behavior w.r.t.
others in the control space.
δ = 3 is used in this work.
A.5. R EWARD
The reward in CARLA environment is consisted of the following components:
[Figure]
• Route distance reward : this reward is computed as the distance traveled along route from the previous to the current time step.
• Collision reward : this is computed as min(collision _ penalty , 0 .
5 ∗ max(0 , episode _ reward)) , where collision _ penalty=20 , episode _ reward denotes the accumulated rewards up to now within the current episode.
Issued only once for a contiguous collision event.
• Red-light reward : this is computed as min(red _ light _ penalty , 0 .
3 ∗ max(0 .
, episode _ reward)) , where red _ light _ penalty=20 .
Issued only once for a contiguous red light violation event.
• Success reward : a success reward of 100 will be received by the agent (ego-car) if its position is within 5 meters to the goal position and its speed is less than 10 − 3 m/s.
A.5. E PISODE T ERMINATION C ONDITIONS
The episode terminates if either of the following conditions are satisﬁed:
• success: the car’s position is within 5 meters to the goal position and its speed is less than 10 − 3 m/s. • the maximum number of allowed time steps K is reached: this number is computed as K = route _ length min _ velocity × ∆ t , where the min _ velocity = 5 m/s and ∆ t = 0 .
05 s.
The state is marked as a non-terminal state, allowing bootstrapping for value learning at this state; • stuck at collision: if the car is involved in the same collision event and does not move more than 1 meter after 100 time steps.
The state is marked as a non-terminal state, allowing bootstrapping for value learning at this state.
A.5. M AP AND E XAMPLE R OUTES
For CARLA driving task, we use Town01, which is a town with representative layout consisting a number of T junctions and intersections.
The layout of Town01 is shown in Figure 9 (a).
The size of the town is about 410m × 344m .
We also spawn 20 other vehicles controlled by autopilot andpedestrians to simulate typical driving conditions involving not only static objects but also dynamic ones.
[Figure]
The goal location is randomly selected from the set of waypoints.
The agent is spawned to a random valid waypoint location at the beginning of each episode.
A waypoint is valid if it is on the road and is at least 10 meters away from any other actor in the world (vehicles and pedestrians).
Some examples routes are shown in Figure 9(b).
As can be observed from the ﬁgure, the routes constructed in this way have a reasonable amount of diversity and cover representative driving scenarios.
For example, different routes could include different number of turns, directions for turning, static objects (as the routes are different) and also different dynamic objects ( e.g. other vehicles) at different space-time locations.
The task of the agent is to navigate to the goal position following the route and stay still after arrived at the goal location.
A. M ORE I MPLEMENTATION D ETAILS A.6. P LAN S AVING , S AMPLING AND G ENERATION
In practice, for plan value learning, for each training iteration, we sample a batch of (state, plan, rewards, next-plan-state) tuple { s it , τ it , r it , s in } Ni =1 from the replay buffer.
More concretely, for each state s i t randomly sampled from the replay buffer, we sample an associated plan τ i t as the actions starting from the current step of s it to a variable number of future steps uniformly sampled in the range of 1 and maximum plan length L , i.e. τ it =
[ a it , · · · , a it + l − 1 ] where l ∼ uniform(1 , L ) .
The reward vector is consisted of the rewards along the plan r it =
[ r it +1 , · · · , r it + l ] .
The plan terminal state s in is the state at the time step after the last step of τ it , i.e. s in =
s it + l .
Therefore, in effect, we use sampled plan for plan value learning.
By constructing plans in this way, we have maximal ﬂexibility of leveraging the action sequences from replay buffer with minimal assumption on the plan distribution.
During training, l commit _ target is set to 0 .
5 L .
Note that in the case of driving (and other similar scenarios), since the plan is a spatial trajectory in the ego-coordinate, and the plan update operator in this case is ρ ( τ ) =
T world2ego ◦ T shift ◦
T ego2world ( τ ) =
T world2ego ◦T shift ( τ (cid:48) )
( c.f.
Table 1).
To enable the construction of plans from action sequences sampled from the replay buffer (not necessarily aligned with the plan boundary during rollout), we need to save the a (cid:48)
=
ω ( τ (cid:48) ) into replay buffer, i.e. D ← D ∪ { ( s t , a (cid:48) t , r ( s t , a t ) , s t +1 ) } .
a (cid:48)
=
ω ( τ (cid:48) ) actually denotes the setpoint in world coordinate.
In this case, since the actions saved in replay buffer are in world coordinate, we can construct plans in world coordinate by by sampling sequences from the replay buffer, in the same way as described above, i.e. τ (cid:48) t
=
[ a t , · · · , a (cid:48) t + l − 1 ] where l ∼ uniform(1 , L ) .
Then the plans in ego-coordinate can be obtained as T world2ego ( τ (cid:48) t ) .
For the standard Gym tasks, since both T world2ego and T world2ego are identity operators, we have τ (cid:48) ≡ τ , meaning we can simply save the current primitive action a = ω ( τ ) into replay buffer.
A scaling factor can be used for the replanning distribution Categorical( z κ ) .
We simply use κ =1 .
For testing the plan generator after training, the mode of the policy is used, by taking the mean of the
Gaussian policy used in the plan generator and taking the mode of Categorical( z ) for replanning.
A.6. P LAN G ENERATOR IN A UTONOMOUS D RIVING
We implement f used in plan generator as follows
[Equation]
] (9) where ∆ a t + i −
=
( a t + i − 1 − a t
+ i − 2 ) .
This actually incorporates a linear prior into f , which is useful for tasks with position-based controls.
A smoothing cost is also applied to the plan to minimize the maximum absolute value of its third order derivative max | ...
τ | (Kyriakopoulos & Saridis, 1988).
In the most basic form, the recurrent plan generator can run at the scale of time-step in order to generate the full plan.
In practice, it can be adjusted according to the actual situation, and running at a coarser scale, which has the advantage of lower-computational cost.
For example, we can let the policy only output a few number ( k ) of anchor points and interpolate between them to get the full plan using a particular form of interpolation methods.
Linear interpolation is used in this work.
A. M ORE R ESULTS A.7. A BLATION R ESULTS ON P LAN L ENGTH AND T ARGET C OMMITMENT L ENGTH
Impact of Plan Length.
The results with different plan lengths are shown in Figure 10(a).
It is observed that the overall the performance is relatively insensitive to plan length, although the performance could be further optimized by tuning this hyper-parameter.
Target Commitment Length.
The results with different target commitment lengths are shown in
Figure 10(b).
The results indicate that the performance of the proposed method is relatively stable across different target commitment lengths, potentially due to its ability in generating plans with adaptive shapes.
[Figure]
A.7. O BSERVATIONAL D ROPOUT : T HE I MPORTANCE OF A DAPTIVE P LANS
Here we compare the proposed method and action repeat under the setting where a ﬁxed replanning duration is used.
For action repeat, this corresponds to the FAR algorithm.
For GPM , we also execute the full plan before switching (denoted as GPM-commit ).
This actually an instance of observational dropout (Freeman et al., 2019), which requires the algorithm to extrapolate controls generated from available observations to those time steps where observations are unavailable.
The results are shown in Figure 11.
This results highlight the importance of adaptive form of plans on performance.
Fixed form of plan as used in FAR might be applicable to certain types of tasks ( e.g. CartPoleSwingUp ).
However, for other tasks, it might be less appropriate and leads to signiﬁcant performance decreases with increasing plan length ( e.g. , LunarLanderContinuous ).
In contrast, the proposed approach shows stable performance across a wide range of plan lengths.
By contrasting the performance of
FAR with GPM-commit , we highlight the importance on the adaptiveness of the plans.
[Figure]
A.7. MPC V . S . GPM
[Figure]
t
[Figure]
MPC has been typically used together with planning for obtaining the control policy (Lowrey et al., 2019).
This strategy is appropriate when a model is given (Lowrey et al., 2019).
However, for exploration in RL,
MPC is less desirable as it essentially switches the plan at every step, not leveraging the temporal property of the plans, as shown on the right by comparing with a variant of using MPC together with GPM (GPM-MPC).
A.7. E XPLORATION T RAJECTORY V ISUALIZATION AND C OMPARISON
[Figure]
v
[Figure]
l o c i t y
[Figure] [Figure]