SPP-RL: S TATE P LANNING P OLICY R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We introduce an algorithm for reinforcement learning, in which the actor plans for the next state provided the current state.
To communicate the actor output to the environment we incorporate an inverse dynamics control model and train it using supervised learning.
We train the RL agent using off-policy state-of-the-art reinforcement learning algorithms: DDPG, TD3, and SAC.
To guarantee that the target states are physically relevant, the overall learning procedure is formulated as a constrained optimization problem, solved via the classical Lagrangian optimization method.
We benchmark the state planning RL approach using a varied set of continuous environments, including standard MuJoCo tasks, safety-gym level 0 environments, and AntPush.
In SPP approach the optimal policy is being searched for in the space of state-state mappings, a considerably larger space than the traditional space of state-action mappings.
We report that quite surprisingly SPP implementations attain superior performance to vanilla state-of-the-art off-policy RL algorithms in a wide class of robotic locomotion environments.
I NTRODUCTION
Research on reinforcement learning (RL) has brought a tremendous number of successful applications in diverse fields of science and technology.
Application areas of RL can be split into two classes: discrete and continuous.
Here, we are interested in continuous simulation environments, mostly in the robotics domain.
Despite the magnitude of successful applications of RL in this domain, two of the main issues: sample efficiency and interpretability of RL trained agents, persist.
These problems have utmost practical importance, especially for mission-critical applications.
The current methods often require a vast amount of experience for training.
The decision-making process of trained agents is not interpretable, sometimes resulting in finding proxy solutions.
Thus, it is vital to research new RL algorithms that may partially solve the mentioned problems.
Traditionally, RL is based on the principle of searching for the optimal policy within the space of state-action mappings.
We propose a new algorithm based on the principle of training an actor (a policy) operating entirely in the state space (state-state mappings).
We call such policies the state planning policies (SPP), whose actions determine desired trajectories in the state-space.
The task of training SPP may initially seem infeasible due to a significantly larger dimension of states than of actions.
Nonetheless, quite surprisingly, we show that the approach is feasible and often leads to significant improvements in average performance for a class of robotic locomotion tasks.
We call our approach State Planning Policy Reinforcement Learning (SPP-RL) .
It is a generic approach for problems specified using continuous environments.
The main building block of SPP-RL – the RL agent can be implemented using virtually any RL algorithm.
We chose to develop our approach using the state-of-the-art off-policy DDPG (Lillicrap et al., 2016), TD3 (Fujimoto et al., 2018), and SAC (Haarnoja et al., 2018a) algorithms.
Note that, in SPP-RL we need another trainable model to communicate the policy output to the environment; as such, we incorporate a learnable inverse dynamics control model (IDM), see Fig.
The overall algorithm optimizes the policy simultaneously with CM .
To ensure that the policy target states satisfy physical and under-actuation constraints, we formulate a constrained optimization objective for policy training.
Our work lies within the category of RL methods that already implemented state-state policies, including work on hierarchical RL like Nachum et al.
(2018), and the D3G algorithm from Edwards et al.
Summary of results.
Although the SPP-RL algorithm searches for the optimal policy within a much larger space, our performance benchmarks revealed that SPP-RL implementations often outperform their vanilla RL counterparts.
Experiments performed in Ant & Humanoid tasks from MuJoCo suite (Todorov et al., 2012; Brockman et al., 2016) reveal that SPP-DDPG outperforms DDPG.
Experiments in Safety-Gym Level 0 environments (Ray et al., 2019) demonstrate that SPP-TD3 and
SPP-SAC outperform by a great margin TD3 and SAC, respectively.
Experiments in AntPush task (Nachum et al., 2018) show that SPP-TD3 outperforms hierarchical RL method HIRO (Nachum et al., 2018) and provides some interpretability of the agent behavior.
We hypothesize that the superior performance of SPP-RL in the tested continuous environments originates in more efficient state-space exploration by state-state policies than traditional state-action policies; here noise is being added to target states rather than actions.
To argue this, we performed series of experiments, including evaluation of a shadow agent utilizing experience from SPP and vanilla replay buffers (Sec. 5.4) and a study of the distributions of states gathered in different replay buffers (App. E.2).
To analyze which features of the algorithm are crucial for superior performance, we report on a thorough ablation study of SPP-TD3 (App. E.1).
We implemented SPP-RL methods as a modular PyTorch library shared as open-source.
SPP-RL algorithms are derived from their vanilla RL counterparts, making extending the library with new
RL algorithms straightforward.
We also share videos with test episodes of the trained agents to accompany benchmark plots (web, 2021).
Last but not least, we demonstrate theoretical convergence of clipped Double Q-learning implemented in SPP-TD3 algorithm within the classical stochastic processes, finite-state/action spaces setting and in environments based on rigid body dynamics model, refer to App. B.3.
1. R ELATED WORK
We present a (non-exhaustive) list of related works; refer to Tab. 1 for a perspective on related work.
The closest approach to ours is the D3G algorithm introduced by Edwards et al.
(2020), which includes state planning policies, and introduces a novel form of the value function defined on statenext state pairs.
There are two main ways our method is distinct.
First, SPP employs the classical formulation of the value function.
Also, we do not include a forward dynamics model nor the cycle loss.
Instead, to guarantee consistency of the policy target-states in SPP, we formulate a constrained optimization problem (compare Fig. 2) solved via Lagrangian optimization.
Our work builds on the classical RL algorithms going back to REINFORCE (Williams, 1992),
Asynchronous Actor-Critic (Mnih et al., 2016), and especially the off-policy actor-critic algorithms including Q-Prop (Gu et al., 2017), DDPG (Lillicrap et al., 2016), SAC (Haarnoja et al., 2018a;b), and
TD3 (Fujimoto et al., 2018).
State planning policies have been used in hierarchical RL (HRL) methods like HIRO (Nachum et al., 2018) and FuN (Vezhnevets et al., 2017).
Contrary to HRL SPP-RL approach does not employ a hierarchy of multiple policies nor state conditioned value functions.
Training predictive models (like IDMs) is fundamental for the model-based RL approach including algorithms: a locally linear latent dynamics model (Watter et al., 2015), model-based planning for discrete and continuous actions (Henaff et al., 2017), model-predictive control (Chua et al., 2018), and model based policy optimization (Janner et al., 2019).
We deployed IDMs for mapping currenttarget states to actions; other applications of IDMs in RL include the context of planning: search on replay buffer (Eysenbach et al., 2019), episodic memory graph (Yang et al., 2020), and topological memory for navigation (Savinov et al., 2018).
Existing many other applications of IDM in context of RL including: policy adaptation during deployment (Hansen et al., 2021), sim to real transfer (Christiano et al., 2016), adversarial exploration (Hong et al., 2020), and curiosity-driven exploration (Pathak et al., 2017).
1. B ACKGROUND
Following the standard setting used in RL literature, we work with infinite horizon Markov decision process (MDP) formalism ( S , A , P, r, ρ 0 , γ ) , where S is a state space , A is a action space , P : S ×
A × S → [0 , 1] is a transition probability distribution , r : S ×
A → R is a reward function , ρ 0 is an initial state distribution , and γ ∈ (0 , 1) is a discount factor .
From now on we assume that the MDP is fixed.
In RL the agent interacts with E in discrete steps by selecting an action a t for the state s t
[Table]
at time t , causing the state transition s t +
= E ( s t , a t ) , as a result the agent collects a scalar reward r t +1 ( s t , a t ) , the return is defined as the sum of discounted future reward R t = (cid:80)
[Table]
γ
[Table]
i
[Table]
r ( s i
[Table]
a i )
[Table]
The goal in RL is to learn a policy that maximizes the expected return from the start distribution.
S TATE P LANNING P OLICY R EINFORCEMENT L EARNING A PPROACH
Our SPP approach is rooted in state-state reinforcement learning, by which we mean setting in which
RL agent is trained to plan goals in the state-space, the approach already employed e.g., in HRL, planning, D3G RL algorithms (see Tab. 1).
In SPP a state planning policy π given the current state s t outputs z t – the desired target state to be reached by the environment in the next step.
Forcing the environment to reach the desired state requires translating the target state to a suitable action a t .
Hence, we employ an additional model capable of mapping the current state-target state pair ( s t , z t ) to the action a t – a (trainable) IDM model.
Ideally, we like to have consistency z t ( s t )
≈ s t
+1 .
The consistency cannot be guaranteed a-priori, is rather achieved in SPP setting by employing a constrained optimization approach.
A diagram illustrating SPP approach is presented in Fig.
We have freedom of choice of the particular RL algorithm (RL agent) and IDMs implementations.
Currently, we use feed-forward neural networks, and RL Agent using implementations of the stateof-the-art off-policy RL algorithms: DDPG (Lillicrap et al., 2016), TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018a).
We present details of SPP-RL implementation in Sec.
4 using as the example SPP-DDPG.
The encountered experiences during the execution of an off-policy RL algorithm are stored in replay buffer D .
The main building block of SPP-RL are the state planning policies, intuitively a state planning policy selects a desired trajectory in the state-space of the environment.
Definition 1.
We call a state planning policy a map π θ : S → P ( S ) parametrized using a vector of parameters θ ∈ R n ( θ ) , and we denote π θ ( z | s ) a probability of the desired target state z ∈ S for the given current state s ∈ S .
We call a deterministic state planning policy a parametrized map π θ : S → S , and we denote π θ ( s )
=
z .
We assume that π has continuous and bounded derivatives with respect to θ .
We will call state planning policy whenever it is clear from the context deterministic/stochastic and omit the parameter subscript π = π θ .
Besides the state planning policy (Def. 1) the second main building block of the overall SPP agent is a model for mapping the current state-target pair ( s t , z t ) to suitable action a t .
Following the existing literature, we call such model the inverse dynamics control model (IDM), or simply the control model .
Definition 2.
For a given MDP ( S , A , P, r, ρ 0 , γ ) .
Let s, z ∈ S .
We define the control model :
CM : S × S
→ A ,
CM ( s, z ) = a, i.e. for the given two states CM computes the action a .
We call s, z
the initial state and the target state respectively, where a informally satisfies arg max b ∈S P ( s, a, b ) ∼ z for stochastic E , or z ≈
E ( s, a ) for deterministic E .
Obviously, in order to work, SPP requires consistency of the target states generated by the policy with the actual next-states of the environment.
We call this property the state consistency property (or simply consistency) of π , refer to Fig.
As it may be intractable to verify SPP for all possible interactions in continuous environments, we are interested in guaranteeing the state consistency for the experiences stored in the replay buffer.
Property 1.
Let D be a replay buffer, CM be an IDM and π be a (SPP) policy.
We say that π has the state consistency property with threshold d > 0 if it holds that
[Equation]
for deterministic π we have z t
= π ( s t ) .
Given ( z t , s t +1 ) , we call distance ∥ z t − s t +1 ∥ 22 the stateconsistency distance (refer Fig. 2).
We will often assume that d is known from context and omit ’with threshold d > 0 ’.
[Figure]
[Figure]
E NSURING S TATE C ONSISTENCY P ROPERTY BY C ONSTRAINED O PTIMIZATION
Our SPP algorithm is utilizing three parametrized models: IDM CM ψ , policy model π θ , and Qfunction model(s) Q ϕ .
In our current implementation, all of the models are feed-forward neural networks.
One way of ensuring the state consistency property (Prop. 1) is to modify the policy training loss, such that the expected values are maximized under fixed state consistency distance penalty.
However, such an approach has many disadvantages, e.g., choosing appropriate learning temperature ( λ ) for the state consistency penalty is a very delicate issue, see the ablation study in
Appendix E.1.
It is easy to notice that setting its value too low would result in π biased towards nonphysical target-states.
On the other hand, setting its value too high would make π overly conservative for off-policy states.
Hence, we find a solution relying on constrained optimization more appealing for the studied problem.
Namely, the objective for policy training is to maximize the sum of discounted rewards assuming a fixed threshold for the state consistency distance.
Definition 3.
Let π be a state planning policy (Def. 1), CM be a control model (Def. 2), D be the replay buffer with experience generated by executing an off-policy RL algorithm.
In particular z t ’s are target states evaluated on-the-fly by π (susceptible to be changed during the course of algorithm), and s t
+1 ’s are E next-states.
Let d > 0 be a fixed hyperparameter.
We define the constrained objective for state planning policy π as follows
[Equation]
where d is a hyperparameter for determining the allowed threshold for the expected divergence of predictions from actual next-states, equation 1a is an expectation over the policy trajectories generated by both of the state planning policy and IDM (trajectory is composed out of tuples ( s t , z t , a t ) ).
The whole optimization process of equation 1a is being performed off-policy, see Sec. 4.
The constrained objective in equation 1 is being solved using the standard Lagrange multiplier method.
The max-min Lagrangian objective L ( π, λ ) for the constrained optimization problem takes the form max
[Equation]
For more details refer to App.
C.1.
A LGORITHM I MPLEMENTATION
We briefly present here details of the SPP Algorithm implementation, more detailed discussion can be found in Appendix C. We implemented SPP-DDPG, SPP-TD3, and SPP-SAC as a modular
Python library within PyTorch framework (Paszke et al., 2019).
All gradient optimization steps were performed using the Adam optimizer by Kingma & Ba (2014).
Many of the algorithmic choices were motivated by the Spinning Up RL on-line resource (Achiam, 2018).
We publish the modular
SPP-RL software package as open-source (web, 2021).
For illustrative purposes, we present the pseudo-code of the full SPP-DDPG algorithm in Algorithm 1.
SPP-SAC and SPP-TD3 algorithms are presented in Appendix C. We provide theoretical derivations concerning SPP-RL algorithms, including convergence in the classical finite setting and policy gradient analogues to vanilla RL in
Appendix B. We emphasize that SPP algorithms are not using any extra samples, i.e. the samples utilized for ICM training are added to the buffer and then reutilized for RL training, and if the buffer is full new samples are not being added anymore.
An important caveat of our policy implementation in SPP-DDPG, not present in the vanilla DDPG, is that the output of π is being normalized in order to reflect the physical bounds.
Contrary to vanilla DDPG where π outputs actions within well-defined uniform bounds, in SPP a suitable normalization of target state π output is being computed online – depends on the past E observations.
Implementation of π is a feed-forward neural network (refer to Appendix D for details) with tangential outputs bounded within [ − 1 , 1] .
Hence, we normalize the output of neural network π by utilizing the current mean and min/max values of the past observations in replay buffer D .
We recompute mean and min/max values after each episode of the algorithm.
Our base implementation of DDPG algorithm is parametrized by the usual hyperAlgorithm
[Table]
do
[Table]
[Table] [Table]
of
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
end end until convergence ; parameters including episode length, update batch size, Polyak averaging parameter ( τ ), actor and critic learning rate l , maximal episode length, number of test episodes, γ .
To ensure that we perform minimization equation 6 within the domain of positive λ values, we optimize the parameter of the softplus function.
All relevant hyper-parameters are provided in Appendix D.
We performed a thorough ablation study of the different features implemented in SPP-TD3 algorithm including the Lagrangian multipliers method (and different values of fixed λ ), target state normalization, state-state critic ( Q ( s, s ′ ) ); we report results in Appendix E.1.
E XPERIMENTAL E VALUATION
To show the feasibility of our method, we performed experiments on a set of benchmarks using continuous environments, most of them having large space dimensions.
We performed all of the reported experiments using the default vector state input.
We show that SPP-RL implementations compare favorably to their vanilla RL counterparts.
As our SPP approach differs considerably from the vanilla off-policy RL, we performed a thorough hyper-parameter sweep from scratch.
We provide the hyper-parameter values from the actual SPP implementations in Appendix D. For the sake of presentation we share videos with example test episodes rendered using the trained actors and high-resolution benchmark plots.
All of our experiments are reproducible, we share the sources, training, evaluation, and trained models online (web, 2021).
All of the reported experiments were run using CPU only, and a single experiment was always run on a single CPU core, i.e., we have not performed collecting experience in parallel.
The experiments were performed on an example machine: AMD Ryzen Tr.
1920X, 64 Gb RAM, Ubuntu OS 18.04.
Example average time of execution of 10 6 steps stands at SPP-DDPG 5 hrs 58 ′
DDPG 3 hrs
(SPP-
SAC 19 hrs 20 ′
SAC 11 hrs.
5. C LASSICAL M U J O C O
First, to measure the performance of SPP-RL we run benchmarks in the MuJoCo control tasks (Todorov et al., 2012) from OpenAI gym (Brockman et al., 2016):
Ant and Humanoid .
The experiments were performed using solely the vector state input.
We present the results in Fig.
We compare the performance of SPP implementations with vanilla RL implementations in terms of average test episodes return.
We emphasize that SPP algorithms do not increase the overall sample efficiency of the RL procedure.
The samples utilized for ICM training are added to the buffer and then reutilized for RL training, and if the buffer is full new samples are not added anymore (refer to Algorithm 1).
The algorithm does not take advantage of a pre-trained ICM before the actual RL training, rather optimizes π and trains ICM simultaneously.
However, as reported in Fig.
3, quite surprisingly, SPP based RL algorithms perform on par (TD3 & SAC case) or significantly better (DDPG case) than corresponding vanilla RL implementations in the MuJoCo environments.
Analyzing the case of (SPP-)DDPG Figs.3a, 3d reveals that SPP significantly outperforms the vanilla implementation, as the vanilla implementation does not converge on those envs (compare with Achiam (2018)).
Analyzing the case of (SPP)-TD3 reveals that for Ant env (Fig.
3b) SPP-TD3 performs on par with vanilla TD3 and outperforms by far D3G – the most related method.
For Humanoid env (Fig. 3e) we see that SPP-TD3 outperformed vanilla TD3, the visible high variance of vanilla TD3 caused by a single run, which did not converge, D3G do not converge in this env.
There is a visible divergence of the results we obtained for D3G in Ant from those reported by Edwards et al.
We used
MuJoCo v.1.5 library version that is known to be compatible with OpenAI gym; it is known that using v.2.0 results in zeroing out a part of observation space (in Ant & Humanoid) (gym, 2019), which may alter the results.
5. S AFETY -G YM (L OCOMOTION T ASKS )
In the next set of benchmarks, we use environments from the safety-gym suite by Ray et al.
Currently, we employed only Level 0 environments (which does not involve the cost function for violating the safety).
We find Level 0 tasks from the safety-gym suite as the perfect ground to study the performance of SPP-RL in robotic locomotion environments, the goal being to steer agents (robots) to solve planar goal-reaching tasks.
Moreover, we concentrate on difficulties arising from higher dimensionality of the state (and actions) space rather than maximizing returns under safety constraints.
The experiments were performed using solely the vector state input.
We leave investigating
[Figure]
the higher-level environments considering the cost function as a topic of future research.
We chose a subset of the most challenging Level 0 tasks, including Car-Push, Doggo-Goal, Doggo-Button environments.
We also create a custom environment (termed Doggo-Columns) based on Doggo-Goal with additional 10 fixed pillars placed in the arena, obscuring the paths toward the goal.
We evaluate our SPP-TD3 implementation against state-of-the-art off-policy algorithms like TD and SAC.
We do not include on-policy algorithms like PPO (Schulman et al., 2017) or TRPO (Schulman et al., 2015), as their performance in the studied environments is inferior to off-policy, refer to
Ray et al.
and the project web-page.
The results presented in Fig.
4 clearly show that SPP-TD3 is superior to vanilla off-policy algorithms within the studied safety-gym environments.
Also, there is a noticeable difference in the learned behavior of the trained agents.
The agents trained using the SPP-RL approach show smarter and more efficient behavior; for instance, the trained using SPP doggo robot learned an efficient gait of moving backward to mark a goal or press a button.
For comparison, we publish videos of the trained agents online (web, 2021).
We argue that the performance boost exhibited by SPP-RL algorithms over vanilla counterparts is due to improved exploration in some sense.
In Sec.
5.4 we show results from evaluating a TD shadow agent, i.e. vanilla TD3 agent utilizing for training some portion of experience from SPPTD3 replay buffer.
In Appendix E.2 we investigate differences in distribution of states collected by both of the methods.
5. H ARDER E XPLORATORY T ASK A NT P USH
In order to demonstrate interpretability aspects of the SPP approach, we describe an experiment in
AntPush environment from Nachum et al.
The experiments were performed using solely the vector state input.
The task is to control the ant such that it reaches the goal.
The goal is hidden within a chamber behind a block.
Therefore, Ant needs to learn to walk around the block and push it to the right first before eventually reaching the goal.
Success is defined as finishing the episode within a radius 5 from the goal.
We benchmark SPP-TD3 against the state-of-the-art
3.0 steps    a v e r age r e t u r n
SPP-TD3TD3 vanilla SAC vanilla SPP-SAC
[Figure]
3.0 steps    a v e r age r e t u r n
SPP-TD3TD3 vanilla SAC vanilla SPP-SACD3G
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
a v e
[Figure]
r e t u r n
SPP
[Figure]
[Figure] [Figure]
50a v e r
[Figure]
u r n
[Figure]
hierarchical RL HIRO method by Nachum et al.
Specifically, we used the implementation by Qin (2021).
Instead of reporting the achieved success rate of a single training run like in Nachum et al.
(2018), which may be spurious if a lucky seed is chosen
, we report the mean and std.dev.
of the AntPush success rate using 10 random seeded training runs.
Our experiments revealed that
HIRO is highly susceptible to the random seed used.
Only a single HIRO agent out of 10 trained using random seeds in total achieved a positive success rate, comparing to 7 out of 10 SPP-TD agents successfully learned to solve the task.
The performance reported in Fig.
5a shows SPP-TD is eventually superior to HIRO.
Example two solution paths are marked on Fig.
5b (blue curves).
The right path is suboptimal as Ant blocks the entrance to the chamber where the goal is.
The left path is optimal, Ant traverses to the left to push the red block away and open the passage towards the goal (green arrow).
Finally, Figs 5c, 5d show the obtained paths in the state space (blue dashed), and the policy target states z t ’s (orange solid), only the first two coordinates corresponding to the position of the Ant body in x, y coordinates are illustrated.
Observe that in the case of the suboptimal path in Fig.
5c, the planned path diverts to the left from the actual path (blue dashed), which indicates that the agent learned and attempted the correct behavior of pushing the red brick away and successfully open the entrance to the goal.
In this case, however, the block is not movable; it is stuck as it was pushed forward before, hence as we see, the actual path in the state-space diverts in the middle.
Figs 5c,5d show that apparently, the policy target states path being more erratic than the actual path in the state space.
Erratic behavior can be mitigated by adjusting the hyperparameter d in equation 1b (the smaller d , the closer the paths will be).
Nonetheless, the policy target paths (orange) in Figs 5c,5d could be potentially used to cluster agent behavior, qualitatively differentiating two example agents executing (sub)optimal path.
This information could then be used to pick appropriate agents for deployment.
[Equation]
E XPLORATION IN SPP-RL
Our experimental evaluation using the safety-gym environments show that SPP-RL implementations outperform by a great margin their vanilla off-policy RL counterparts (TD3 and SAC) in terms of the average returns (See Fig.4).
The performance boost is also visible in case of DDPG in MuJoCo
Ant (Figs. 3a, 3d).
In this section, we argue the performance boost of SPP-RL compared to the vanilla RL counterparts.
Our intuition is that exploration performed in the target-state space rather than in the action space may be more efficient in some cases.
Exploration using SPP policies results in more viable experience being collected in the replay buffer, leading to more efficient Actor & Critic training.
It is also possible that the constrained optimization induces some kind of curriculum.
To confirm the mentioned intuition, we evaluated the performance of a TD3 shadow agent i.e., a vanilla TD3 Actor&Critic trained using (partially) experience collected by the SPP-TD3 agent.
Both of the agents
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
t u
[Figure]
n
[Figure]
were trained in parallel.
The TD3 shadow agent updates were performed using samples drawn from two of the replay buffers.
The replay buffers of the SPP-TD3 and TD3 agent were used according to a 50/50 ratio.
The results are presented in Fig.
Such TD3 shadow agent outperforms vanilla
TD3, and eventually, its performance matches SPP-TD3 agent’s in all of the studied safety-gym environments, excluding Doggo-Button.
Additionally, to address the question of how the distribution of experiences gathered by vanilla
RL/SPP-RL compares to each other, we performed an additional experiment in Appendix E.2.
We present plots of the discretized distributions of states encoded using a random encoder and crossentropy of two distributions w.r.t.
the quantity of gathered experience.
[Figure]
a v e r age r e t u r n
SPP
[Figure]
TD
[Figure]
TD3 TD
[Figure]
C ONCLUSIONS
We presented the State Planning Policy Reinforcement Learning (SPP-RL) – an approach for reinforcement learning, where the policy is selecting target states.
Experiments performed on continuous benchmark environments often show the superior performance of SPP-RL compared to state-of-theart off-policy RL algorithms.
There are various avenues for future work pertaining to this research.
One path is to include in our approach physically informed control models.
Another, important path of work is to implement a long-term policy planning method scheme and application in the safety
RL setting of SPP approach.
R EPRODUCIBILITY S TATEMENT
To ensure reproducibility of our results we published a supplementary material web page available at https://sites.google.com/view/spprl, there a link to a public github repository.
The repository contains the sources, training & evaluation scripts, and trained models.
The webpage also contains the benchmark plots in higher resolution, and also include videos with agents trained using different methods in the considered environments.
A A PPENDIX
Videos visualizing the trained agents, and link to the git software repository with SPP-RL algorithms implementations are available online https://sites.google.com/view/spprl.
B T HEORY OF S TATE P LANNING P OLICY RL A LGORITHMS
In this section we establish foundations of SPP-RL theory.
B. S TATE - VALUE AND A CTION - VALUE F UNCTIONS
In order to employ in SPP RL agents based on the state-of-the-art RL actor-critic algorithms, we start with defining the classical state-value and action-value functions in SPP setting.
Recall R t = (cid:80)
Ti = t γ ( i − t )
r ( s i , a i ) is the sum of discounted future rewards.
The classical Q -function is
[Equation]
however, in our setting we use state planning policies (Def. 1) and we do not directly sample an action using the policy a i ∼ π .
Instead we sample a target state
z i ∼ π .
Then the actual action a i is being computed a i = CM ( s i , z i ) .
Hence, the Q -function in SPP setting can take the form
[Equation]
i.e. the expected value of following the trajectory planned by the policy π in the state-space, and using IDM CM .
As we use deterministic IDM map CM : ( s t , z t ) (cid:55)→ a t , we further reformulate equation 2 into
[Equation]
Analogously we define the state-value function V π, CM
[Equation]
Observe that if the parameters vector ψ of CM is fixed, both of the formulations equation 2 and equation 3 are equivalent as CM maps ( s t , z t )
(cid:55)→ a t .
However, in SPP CM is being trained, and its parameters are being changed during the execution of the full SPP algorithm.
Therefore, we find the formulation equation 3 more suitable, as the established theory proves convergence of the Bellman iterates applied to equation 3 towards the optimal Q .
We evaluated the first variant of Q -function equation 2 in the ablation study presented in Appendix E.1 and it does not perform well.
For the Q -function equation 3 we show theoretical convergence within the established framework in App.
B.3.
B. P OLICY G RADIENT T HEOREM
In this section, for the sake of completeness we include the fundamental Policy Gradient Theorem in the setting of SPP policies.
We provide a formula for the gradient of the standard reward function in the deterministic policy gradient setting (Silver et al., 2014), which is applied in DDPG off-policy
RL algorithm (Lillicrap et al., 2016), and also generalized in SAC off-policy algorithm (Haarnoja et al., 2018a) for stochastic policies.
Observe that the standard reward function in the deterministic policy gradient setting ( π and CM are deterministic) is given by
[Equation]
(5) where d β ( s ) is the stationary discounted state distribution.
Let the reward in the deterministic setting be given by equation 5.
Assume CM is deterministic and differentiable, rest of assumptions like in (Silver et al., 2014, Thm. 1).
Then, the formula for the standard policy gradient follows from (Silver et al., 2014, Thm. 1)
[Equation]
It is a straightforward extension of the formula given in (Silver et al., 2014, Thm. 1), applying one additional step of the chain rule.
Observe that in our algorithm we compute the policy gradient of the Lagrangian objective L ( π, λ ) in the deterministic setting, i.e.,
[Equation]
where D is an experience replay buffer (given), and J is the standard reward function in equation 5.
The gradient w.r.t.
θ of the second term satisfies
[Equation]
by applying the Leibniz integral rule to exchange order of derivative and integration, and continuity of π θ and its derivative.
In particular, note that the next state s ′ is sampled from the buffer and thus does not depend on the policy.
Therefore, it holds that
[Equation]
E s ∼ d β (cid:104) ∇ θ π θ ( s ) · ∇ z CM ( s, z ) · ∇ a Q ( s, a )
| z
= π θ ( s ) ,a = CM ( s,z ) (cid:105) − λ
E ( s,s ′ ) ∈D (cid:20) ∇ θ π θ ( s ) · ∇ z
∥ s ′ − z ∥ 22 (cid:12)(cid:12)(cid:12) z
= π ( s ) (cid:21) .
B. C ONVERGENCE OF Q-L EARNING IN SPP SETTING
In this section we argue convergence of Q-learning in SPP-TD3 algorithm within a finite setting using the existing framework of convergent stochastic processes based on the classic lemma by
Bertsekas (2005).
Used in Singh et al.
(2000) to prove convergence of SARSA algorithm, and subsequently in Fujimoto et al.
(2018) to prove convergence of TD3 algorithm Fujimoto et al.
We provide the lemma here for completeness.
Lemma B.1.
Consider a stochastic process ( α t , ∆ t , F t ) , t ≥ 0 , where α t , ∆ t , F t : X → R , which satisfies the equations
[Equation]
Let P t be a sequence of increasing σ -fields such that α 0 , ∆ 0 are P 0 -measurable and α t , ∆ t and
F t − 1 are P t -measurable, t = 1 , 2 , . . . .
Assume that the following hold
the set of possible states X is finite.
≤ α t ( x ) ≤ 1 , (cid:80) t α t ( x ) = ∞ , (cid:80) t
α 2 t ( x ) < ∞ w.p. 1 . 3.
∥ E { F t ( · ) | P t ∥ W ≤ κ ∥
∆ t ∥ W + c t , where κ ∈ [0 , 1) and c t converges to zero w.p. 1 . 4.
Var { F t ( x ) | P t } ≤ K (1 + ∥
∆ t ∥ W ) 2 , where K is some constant.
Then, ∆ t converges to zero with probability one.
The proof can be found e.g. in Jaakkola et al.
; Singh et al.
Based on Lemma B.1 Fujimoto et al.
(2018) show the convergence of Clipped Double Q-learning to the optimal value function Q ∗ , as defined by the Bellman optimality equation w.p. 1 .
Theorem B.2 (Thm. 1 in Fujimoto et al.
Given the following conditions:
Each state action pair is sampled an infinite number of times.
The MDP is finite.
γ ∈ [0 , 1) .
Q values are stored in a lookup table.
Both Q A and Q B receive an infinite number of updates.
The learning rates satisfy α t ( s, a ) ∈ [0 , 1] , (cid:80) t α t ( s, a ) = ∞ , (cid:80) t ( α t ( s, a )) 2 < ∞ with probability 1 and α t ( s, a ) = 0 , ∀ ( s, a ) ̸
= ( s t , a t ) .
Var [ r ( s, a )] < ∞ , ∀ s, a .
Then Clipped Double Q-learning will converge to the optimal value function Q ∗ , as defined by the
Bellman optimality equation with probability 1 .
We can apply Thm.
B.2 to show convergence of Q learning in SPP-TD3 algorithm.
First, observe that SPP-TD3 utilizes the same Q -functions, hence, the assumptions 2 , . . .
, 7 can be transferred to
SPP-TD3 setting.
However, the policy part is different.
Let us focus our attention to the assumption 1 of Thm.
B.2.
Observe that in the classical setting the assumption 1 is easily realizable by employing a policy performing either fully random exploration or ε -greedy exploration.
In the studied SPP setting the situation is much more subtle.
We do not have a-priori guarantees that the such an assumption is satisfied.
Observe that even a randomly exploring policy may not realize such assumption.
It will not be satisfied if for example the IDM is degenerate and maps each pair of states to the same action.
However, if we show
• for each pair of reachable states there is a unique action that results in the transition of the previous state to the next state, • the SPP policy outputs only realizable target states.
then assumption 1 of Thm.
B.2 can also be satisfied in SPP-RL setting by employing a fully random or ε -greedy SPP policy.
The first condition above may not be true for general environments, as we can easily imagine an environment, where a particular state may be reached from another state by applying different actions.
However, in the case of rigid body environments we can demonstrate that the desired property is true.
We formalize this in the following lemma.
Lemma B.3.
Let E be a (deterministic) environment based on the rigid body dynamics model, i.e. the state transitions are computed using a time-wise discretization of a continuous and differentiable dynamics model.
Let s, s ′ ∈ S be a pair of reachable states in environment E .
If there exists a, a ′ ∈ A , such that E ( s, a ) =
E ( s, a ′ )
=
s ′ , i.e. actions a and a ′ result in the same state transition, then it holds that a = a ′ .
Proof.
It is known that the dynamics of rigid bodies is governed by the laws of kinematics and by the application of Newton’s second law (Tsai, 1999).
Namely, the equations governing the dynamics of rigid bodies in (time-dependent) generalized position-velocity coordinates ( q, v ) , q : R → R n , v : R → R n take the following form
[Equation]
where M : R n → R n × n is the positive definite, symmetric inertia matrix, k : R n × R n → R n is the vector of external and contact forces, ρ : : R n × R n → R n is the sum of friction components, τ : R →
R n is the applied external force (Pfeiffer & Glocker; Todorov et al., 2012).
Hence, there is a one-to-one mapping of state tuple ( q, v, dvdt , ρ ) and the external force τ (torque).
The one-to-one mapping still holds for realizable environment E based on simulation, where infinitesimal acceleration dv is approximated by the finite difference v t +1 − v t , dt is a fixed time-step, ρ is calculated from the Cartesian body position and the fixed domain constraints, external force τ is calculated from input action a by a linear transformation.
We conclude with the following corollary demonstrating the convergence of Q learning in SPP-TD algorithm within the finite setting.
Corollary B.3.1.
Let
• E be a (deterministic) environment based on the rigid body dynamics model,
i
e. the state transitions are being generated by a time-wise discretization of a continuous and differentiable dynamics model, • the experiences utilized for Q updates be gathered using a random/ ε -greedy SPP (deterministic) policy π , • the target states form π be realizable,
i
e. if z = π ( s ) for s ∈ S , then it holds that z ∈ S , • IDM CM be implemented as a lookup table and receive infinite updates.
Moreover, assume 2 , . .
, 7 of Thm.
B.2.
Then, clipped Double Q-learning in SPP-TD3 will converge to the optimal value function Q ∗ , as defined by the Bellman optimality equation, with probability 1 .
Proof.
From Lemma B.3 it follows that for any pair of reachable states s, s ′ of environment E there exists a unique action a resulting in transition E ( s, a ) = s ′ .
Hence, as CM receives infinite updates and is implemented as a lookup table it will learn to map any of the policy target states (realizable states) to the unique action.
It follows that each state action will be sampled infinite number of times in the limit.
As the rest of Thm.
B.2 assumptions are assumed we obtain the claim.
C SPP-RL A LGORITHM D ETAILS C. L AGRANGIAN O PTIMIZATION
To solve the constrained optimization problem we use the idea of Lagrange multipliers going back to the classical calculus.
Lagrange multipliers were already applied in the RL context in Ray et al.
(2019), where authors developed on-policy RL algorithms for safety RL.
This method uses an adaptive penalty coefficient to enforce constraints.
We find this method plausible, as it translates a constrained objective into solving a classical max-min problem that can be incorporated within any RL algorithm.
The max-min Lagrangian objective L ( π, λ ) for the constrained optimization problem equation 1 takes the form
[Equation]
Intuitively, when the constraint equation 1b is satisfied, then the min w.r.t.
λ is attained for λ = 0 , and causes λ to decrease.
On the other hand, if the constraint is not satisfied, then the min is attained for λ = ∞ , and hence causing λ to increase, the constraint satisfaction receives more weight in the overall max-min objective, eventually forcing the constraint to be fulfilled.
We solve the max-min Lagrangian objective for SPP policy using simultaneous gradient ascent w.r.t.
the policy parameters θ , descent w.r.t.
the Lagrange multiplier λ , updates.
Such an approach can be viewed as an instance of a primal-dual algorithm for constrained MDP.
It is known in the literature that time-steps of Actor, Critic & dual variable gradient updates should be adjusted separately to guarantee convergence, refer to Borkar (2005) and references therein.
The SPP algorithms use three learning rates: actor ( l θ ), critic ( l ϕ ), and dual variable ( l λ ), which were adjusted using a hyper-parameter optimization.
The actor and critic learning rates were equal in all of the studied algorithms, whereas the dual variable was fixed in all algorithms to l λ = 0 . 0001 .
C. SPP-SAC A LGORITHM
The original DDPG algorithm, which is the base of SPP-DDPG presented in Algorithm 1 aims at training a deterministic policy, as contrary to for example
SAC algorithm by Haarnoja et al.
(2018a;b) that trains a stochastic policy.
Its main principle is rooted in the very successful deep
Q -learning method, originally applied for solving the Atari benchmark problem (Mnih et al., 2013).
However, DDPG is suitable for continuous action spaces like in the MuJoCo tasks.
SAC is an algorithm that combines ideas of DDPG and soft Q-learning Haarnoja et al.
(2017) utilizing the advantages of both approaches, SPP-SAC is presented in Algorithm 2.
[Table]
D is not
[Table]
then
[Table] [Table]
do
[Table]
t +1 , ˆ a t +1 ) ;
Compute ϕ
= ϕ − l ϕ · ∇ ϕ 1 |B| (cid:80) B (cid:16) y − Q π, CM ϕ ( s t , a t ) (cid:17) 2 ;
Compute θ
=
θ
+ l θ · ∇ θ 1 |B| (cid:80) B (min i =1 , 2 Q π, CM ϕ ( s t , ˜ a θ,t ( s t ))
− α log π θ (˜ z θ,t ( s t ) | s t ) , where ˜ z θ,t ( s t ) ∼ π ( z t | s t ) is differentiable wrt θ ;
Compute θ = θ − l θ λ |B| · ∇ θ (cid:80) B
∥ s t +1 − ˜
z θ,t ( s t ) ∥ 22 ;
Update λ = λ + l λ (cid:16) 1 |B| (cid:80) B
∥ s t +1 − ˜
z θ,t ( s t ) ∥
22 − d (cid:17) ;
Update target networks ϕ targ,i = (1 − τ )
ϕ targ,i + τϕ i , for i =
1,2; end end until convergence ;
C. SPP-TD3 A LGORITHM
Twin Delayed Policy Gradient (TD3) (Fujimoto et al., 2018) is an improved DDPG algorithm using several tricks, which can be summarized as Clipped Double-Q Learning, Delayed Policy Updates,
Target Policy Smoothing.
SPP-TD3 algorithm is presented in Algorithm 3.
[Table]
D H YPERPARAMETERS U SED IN E XPERIMENTS
We present hyper-parameters used in our experiments in Table 2 for the SPP-DDPG, in Table 6 for the SPP-SAC.
SPP-TD3 hyper-parameters were dependent on the set of benchmarks, and hyperparameters specific for MuJoCo environments are in Tab. 3, for SafetyGym in Tab. 4, and for AntPush in Tab. 5.
In SPP-DDPG implementation we used the following architectures:
• Actor: dim( S ) −→ 256 ReLU −→ 256 ReLU −→ dim( A ) tanh ; • Critic: dim( S )
+ dim( A ) −→ 256 ReLU −→ 256 ReLU → 1 ; • CM : 2 · dim( S ) −→ 100 tanh −→ 50 tanh −→ dim( A ) tanh;
In SPP-TD3 implementation we used the following architectures:
• Actor: dim( S ) −→ 256 ReLU −→ 256 ReLU −→ dim( A ) tanh ; • Critic: dim( S )
+ dim( A ) −→ 256 ReLU −→ 256 ReLU → 1 ; • CM : 2 · dim( S ) −→ 100 tanh −→ 50 tanh −→ dim( A ) tanh;
[Table] [Table]
In SPP-SAC implementation we used the following architectures:
• Actor: dim( S ) −→ 256 ReLU −→ 256 ReLU −→ dim( A ) tanh ; • Critic: dim( S )
+ dim( A ) −→ 256 ReLU −→ 256 ReLU → 1 ; • CM : 2 · dim( S ) −→ 64 tanh −→ 32 tanh −→ dim( A ) tanh
[Table] [Table]
[Table]
E A DDITIONAL E XPERIMENTS E. A BLATION S TUDY
We performed a thorough ablation study of the features that we implemented in the SPP-TD3 algorithm presented in Sec.
C. Ablation study was performed in Ant (Fig. 7a) and Doggo-Goal (Fig. 7b).
We explain the meaning of particular ablations visible in Fig.
• SPP-TD3 less init.
sampl.
means that less randomly generated experience the algorithm added to the replay buffer at the beginning of executing Alg.
3 (the first repeat until block).
In some cases, like SPP-TD3 for Doggo-Goal, see Table
we choose to include a large number of random samples in the buffer (400000) .
However, this has no considerable effect on performance.
In fact, the SPP-RL agent utilizing the number of random samples the same as vanilla RL performs better initially.
Only in the long term, slightly losing to the agent utilizing more experience.
This shows that the performance boost of SPP-RL still persists when feeded the same number of initial random samples as in the corresponding vanilla RL method.
• SPP-TD3 w/o state consistency the state consistency defined in Property 1 is not introduced in the algorithm; therefore, there is no incentive on the policy to choose as targets realizable states.
Therefore it demonstrates that the SPP approach is more than just a policy reparametrization.
• SPP-TD3 Q ( s, s ′ )
critic
(state-state critic) replaced the traditional target Q function computation using state-action pairs with state-next state pairs.
Hence the step of computing target actions is omitted.
As a result, the algorithm did not converge in several cases and exhibited a significant variation of the obtained returns between different runs.
This is consistent with the observed performance of D3G algorithm in Ant (Fig. 3b).
We do not show this ablation in Doggo-Goal, as the algorithm did not converge in this environment.
• SPP-TD3 w/o state normalization π is not being normalized using the current mean and min/max values of the past observations in replay buffer D .
• SPP-TD3 const.
λ = 1 the Lagrangian multipliers method is not used.
The algorithm uses the constant value λ
= 1 when performing policy updates.
We chose this particular λ value, as it tends to work well in Ant, whereas in Doggo-Goal results in lack of convergence; this shows how delicate the matter of choosing appropriate λ (when fixed) per given environment is.
• SPP-TD3 const.
λ
= 0 .
1 or λ = 0 .
5 the Lagrangian multipliers method is not used, the algorithm uses the constant value λ = 0 .
1 or = 0 .
5 respectively when performing policy updates.
The particular λ values resulted in much better performance in Doggo-Goal than λ = 1 (which works in Ant).
Observe that λ = 0 .
1 results in even better performance than the base SPP-TD3 implementation utilizing Lagrangian multipliers.
However, employing the Lagrange multipliers provides a natural way of solving the constrained objective optimization
[Figure]
and the hyper-parameter d is adjusted once for all the environments.
[Figure]
[Figure] [Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
As argued in Sec.
5.4 the performance boost visible in SPP-RL vs. vanilla RL approaches is presumably due to more efficient exploration performed by SPP-RL algorithms vs. vanilla RL approaches.
One empirical argument given concerns the performance of the so-called TD3 shadow agent, i.e., vanilla TD3 agent implementation utilizing (partially) the experience gathered by SPP-TD3 agent to perform its Actor&Critic updates.
Here we provide empirical evidence that the distribution of observations in replay buffers gathered by SPP-RL and vanilla RL implementations differs considerably.
In Fig.
8 we present an empirical study of distributions of states gathered in SPP-SAC and vanilla SAC replay buffers for the Doggo-Goal task.
The state-space in this task has 72 dimensions.
Hence to make it amenable to visual investigation and entropy computation, we encode the state vectors using a random encoder.
The encoder architecture that we used for this task is just initialized (default PyTorch ini) and not trained architecture: 72 → 20 tanh → 10 tanh → 2 .
Fig.show plots of discretized state distributions gathered by example run of vanilla SAC and SPP-SAC respectively using the Doggo-Goal environment and encoded using the random encoder.
We also compute cross-entropy to quantify the difference between those distributions as the training of both algorithms progresses and the replay buffer is being filled up.
Observe that the cross-entropy is clearly increasing as the replay buffer is being filled up, which suggests that vanilla RL and SPP-RL algorithms gather different observation distributions in the replay buffer.
[Figure] [Figure]
[Figure] [Figure]