A Framework for Predictable Actor-Critic Control
Abstract
Reinforcement learning (RL) algorithms commonly provide a one-action plan per time step.
Doing this allows the RL agent to quickly adapt and respond to stochastic environments yet it restricts the ability to predict the agent’s future behavior.
This paper proposes an actor-critic framework that predicts and follows an n -step plan.
Committing to the next n actions presents a trade-off between behavior predictability and reduced performance.
In order to balance this trade-off, a dynamic plan-following criteria is proposed for determining when it is too costly to follow the preplanned actions and a replanning procedure should be initiated instead.
Performance degradation bounds are presented for the proposed criteria when assuming access to accurate state-action values.
Experimental results, using several robotics domains, suggest that the performance bounds are also satisfied in the general (approximation) case on expectancy.
Additionally, the experimental section presents a study of the predictability versus performance degradation tradeoff and demonstrates the benefits of applying the proposed plan-following criteria.
Introduction
Deep reinforcement-learning (RL) algorithms are considered state of the art for solving Markov decision processes [12, 28, 20].
Such algorithms can perform at, and even surpass, human-level control in various domains.
Examples include robotics [12, 8], traffic management [1, 2], autonomous driving [17, 27] and energy management [10, 33].
Common RL algorithms train a policy which, given the current state of the environment, returns a single action to be applied at the current time step.
While allowing the RL agent flexibility to quickly react to changes in stochastic environments, such a one-step planning horizon limits the ability to predict the agent’s behavior.
This is a limiting characteristic of common RL algorithms, as predictability was shown to be beneficial in multiagent domains [34, 30, 18, 7, 26] and for regulating an RL-agent’s performance in safety-critical tasks [3], e.g. a human operator overseeing an autonomous driving controller.
Additionally, it is shown to be helpful for an agent to learn a stable policy by relying on its own predicted plan [24].
In this paper we take a first step toward reconciling state-of-the-art RL with n -step planning while bounding the potential performance degradation.
The proposed approach, denoted predictable actor-critic (PrAC), trains an environment model approximator which is used to produce and store imaginary plans [24].
At each time step, PrAC uses a state-action ( Q ) value approximator to evaluate the performance degradation affiliated with following the (previously computed) imaginary plan.
If the expected future discounted reward is reduced by more than some threshold, ϵ , a replanning operation is performed and a new (presumably better) plan is followed.
The proposed approach is illustrated in Figure 1.
We show that, under simplifying assumptions, applying PrAC on top of a static policy would not degrade the expected sum of discounted rewards by more than ϵ 1 − γ , where γ is the domain discount factor.
Our experimental results show that in some domains, such as LunarLander [4], Walker, and Hopper [6],
PrAC results in little or no performance degradation while predicting an average of 12, 5, and 4 actions into the future, respectively.
Meanwhile for other domains, such as Cheetah and Humanoid [6], PrAC incurs substantial ( ∼ 22%) performance degradation to produce an average of only 3 and 2 forecasted actions, respectively.
1.1 Preliminaries
This paper addresses control problems modeled as Markov Decision Processes (MDPs) [23], each with state space, S , action space, A , transition probabilities, P , reward function, R , and discount factor, γ .
An agent is assumed to start from state s 0 and select action a 0 according to a policy , π .
The policy is commonly assumed to be stochastic (soft policy) [12], i.e., mapping states to a distribution over actions.
Given a soft policy, an action is selected by sampling the affiliated distribution, a t ∼ π ( s t ) .
Based on the chosen action and transition probability, P ( s t +1 | s t , a t ) , the agent receives a reward, r t ∼ R ( s t , a t , s t +1 ) , from the environment and observes the next state, s t +1 .
A transition is a tuple of the form ( s t , a t , r t , s t +1 ) .
A set of consecutive transitions generates a (stochastic) trajectory, τ = ( s 0 , a 0 , r 0 , s 1 , s 1 , a 1 , r 1 , s 2 , . . . ) .
A solution for a given MDP is a policy, π ∗ = arg max π V π ( s 0 ) , where V π ( s ) =
E τ ∼ π | s
= s [ (cid:80) ∞
t =0 γ t r t ] is called the state value function.
In this paper, we quantify an algorithm’s performance as V ( s 0 ) .
A state-action ( Q ) value is defined over a policy, π , as Q π ( s, a ) =
E τ ∼ π | s
=
s,a 0 = a [ (cid:80) ∞ t =
γ t r t ] .
Q ∗ is the Q value obtained under an optimal policy π ∗ .
State-of-the-art RL algorithms [12] use an actor-critic framework where a state-action value approximator ( Q ) is trained.
Following the Policy-Gradient Theorem [35], the approximated Q values are used to compute and apply a policy gradient step with respect to the expected sum of discounted rewards.
Related work
This paper proposes extending the common actor-critic framework to compute and follow n -step plans.
In order to do so, we suggest training a model approximator for the underlying state transition probabilities ( P ).
Training a model of the environment, i.e., approximating P ( s ′ | s, a ) based on observed transitions, is commonly done as part of a model-based RL [15, 36] approach.
Such a model can be used to produce imaginary trajectories [24] that improve training sample efficiency in many domains.
The imagined trajectory is a simulated future trajectory based on the learned model and current policy.
We differentiate between the imaginary trajectory, which is a set of predicted ( s, a, r, s ′ ) transitions, and the imaginary plan, which is the affiliated sequence of actions in the imaginary trajectory.
For computing the imaginary trajectory, Racaniere et al.
2017 trained an environment model with a recurrent architecture.
The proposed environment model extended on the notion of action-conditional next-step predictors [22, 5, 19], which predicts the next state, and potentially the reward, received after following a given action at a given state (or history of states).
It is important to note that modeling errors can compound over long trajectories resulting in an unlikely imaginary trajectory and an affiliated risky imaginary plan.
Such modeling errors were shown to be common in complex domains [31, 32].
In order to reduce environment modeling errors, Ke et al.
2018 proposed building a latent-variable autoregressive model [11] by leveraging recent ideas in variational inference [37].
Another recent approach for reducing modeling errors [21] suggests training a prediction model, ˆ f ϕ ( s t , a t ) , that outputs not the next state, s t +1 , but the resulting change to the current state, s t + a − s t .
The affiliated ( l 2 ) loss function is defined as L ϕ ( s t , a t , s t +1 )
=
∥ ( s t +1 − s t ) − ˆ f ϕ ( s t , a t ) ∥ 2 ; see Eq2 in Nagabandi et al.
Kim et al. 2020 proposed to utilize imaginary plans towards intention sharing in multiagent RL scenarios.
It was demonstrated that broadcasting agents’ future intentions (the imagined plans) improve coordination and overall performance over sharing just the current and past states of agents [9, 29, 14, 7].
Predictable actor-critic control
As stated above, Racaniere et al.
2017 used imagined trajectories to provide context to model-based algorithms.
However, using those trajectories as a planned control sequence presents a challenge.
Following the noisy predictions of imaginary trajectories may reduce final performance or prevent policy convergence altogether.
This noise or variance in the imagined trajectory is determined by the two components which produce it: the agent’s policy and the model of the environment.
When the policy is suboptimal or the model is inaccurate, approximation of the trajectory may fail to correctly predict the future.
Even with an optimal policy and fully known model, a stochastic environment could push the agent off a predicted trajectory.
As such, new predictions must be produced when the current one is evaluated as insufficient.
Perpetually changing the predicted (imaginary) trajectory, however, does not lend well to predicting which actions an agent will take in the future.
In our approach, Predictable Actor-Critic (PrAC), we suggest a method of merging these two goals in a generalized framework by reviewing the current plan at each new state and preserving planned steps for as long as some performance degradation threshold is met.
A predictable control plan is therefore a trade-off between maximizing the length of predicted plans and minimizing performance degradation.
We utilize values learned by the critic component of a Q -learning actor-critic RL algorithm to evaluate trajectories imagined through an approximated model.
Algorithm 1 presents the PrAC framework when applied on top of Soft Actor-Critic (SAC) [12].
PrAC can be extended to other algorithms which learn and store Q-values such as DQN [20].
However, studying the impact of PrAC in such cases is beyond the scope of this paper.
We note that SAC might seem incompatible with PrAC as it does not converge on exact Q values but some combination of the
Q values and a bias towards high entropy in the Q -value distribution.
However, if the temperature parameter ( α ) in SAC is reduced over time to zero, as proposed by Haarnoja et al.
2018, then the learned Q -values are eventually unbiased.
PrAC receives, as input, a performance degradation tolerance value, ϵ .
In line 1, PrAC starts by initializing three function approximators, a policy ( π θ ), an action-value function ( Q ψ ), and a transition function ( P ϕ ).
At the beginning of each episode a new plan is imagined (Line 4).
The initial plan is a simple rollout of the current policy ( π ) on top of the current model approximation ( P ).
Next, for each environment step, a first-in-first-out action is retrieved from the imagined plan queue (Line 6) and executed (Line 7).
Once the next state is observed, a replan operation is initiated, where the current imaginary plan is reevaluated and adjusted (Line 10).
Finally, in Lines 11–14, the policy, state-action approximator, and model approximator are updated using gradient descent.
As presented, the policy ( π ) and state-action approximator ( Q ) updates follow the SAC, entropy-adjusted loss function.
As such, these include a learning rate parameter, λ Q and λ π , respectively, as well as the temperature parameter, α .
We note that these update rules are not inherent to PrAC and thus refer the reader to Haarnoja et al. 2018 which provides full details regarding these parameters and suggested value assignments.
The gradient step update for the model approximator ( P ) follows the model loss function proposed in Nagabandi et al.
2018 along with a learning rate parameter, λ P .
Algorithm 2 details how an imaginary plan is evaluated and updated within the Replan procedure.
As input, it takes the current state of the environment ( s ), the current imagined plan ( plan ), the policy ( π ), state-action function ( Q ), environment model ( P ), and performance degradation tolerance parameter ( ϵ ).
Each action in the current imagined plan, starting from the earliest planned action and moving forwards in time, is checked against the plan-following criteria.
This criteria determines whether an action and its succeeding plan, should still be considered as the projected plan.
The proposed criteria compares the state-action value of the preplanned (imaginary) action against the state-action value of an action drawn from the behavior policy ( π ).
For an action to remain in the imaginary (predicted) plan its value must be within ϵ of the value that a newly recalculated imagined plan would have taken.
The tolerance parameter ϵ bounds the acceptable degradation in performance for facilitating a predictable control plan.
The criteria is checked for each action in Lines 2-4; if passed, the plan is preserved in Line 6.
The imaginary plan validation process continues as the following state is sampled from the approximated environment model.
It should be noted that the model may predict different state transitions than it predicted when the plan was originally formed.
So long as the planed action meets the criteria, no adjustment is necessary.
However, once an action is found in violation of the criteria then the future plan from that point onward should be recomputed.
The preservation of the plan up to this action provides the predictable portion of the plan.
In Algorithm 2, the imagination core function from Racaniere et al.
2017 is used in Lines 8-11.
An action is first sampled from the policy at the final state in the current imaginary plan.
Then the state transition is predicted by the environment model given both the state and action.
This process could be repeated to produce an arbitrary length imaginary plan.
Line 8 of Algorithm 2 presents a method for slowly building up the length of the imaginary plan ( step beyond the current plan length per step) as we can consistently follow the plan that we have.
However, if we break from our current plan, this logic resets our plan length.
This approach is preferable for computational reasons.
In cases where computational resources are sufficient, one can set a constant n -step prediction horizon.
This would always result in a constant length imaginary plan, even when the model approximation is not reliable enough to allow following lengthy plans.
In such cases, the plan following criteria will often truncate the imaginary plan.
Input : Suboptimality tolerance ϵ 1 Initialize: policy ( π ) parameters θ , action-value function ( Q ) parameters ψ , model approximation ( P ) parameters ϕ , empty replay buffer D 2 for each episode do reset environment: s ← s plan ← REPLAN( s, plan := empty array , π θ , P ϕ , Q ψ , ϵ ) for each environment step do a ← dequeue ( plan ) execute a and observe r, s ′ store s, a in D s ← s ′ plan ← REPLAN( s, plan, π θ , P ϕ , Q ψ , ϵ ) // reuses as much of the old plan as possible for each gradient step do ψ = ψ − λ Q ∇ ψ [ Q ( s t , a t ) − ( r t + γ ( Q ( s t +1 , a t +1 )
− α log π ( a t +1 , a t +1 )))]
θ = θ − λ π ∇ θ
[ α log π ( a t | s t ) − Q ( s t , a t )]
ϕ = ϕ − λ P ∇ ϕ
[ P ( s t , a t ) − s t +1 ]
Input : state s , current plan plan , policy π , action-value function Q , model approximation P , performance degradation tolerance ϵ
Output : a new plan plan ′ an empty array plan ′ do a ← plan ] if Q ( s, a ) + ϵ < Q ( s, π ( ·| s )) then break // ϵ -suboptimal.
Abort.
append a to plan ′
s ∼ P ( · s a ) 8 for 2 iterations do a ∼ π ( ·| s ) append a to plan ′
s ∼ P ( ·| s, a ) 12 return plan ′
3.1 Performance bounds
We show that, under simplifying assumptions, the proposed plan-following criterion bounds the performance degradation incurred by PrAC.
Lemma 1.
Assuming known state-action ( Q ) values for a policy π , and a discount factor, 0 < γ < 1 , applying PrAC on top of π would not degrade the sum of discounted rewards by more than ϵ 1 − γ .
Proof.
At every state, s , PrAC applies action, a p , which satisfies:
[Equation]
By definition:
[Equation]
Combining 1 and 2 yields:
[Equation]
Equation 3 results in a recursive definition over successive state values V π ( s ) and V π ( s ′ ) .
Considering the PrAC state-value definition, V p ( s )
=
E τ ∼ PrAC | s
= s (cid:80) t γ t R ( s t , a p ) , and expanding the recursive term results in:
[Equation]
Assuming γ < 1 , the performance degradation of PrAC can be bounded by (cid:80) ∞ t =
γ t
ϵ
=
ϵ 1 − γ .
In many realistic scenarios, it is not reasonable to assume that Q π is known but that it is only approximated.
Nonetheless, we present empirical results showing that this bound also holds, in expectancy, for cases where Q π is only approximated, as is the case in actor-critic algorithms.
Experimental study
The experimental study is designed towards the following objectives:
Investigate to what extent does the theoretical performance bound for PrAC hold when Q -values are approximated.
Present the trade-off between performance degradation and predicted plan length in several domains, both for the case when we use PrAC during training and the case where we train the baseline policy and then apply PrAC. [Table]
4.1 Domains
Results for six continuous-action domains are presented in this section.
These include:
HumanoidBulletEnv-v0, HopperBulletEnv-v0, HalfCheetahBulletEnv-v0, Walker2DBulletEnv-v0, and AntBulletEnv-v0.
These domains were selected to be similar to those used to study SAC [12].
Additionally, we present results for the LunarLanderContinuous-v2.
A snapshot from all six domains can be seen in Figure 2.
The domains included here have state spaces which are vectors of high-level features (i.e. not images).
For the PyBullet robotic domains, the state space consists of physics descriptors for the agent.
For example, in the Ant domain, the state space consists of the position and orientation of the torso and the joint angles, the velocity of the agent, and the external forces applied to each of the links at the center of mass.
The action spaces in all environments are continuous.
In the PyBullet robotic domains, the action spaces are the magnitude of torque applied to each actuator.
The state and action space sizes for each domain is given in Table 1.
4.2 Hyper-parameter settings
The reported PrAC implementation utilizes the open-source Stable Baselines [13] implementation of
SAC.
Stable Baselines maintains a well-documented library of benchmark reinforcement-learning algorithms.
The code for our experiments is available at github.com/josiahcoad/AFRL.
In our experiments many hyper-parameters were constant between the domains and were selected to match the tuned parameters made available in the Stable Baselines Zoo [25].
These were as follows:
A 3e-4 learning rate for both the actor ( λ π ) and the critic ( λ Q ), a 1e6 buffer size, 100 random actions before starting learning, a minibatch size for each gradient update of 256, a soft polyak update coefficient ( τ ) of 0.005, a discount factor ( γ ) of 0.98, and a training frequency of every step with one gradient step per training and a target network update frequency of every step.
The entropy coefficient ( α ) in SAC, equivalent to the inverse of reward scale in the original SAC paper, was dynamically adjusted using the Stable Baselines implementation.
The Humanoid and Cheetah domains required some parameters to be adjusted from other domains.
In the Humanoid domain a batch size of 64 was used and the number of random acts pre-training was raised to 1,000.
In the Cheetah domain the number of random acts pre-training was raised to 10,000, learning rate ( λ π , λ Q ) set to 7.3e-4, a 3e-5 buffer size, τ = 0 .
02 , and an update frequency of once every 8 steps with 8 gradient steps.
The environment model for all domains uses a small network with 4 hidden layers, each with 64 nodes and relu activation functions.
The Adam optimizer with learning rate ( λ P ) of 1e-4 was used to optimize parameters.
4.3 Computing resources
Modest compute was necessary to conduct our experiments.
Amazon EC2 type t3.large instances were used.
This type contains 2 vCPUs and 2 GB of memory.
Experiments were run on the Amazon
Linux 2 operating system and require the following open-source Python packages: NumPy (1.19.5),
Gym (0.18.0), Torch (1.8.1), PyBullet (3.1.4) and Stable Baselines3 (1.1.0).
4.4 Performance bounds
We start by investigating the performance degradation introduced by PrAC as a function of the tolerance parameter ϵ .
The theoretical analysis in Section 3 relies on several limiting assumptions.
However, we find that in practice this bound still holds in empirical evaluation.
In order to present a clearer trend over several domains, we report the normalized performance for PrAC where the normalized performance is defined as follows.
Definition 1 (Normalized performance) .
[Equation]
V rand is the expected sum of discounted returns following a random policy.
V π ∗ is the expected sum of discounted returns following a Q -greedy policy, i.e., π ∗ ( s ) =
arg max a Q ( s, a ) .
In the left-hand side of Figure 3, the theoretical bound is plotted in yellow as a function of the ϵ coefficient on the horizontal axis with γ = 0 .
The lower-bound formula derived from Section 3 is used here.
The bound decreases as ϵ increases.
On the vertical axis, the performance of a random baseline (red) and the SAC policy π is plotted.
The , represents the normalized performance of PrAC.
In Lander, Hopper and Walker environments, we observe that the forecast decreases very little while providing a multistep action plan.
In the remaining environments, Humanoid-0.1 is stopped early as it fails to improve beyond random performance. Humanoid, Cheetah and Ant, performance degrades significantly at higher coefficients.
We speculate these environments are more difficult to model due to an increased state space size.
Nevertheless, all environments respect the theoretical bounds in expectation.
The hyper-parameter, epsilon, introduced with PrAC has the intuition that ϵ = 0 reduces to a plan generator with no additional care given to retaining the current plan, e.g. the type of plan generators in [24] and [18].
We should expect a very small forecast with ϵ = 0 .
On the other hand, ϵ = ∞ is a plan generator which never deviates from the current plan and thus would have a high forecast and low performance.
The epsilon coefficient equal to ϵ/ ( V π ( s 0 ) − V rand ( s 0 )) is used to generalize the epsilon hyper-parameter better across domains.
4.5 Predictability-performance trade-off
The design of PrAC leads to a trade-off between agent performance and plan reliability.
We seek to characterize this relationship through a series of experiments that showcase both use cases for PrAC.
PrAC can be applied both during training and after training.
This makes PrAC a versatile addition to any RL agent which learns Q-values.
If predictability is desired during training, then training- with -PrAC should be considered, otherwise training- then -PrAC can be used to apply PrAC to an already fully trained agent.
We analyze and report results on both training- with -PrAC and training- then -PrAC.
Forecast : A measure (scalar) of predictability or how far into the future PrAC provides a reliable plan.
This value is computed per time step and in hindsight.
When applying action a t , forecast t is defined as f t = t + 1 − i , where a t was originally determined during the Replan procedure (Algorithm 2) at time step i .
The forecast value of a full episode (of length T ) is defined as 1 T (cid:80) Tt =
f t .
In Figure 4 the learning curves of 3 representative domains are presented when PrAC’s imagined plans are used while the agent is learning.
Learning curves for decreasing values of ϵ are shown against the learning curve of the baseline Soft Actor-Critic algorithm.
In the LunarLander domain (the leftmost plot in Figure 4), values smaller than 0 .
01 achieve the domain goal of a 200 point reward.
The rate of reaching this reward is similar to the baseline for each as well.
However, PrAC offers a plan forecast of about 5 during this training which provides additional predictability.
When the epsilon coefficient is increased to 0 .
1 , forecast length is increased dramatically; but the agent does not replan appropriately, so the episode reward fails to reach the final baseline episode reward.
A similar trend is observed in the Cheetah domain (the center plot in Figure 4).
When the ϵ coefficient is large, e.g. 0.1, episode reward is greatly reduced with an improvement in forecasting.
Using
PrAC with this coefficient is impractical.
In this domain, we start to observe the trade-off at epsilon coefficients of 0 . 01 and 0 .
For a small incurred reward penalty, a forecast is possible.
We speculate returns with epsilon = 0 .
001 are slightly higher than the baseline due to randomization in the training process.
With humanoid (the rightmost plot in Figure 4), a large epsilon coefficient obstructs the policy from learning.
The episode reward stays consistent with that of a random agent.
However, for smaller values of the epsilon coefficient, the agent does improve and obtains a final episode reward within 75% of the baseline agent learned policy.
Overall, we observe a trend across the studied environments—decreased performance yields improved forecasting.
This trade-off occurs in both training-with-PrAC and training-then-PrAC.
Summary
In this work, we have proposed an easily adopted method to allowing for plan forecasting via actionvalue evaluations.
We prove a lower bound for the performance degradation incurred for following a forecasted plan, under some assumptions.
Empirically, we found that the method respects these bounds in expectation when biases are introduced resulting from model approximation errors.
Our empirical study also shows a trade-off between plan consistency and performance degradation in a number of environments.
Our study suggests the proposed method, Predictable Actor-Critic (PrAC), could be used to produce n -step plans from one-step reinforcement learning methods with bounded degradation.