T ARGET E NTROPY A NNEALING FOR D ISCRETE S OFT A CTOR –C RITIC
A BSTRACT
Soft Actor-Critic (SAC) is considered the state-of-the-art algorithm in continuous action space settings.
It uses the maximum entropy framework for efﬁciency and stability, and applies a heuristic temperature Lagrange term to tune the temperature α , which determines how ”soft” the policy should be.
It is counter-intuitive that empirical evidence shows SAC does not perform well in discrete domains.
In this paper we investigate the possible explanations for this phenomenon and propose Target Entropy Scheduled SAC (TES-SAC), an annealing method for the target entropy parameter applied on SAC.
Target entropy is a constant in the temperature Lagrange term and represents the target policy entropy in discrete SAC.
We compare our method on Atari 2600 games with different constant target entropy SAC, and analyze on how our scheduling affects SAC.
I NTRODUCTION
Deep reinforcement learning (RL) algorithms are capable of learning good behavior policies in a wide range of environments (Silver et al., 2016;
Mnih et al., 2013; Schulman et al., 2015; Gu et al., 2017).
In environments with continuous action spaces, Soft Actor–Critic (SAC) (Haarnoja et al., 2018b) has been shown to learn robustly for control tasks in simulated environments and real-world robots (Haarnoja et al., 2018a).
Motivated by the Maximum-Entropy (MaxEnt) RL theory (Ziebart, 2010; Todorov, 2008; Olsson et al., 2005; Toussaint, 2009;
Fox et al., 2016; Levine, 2018), SAC simultaneously trains a critic to evaluate the free-energy (value–entropy tradeoff) of an actor policy, and the actor to imitate the softmax policy induced by the critic.
To set the value–entropy tradeoff coefﬁcient, the temperature α , SAC optimizes an objective in which α is the Lagrange multiplier for the constraint that the average actor’s entropy H is no less than a given target entropy ¯ H .
SAC in discrete action spaces has not shown the same success as in continuous action spaces (Christodoulou 2019; and see its implementation in the RLlib Python package, Liang et al.
Christodoulou (2019) suggests setting the target entropy ¯ H to 98% of the maximum possible policy entropy.
While this conﬁguration can lead to stable learning of good policies in some environments, in most scenarios it greatly underperforms.
In most environments, there simply is no good policy that satisﬁes this target entropy constraint, and reaching a high temperature α that induces a high-entropy actor is detrimental to achieving high policy value, since it would be close to random.
On the other hand, setting a low constant target entropy restraint forces SAC to quickly decrease α at early stage, and consequently with α → 0 the MaxEnt learning objective returns to the classic RL learning objective.
The actor would attempt to learn the logits of a deterministic greedy policy, and the critic will try to evaluate it.
Because the logits of a deterministic policy saturate, this algorithm is prone to early overﬁtting that is difﬁcult to “unlearn”.
Note that this issue is less severe in continuous action spaces, where the greedy action (the mean of the Gaussian policy distribution) is largely decoupled from the policy entropy (completely determined by the policy variance).
Selecting the correct value for the target entropy is evidently essential for obtaining good results in SAC.
Using a constant target entropy throughout training requires potentially heavy ﬁne-tuning for best performance.
Alternatively, we look into automated tuning methods for the target entropy value.
In this paper, we propose the Target Entropy Scheduled SAC (TES-SAC), a heuristic scheduling method applied on SAC to reach appropriate temperature values during training by gradually dropping the target entropy ¯ H , using the policy entropy as a signal.
We initialize the target entropy to be the maximum possible entropy, log | A | , and decrease it by a constant factor as soon as the average policy entropy stabilizes around the target.
In this way, the average policy entropy will be high when training starts, and decreases gradually as training continues, reducing early overﬁtting to the insufﬁciently trained critic.
While a similar approach can be applied to Soft Q-Learning (SQL) (Fox et al., 2016; Haarnoja et al., 2017), a MaxEnt RL algorithm in which the policy is directly computed from a value network, we ﬁnd that an actor policy network can stabilize training.
Intuitively, a sudden change in the temperature α in SQL immediately causes a shift in the softmax policy, which creates a fast moving target for the value network training process.
In contrast, the policy network in SAC takes time to adapt to a changing temperature, creating a more stable target for the value network.
We analyze the problems with using a constant target entropy in SAC and propose TES-SAC in
Section 3.
We experiment on the Atari 2600 benchmark to compare our scheduled target entropy method with constant target entropy discrete SAC in Section 4 (Mnih et al., 2013).
Note that we are not trying to ﬁne-tune TES-SAC to present state-of-the-art performance, but rather to provide empirical evidence that TES-SAC solves some of the issues with SAC using a constant target entropy in discrete settings.
We also discuss the possibility of applying our target entropy scheduling method to SQL in comparison with SAC in Section 4.4.
P RELIMINARIES 2. N OTATION
We mainly focus on discrete action spaces, with a Markov decision process (MDP) deﬁned by ( S , A , p, r ) , where S is the state space, A is the discrete action space, p ( s (cid:48) | s, a ) is the state transition probability for current state s ∈ S , action a ∈ A , and next state s (cid:48) ∈ S , and r ( s, a ) is the reward given an action-state pair.
We want to learn a stochastic policy π ( a | s ) that outputs action probability given a state.
An optimal policy π ∗ should maximize the expected discounted return
R = (cid:80) t ≥ 0 γ t r ( s t , a t ) , where γ is a discount factor with range [0, 1].
2. S OFT A CTOR -C RITIC
Different from the above standard RL objective, Soft Actor-Critic (Haarnoja et al., 2018b) uses an entropy-regularized objective (Ziebart, 2010)
[Equation]
where p π ( s t , a t ) is the distribution over the state and action at time t induced by rolling out the policy π in the MDP, H [ π ( ·| s t )] is the policy entropy at state s t , and the temperature α controls the trade-off between the entropy term and the expected rewards.
When α → 0 , the maximum entropy objective becomes the standard RL objective.
To maximize this objective, SAC iterates between (1) updating a critic value function Q θ ( s, a ) to match a soft Bellman backup target, and (2) minimizing the Kullback–Leibler (KL) divergence between an actor π φ and the soft-greedy policy.
Soft Bellman backup.
SAC updates the soft Q-function, parametrized by θ , by minimizing the soft
Bellman error for a state s , an action a taken in s , the obtained reward r , and the next state s (cid:48)
[Equation]
(2) where the next-step target value
[Equation]
is computed from a target network Q ¯ θ copied periodically from the critic Q θ .
The experience ( s, a, r, s (cid:48) ) is obtained by sampling a replay buffer replenished by rollouts of the actor π φ .
Policy update.
The policy network, parametrized by φ is a distillation of the softmax policy induced by the Q-function, and is updated by minimizing the KL-divergence
[Equation]
over the parametric family of policies π φ , which in the continuous case is a Gaussian action distributions with mean and log-variance generated by a neural network.
The partition function Z θ ( s ) is a normalizer that can be ignored because it is action-independent.
The resulting actor loss for a state s sampled from the replay buffer is
[Equation]
SAC uses the temperature Lagrange term to tune the temperature α (Haarnoja et al., 2018c).
Equation (1) can be viewed as the Lagrangian of the objective to ﬁnd a policy with maximum expected return that at the same time satisﬁes an entropy constraint
[Equation]
where ¯ H is an externally selected threshold expected entropy.
While the standard formulation (1) omits the constant ¯ H , optimizing for the temperature α involves minimizing
[Equation]
where ¯ H is thus called the target entropy .
The intuition is for the temperature to dynamically increase or decrease to encourage the policy entropy to approach the target entropy.
SAC can be straightforwardly applied to discrete action spaces (Christodoulou, 2019).
One improvement that can be obtained in the discrete case is to directly calculate policy-expected values in (3), (5), and (7) as E a ∼ π f ( s, a ) = (cid:80) a π ( a | s ) f
( s, a ) .
This change reduces the variance by calculating the true expectation instead of sampling from the policy.
In addition, while the policy action distribution in continuous Soft Actor-Critic is limited to a speciﬁc parametric family, usually Gaussian distributions, in discrete SAC we can represent any categorical action distribution by generating its logits.
T ARGET E NTROPY S CHEDULED S OFT A CTOR C RITIC
In this section, we ﬁrst analyze the drawbacks of constant target entropy, then we present Target
Entropy Scheduled SAC (TES-SAC) and explain in detail how we use this method to tune the policy entropy.
Intuitively, in early stages of training, the policy is relatively random; as training proceeds it becomes increasingly deterministic.
We should therefore drop the target entropy in a proper way that is neither too fast nor too slow: if we drop it too fast, the policy will be deterministic when the training is still in an early stage, and if too slow, the policy will remain stochastic for an undesirably long time.
3. SAC WITH C ONSTANT T ARGET E NTROPY
In the original SAC temperature Lagrange term, target entropy is set to be a constant.
In this section, we describe the issues with using a constant target entropy, and why the choice of the constant is extremely environment dependent.
The temperature Lagrange term suggests that the policy entropy will try to approach the target entropy.
Therefore when the target entropy is set to be a large constant, correspondingly the policy entropy will also be high.
A policy with high policy entropy is similar to a random policy, which is undesirable in most cases.
If we set the target entropy too small, the policy entropy will quickly drop, trying to hit the target, and become overly deterministic in an early stage of training.
Moreover, if the target entropy is setted lower than the minimum achievable entropy level (see Section 4.2), α will exponentially decrease to zero.
The soft Bellman loss in equation 2 and policy loss in equationshow that when α is small, the entropy term disappears, indicating the actor would learn the logits of a deterministic greedy policy, and the critic would evaluate it.
The algorithm would easily overﬁt in an early stage and would be hard to “unlearn”, because the logits of a deterministic policy would saturate.
As we explain in more detail in Section 4.4, we still use a policy network in spite of this problem because it stabilizes the policy when dropping the target entropy.
When dropping the target entropy, α will also drop abruptly, which will cause an immediate shift in the softmax policy, but a policy network will still need time to learn that shift which gives the value network a more stable target.
Why not choose a constant target entropy that is neither too high nor too low?
It is possible that certain constant target entropies perform well in some environments.
Such a constant, however, will be extremely dependant on the dynamics of the environment and sensitive to noise.
Tuning the best constant target entropy for each environment is expensive in computation and data.
Our TES-SAC tries to solve the above problems.
3. E XPONENTIAL M OVING W INDOW S CHEDULING
Our TES-SAC scheduling method checks if the policy entropy has become stable and if it has approached the target entropy.
If the policy entropy satisﬁes these requirements, we shrink the current target entropy by a factor.
When calculating the policy entropy for the current iteration, we have
[Equation]
where e i stands for the policy entropy for iteration i , π ( a | s ) is the policy action distribution, and E b is the expectation over the mini-batch.
We record the exponential moving mean ˆ µ i and the exponential moving standard deviation ˆ σ i of the policy entropy, and calculate the exponential moving window of the policy entropy (Finch, 2009) such that
[Equation]
Here i indicates the sequence in the exponential moving window.
If ˆ µ i is close to the target entropy within the mean threshold, and ˆ σ i is smaller than the standard deviation threshold, we multiply the target entropy by a constant factor.
We describe this scheduling process in Algorithm 1, Appendix
B.
E XPERIMENTS
We show through empirical experiments that TES-SAC can learn faster for some environments and provide domain-generalized hyperparameters compared to constant target entropy.
For each environment, we compare our TES-SAC method with SAC with constant target entropy ¯
H = C
· log | A | , where C ∈ { 0 . 98 , 0 . 5 , 0 . 01 } .
We experiment on several classical control tasks, as well as the Atari 2600 games.
4. O VERALL P ERFORMANCE
Figure 1 shows the normalized results over different Atari games, and Figure 2 shows learning curves for 4 environments.
We also include a performance table for all 24 environments we experimented on in Appendix E. We ﬁnd that TES-SAC signiﬁcantly outperforms constant target entropy in Hero,
BattleZone, BankHeist, and LunarLander.
In environments including Assault, MsPacman, Freeway, and Krull, TES-SAC performs similar to the best ﬁxed target entropy SAC.
Our Target Entropy
Scheduled SAC outperforms constant target entropy in 8 out of 24 environments, which is greater than any other constant target entropy.
Theses results suggest that target entropy scheduling can learn better in some environments, and has hyperparameters that are more robust to different domain dynamics than constant target entropy.
[Figure]
The changes in the temperature α determines SAC’s performance.
Intuitively, a high target entropy induces α to increase and leads to a more stochastic policy, and vice versa.
In this section we visualize the α dynamics under different constant target entropy conﬁgurations using environments
MsPacman and Battlezone.
Figure 3 shows log α and the policy entropy comparison of low target entropy ( ¯ H = 0 .
01 · log |A| ) and high target entropy ( ¯ H = 0 .
5 · log |A| ) for MsPacman and BattleZone.
In MsPacman, (1) temperature α exponentially decreases when ¯ H = 0 .
01 · log | A | , and (2) α slowly decreases when ¯ H = 0 .
5 · log | A | .
(1) happens because the policy entropy cannot drop to the target entropy.
This makes sense in that some environments have a minimum entropy level not close to zero, indicating that the environment has some actions that are really similar — the impact to the environment from taking one of them is barely different than taking another.
For those environments whose minimum entropy is above the target entropy ( ¯ H = 0 .
01 · log |A| in our experiments), the policy entropy will never reach the target entropy, which makes the temperature α drop exponentially fast.
In MsPacman which has 9 actions, for example, the target entropy is ¯ H = 0 .
01 · log 9 = 0 . 022 .
The policy entropy, however, approaches 0.1 without decreasing further, which makes α keep decreasing.
This minimum entropy level can vary greatly among different environments.
BattleZone is an example the minimum entropy level is close to zero, so even if we set ¯ H = 0 .
01 · log |A| , α does not exponentially decrease like in MsPacman.
When the algorithm can quickly reach the minimum entropy level, α starts decreasing rather slowly.
This usually happens when we set a slightly larger ﬁxed target entropy, for example 0 .
5 · log |A| as used in our experiments.
In these cases, policy entropy successfully reaches the target entropy.
Without dropping the target entropy, we do not see long-term α decrease.
These two α patterns, of course, are sensitive to environment dynamics.
Our scheduling tends to heuristically choose the better α pattern for each environment.
This makes our scheduling much more robust and domain-generalized.
[Figure]
In our experiments, we observe that the standard deviation(std) threshold is a relatively more sensitive hyperparameter compared to average threshold and the target decrease factor.
The speed at which the target entropy decreases is largely dependant on the standard deviation threshold.
We design an experiment to show how different std thresholds will affect the performance.
Figure 4 is an example in MsPacman, where we use three different std thresholds: 0 . 03 , 0 . 05 , and 0 .
We can see that different std thresholds do change the speed of target entropy decrease, but don’t actually affect the performance much.
This suggests that our hyperparameters require little tuning.
4.3. F IXED - STEP S CHEDULING V.S. E XPONENTIAL W INDOW S CHEDULING
[Figure]
We compare ﬁxed-step scheduling with our scheduling method to justify why we use policy entropy as a signal for dropping the target entropy.
Fixed-step scheduling drops the target entropy every certain number of experience steps.
This means ﬁxed-step scheduling requires potentially heavy tuning for best performance.
Figure 5 shows the average return, log α value, and policy entropy in the
MsPacman environment when we evenly drop the target entropy four times within 1M experience steps to be [0 .
98 log |A| , 0 .
75 log |A| , 0 .
5 log |A| , 0 .
25 log |A| , 0 .
01 log |A| ] .
TES-SAC signiﬁcantly outperforms ﬁxed-step scheduling, and the α value under ﬁxed-step scheduling becomes ridiculously high in early training.
We share our ablation study results on more environments in Appendix C.
4. A PPLYING T EMPERATURE S CHEDULING TO S OFT Q-L EARNING
The scheduling method we proposed for SAC can potentially be applied to Soft Q-Learning as well.
In SQL, we approximate the policy as a Q-Function-based soft-greedy policy (Fox et al., 2016):
[Equation]
We can then apply the target entropy scheduling in Section 3.2 to calculate α using the temperature
Lagrange term with this soft-greedy policy similar to equation 7:
[Equation]
[Equation]
A Q-Function-based policy is very unstable when adjusting the target entropy, however, because α will abruptly decrease when the target entropy drops.
This will cause a sudden shift in the softgreedy policy described in equation 11.
Figure 6 visualizes this instability using the Atarigame Seaquest as an example.
[Equation]
[Figure]
The cause for this difference in applying our target entropy scheduling to SAC and SQL is that in
SAC there is a policy network π φ instead of a Q-Function-based soft-greedy policy.
This means we can directly calculate the policy entropy using equation 8 with the policy network π φ ( a | s ) .
A policy network will effectively stabilize the system, and somewhat tolerates the fast change in temperature since it will take some time to learn this change.
C ONCLUSION
In this article, we analyze some issues with applying SAC to discrete action spaces with a constant target entropy: such a target entropy would be extremely environment dependent and require ﬁnetuning.
We present TES-SAC as an alternative, a heuristic method to schedule the target entropy for discrete Soft Actor Critic by observing the dynamics in the policy entropy.
We show empirical evidence that this scheduling method requires little tuning, is more robust, and generally outperforms
SAC with a constant target entropy.
We also explain why this target entropy scheduling method will not be as effective when applied to SQL.
We have not yet attempted to tune our method towards state-of-the-art performance, because our scheduling is rather heuristic and is not backed by strong theory to make every part incontrovertible.
Nevertheless, we believe this work is an important step forward in terms of annealing the temperature in soft actor critic.
For future work we are interested in applying a similar method to continuous SAC target entropy scheduling.
Although continuous SAC already demonstrates state-of-the-art performance using a constant target entropy, our results suggest that integrating target entropy scheduling could potentially improve the performance.
A H YPERPARAMETERS
[Table]
B A LGORITHM FOR T ARGET E NTROPY S CHEDULE
Input : current policy entropy: e t
Parameters : exponential window discount λ , avg threshold ¯ µ , std threshold ¯ σ , discount factor k , total conditioned num T , initial target entropy ¯ H 0 .
Output : current target entropy ¯ H 1: Let ˆ µ = ¯ H 0 , ˆ σ = 0, i = 0, ¯ H = ¯ H2: for each timestep do 3: δ =
e t − ˆ µ 4: ˆ µ =
ˆ µ + (1 − λ ) ∗ δ 5: ˆ
σ 2 = λ ∗ (ˆ σ
+ (1 − λ ) ∗ δ 2 ) 6: ˆ σ = √ ˆ σ7: if not ( ¯ H − ¯ µ < ˆ µ < ¯ H + ¯ µ ) or ˆ σ > ¯ σ then 8: return ¯ H 9: end if 10: i
=
i +11: if i ≥ T then 12: i =13: ¯ H =
¯ H ∗ k 14: return ¯ H 15: end if 16: end for
C A BLATION S TUDY ON MORE ENVIRONMENTS C. D IFFERENT S TANDARD D EVIATION T HRESHOLDS
[Figure]
[Figure]
D N ORMALIZED PERFORMANCE
E P ERFORMANCE T ABLE
[Table]