T EMPORAL -D IFFERENCE V ALUE E STIMATION VIA U NCERTAINTY -G UIDED S OFT U PDATES
A BSTRACT
Temporal-Difference (TD) learning methods, such as Q-Learning, have proven effective at learning a policy to perform control tasks.
One issue with methods like
Q-Learning is that the value update introduces bias when predicting the TD target of a unfamiliar state.
Estimation noise becomes a bias after the max operator in the policy improvement step, and carries over to value estimations of other states, causing Q-Learning to overestimate the Q value.
Algorithms like Soft Q-Learning (SQL) introduce the notion of a soft-greedy policy, which reduces the estimation bias via soft updates in early stages of training.
However, the inverse temperature β that controls the softness of an update is usually set by a hand-designed heuristic, which can be inaccurate at capturing the uncertainty in the target estimate.
Under the belief that β is closely related to the (state dependent) model uncertainty, Entropy Regularized Q-Learning (EQL) further introduces a principled scheduling of β by maintaining a collection of the model parameters that characterizes model uncertainty.
In this paper, we present Unbiased Soft Q-Learning (UQL), which extends the work of EQL from two action, finite state spaces to multi-action, infinite state space Markov Decision Processes.
We also provide a principled numerical scheduling of β , extended from SQL and using model uncertainty, during the optimization process.
We show the theoretical guarantees and the effectiveness of this update method in experiments on several discrete control environments.
I NTRODUCTION
Reinforcement Learning (RL) algorithms learn a control policy that maximizes the expected discounted sum of future rewards (the policy value ) from state, action, and reward experience collected through interactions with an environment.
Temporal-Difference (TD) RL methods (Sutton & Barto, 2018) maintain value estimates, and iteratively improve them by combining experienced short-term rewards with estimates of the following long-term values.
Q-Learning (Watkins & Dayan, 1992) is a popular TD method that learns state–action value estimates ( Q function ), by assuming for target value estimates that a greedy policy will be used in the following state.
Although Q-learning asymptotically converges to the optimal policy, it is known to be positively biased and overestimate the value before convergence (Thrun & Schwartz, 1993).
This bias is detrimental to efficient learning, because it can cause optimal actions to appear suboptimal, delaying the collection of experience relevant to the agent’s optimal policy.
Several approaches have been proposed to mitigate the overestimation in TD learning, including Double Q-Learning (van Hasselt et al., 2015), which attempts to correct this bias by using two Q functions; Soft Q-Learning (SQL) (Fox et al., 2016; Haarnoja et al., 2017), in which value updates use a softmax policy for target value estimation; and Maxmin (Lan et al., 2020), which estimates the target value as the minimum of several Q functions.
None of these methods eliminate the bias completely.
On the other hand, it should be noted that the role that unbiased estimation could play in improving reinforcement learning remains unclear (Lan et al., 2020).
Fox et al.
(2016) showed that in SQL there exists a particular state-dependant inverse-temperature parameter β of the softmax operator that makes the target value estimates unbiased.
However, this unbiased-update β value is generally unknown.
Fox (2019) suggests that β is inversely related to the variance of the estimate’s parameter distribution by deriving a closed-form expression for
MDPs with two actions.
However, it is not obvious how to generalize this expression to multi-action
Markov Decision Processes and neural network Q function approximator representations.
In this work, we estimate β by maintaining an ensemble of Q functions to approximate the uncertainty in Q.
This estimated β provides a principled temperature schedule that is “softer” when uncertainty is high but gradually becomes “harder” as training proceeds.
We experiment in Atari environments to show the effectiveness of our method in practical discrete action space environments, and find that our method outperforms the Rainbow DQN (Hessel et al., 2018) and PPO (Schulman et al., 2017) algorithms in most Atari environments.
This work shows that incorporating awareness of model uncertainty can help to provide a target with less bias, which leads to faster policy improvement.
To summarize, our contributions are as follows:
• We provide a principled numerical method for estimating a β term that makes the maximum-entropy target value approximately unbiased in SQL.
• We provide extensive experimental results demonstrating that reducing bias in SQL via our estimated β improves performance.
• We provide a proof of convergence of our method.
P RELIMINARIES
We consider a Markov Decision Process (MDP) with transition probability p ( s ′ | s, a ) .
An agent controls the process using policy π ( a | s ) .
A reward r that only depends on s and a is observed after an action is taken.
This work focuses on MDPs with discrete action spaces, a ∈ { 0 , 1 , ...
d − 1 } , where d is the size of the discrete action space and varies in different environments.
We denote trajectory as a sequence of states, actions, and rewards ξ =
( s 0 , a 0 , r 0 , s 1 . . . ) .
An RL algorithm should discover a control policy that maximizes expected return of trajectory R ( ξ )
= (cid:80) t ≥
γ t r t , where t is the time step in an interaction process and 0 ≤ γ ≤ 1 is a discount factor.
Value-based methods infer a control policy by maintaining a state-action value function of some policy π ,
[Equation]
Q-Learning (Watkins & Dayan, 1992) learns the optimal value of each state action by minimizing the Temporal-Difference loss via the stochastic update,
[Equation]
This update is guaranteed to converge to the optimal Q ∗ , since the Bellman operator for Q-Learning,
[Equation]
is a contraction in L ∞ .
However, in early stages of training, our estimate Q can have significant errors (“noise”).
By using this noisy estimate, Q-Learning results in biased estimation of the next state value V ( s ) = max a Q ( s, a ) (Thrun & Schwartz, 1993).
The Q-Learning target estimation bias is a consequence of Jensen’s inequality, since
[Equation]
Training with this target estimate will propagate the bias on to preceding states due to the nature of the Bellman backup (Kumar et al., 2019; 2020).
Deep Q-Networks (Mnih et al., 2013; 2015) effectively incorporate the same biased estimate, training a neural network on squared error between the current Q and the Bellman target: [Equation] −
[Equation]
θ [Equation] [Equation] Instead, we would prefer an estimator of Q with an unbiased estimate of the optimal value:
[Equation]
A greedy policy a = arg max a Q ( s, a ) selects the best action a given its current knowledge of
Q ( s, a ) as the target estimate for the Bellman backup.
This is problematic because in early stages of learning, the Q ( s, a ) estimates are incorrect.
Fox et al.
(2016) proposed an additional cost term, which combined the expected return induced by the policy with a penalty for deviating from an agnostic prior policy π 0 , i.e.,
[Equation]
The state-action free energy function (Fox et al., 2016) evaluated at the soft-optimal policy can be written as V β ( s ) , the mellowmax (Asadi & Littman, 2017) over the action value of the next state.
[Equation]
V β ( s ) is the soft-optimal value function under maximum-entropy notion of optimality, where β is the inverse temperature.
Fox et al.
(2016) also proved that under a specific scheduling of β , controlling the level of trade-off between a deterministic policy ( β → ∞ ) and an agnostic prior policy ( β = 0 ) for selecting the distribution over actions to evaluate the next state, the soft-optimal value V β ( s ) of the maximum-entropy objective, is an unbiased estimator of the next state value.
The soft-greedy policy is the policy π ( a | s ) that maximizes V β ( s ) ,
[Equation]
Selecting the action soft-greedily using a good β that captures the uncertainty in current estimates mitigates the positive bias caused by overestimating the action value.
Previous work has also shown that selecting a good schedule of β for balancing uncertainty and optimality is crucial for the success of maximum-entropy RL algorithms (Fox et al., 2016;
Haarnoja et al., 2018; Fox, 2019; Hu et al., 2021; Xu et al., 2021).
We set to find the inverse temperature β using model uncertainty.
W HY R EDUCE E STIMATION B IAS
RL with unbiased value estimation is not necessarily the best way to do exploration (Fox et al., ; Lan et al., 2020), but it is important to differentiate between exploration bonus and biased estimation.
“Optimism under uncertainty” refers to the exploration bonus in selecting an action when that state-action pair is not familiar to the agent.
In contrast, bias corresponds to systematic errors in the value estimates themselves.
Since there is no reason to believe that this bias is somehow well-chosen to balance exploration and exploitation, it makes sense to attempt to decouple these two issues by performing unbiased estimation; then exploration bonuses can be incorporated in a more coherent way.
The TD estimation bias may also arise from multiple sources, some of which may be outside our control.
The bias we attempt to eliminate in this work is that induced by the max operator in the TD next-state value estimate.
If the agent is insufficiently familiar with the the next state and actions, the estimate can be noisy or wrong.
For example, our initial Q values can be biased by initialization.
If we have never encountered a state, it’s value estimate is determined by generalization, which is known to introduce optimistic bias (Thrun & Schwartz, 1993).
Even if we can estimate these values in a less biased way, our estimate errors will combine with the randomness inherent in the MDP, and create a large Jensen’s gap between the Q target estimate and the true state value as we propagate our estimates through the backward recursive structure of the Bellman updates (Lee et al., 2020; Kumar et al., 2019; 2020).
This motivates us to examine the connection between our update policy and our uncertainty in Q.
U NBIASED V ALUE E STIMATION 4. R ELATIONSHIP BETWEEN M ODEL U NCERTAINTY AND P OLICY E NTROPY
Making decisions and learning with imperfect decision makers is well studied, e.g., Rubin et al.
Ideally, we want a deterministic policy and “hard max” update only when the model is certain of its target estimate given the observed data, and the update and policy should be more stochastic when the target estimate is more uncertain.
In maximum-entropy RL frameworks (Ziebart, 2010), the policy entropy that determines the greediness of each update is controlled by an inverse temperature β , analogously to the thermodynamic temperature of a system in a Boltzmann distribution.
Unlike many maximum-entropy RL methods that use a constant temperature or a heuristic temperature schedule, we propose to estimate the β ( s ) that yields an unbiased update during training.
Then, we select the next state action soft-greedily and perform a soft-optimal target update that is unbiased in expectation as in SQL (Fox et al., 2016):
[Equation]
4. U NBIASED S OFT U PDATE VIA M ODEL U NCERTAINTY
Fox et al.
(2016) showed the existence of a state-conditioned inverse temperature β Q ( s ) , for which the state value function V β ( s ) is an unbiased estimator of max a E
[ Q ( s, a )] .
Specifically, assuming our current estimate Q is unbiased, there exists a temperature β ( s ) whose maximum-entropy policy satisfies,
[Equation]
and so does not accumulate value estimation bias.
In practice, we propose to estimate these expectations using a finite collection of functions { Q ( i ) } ; by defining the discrepancy function
[Equation]
where ˆ E denotes the empirical average over the finite collection { Q ( i ) } , we can solve for the root of f numerically to estimate β ( s ) .
The resulting β depends on the model uncertainty — if the collection { Q ( i ) } disagree on which action should achieve the maximum value (or the greedy policy), E
Q [ V β ( s )] − max a E Q [ Q ( s, a )] can be large, resulting in β ( s ) close to 0.
In practice, approximating the desired quantity max a E Q [ Q ( s, a )] with our empirical average max a ˆ E [ Q ( s, a )] still creates a slight positive bias, since by Jensen’s inequality,
[Equation]
To compensate, we correct the inverse temperature by multiplying by some κ ∈ (0 , 1) .
L EARNING IN F INITE S TATE S PACE MDP S
We first consider finite state MDPs, in which Q ( s, a ) is represented in a tabular form.
We give convergence properties of our ensemble approach, and illustrate its behavior on an example domain.
5. U NBIASED S OFT U PDATE
In a tabular representation, given an ensemble { Q ( i ) } , the update rule of Unbiased Soft Q-Learning (UQL) on each ensemble member Q ( i ) using experience ( s, a, r, s ′ ) is given by,
[Equation]
We can define the Bellman operator at temperature w = [Equation] Q [Equation]
s ) by,
[Equation]
This form of soft update addresses uncertainty through scheduling an increasing β with more training as a result of decreased model uncertainty.
The mellowmax operator smoothly degrades to max when model uncertainty is lowered during training.
This allows convergence of our method to the same optimal fix point Q ∗ as Q-Learning.
Combining the above analysis, following Fox et al.
; Asadi & Littman (2017) we can then show,
Lemma 5.1.
The Bellman operator B w is a contraction between Q ( i ) and Q ( j ) for all i, j in L ∞ :
[Equation]
Proof.
See Appendix A.
In practice, we update the ensemble members Q ( k ) using a stochastic approximation to the Bellman operator resulting from our observed reward r and new state s ′ .
We therefore adapt the proof of convergence for Q-learning from Jaakkola et al.
(1994) to show that,
Theorem 5.2.
If S and A are finite sets, and the learning rate α t ( s, a ) obeys (cid:80) t α t ( s, a ) = ∞ and (cid:80) t α 2 t ( s, a ) < ∞ , then the Unbiased Soft Q-Learning ensemble converges to Q ( i ) =
Q ( j ) =
Q ∗ for all i, j with probability one via the stochastic update rule in Equation 1.
Proof.
See Appendix A.
The full algorithm is given in Algorithm 1.
Initialize replay memory D to capacity N
Initialize K parameterized state-action value function { Q ( k ) } ; save ¯ Q ( k ) =
Q ( k ) for all k
Initialize β search range B =
[ a, b ] , correction constant κ for t = 1 , . .
, T do
Select action a t according to arbitrary policy π
Execute action a t and observe reward r t and next state observation s t +
Store transition ( s t , a t , r t , s t +1 ) in D s t ← s t +1 for k = 1 , . . .
K do
Sample random minibatch of transitions ( s j , a j , r j , s ′ j ) from D f Q,s ( β ) =
ˆ E Q (cid:2) V β ( s ) (cid:3) − max a ˆ E Q (cid:2) Q ( s, a ) (cid:3) β s ′
j ← BinarySearch ( f s ′
j ; ¯ Q , B ) y ( k ) j = (cid:40)
r j , if s ′
j is terminal state .
r j + γV ( k ) κβ s ′ j
( s ′ j ) , otherwise .
Update Q ( k ) with TD error y ( k ) j − Q ( k ) ( s j , a j )
Every once in a while: ¯ Q ( k ) ← Q ( k ) for all k
5. V ISUALIZING THE E STIMATION B IAS IN E ARLY S TAGE T RAINING
To analyze the status of this learning algorithm at each specific state, we tested our method in a
Gridworld domain (Figure 1a).
The black tiles are walls; colored tiles are valid states.
At each state, the agent may move in any of 8 different directions.
Then, environment dynamics may cause the agent to slip into to a neighboring state of the desired one with some probability.
We maintain an ensemble of Q tables for our method.
To eliminate the influence of an exploration policy, we perform stochastic updates on the Q ensemble with uniformly sampled s and a and then sample r and s ′ according to the MDP dynamics. [Figure]
This toy example highlights the difference between Q-Learning and Unbiased Soft Q-Learning in terms of reducing bias propagation.
In Figure 1b, we see that Q-Learning overestimates the value.
After estimating the inverse temperature β , and updating each ensemble member with correction κ = 0 .
5 , the estimation bias is significantly decreased in early stage training, leading to faster convergence.
The positive bias effect increases with the value of κ used in the update (see Figure 1c).
5. P OLICY Q UALITY IN E ARLY S TAGE T RAINING
[Figure]
Reducing the bias in our estimates of Q can also significantly improve our estimated policy during training.
In particular, since the max operator of Q-Learning overestimates the value in early stages, the positive bias may swamp the true relative values of subsequent states, making the policy converge more slowly to its optimum.
For example, in the Gridworld domain, we can see that, in addition to the improved quality of the value estimates of each state, the estimated policy of most states also converges faster for UQL (Figure 2c) than using standard Q-Learning (Figure 2b).
L EARNING IN H IGH -D IMENSIONAL C ONTINUOUS S TATE S PACE MDP S
In MDPs with extremely many or even continuous state values, one can approximate the state-action value function Q using a parameterized model such as a deep neural network (DNN).
Our approach naturally extends to this environment by using ensembles of neural approximations.
6. U NBIASED S OFT Q-L EARNING WITH E NSEMBLE OF F UNCTION A PPROXIMATORS
We use an ensemble of DNNs to learn to approximate Q values of each high-dimensional stateaction pair (Algorithm 1), which characterizes the uncertainty of value predictions.
As is typical, our loss function is the mean squared Bellman error of a batch of transitions sampled from the replay buffer.
Previous methods (Lan et al., 2020; Lee et al., 2020) have shown the effectiveness of ensemble learning in combating the bias in value estimation.
Lan et al.
(2020) takes the minimum estimate over the ensemble in order to learn the value more pessimistically.
Ensemble methods come with a natural notion of model uncertainty, often reflected through prediction entropy or variance.
An example is Lee et al.
(2020), which improves sample efficiency by updating the value estimator with a Bellman operator weighted by a quantified estimation of model uncertainty, which is a value related to the output variance of Q networks.
In this paper, we use an ensemble to approximate the model output distribution.
We update the ensemble iteratively according to the Unbiased Soft
Update Equation 1, and compare the results with widely used update rules.
6. A DDITIONAL I MPROVEMENTS
To further improve the performance of UQL, we included several helpful adaptations that fit easily in our ensemble Soft Q-Learning framework.
Upper Confidence Exploration (Audibert et al., 2009;
Auer et al., 2002), Prioritized Experience Replay (Schaul et al., 2015), and Dueling Networks (Wang et al., 2016).
There are also methods that work well, and are used in Rainbow DQN (Hessel et al., 2018), that are not included in our method, such as Double DQN (van Hasselt et al., 2015), Distributional Q (Bellemare et al., 2017), Multi-step value estimation (Sutton, 1988; Sutton & Barto, 2018), and Noisy networks (Fortunato et al., 2017).
It remains an open question if these can be incorporated to improve UQL.
E XPERIMENTS
In order to understand the effect of unbiased estimation in theoretical and practical settings, we evaluate our unbiased update in Atari environments with neural network Q function representations.
We design our experiments to answer the following questions:
• Can unbiased soft updates improve performance?
• How well does unbiased soft update control value estimation bias?
• Does temperature reflect model uncertainty?
• How do different correction constants affect performance and bias?
7. O VERALL P ERFORMANCE
Figure 3 shows the average performance of
UQL across 25 Atari environments, normalized by random (0) and human performance scores (1).
PPO and Rainbow DQN values are as reported in Kaiser et al.
Detailed performance values are given in in Appendix E.
7. C OMPARISON TO R AINBOW DQN
UQL executes 1 update for each ensemble member for every 2 transitions sampled from the environment.
The temperature used in this experiment is not accounting for finite ensemble size.
A modified version of Rainbow, which updates with the exact same frequency (5 updates for every 2 transitions sampled) as UQL.
This shows that the performance increase from more frequent gradient update is limited.
Results in Appendix B.
Prioritized replay, Dueling Networks are not used in these experiments.
7. T HE EFFECT OF UNBIASED SOFT UPDATE
To visualize the accuracy of the value prediction that unbiased soft updates gives and the quality of the trained policy compare to other update methods for learning Q networks, we experimented in 4 Atari domains (shown in Figure 4).
We use ˆ E ξ ∼ p πθ ( ξ )
[ V θ ( s 0 ) − R ( ξ )] as a proxy to the overall estimation bias of the Q networks.
We can see that in this experiment, lower estimation is linked to better performance.
To isolate the effect of the exploration strategy, these experiments use an experimental setting that shares the replay buffer between algorithms, in order to eliminate any influence on the exploration quality induced by learned Q values.
In each training iteration, each algorithms samples 1 transition using epsilon-greedy exploration policy from a copy (assigned to that algorithm) of the Atari game, and stores to the shared replay buffer.
Then, each algorithm samples a batch uniformly from the shared replay buffer and takes a gradient step on its Q network using target estimates generated by that algorithm.
UQL is still able to outperform in terms of performance and bias, results shown in
Appendix B.
7. U NBIASED T EMPERATURE AND M ODEL U NCERTAINTY
[Figure] [Figure]
To visualize the state unbiased temperature, we perturbed a normal state representation framestack of Seaquest (Figure 5a) into a state that the learned agent has never seen (Figure 5b, 5c, 5d) with increasing insanity.
We compute the temperature of each state using a model trained with UQL for 5.1M steps in these 4 states.
The action distribution selected to evaluate the target estimate will be closer to an arg max over action in Figure 5a with which the agent is familiar with, and closer to uniform in Figure 5d with which the agent is unfamiliar.
7. U SING A D IFFERENT C ORRECTION C ONSTANT
In Figure 6, we run the algorithm with w = ( κβ Q θ ( s ))
− 1 as the temperature for update with κ ∈ {∞ , 2 , 1 , 0 . 5 } on 5 ensemble members.
Q-Learning is equivalent to using κ = ∞ .
From this experiment, we can see that bias is monotonic in β .
A correction constant κ ∈ (0 , 1) may be tuned for performance to counter overestimation of Q from a finite collection of approximators.
We can see that lower bias is linked to better performance, but the strength of this correlation can be affected by other factors of training like exploration.
7. U SING A D IFFERENT N EXT -S TATE R EDUCTION O PERATOR
In Figure 9, we use an alternative target estimates.
Softmax of the next state V β ( s ) =
E a ∼ π β ( ·| s )
[ Q ( s, a )] , which evaluates the next state value as the expected Q value following Boltzman distribution over actions ( π β ( a | s ) def = π 0 ( a | s )exp( βQ ( s,a )) [Equation] s ) [Equation]
( βQ
[Equation]
s, [Equation] a )) ). We use these estimators to substitute the maximum-entropy target estimate and search the parameter β updates.
The baseline method “mean” updates the ensemble member Q ( i ) with TD error r + γ max a ˆ E Q [ Q ( s ′ , a ′ )]
− Q ( i ) ( s, a ) .
D ISCUSSION AND F UTURE W ORK
UQL is based on the temperature scheduling of EQL (Fox, 2019), but gives a tractable numerical solution to find the temperature of the maximum-entropy target estimate used in SQL, which is more theoretically motivated than EQL.
In this work, we correct the bias from estimating the value using max a ˆ E Q [ Q ( s, a )] with κ , which can be environment and exploration dependent.
Therefore, future work on a better approximation of the unbiased target will be potentially helpful to derive a better algorithm.
It is also clear that less estimation bias does not necessarily lead to better performance, which depends not only the value estimation, but also on the exploration strategy.
It will be also beneficial to have more theoretical results on the properties of an exploration strategy during its interaction with a noisy estimate of Q. It would also be interesting to conduct experiments that show the reason why just using ensemble estimate mean in neural network Q representation works well already.
It is obvious that the success of this method comes partially from this mean target estimate.
A C ONVERGENCE OF U NBIASED S OFT Q-L EARNING
In this section we prove the convergence of each ensemble member Q ( k ) in ensemble Q , to the optimal Q ∗ , with probability 1, under the unbiased soft update rule,
[Equation]
where we define the mellowmax operator with temperature w as
[Equation]
The temperature w ( Q, s ) is a function of the ensemble’s estimates at state s , specifically the value of w that ensures,
[Equation]
We assume a discrete action set with A actions.
At a high level, we first show that the temperature-annealed Bellman operator exhibits a contraction property, then use this to show that for any sequence of temperatures { w t } , our ensemble members { Q ( i ) } converge together.
Finally, we show that, under our temperature selection strategy, this convergence implies that all members converge to Q ∗ .
We first state a few useful properties of the mellowmax function.
Mellowmax is closely related to the temperature-annealed log (cid:80) exp function (e.g., Liu & Ihler (2011)), from which we can immediately see that mellowmax a ; w is convex in w , with gradient ∂∂w mellowmax a ; w Q ( a ) =
H ( p w ( a ))
+ E p w
[log π 0 ] = − D ( p w ∥ π 0 ) where H ( · ) is the entropy of the annealed Gibbs distribution p w ( a )
∝ π 0 ( a ) exp( Q ( a ) /w ) with base measure π 0 > 0 and D is the Kullback-Leibler divergence.
The temperature extremes of mellowmax are,
[Equation]
Correspondingly, the limiting Gibbs distributions are p ∞ =
π 0 and p 0 a distribution that puts uniform weight on the maxima of Q ( a ) .
Finally, we have that
[Equation]
i.e., mellowmax is monotonic decreasing in w , and that commutation results in a bound:
[Equation]
with equality if w = w ′ .
Now, define the Bellman operator B w and denote the optimal fixed point for temperature w as Q ∗ w , so that: [Equation]
[Equation]
Then, we can show that,
Proof.
Lemma 5.1.
Following e.g., Fox et al.
(2016), let
[Equation]
Then, for any s ′ , a ′ ,
[Equation]
and similarly,
[Equation]
Therefore
[Equation]
Applying this to the definition of the Bellman updates on Q ( i ) and Q ( j ) gives,
[Equation]
To examine the convergence of our ensemble, we make use of a theorem from Jaakkola et al.
Theorem A.1 (Jaakkola et al.
The random process { ∆ ( k,k ′ ) t } in form ∆ ( k,k ′ ) t + = (1 −
[Equation]
taking values in R converges to w .p.
1 if, The number of states S and actions A are finite; 2.
The learning rate obeys (cid:80) t α t ( s, a ) = ∞ and (cid:80) t α 2 t ( s, a ) < ∞ ; 3.
∥ E [ F ( k,k ′ )
t | P t ] ∥ W < γ ∥ ∆ t ∥ W , where γ ∈ (0 , 1) ; 4.
Var[
[Equation]
for some weighted max norm ∥ · ∥ W and where P t denotes the past history of the algorithm.
Now we can show that our ensemble members converge together under a standard stochastic update on each state s :
Proof.
Theorem 5.2.
Our stochastic update has the form,
[Equation]
for some sequence of temperatures w t , which may depend on the current ensemble.
For each pair of ensemble members i, j , define
[Equation]
The stochastic update can be rewritten as
[Equation]
suggesting the definition,
[Equation]
for which it is easy to see
that
[Equation]
where the inequality follows from Lemma 5.1.
Finally, as in Jaakkola et al.
(1994), the required variance condition holds since Var[ r ( s, a )] is bounded and F depends at most linearly on the Q ( i ) t .
Thus the conditions of Theorem A.1 are satisfied.
Finally, we show that as the ensemble members { Q ( i ) } converge together, our temperature w t → 0 , and thus all members converge to Q ∗ .
Suppose we have a collection of functions { Q ( i ) ( a ) } with,
[Equation]
We note that the ensemble average can also be denoted as a mellowmax , since 1 | Q | (cid:80) i Q ( i ) ( s, a )
=
mellowmax i,π 0 ; ∞
Q ( i ) ( s, a ) with uniform π 0 ; we abbreviate as mmax i ; ∞ for convenience.
Our temperature w t is thus defined by the value w ∗ that satisfies,
[Equation]
This w ∗ must exist since mellowmax is monotone in w and,
[Equation]
the value w ∗ is unique if the left and right terms are not equal.
If ∥ Q ( i ) − Q ( j ) ∥ ∞ ≤ ϵ , we can easily see that
[Equation]
since, defining a i = arg max a Q ( i ) ( a ) ,
[Equation]
where Q (1) can be any of the functions in the set { Q ( i ) } .
Thus we have that our optimal temperature’s mellowmax value is close to that of max :
[Equation]
Now, since mmax i ; ∞ mmax a ; w Q ( i ) ( a ) is decreasing in w , we will have 0 ≤ w ∗ ≤ w if mmax i ; ∞ mmax a ; w Q ( i )
[Equation]
a ) ≤ mmax i ; ∞ mmax a
[Equation]
0 Q [Equation]
[Equation]
a
[Equation]
2 ϵ ≤ mmax [Equation] mmax i ; ∞ Q ( i ) ( a ) So we need to find a value of w that reduces the mellowmax value by at least 2 ϵ .
Assume that each Q ( i ) is non-constant, i.e., max a Q ( i ) ( a ) − min a Q ( i ) ( a ) = δ > 0 .
This ensures that w ∗ is unique, and (equivalently) that the derivative of mellowmax at w = 0 is non-zero 1 ; in particular, if only one action value Q ( a ′ ) is smaller than the others, the slope of mellowmax will be lim w → 0 H ( p w ) + E p w [log π 0 ] < max a log[1 − π 0 ( a )]
(since p w will place zero mass on some action a ).
Thus, as ϵ → 0 , we have
[Equation]
Therefore, the ensemble members { Q ( i ) } converge to each other, and as they do, w → 0 , so that all
Q ( i ) are converging to Q ∗ , the optimal Q -function under the standard Bellman recursion.
[Figure]
A BLATION S TUDIES
C H YPERPARAMETER C ONFIGURATIONS
All results in Table E is generated using the hyperparameter configuration from Lee et al.
(2020), except hyperparameters for κ and searching β .
No tuning is done on the hyperparameters.
→ denotes linear schedule
[Table] [Table]
D P ERFORMANCE N ORMALIZED BY H UMAN S CORE
Performance at 500k interactions.
The results of UQL ( κ = 1 and κ = 0 . 5 ) show the average score and standard deviation of 5 runs.
For Random, we report the numbers reported in Lee et al.
For PPO, Rainbow, we report the number reported in Kaiser et al.
[Figure]
[Table]
hop
[Table]
E P ERFORMANCE C OMPARE TO H UMAN AT 500 K INTERACTION
[Table]
Chopper-Command 811.
CrazyClimber 10780.
DemonAttack 152.
Freeway 0.
Frostbite 65.
Gopher 257.
Hero 1027.
[Table] [Table]
F P ERFORMANCE C OMPARE TO H UMAN AT 100 K INTERACTION
[Table]