O RCHESTRATED V ALUE M APPING FOR R EINFORCEMENT L EARNING
A BSTRACT
We present a general convergent class of reinforcement learning algorithms that is founded on two distinct principles: (1) mapping value estimates to a different space using arbitrary functions from a broad class, and (2) linearly decomposing the reward signal into multiple channels.
The ﬁrst principle enables incorporating speciﬁc properties into the value estimator that can enhance learning.
The second principle, on the other hand, allows for the value function to be represented as a composition of multiple utility functions.
This can be leveraged for various purposes, e.g. dealing with highly varying reward scales, incorporating a priori knowledge about the sources of reward, and ensemble learning.
Combining the two principles yields a general blueprint for instantiating convergent algorithms by orchestrating diverse mapping functions over multiple reward channels.
This blueprint generalizes and subsumes algorithms such as Q-Learning ,
Log Q-Learning , and Q-Decomposition .
In addition, our convergence proof for this general class relaxes certain required assumptions in some of these algorithms.
Based on our theory, we discuss several interesting conﬁgurations as special cases.
Finally, to illustrate the potential of the design space that our theory opens up, we instantiate a particular algorithm and evaluate its performance on the Atari suite.
I NTRODUCTION
The chief goal of reinforcement learning (RL) algorithms is to maximize the expected return (or the value function) from each state (Szepesv ´ ari, 2010; Sutton & Barto, 2018).
For decades, many algorithms have been proposed to compute target value functions either as their main goal (criticonly algorithms) or as a means to help the policy search process (actor-critic algorithms).
However, when the environment features certain characteristics, learning the underlying value function can become very challenging.
Examples include environments where rewards are dense in some parts of the state space but very sparse in other parts, or where the scale of rewards varies drastically.
In the Atari 2600 game of Ms. Pac-Man, for instance, the reward can vary from 10 (for small pellets) to as large as 5000 (for moving bananas).
In other games such as Tennis, acting randomly leads to frequent negative rewards and losing the game.
Then, once the agent learns to capture the ball, it can avoid incurring such penalties.
However, it may still take a very long time before the agent scores a point and experiences a positive reward.
Such learning scenarios, for one reason or another, have proved challenging for the conventional RL algorithms.
One issue that can arise due to such environmental challenges is having highly non-uniform action gaps across the state space.
In a recent study, van Seijen et al.
(2019) showed promising results by simply mapping the value estimates to a logarithmic space and adding important algorithmic components to guarantee convergence under standard conditions.
While this construction addresses the problem of non-uniform action gaps and enables using lower discount factors, it further opens a new direction for improving the learning performance: estimate the value function in a different space that admits better properties compared to the original space.
This interesting view naturally raises theoretical questions about the required properties of the mapping functions, and whether the guarantees of convergence would carry over from the basis algorithm under this new construction.
One loosely related topic is that of nonlinear Bellman equations.
In the canonical formulation of
Bellman equations (Bellman, 1954; 1957), they are limited in their modeling power to cumulative rewards that are discounted exponentially.
However, one may go beyond this basis and redeﬁne the
Bellman equations in a general nonlinear manner.
In particular, van Hasselt et al.
(2019) showed that many such Bellman operators are still contraction mappings and thus the resulting algorithms are reasonable and inherit many beneﬁcial properties of their linear counterparts.
Nevertheless, the application of such algorithms is still unclear since the ﬁxed point does not have a direct connection to the concept of return.
In this paper we do not consider nonlinear Bellman equations.
Continuing with the ﬁrst line of thought, a natural extension is to employ multiple mapping functions concurrently in an ensemble, allowing each to contribute their own beneﬁts.
This can be viewed as a form of separation of concerns (van Seijen et al., 2016).
Ideally, we may want to dynamically modify the inﬂuence of different mappings as the learning advances.
For example, the agent could start with mappings that facilitate learning on sparse rewards.
Then, as it learns to collect more rewards, the mapping function can be gradually adapted to better support learning on denser rewards.
Moreover, there may be several sources of reward with speciﬁc characteristics (e.g. sparse positive rewards but dense negative ones), in which case using a different mapping to deal with each reward channel could prove beneﬁcial.
Building upon these ideas, this paper presents a general class of algorithms based on the combination of two distinct principles: value mapping and linear reward decomposition.
Speciﬁcally, we present a broad class of mapping functions that inherit the convergence properties of the basis algorithm.
We further show that such mappings can be orchestrated through linear reward decomposition, proving convergence for the complete class of resulting algorithms.
The outcome is a blueprint for building new convergent algorithms as instances.
We conceptually discuss several interesting conﬁgurations, and experimentally validate one particular instance on the Atari 2600 suite.
V ALUE M APPING
We consider the standard reinforcement learning problem which is commonly modeled as a Markov decision process (MDP; Puterman (1994))
M = ( S , A , P, R, P 0 , γ ) , where S and A are the discrete sets of states and actions, P ( s (cid:48) | s, a ) .
= P [ s t +
= s (cid:48) | s t = s, a t = a ] is the state-transition distribution,
R ( r | s, a, s (cid:48) ) .
=
P [ r t = r
| s t = s, a t = a, s t +
= s (cid:48) ]
(where we assume r ∈
[ r min , r max ] ) is the reward distribution, P 0 ( s ) .
= P [ s 0 =
s ] is the initial-state distribution, and γ ∈ [0 , 1] is the discount factor.
A policy π ( a | s ) .
= P [ a t = a | s t = s ] deﬁnes how an action is selected in a given state.
Thus, selecting actions according to a stationary policy generally results in a stochastic trajectory.
The discounted sum of rewards over the trajectory induces a random variable called the return .
We assume that all returns are ﬁnite and bounded.
The state-action value function Q π ( s, a ) evaluates the expected return of taking action a at state s and following policy π thereafter.
The optimal value function is deﬁned as Q ∗ ( s, a ) .
= max π Q π ( s, a ) , which gives the maximum expected return of all trajectories starting from the state-action pair ( s, a ) .
Similarly, an optimal policy is deﬁned as π ∗ ( a | s ) ∈ arg max π Q π ( s, a ) .
The optimal value function is unique (Bertsekas & Tsitsiklis, 1996) and can be found, e.g., as the ﬁxed point of the Q-Learning algorithm
(Watkins, 1989; Watkins &
Dayan, 1992) which assumes the following update:
[Equation]
where α t is a positive learning rate at time t .
Our goal is to map Q to a different space and perform the update in that space instead, so that the learning process can beneﬁt from the properties of the mapping space.
We deﬁne a function f that maps the value function to some new space.
In particular, we consider the following assumptions:
Assumption 1 The function f ( x ) is a bijection (either strictly increasing or strictly decreasing) for all x in the given domain D
= [ c 1 , c 2 ] ⊆ R .
Assumption 2 The function f ( x ) holds the following properties for all x in the given domain D
= [ c 1 , c 2 ] ⊆ R :
f is continuous on [ c 1 , c 2 ] and differentiable on ( c 1 , c 2 ) ;
| f
(cid:48) ( x ) | ∈
[ δ 1 , δ 2 ] for x ∈ ( c 1 , c 2 ) , with 0 < δ 1 < δ 2 < ∞ ; 3.
f is either of semi-convex or semi-concave.
We next use f to map the value function, Q ( s, a ) , to its transformed version, namely
[Equation]
Assumption 1 implies that f is invertible and, as such, Q ( s, a ) is uniquely computable from (cid:101) Q ( s, a ) by means of the inverse function f − 1 .
Of note, this assumption also implies that f preserves the ordering in x ; however, it inverts the ordering direction if f is decreasing.
Assumption 2 imposes further restrictions on f , but still leaves a broad class of mapping functions to consider.
Throughout the paper, we use tilde to denote a “mapped” function or variable, while the mapping f is understandable from the context (otherwise it is explicitly said).
2. B ASE A LGORITHM
If mapped value estimates were naively placed in a Q-Learning style algorithm, the algorithm would fail to converge to the optimal values in stochastic environments.
More formally, in the tabular case, an update of the form (cf.
Equation 1)
[Equation]
converges 2 to the ﬁxed point (cid:101) Q (cid:12) ( s, a ) that satisﬁes
[Equation]
Let us deﬁne the notation Q (cid:12) ( s, a ) .
=
f − 1 (cid:0) (cid:101) Q (cid:12) ( s, a ) (cid:1) .
If f is a semi-convex bijection
[Equation]
f − 1 will be semi-concave and Equation 4 deduces
[Equation]
where the third line follows Jensen’s inequality.
Comparing Equation 5 with the Bellman optimality equation in the regular space, i.e. Q ∗ ( s, a ) =
E s (cid:48) ,r ∼ P,R [ r + γ max a (cid:48)
Q ∗ ( s (cid:48) , a (cid:48) )] , we conclude that the value function to which the update rule (3) converges overestimates Bellman’s backup.
Similarly, if f is a semi-concave function, then Q (cid:12) ( s, a ) underestimates Bellman’s backup.
Either way, it follows that the learned value function deviates from the optimal one.
Furthermore, the Jensen’s gap at a given state s — the difference between the left-hand and right-hand sides of Equation 5 — depends on the action a because the expectation operator depends on a .
That is, at a given state s , the deviation of Q (cid:12) ( s, a ) from Q ∗ ( s, a ) is not a ﬁxed-value shift and can vary for various actions.
Hence, the greedy policy w.r.t.
(with respect to) Q (cid:12) ( s, · ) may not preserve ordering and it may not be an optimal policy either.
In an effort to address this problem in the spacial case of f being a logarithmic function, van Seijen et al.
(2019) observed that in the algorithm described by Equation 3, the learning rate α t generally conﬂates two forms of averaging: (i) averaging of stochastic update targets due to environment stochasticity (happens in the regular space), and (ii) averaging over different states and actions (happens in the f ’s mapping space).
To this end, they proposed to algorithmically disentangle the two and showed that such a separation will lift the Jensen’s gap if the learning rate for averaging in the regular space decays to zero fast enough.
Building from Log Q-Learning (van Seijen et al., 2019), we deﬁne the base algorithm as follows: at each time t , the algorithm receives (cid:101) Q t ( s, a ) and a transition quadruple ( s, a, r, s (cid:48) ) , and outputs (cid:101) Q t +1 ( s, a ) , which then yields Q t +1 ( s, a ) .
=
f − 1 (cid:0) (cid:101) Q t +1 ( s, a ) (cid:1) .
The steps are listed below
[Equation]
Here, the mapping f is any function that satisﬁes Assumptions 1 and 2.
Remark that similarly to the Log Q-Learning algorithm, Equations 9 and 10 have decoupled averaging of stochastic update targets from that over different states and actions.
R EWARD D ECOMPOSITION
Reward decomposition can be seen as a generic way to facilitate (i) systematic use of environmental inductive biases in terms of known reward sources, and (ii) action selection as well as value-function updates in terms of communication between an arbitrator and several subagents, thus assembling several subagents to collectively solve a task.
Both directions provide broad avenues for research and have been visited in various contexts.
Russell & Zimdars (2003) introduced an algorithm called
Q-Decomposition with the goal of extending beyond the “monolithic” view of RL.
They studied the case of additive reward channels, where the reward signal can be written as the sum of several reward channels.
They observed, however, that using Q-Learning to learn the corresponding Q function of each channel will lead to a non-optimal policy (they showed it through a counterexample).
Hence, they used a Sarsa -like update w.r.t.
the action that maximizes the arbitrator’s value.
Laroche et al.
(2017) provided a formal analysis of the problem, called the attractor phenomenon, and studied a number of variations to Q-Decomposition .
On a related topic but with a different goal, Sutton et al.
(2011) introduced the Horde architecture, which consists of a large number of “demons” that learn in parallel via off-policy learning.
Each demon estimates a separate value function based on its own target policy and (pseudo) reward function, which can be seen as a decomposition of the original reward in addition to auxiliary ones.
van Seijen et al.
(2017) built on these ideas and presented hybrid reward architecture ( HRA ) to decompose the reward and learn their corresponding value functions in parallel, under mean bootstrapping .
They further illustrated signiﬁcant results on domains with many independent sources of reward, such as the Atari 2600 game of Ms. Pac-Man.
Besides utilizing distinct environmental reward sources, reward decomposition can also be used as a technically-sound algorithmic machinery.
For example, reward decomposition can enable utilization of a speciﬁc mapping that has a limited domain.
In the Log Q-Learning algorithm, for example, the log( · ) function cannot be directly used on non-positive values.
Thus, the reward is decomposed such that two utility functions (cid:101) Q + and (cid:101) Q − are learned for when the reward is non-negative or negative, respectively.
Then the value is given by Q ( s, a ) =
exp (cid:0) (cid:101) Q
+ ( s, a ) (cid:1) − exp (cid:0) (cid:101) Q − ( s, a ) (cid:1) .
The learning process of each of (cid:101) Q + and (cid:101) Q − bootstraps towards their corresponding value estimate at the next state with an action that is the arg max of the actual Q , rather than that of (cid:101) Q + and (cid:101) Q − individually.
We generalize this idea to incorporate arbitrary decompositions, beyond only two channels.
To be speciﬁc, we are interested in linear decompositions of the reward function into L separate channels r ( j ) , for j = 1 . . .
L , in the following way:
[Equation]
with λ j ∈ R .
The channel functions r ( j ) map the original reward into some new space in such a way that their weighted sum recovers the original reward.
Clearly, the case of L = 1 and λ 1 = 1 would retrieve the standard scenario with no decomposition.
In order to provide the update, expanding from Log Q-Learning , we deﬁne (cid:101) Q ( j ) for j = 1 . . .
L , corresponding to the above reward channels, and construct the actual value function Q using the following:
[Equation]
We explicitly allow the mapping functions, f j , to be different for each channel.
That is, each reward channel can have a different mapping and each (cid:101) Q ( j ) is learned separately under its own mapping.
Before discussing how the algorithm is updated with the new channels, we present a number of interesting examples of how Equation 11 can be deployed.
As the ﬁrst example, we can recover the original Log Q-Learning reward decomposition by considering L = 2 , λ 1 = +1 , λ 2 = − 1 , and the following channels:
[Equation]
Notice that the original reward is retrieved via r t = r (1) t − r (2) t .
This decomposition allows for using a mapping with only positive domain, such as the logarithmic function.
This is an example of using reward decomposition to ensure that values do not cross the domain of mapping function f .
In the second example, we consider different magniﬁcations for different sources of reward in the environment so as to make the channels scale similarly.
The Atari 2600 game of Ms. Pac-Man is an example which includes rewards with three orders of magnitude difference in size.
We may therefore use distinct channels according to the size-range of rewards.
To be concrete, let r ∈
[0 , 100] and consider the following two conﬁgurations for decomposition (can also be extended to other ranges).
Conﬁguration 1:
[Equation]
if r t ≤0 .
1 r t if r t ∈ (1 , 10] r t >r (3) t :=
(cid:26)if r t ≤0 .
01 r t if r t ∈ (10 , 100]
Conﬁguration 2: λ 1 = 1 , λ 2 = 9 , λ 3 =r (1) t := (cid:26) r t if r t ∈ [0 , 1]r t >r (2) t :=   if r t ≤( r t − 1) / 9 if r t ∈ (1 , 10] r t >r (3) t :=
(cid:26)if r t ≤( r t − 10) / 90 if r t ∈ (10 , 100]
Each of the above conﬁgurations presents certain characteristics.
Conﬁguration 1 gives a scheme where, at each time step, at most one channel is non-zero.
Remark, however, that each channel will be non-zero with less frequency compared to the original reward signal, since rewards get assigned to different channels depending on their size.
On the other hands, Conﬁguration 2 keeps each channel to act as if there is a reward clipping at its upper bound, while each channel does not see rewards below its lower bound.
As a result, Conﬁguration 2 fully preserves the reward density at the ﬁrst channel and presents a better density for higher channels compared to Conﬁguration 1.
However, the number of active channels depends on the reward size and can be larger than one.
Importantly, the magnitude of reward for all channels always remains in [0 , 1] in both conﬁgurations, which could be a desirable property.
The ﬁnal point to be careful about in using these conﬁgurations is that the large weight of higher channels signiﬁcantly ampliﬁes their corresponding value estimates.
Hence, even a small estimation error at higher channels can overshadow the lower ones.
Over and above the cases we have presented so far, reward decomposition enables an algorithmic machinery in order to utilize various mappings concurrently in an ensemble.
In the simplest case, we note that in Equation 11 by construction we can always write r ( j ) :
= r and
L (cid:88)
j = λ j := 1 .
That is, the channels are merely the original reward with arbitrary weights that should sum to one.
We can then use arbitrary functions f j for different channels and build the value function as presented in Equation 12.
This construction directly induces an ensemble of arbitrary mappings with different weights, all learning on the same reward signal.
More broadly, this can potentially be combined with any other decomposition scheme, such as the ones we discussed above.
For example, in the case of separating negative and positive rewards, one may also deploy two (or more) different mappings for each of the negative and positive reward channels.
This certain case results in four channels, two negative and two positive, with proper weights that sum to one.
O RCHESTRATION OF V ALUE -M APPINGS USING D ECOMPOSED R EWARDS 4. A LGORITHM
To have a full orchestration, we next combine value mapping and reward decomposition.
We follow the previous steps in Equations 6 –10, but now also accounting for the reward decomposition.
The core idea here is to replace Equation 6 with 12 and then compute (cid:101) Q ( j ) for each reward channel in parallel.
In practice, these can be implemented as separate Q tables, separate Q networks, or different network heads with a shared torso.
At each time t , the algorithm receives all channel outputs (cid:101) Q ( j ) t , for j = 1 . . .
L , and updates them in accordance with the observed transition.
The complete steps are presented in Algorithm 1.
A few points are apropos to remark.
Firstly, the steps in the for-loop can be computed in parallel for all L channels.
Secondly, as mentioned previously, the mapping function f
j may be different for each channel; however, the discount factor γ and both learning rates β f,t and β reg,t are shared among all the channels and must be the same.
Finally, note also that the action ˜ a t +1 , from which all the channels bootstrap, comes from arg max a (cid:48)
Q t ( s t +1 , a (cid:48) ) and not the local value of each channel.
This directly implies that each channel-level value Q ( j ) =
f − 1 ( (cid:101) Q j ) does not solve a channel-level
Bellman equation by itself.
In other words, Q ( j ) does not represent any speciﬁc semantics such as expected return corresponding to the rewards of that channel.
They only become meaningful when they compose back together and rebuild the original value function.
4. C ONVERGENCE
We establish convergence of Algorithm 1 by the following theorem.
Algorithm 1:
Input: (at time t ) (cid:101) j s t , a t r t , s t +
Output:
Q j ) t
+ j
L
Compute r t ( j ) for j = 1 . . .
L begin
[Equation]
for j = 1 to L do
U ( j ) t := r ( j ) t + γf − 1 j (cid:16) (cid:101) Q ( j ) t ( s t +
[Equation]
˜ a t
[Equation]
) (cid:17) ˆ U ( j ) t := f
[Equation] [Equation]
− (cid:101) Q ( j ) t ( s t , a t ) (cid:17) end end
Theorem 1 Let the reward admit a decomposition as deﬁned by Equation 11, Q t ( s t , a t ) be deﬁned by Equation 12, and all (cid:101) Q ( j ) t ( s t , a t ) updated according to the steps of Algorithm 1.
Assume further that the following hold:
All f j ’s satisfy Assumptions 1 and 2; 2.
TD error in the regular space (second term in line 4 of Algorithm 1) is bounded for all j ; 3.
(cid:80) ∞ t =0 β f,t · β reg,t = ∞ ; 4.
(cid:80) ∞ t =
( β f,t · β reg,t ) 2 < ∞ ; 5.
β f,t · β reg,t → 0 as t → ∞ .
Then, Q t ( s, a ) converges to Q ∗ t ( s, a ) with probability one for all state-action pairs ( s, a ) .
The proof follows basic results from stochastic approximation theory (Jaakkola et al., 1994; Singh et al., 2000) with important additional steps to show that those results hold under the assumptions of
Theorem 1.
The full proof is fairly technical and is presented in Appendix A.
We further remark that Theorem 1 only requires the product
β f,t · β reg,t to go to zero.
As this product resembles the conventional learning rate in Q-Learning , this assumption is no particular limitation compared to traditional algorithms.
We contrast this assumption with the one in the previous proof of Log Q-Learning which separately requires β reg to go to zero fast enough.
We note that in the case of using function approximation, as in a DQN -like algorithm (Mnih et al., 2015), the update in line 5 of Algorithm 1 should naturally be managed by the used optimizer, while line 4 may be handled manually.
This has proved challenging as the convergence properties can be signiﬁcantly sensitive to learning rates.
To get around this problem, van Seijen et al.
(2019) decided to keep β reg,t at a ﬁxed value in their deep RL experiments, contrary to the theory.
Our new condition, however, formally allows β reg,t to be set to a constant value as long as β f,t properly decays to zero.
A somewhat hidden step in the original proof of Log Q-Learning is that the TD error in the regular space (second term in Equation 9) must always remain bounded.
We will make this condition explicit.
In practice, with bounds of the reward being known, one can easily ﬁnd bounds of return in regular as well as f j spaces, and ensure boundness of U ( j ) t − f − 1 j ( (cid:101) Q ( j ) t ) by proper clipping.
Notably, clipping of Bellman target is also used in the literature to mitigate the value overﬂow issue (Fatemi et al., 2019).
The scenarios covered by Assumption 2, with the new convergence proof due to Theorem 1, may be favorable in many practical cases.
Moreover, several prior algorithms such as
Q-Learning , Log Q-Learning , and Q-Decomposition can be derived by appropriate construction from Algorithm 1.
4. R EMARKS T IME - DEPENDENT C HANNELS
The proof of Theorem 1 does not directly involve the channel weights λ j .
However, changing them will impact Q t , which changes the action ˜ a t +1 in line 2 of Algorithm 1.
In the case that λ j ’s vary with time, if they all converge to their ﬁnal ﬁxed value soon enough before the learning rates become too small, and if additionally all state-action pairs are still visited frequently enough after λ j ’s are settled to their ﬁnal values, then the algorithm should still converge to optimality.
Of course, this analysis is far from formal; nevertheless, we can still strongly conjecture that an adaptive case where the channel weights vary with time should be possible to design.
S LOPE OF M APPINGS
Assumption 2 asserts that the derivative of f j must be bounded from both below and above.
While this condition is sufﬁcient for the proof of Theorem 1, we can probe its impact further.
The proof basically demonstrates a bounded error term, which ultimately converges to zero under the conditions of Theorem 1.
However, the bound on this error term (see Lemma 2 in Appendix A) is scaled by δ max = max j δ ( j ) , with δ ( j ) being deﬁned as
[Equation]
where δ ( j )and δ ( j )are deﬁned according to Assumption 2 ( 0 < δ ( j )≤ | f (cid:48)
j ( x ) | ≤ δ ( j ) 2 ).
In the case of f j being a straight line δ ( j ) = 0 , thus no error is incurred and the algorithm shrinks to
Q-Learning .
An important extreme case is when δ ( j )is too small while δ ( j )is not close to δ ( j ) 1 .
It then follows from Equation 15 that the error can be signiﬁcantly large and the algorithm may need a long time to converge.
This can also be examined by observing that if the return is near the areas where f (cid:48)
j is very small, the return may be too compressed when mapped.
Consequently, the agent becomes insensitive to the change of return in such areas.
This problem can be even more signiﬁcant in deep RL due to more complex optimization processes and nonlinear approximations.
The bottom-line is that the mapping functions should be carefully selected in light of Equation 15 to avoid extremely large errors while still having desired slopes to magnify or suppress the returns when needed.
This analysis also explains why logarithmic mappings of the form f ( x ) = c · log( x + d ) (as investigated in the context of Log Q-Learning by van Seijen et al.
present unfavorable results in dense reward scenarios; e.g. in the Atari 2600 game of Skiing where there is a reward at every step.
In this expression c is a mapping hyperparameter that scales values in the logarithmic space and
d is a small positive scalar to ensure bounded derivatives, where the functional form of the derivative is given by f (cid:48)
( x ) = cx + d .
Hence, δ 2 = cd , whereas δ 1 can be very close to zero depending on the maximum return.
As a result, when learning on a task which often faces large returns, Log Q-Learning operates mostly on areas of f where the slope is small and, as such, it can incur signiﬁcant error compared to standard Q-Learning .
See Appendix B for a detailed illustration of this issue, and Appendix D for the full list of reward density variations across a suite of 55 Atari 2600 games.
E XPERIMENTAL R ESULTS
In this section, we illustrate the simplicity and utility of instantiating new learning methods based on our theory.
Since our framework provides a very broad algorithm class with numerous possibilities, deep and meaningful investigations of speciﬁc instances go far beyond the scope of this paper (or any single conference paper).
Nevertheless, as an authentic illustration, we consider the LogDQN algorithm (van Seijen et al., 2019) and propose an altered mapping function.
As discussed above, the logarithmic mapping in LogDQN suffers from a too-small slope when encountering large returns.
We lift this undesirable property while keeping the desired magniﬁcation property around zero.
Speciﬁcally, we substitute the logarithmic mapping with a piecewise function that at the break-point x = 1 − d switches from a logarithmic mapping to a straight line with slope c (i.e. the same slope as c · log( x + d ) at x = 1 − d ):
[Equation]
We call the resulting method LogLinDQN , or Logarithmic-Linear DQN .
Remark that choosing x = 1 − d as the break-point has the beneﬁt of using only a single hyperparameter c to determine both the scaling of the logarithmic function and the slope of the linear function, which otherwise would require an additional hyperparameter.
Also note that the new mapping satisﬁes Assumptions 1 and 2.
We then use two reward channels for non-negative and negative rewards, as discussed in the ﬁrst example of Section 3 (see Equation 13), and use the same mapping function for both channels.
Our implementation of LogLinDQN is based on Dopamine (Castro et al., 2018) and closely matches that of LogDQN , with the only difference being in the mapping function speciﬁcation.
Notably, our
LogLin mapping hyperparameters are realized using the same values as those of LogDQN ; i.e. c = 0 .
5 and d ≈ 0 .
We test this method in the Atari 2600 games of the Arcade Learning Environment ( ALE ) (Bellemare et al., 2013) and compare its performance primarily against LogDQN and DQN (Mnih et al., 2015), denoted by “ Lin ” or “( Lin ) DQN ” to highlight that it corresponds to a linear mapping function with slope one.
We also include two other major baselines for reference:
C51 (Bellemare et al., 2017) and Rainbow (Hessel et al., 2018).
Our tests are conducted on a stochastic version of Atari 2600 using sticky actions (Machado et al., 2018) and follow a uniﬁed evaluation protocol and codebase via the Dopamine framework (Castro et al., 2018).
Figure 1 shows the relative human-normalized score of LogLinDQN w.r.t.
the worst and best of
LogDQN and DQN for each game.
These results suggest that LogLinDQN reasonably uniﬁes the good properties of linear and logarithmic mappings (i.e. handling dense or sparse reward distributions respectively), thereby enabling it to improve upon the per-game worst of LogDQN and DQN
[Figure]
(top
and perform competitively against the per-game best of the two (bottom panel) across a large set of games.
Figure 2 shows median and mean human-normalized scores across a suite of
Atari 2600 games.
Our LogLinDQN agent demonstrates a signiﬁcant improvement over most baselines and is competitive with Rainbow in terms of mean performance.
This is somewhat remarkable provided the relative simplicity of LogLinDQN , especially, w.r.t.
Rainbow which combines several other advances including distributional learning, prioritized experience replay, and n-step learning.
C ONCLUSION
In this paper we introduced a convergent class of algorithms based on the composition of two distinct foundations: (1) mapping value estimates to a different space using arbitrary functions from a broad class, and (2) linearly decomposing the reward signal into multiple channels.
Together, this new family of algorithms enables learning the value function in a collection of different spaces where the learning process can potentially be easier or more efﬁcient than the original return space.
Additionally, the introduced methodology incorporates various versions of ensemble learning in terms of linear decomposition of the reward.
We presented a generic proof, which also relaxes certain limitations in previous proofs.
We also remark that several known algorithms in classic and recent literature can be seen as special cases of the present algorithm class.
Finally, we contemplate research on numerous special instances as future work, following our theoretical foundation.
Also, we believe that studying the combination of our general value mapping ideas with value decomposition (Tavakoli et al., 2021), instead of the the reward decomposition paradigm studied in this paper, could prove to be a fruitful direction for future research.
R EPRODUCIBILITY S TATEMENT
We release a generic codebase, built upon the Dopamine framework (Castro et al., 2018), with the option of using arbitrary compositions of mapping functions and reward decomposition schemes as easy-to-code modules.
This enables the community to easily explore the design space that our theory opens up and investigate new convergent families of algorithms.
This also allows to reproduce the results of this paper through an accompanying conﬁguration script.
The source code can be accessed at: https://github.com/microsoft/orchestrated-value-mapping .
A P ROOF OF T HEOREM
We use a basic convergence result from stochastic approximation theory.
In particular, we invoke the following lemma, which has appeared and proved in various classic texts; see, e.g., Theoremin Jaakkola et al.
(1994) or Lemma 1 in Singh et al.
Lemma 1 Consider an algorithm of the following form:
[Equation]
with x being the state variable (or vector of variables), and α t and ψ t denoting respectively the learning rate and the update at time t .
Then, ∆ t converges to zero w.p. (with probability) one as t → ∞ under the following assumptions:
The state space is ﬁnite; 2.
(cid:80) t α t = ∞ and (cid:80) t
α 2 t < ∞ ; 3.
|| E { ψ t ( x ) | F t }|| W ≤ ξ || ∆ t ( x ) || W , with ξ ∈ (0 , 1) and || · || W denoting a weighted max norm; 4.
Var { ψ t ( x ) | F t } ≤ C (1 + || ∆ t ( x ) || W ) 2 , for some constant C ;
where F t is the history of the algorithm until time t .
Remark that in applying Lemma 1, the ∆ t process generally represents the difference between a stochastic process of interest and its optimal value (that is Q t and Q ∗ t ), and x represents a proper concatenation of states and actions.
In particular, it has been shown that Lemma 1 applies to Q-Learning as the TD update of Q-Learning satisﬁes the lemma’s assumptions 3 and 4 (Jaakkola et al., 1994).
We deﬁne Q ( j ) t ( s, a ) :
= f − 1 (cid:16) (cid:101) Q ( j ) t ( s, a ) (cid:17) , for j = 1 . . .
L .
Hence,
Q t (
[Equation]
We next establish the following key result, which is core to the proof of Theorem 1.
The proof is given in the next section.
Lemma
Following Algorithm 1, for each channel j ∈ { 1 , . . .
, L } we have
[Equation]
with the error term satisfying the followings:
Bounded by TD error in the regular space with decaying coefﬁcient
[Equation]
where δ ( j ) =
δ ( j ) 2 / δ ( j ) 1 − 1 is a positive constant; 2.
For a given f j , e ( j ) t does not change sign for all t (it is either always non-positive or always non-negative); 3.
e ( j ) t is fully measurable given the variables deﬁned at time t .
From Lemma
[Equation]
it follows that for each channel:
[Equation]
with e ( j ) t converging to zero w.p. one under condition 4 of the theorem
[Equation]
and U ( j ) t deﬁned as:
[Equation]
Multiplying both sides of Equation 20 by λ j and taking the summation
[Equation]
we write:
[Equation]
Hence, using Equation 17 we have:
[Equation]
Deﬁnition of ˜ a t +1 deduces that
[Equation]
By deﬁning e t := (cid:80)
Lj =
λ j e ( j ) t , we rewire Equation 21 as the following:
[Equation]
This is a noisy Q-Learning algorithm with the noise term decaying to zero at a quadratic rate w.r.t.
the learning rate’s decay; more precisely, in the form of ( β reg,t · β f,t ) 2 .
Lemma 1 requires the entire update to be properly bounded (as stated in its assumptions 3 and 4).
It has been known from the proof of Q-Learning (Jaakkola et al., 1994) that TD error satisﬁes these conditions, i.e. r t + γ max a (cid:48)
Q t ( s t +1 , a (cid:48) )
− Q t ( s t , a t ) satisﬁes assumptions 3 and 4 of Lemma 1.
To prove convergence of mapped Q-Learning , we therefore require to show that | e t | also satisﬁes a similar property; namely, not only it disappears in the limit, but also it does not interfere intractably with the learning process during training.
To this end, we next show that as the learning continues, | e t | is indeed bounded by a value that can be arbitrarily smaller than the TD error.
Consequently, as
TD error satisﬁes assumptions 3 and 4 of Lemma 1, so does | e t | , and so does their sum.
Let δ max = max j δ ( j ) , with δ ( j ) deﬁned in Lemma 2.
Multiplying both sides of Equation 19 by λ j and taking the summation over j , it yields:
[Equation]
The second line follows from Lemma 2.
If TD error in the regular space is bounded, then (cid:12)(cid:12)(cid:12) r ( j ) t + γ Q ( j ) t
( s t +1 , ˜ a t +1 ) − Q ( j ) t ( s t , a t ) (cid:12)(cid:12)(cid:12) ≤ K ( j ) for some K ( j ) ≥ 0 .
Hence, Equation 23 induces:
[Equation]
with K = (cid:80)
Lj [-=
-]=1 | λ j | · K ( j ) ≥ 0 ; thus, | e t | is also bounded for all t .
As (by assumption) β reg,t · β f,t converges to zero, we conclude that there exists T ≥ 0 such that for all t ≥ T we have
[Equation]
for any given ξ ∈ (0 , 1] .
Hence, not only | e t | goes to zero w.p. one as t → ∞ , but also its magnitude always remains upperbounded below the size of TD update with any arbitrary margin ξ .
Since TD update already satisﬁes assumptions 3 and 4 of Lemma 1, we conclude that with the presence of e t those assumptions remain satisﬁed, at least after reaching some time T where Equation 25 holds.
Finally, Lemma 2 also asserts that e t is measurable given information at time t , as required by
Lemma 1.
Invoking Lemma 1, we can now conclude that the iterative process deﬁned by Algorithmconverges to Q ∗ t w.p. one.
P ROOF OF L EMMAP
[Equation]
Our proof partially builds upon the proof presented by van Seijen et al.
To simplify the notation, we drop j in f j , while we keep j in other places for clarity.
By deﬁnition we have (cid:101) Q ( j ) t ( s, a ) = f (cid:16) Q ( j ) t ( s, a ) (cid:17) .
Hence, we rewrite Equations 3, 4, and 5 of
Algorithm 1 in terms of Q ( j ) t :
[Equation] [Equation]
The ﬁrst two equations yield:
[Equation]
By applying f − 1 to both sides of Equation
[Equation]
we get:
[Equation]
which can be rewritten as:
[Equation]
where e ( j ) t is the error due to averaging in the mapping space instead of in the regular space:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
We next analyze the behavior of e ( j ) t under the Theorem 1’s assumptions.
To simplify, let us introduce the following substitutions:
[Equation]
The error e ( j ) t can be written as
[Equation]
We remark that both v and w lie between a and b .
Notably, e ( j ) t has a particular structure which we can use to bound w − v .
See Table 1 for the ordering of v and w for different possibilities of f .
We deﬁne three lines g 0 ( x ) , g 1 ( x ) , and g 2 ( x ) such that they all pass through the point ( a, f ( a )) .
As for their slopes, g 0 ( x ) has the derivative f (cid:48) ( a ) , and g 2 ( x ) has the derivative f (cid:48)
( b ) .
The function g 1 ( x ) passes through point ( b, f ( b )) as well, giving it derivative ( f ( a ) − f ( b )) /
( a − b ) .
See Figurefor all the possible cases.
We can see that no matter if f is semi-convex or semi-concave
and if it is increasing or decreasing these three lines will sandwich f over the interval [ a, b ] if b ≥ a , or similarly over [ b, a ] if a ≥ b .
Additionally, it is easy to prove that for all x in the interval of a and b , either of the following holds:
[Equation]
or
[Equation]
The ﬁrst one is equivalent to
[Equation]
35) while the second one is equivalent to
[Equation]
From the deﬁnition of g 1 it follow that in all the mentioned possibilities of f combined with either of a ≥ b or b ≥ a , we always have g 1 ( v ) =
˜ w and g − 1 1 ( ˜ w ) = v .
Hence, plugging ˜ w in Equationand Equation 36 (and noting that f − 1 ( ˜ w ) = w ) deduces
[Equation]
) or
[Equation]
Either way, regardless of various possibilities for f as well as a and b , we conclude that
[Equation]
[Figure]
From deﬁnition of the lines g
and g 2 , we write the line equations as follows:
[Equation]
Applying these equations on the points ( g − 1 0 ( ˜ w ) , ˜ w ) and ( g − 1 2 ( ˜ w ) , ˜ w ) , respectively
[Equation]
it yields:
[Equation]
f
[Equation]
(cid:48)
[Equation]
which deduce
[Equation]
Plugging the above in Equation
[Equation]
it follows:
[Equation] [Equation]
We next invoke the mean value theorem , which states that if f is a continuous function on the closed interval [ a, b ] and differentiable on the open interval ( a, b ) , then there exists a point c ∈ ( a, b ) such that f ( b ) − f
( a ) =
f (cid:48) ( c )( b − a ) .
Remark that based on Assumption 2, c would satisfy f (cid:48)
j ( c ) ≤ δ ( j ) 2 , also that 1 f (cid:48) ( b ) − 1 f (cid:48) ( a ) ≤ 1 δ ( j ) 1 − 1 δ ( j ) 2 .
Hence,
[Equation]
From Equation
[Equation]
it follows that
[Equation]
We therefore can write
[Equation]
where δ ( j ) =
δ ( j ) 2 ( 1 δ ( j ) 1 − 1 δ ( j ) 2 ) is a positive constant.
This completes the proof for the ﬁrst part of the lemma.
P ART
For this part, it can be directly seen from Figure
that for a given f j , the order of w (cid:48) , w , v , and v (cid:48) is ﬁxed, regardless of whether a ≥ b or b ≥ a (in Figure 3, compare each plot A, B, C, and D with their counterparts at the bottom).
Hence, the sign of e ( j ) t = w − v will not change for a ﬁxed mapping.
P ART
Finally, we note that by its deﬁnition, e ( j ) t comprises quantities that are all deﬁned at time t .
Hence, it is fully measurable at time t , and this completes the proof.
B D ISCUSSION ON L OG VERSUS L OG L IN
As discussed in Section 4.3 (Slope of Mappings), the logarithmic ( Log ) function suffers from a toolow slope when the return is even moderately large.
Figure 4 visualizes the impact more vividly.
The logarithmic-linear ( LogLin ) function lifts this disadvantage by switching to a linear ( Lin ) function for such returns.
For example, if the return changes by a unit of reward from 19 to 20 , then the change will be seen as 0 .
05 in the Log space (i.e. log(20) − log(19) ) versus 1 . 0 in the LogLin space; that is, Log compresses the change by 95% for a return of around 20 .
As, in general, learning subtle changes is more difﬁcult and requires more training iterations, in such scenarios normal DQN (i.e. Lin function) is expected to outperform LogDQN .
On the other hand, when the return is small (such as in sparse reward tasks), LogDQN is expected to outperform DQN .
Since LogLin exposes the best of the two worlds of logarithmic and linear spaces (when the return lies in the respective regions), we should expect it to work best if it is to be used as a generic mapping for various games.
[Figure]
C E XPERIMENTAL D ETAILS
The human-normalized scores reported in our Atari 2600 experiments are given by the formula (similarly to van Hasselt et al.
[Equation]
where score agent , score human , and score random are the per-game scores (undiscounted returns) for the given agent, a reference human player, and random agent baseline.
We use Table 2 from Wang et al.
(2016) to retrieve the human player and random agent scores.
The relative human-normalized score of LogLinDQN versus a baseline in each game is given by (similarly to Wang et al.
[Equation]
where score LogLinDQN and score baseline are computed by averaging over the last 10% of each learning curve (i.e. last 20 iterations).
The reported results are based on three independent trials for LogLinDQN and LogDQN , and ﬁve independent trials for DQN .
D A DDITIONAL R ESULTS
Figure 5 shows the relative human-normalized score of LogLinDQN versus LogDQN (top panel) and versus ( Lin ) DQN (bottom panel) for each game, across a suite of 55 Atari 2600 games.
LogLinDQN signiﬁcantly outperforms both LogDQN
and ( Lin ) DQN on several games, and is otherwise on par with them (i.e. when LogLinDQN is outperformed by either of LogDQN or ( Lin ) DQN , the difference is not by a large margin).
Figure 6 shows the raw (i.e. without human-normalization) learning curves across a suite of 55 Atari 2600 games.
Figures 7, 8, and 9 illustrate the change of reward density (measured for positive and negative rewards separately) at three different training points (
before
training begins,
after
iteration 5 , and after iteration 49 ) across a suite of 55 Atari 2600 games.
[Figure]
[Figure] [Figure]