O RCHESTRATED V ALUE M APPING FOR R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a general convergent class of reinforcement learning algorithms that is founded on two distinct principles: (1) mapping the value function into a different space via arbitrary functions from a broad class and (2) linearly decomposing the reward signal into multiple channels.
The ﬁrst principle enables asserting speciﬁc properties on the value function that can enhance learning.
The second principle, on the other hand, allows for the value function to be represented as a composition of multiple utility functions.
This can be leveraged for various purposes, including dealing with highly varying reward scales, incorporating a priori knowledge about the sources of reward, and ensemble learning.
Combining the two principles yields a general blueprint for instantiating convergent algorithms by orchestrating diverse mapping functions over multiple reward channels.
This blueprint generalizes and subsumes algorithms such as classical Q-Learning , Log Q-Learning , and Q-Decomposition .
Moreover, our convergence proof for this general class relaxes certain required assumptions in some existing algorithms.
Using our theory we discuss several interesting conﬁgurations as special cases.
Finally, to illustrate the potential of the design space that our theory opens up, we instantiate a particular algorithm and evaluate its performance on the suite of Atari 2600 games.
I NTRODUCTION
The chief goal of reinforcement learning (RL) algorithms is to maximize the expected return — the value function — from each state (Sutton & Barto, 2018; Szepesv ´ ari, 2010).
For decades, many algorithms have been proposed to compute the value function either as their main goal (critic-only algorithms) or as the means to help the policy search process (actor-critic algorithms).
However, when the environment comprises certain properties, learning the induced value function may become very challenging (van Seijen et al., 2019).
Examples include environments where rewards are dense in some parts of the state-space but very sparse in other parts, or when the size of rewards may vary dramatically.
In the Atari 2600 game of Ms. Pac-Man, for instance, the reward can vary fromfor small pellets to as large as 5000 for moving bananas.
In other games like Tennis, as the agent acts randomly, it easily loses the game, causing negative rewards with high frequency.
However, when the agent learns to capture the ball (hence no negative reward anymore), it may still take a very long time before the agent gains a score and see a positive reward.
Such learning scenarios, for one reason or another, have proved challenging for the conventional RL algorithms.
One issue that may rise due to these environmental challenges is for the action gap — the optimal value difference between the best and second-best actions (Farahmand, 2011) — to become significantly non-uniform.
A recent study (van Seijen et al., 2019) showed promising results by simply mapping the value function to a logarithmic space and add important algorithmic steps to guarantee the convergence under generic conditions.
While this construction addresses the problem of nonuniform action-gap and facilitates the use of smaller discount factors, it further opens a new direction to improve the learning: learn the value function in a different space that admits better properties compared to the original space.
This interesting view naturally raises theoretical questions about the required properties of the functions that map the value to the learning space, and whether the algorithm still guarantees to converge to the optimal value function under this new construction.
A loosely related topic is to presume a nonlinear Bellman equation.
Originally, the Bellman equation of optimality (Bellman, 1954; 1957) is a direct result of deﬁning return as a discounted sum of future rewards.
However, one may ignore this basis, and re-deﬁne Bellman equation in a generic non-linear way.
In particular, van Hasselt et al.
(2019) have shown that the new Bellman operator is still a contraction mapping and therefore the resulting algorithms may be well-behaved.
The application of such algorithms is still unclear since the ﬁxed-point does not have direct connection to the concept of return.
In this paper, we do not consider nonlinear Bellman equations.
Continuing with the ﬁrst line of thought, a natural extension is to use different mappings concurrently as an ensemble, each of which adding different beneﬁts, just like having various expertise in the board that runs a company.
This may also be seen as a way of separation of concerns (van Seijen et al., 2016).
Ideally, we may wish to dynamically modify the inﬂuence of different mappings as the learning advances.
For example, the agent may start with mappings that facilitate learning from sparse rewards, then as the agent learns to collect more rewards and they get denser, it gradually switches to mappings that simpliﬁes learning on dense rewards.
Of note, if there are known sources of rewards with known, speciﬁc characteristics (say sparse positive rewards, but dense negative ones), same approach may prove beneﬁcial.
This has been the backbone for reward channeling architectures such as (van Seijen et al., 2017).
In this paper, building upon the aforementioned ideas, we present two orthogonal directions: generic value mapping and reward decomposition.
We then combine the two and build a broad algorithm class, which can be instantiated to generate various interesting agents.
We present the convergence results for the complete version of the algorithm.
It is worth noting that this algorithm may be best seen as a blueprint for constructing other convergent algorithm classes.
V ALUE M APPING
We consider a general reinforcement learning problem, modeled as a Markov Decision Process (MDP) (Puterman, 1994)
M = ( S , A , P, r, γ ) , where S and A are the discrete sets of states and actions; P : S ×
A × S → [0 , 1] is a transition function that deﬁnes the transition probability from state s to s (cid:48)
if action a is taken; r : S × A × S → [ r min , r max ] is a reward function and γ ∈ [0 , 1] denotes a scalar discount factor.
A policy π ( s, a ) = P [ A t = a | S t = s ] deﬁnes how an action is selected in a given state.
Therefore, selecting actions according to a stationary policy in general results in a stochastic trajectory.
The discounted sum of rewards over the trajectory induces a random variable, called return .
We assume that all returns are ﬁnite and bounded.
A trajectory is therefore called optimal if it induces maximum return.
A state-action value function Q π ( s, a ) is deﬁned in conjunction with a policy π .
Q π ( s, a ) induces the expected return of taking action a at state s and following π thereafter.
The optimal value function is deﬁned as Q ∗ ( s, a ) =
max π Q π ( s, a ) , which is the maximum expected return of all trajectories starting from ( s, a ) .
Similarly, an optimal policy is deﬁned as π ∗ ( s, a ) =
arg max π Q π ( s, a ) .
The optimal value function is unique (Bertsekas & Tsitsiklis, 1996) and can be found for example as the ﬁxed-point of the Q-Learning algorithm (Watkins, 1989; Watkins & Dayan, 1992), given by the following update rule:
[Equation]
where α t is the learning rate (step size) parameter deﬁned at time t .
Our goal is to ﬁrst map Q to a different space, and then perform the update, so that the learning process can beneﬁt from the properties of the mapped space.
We deﬁne a function f that maps the value function to some new space.
In particular, we consider the following assumptions:
Assumption 1 The function f ( x ) is a bijection (either strictly increasing or strictly decreasing) for all x in the given domain D
= [ c 1 , c 2 ] ⊆ R .
Assumption 2 The function f ( x ) holds the following properties for all x in the given domain D
= [ c 1 , c 2 ] ⊆ R :
f is continuous on [ c 1 , c 2 ] and differentiable on ( c 1 , c 2 ) ; 2.
| f (cid:48)
( x ) | ∈
[ δ 1 , δ 2 ] for x ∈ ( c 1 , c 2 ) , with 0 < δ 1 < δ 2 < ∞ ; 3.
f is either of semi-convex or semi-concave.
We next use f to map the value function, Q ( s, a ) , to its transformed version, namely
[Equation]
Assumption 1 induces that f ( x ) is invertible, which we denote by f − 1 ( x ) .
Additionally, having known either of Q ( s, a ) or (cid:101) Q ( s, a ) , the other one is uniquely computable.
Of note, this assumption also implies that f preserves ordering; however, it will change the direction of ordering if f is decreasing.
Assumption 2 imposes more restriction on f ; nevertheless, it is still a broad class of functions to consider.
Throughout the paper, we use tilde notation to refer to the “mapped” function or variable, while the mapping f is understandable from the context (otherwise it is explicitly said).
2. B ASE A LGORITHM
If the mapped value is naively placed in a Q-Learning -like algorithm, it will not converge to the optimal values in stochastic environments.
More formally, for a tabular MDP, an update of the following form: (compare it with equation 1)
[Equation]
converges 1 to the ﬁxed point (cid:101) Q (cid:12) ( s, a ) that satisﬁes
[Equation]
where s (cid:48) denotes the next state.
Let us deﬁne the notation Q (cid:12) ( s, a ) .
=
f − 1 (cid:16) (cid:101) Q (cid:12) ( s, a ) (cid:17) .
If f is a semi-convex bijection, f − 1 will be semi-concave and equation 4 deduces
[Equation]
where the third line follows Jensen’s inequality.
Comparing (5) with the Bellman’s optimality equation in the regular space, i [Equation] e. Q ∗
[Equation]
= [Equation] s (cid:48) , a (cid:48) )] ; we conclude that the value function to which (3) converges over-estimates Bellman’s backup.
Similarly, if f is a semi-concave function, then Q (cid:12) ( s, a ) under-estimates Bellman’s backup.
Either way, it follows that the learned value function deviates from the optimal one.
Further, the Jensen’s gap at a given state s — the difference between left- and right-hand-side of (5) — depends on the action a because the expectation operator depends on a .
That is, at a given state s , the deviation of Q (cid:12) ( s, a ) from Q ∗ ( s, a ) is not a ﬁxed-value shift and can vary for various actions.
Hence, the greedy policy w.r.t.
Q (cid:12) ( s, · ) may not preserve ordering and it may not be an optimal policy either.
In an effort to address this problem in the spacial case of f being a logarithmic function, van Seijen et al.
(2019) observed that in the algorithm described by equation 3, the step-size α generally conﬂates two forms of averaging: (i) averaging of stochastic update targets due to environment stochasticity (happens in the regular space), and (ii) averaging over different states/actions (happens in the f space).
Moreover, they proposed to algorithmically separate the two and showed that such a separation will lift the Jensen gap between Q (cid:12) ( s, a ) and Q ∗ ( s, a ) if the learning rate of the regular space decays to zero fast enough.
Building from Log Q-Learning (van Seijen et al., 2019), we deﬁne the base algorithm as the following: at time t it receives (cid:101) Q t ( s, a ) and a transition tuple ( s, a, r, s (cid:48) ) as deﬁned previously, and outputs (cid:101) Q t +1 ( s, a ) , which then yields Q t +1 ( s, a ) =
f − 1 ( (cid:101) Q t +1 ( s, a )) .
The steps are listed below
[Equation]
Here, the mapping f is any function that satisﬁes Assumptions 1 and 2.
Remark that similarly to the Log Q-Learning algorithm, (9) and (10) have decoupled averaging of stochastic update targets from that of different state-actions.
R EWARD D ECOMPOSITION
Reward decomposition can be seen as a generic way to facilitate (i) systematic use of environmental inductive biases in terms of known reward sources, and (ii) action selection as well as value-function updates in terms of communication between an arbitrator and several subagents; hence, assembling several subagents to collectively solve a task.
Both directions provide broad avenues for research and have been visited in various contexts.
Russell & Zimdars (2003) introduced an algorithm called
Q-Decomposition with the goal of extending the “monolithic” view of RL.
They studied the case of additive reward channels, where the reward signal can be written as the sum of several reward channels.
They observed, however, that using Q-Learning to learn the corresponding Q function of each channel will lead to a non-optimal policy (they showed it through a counter example).
Hence, they used a Sarsa -like update with respect to the action that maximizes arbitrator’s value.
Laroche et al.
(2017) provided a formal analysis of the problem, called attractor phenomenon, and studied a number of variations to Q-Decomposition .
In a related topic but with a different goal, Sutton et al.
(2011) introduced the Horde architecture, which consists of a large number of ‘demons’ that learn in parallel via off-policy learning.
Each demon trains a separate general value function ( GVF ) based on its own policy and (pseudo) reward function, which can be seen as a decomposition of the original reward in addition to auxiliary ones.
van Seijen et al.
(2017) built on these ideas and presented an architecture, called HRA , to decompose the reward and learn their corresponding value functions in parallel, under mean bootstrapping .
They further illustrated signiﬁcant results on domains with many sources of independent rewards, such as the Atari 2600 game of Ms. Pac-Man.
Besides utilizing distinct environmental reward sources, reward decomposition can also be used as a technically-sound algorithmic machinery.
For example, reward decomposition can enable utilization of a speciﬁc mapping that has a limited domain.
In the Log Q-Learning algorithm, for example, the log( · ) function cannot be directly used on non-positive values.
The reward is therefore decomposed such that two utility functions (cid:101) Q + and (cid:101) Q − are learned for when the reward is non-negative or negative, respectively.
Then the value is given by Q ( s, a ) =
exp( (cid:101) Q
+ ( s, a ))
− exp( (cid:101) Q − ( s, a )) .
The learning process of each of (cid:101) Q + and (cid:101) Q − bootstraps towards the value of next state with an action which is the arg max of the actual Q , rather than that of (cid:101) Q + and (cid:101) Q − .
We generalize this idea to incorporate arbitrary decomposition beyond only two channels.
We are particularly interested in the following form of decomposition: Let the reward function be decomposed into L separate channels r ( j ) , j = 1 . . .
L , as the following:
[Equation]
with λ j ∈ R .
The channel functions r ( j ) map the original reward into some new space in such a way that their weighted sum recovers the original reward.
Clearly, the case of L = 1 and λ 1 = 1 would retrieve the normal scenario with no decomposition.
In order to provide the update, expanding from
Log Q-Learning , we then deﬁne (cid:101) Q ( j ) for j = 1 . . .
L corresponding to the above reward channels, and construct the actual value function by the following:
[Equation]
We explicitly allow the mapping functions, f j ’s, to be different.
That is, each reward channel can have a different mapping and each (cid:101) Q ( j ) is then learned separately under its own mapping.
Before discussing how the algorithm is updated with the new channels, we present a number of interesting examples of how (11) can be deployed.
As the ﬁrst example, we can recover the original Log Q-Learning reward channels, by considering
L = 2 , λ 1 = +1 , λ 2 = − 1 , and the following channels:
[Equation]
Note the original reward is retrieved by r t = r (1) t − r (2) t .
This decomposition allows for using a mapping with only positive domain, like the logarithmic function.
This is an example of using reward decomposition to ensure that values do not cross the domain of f .
In the second example, we consider different magniﬁcations for different sources of reward in the environment so as to make the channels scale similarly.
The Atari 2600 game of Ms. Pac-Man is an example, which includes rewards with three orders of magnitude difference in size.
We may therefore use distinct channels according to the size-range of rewards.
To be concrete, let r ∈
[0 , 100] and consider the following two conﬁgurations for decomposition (can be extended to other ranges).
Conﬁguration 1:
[Equation]
if r t ≤0 .
1 r t if r t ∈ (1 , 10] r t >r (3) t :=
(cid:26)if r t ≤0 .
01 r t if r t ∈ (10 , 100]
Conﬁguration 2: λ 1 = 1 , λ 2 = 9 , λ 3 =r (1) t := (cid:26) r t if r t ∈ [0 , 1]r t >r (2) t :=   if r t ≤( r t − 1) / 9 if r t ∈ (1 , 10] r t >r (3) t :=
(cid:26)if r t ≤( r t − 10) / 90 if r t ∈ (10 , 100]
Each of the above conﬁgurations presents certain characteristics.
Conﬁguration 1 gives a scheme where at each time step, at most one channel is non-zero.
Remark, however, that each channel will be non-zero with less frequency compared to the original reward, since the rewards are distributed over various channels, depending on their size.
On the other hands, Conﬁguration 2 keeps each channel to act as if there is a reward clipping at its upper bound, while each channel does not see rewards below its lower bound.
As a result, it fully preserves the reward density at the ﬁrst channel, and presents a better density for higher channels compared to Conﬁguration 1.
However, the number of active channels depends on the reward size and can be larger than one.
Importantly, the magnitude of all reward channels always remains in [0 , 1] in both conﬁgurations, which could be a desired property.
The ﬁnal point to be careful about in using these conﬁgurations is that the large weight of higher channels signiﬁcantly ampliﬁes their values; hence, even a small error in higher channels can shadow out the lower ones.
Over and above the cases we presented so far, reward decomposition enables an algorithmic machinery in order to utilize various mappings concurrently as an ensemble.
In the simplest case, we note that in equation (11), by construction we can always write
[Equation]
j ) t ( s t , a t , s t +1 ) = r t ( s t , a t , s t +1 ) and L (cid:88)
j
[Equation] [Equation]
begin
Q t ( s t , a t ) :
= (cid:80)
{+Lj =
+}
λ j f − 1 j (cid:16) (cid:101) Q ( j ) t ( s t , a t ) (cid:17) ˜ a t +1 :=
arg max a (cid:48) (cid:16) Q t ( s t +1 , a (cid:48) ) (cid:17) for do
U ( j ) t := r ( j ) t + γf − 1 j (cid:16) (cid:101) Q ( j ) t ( s t +1 , ˜ a t +1 ) (cid:17) ˆ U ( j ) t := f − 1 j (cid:16) (cid:101) Q ( j ) t ( s t , a t ) (cid:17) + β reg,t (cid:16) U ( j ) t − f − 1 j (cid:16) (cid:101) Q ( j ) t ( s t , a t ) (cid:17)(cid:17) (cid:101) Q ( j ) t +1 ( s t , a t ) :
= (cid:101) Q ( j ) t ( s t , a t ) + β f,t (cid:16)
f j (cid:16) ˆ U ( j ) t (cid:17) − (cid:101) Q ( j ) t ( s t , a t ) (cid:17) end end
That is, the channels are merely the original reward with arbitrary weights, which should sum to one.
We can then use arbitrary functions f j for different channels and build the value function as presented in equation (12).
This construction directly induces an ensemble of arbitrary mappings with different weights, all would learn on the same reward signal.
More broadly, this can potentially be combined with any other decomposition schemed, such as the ones we discussed above.
For example, in the case of separating negative and positive rewards, one may also deploy two (or more) different mappings for each of the negative and positive reward channels.
This certain case results in four channels: two negative and two positive with proper weights that sum to one.
O RCHESTRATION OF V ALUE -M APPINGS USING D ECOMPOSED R EWARDS 4. A LGORITHM
To have a full orchestration, we next combine value mappings and reward decomposition.
We follow the previous steps in equations (6) to (10), but now also accounting for the reward decomposition.
The core idea here is to replace (6) with (12) and then compute (cid:101) Q ( j ) for each reward channel in parallel.
In practice, these can be implemented as separate Q tables, separate Q networks, or different network heads with a shared torso (the shared neural network may be loosely interpreted as a feature extractor).
At time t , the algorithm receives all the channel outputs (cid:101) Q ( j ) t for j = 1 . . .
L , and updates them in accordance with the observed transition.
The complete steps are presented in Algorithm 1.
A few points are apropos to remark.
First, the three steps of the for-loop can be computed in parallel for all the L channels (i.e. each of Q table or head).
Second, as mentioned previously, the mapping function f
j may be different for each channel; however, the discount factor γ and both learning rates β f,t and β reg,t are shared among all the channels and must keep the same.
Finally, note also that the action ˜ a t +1 , from which all the channels bootstrap, comes from arg max a (cid:48)
Q t ( s t +1 , a (cid:48) ) and not the local value of each channel.
This directly implies that each channel-level value Q ( j ) =
f − 1 ( (cid:101) Q j ) does not solve a channel-level Bellman equation by itself.
In other words, Q ( j ) does not represent any speciﬁc semantics such as expected return corresponding to the rewards of that channel.
They only become meaningful when they compose back together and rebuild the original value function.
4. C ONVERGENCE
Let Q t ( s t , a t ) be deﬁned by equation (12) and
all (cid:101) Q ( j ) t ( s t , a t ) are updated according to the steps of
Algorithm 1.
We establish convergence of Algorithm 1 by the following theorem.
Theorem 1 Let the reward admit a decomposition as deﬁned by (11), Q t ( s t , a t ) be deﬁned by (12), and all (cid:101) Q ( j ) t ( s t , a t ) are updated according to the steps of Algorithm 1.
Assume further that the following hold:
All f j ’s hold Assumptions 1 and2.
TD error in the regular space (the 2nd term of line 4 in Algorithm 1) is bounded for all j 3.
(cid:80) ∞ t =0 β f,t · β reg,t = ∞ 4.
(cid:80) ∞ t =
( β f,t · β reg,t ) 2 < ∞ 5.
β f,t · β reg,t → 0 as t → ∞
Then Q t ( s, a ) converges to Q ∗ t ( s, a ) with probability one for all state and actions ( s, a ) .
The proof follows basic results from stochastic approximation theory (Jaakkola et al., 1994; Singh et al., 2000) with important additional steps to show that those results hold under the assumptions of
Theorem 1.
The full proof is fairly technical and is presented in Appendix A.
We further remark that Theorem 1 only requires the product
β f,t · β reg,t to go to zero.
As this product resembles the conventional learning rate in Q-Learning , this assumption is no particular limitation compared to traditional algorithms.
We contrast this assumption with the one in the previous proof of Log Q-Learning , which requires β reg to go to zero fast enough.
We note that in the case of using function approximation, as in a DQN -like algorithm (Mnih et al., 2015), the update in line 5 of Algorithm 1 should naturally be managed by the used optimizer, while line 4 may be handled manually.
This has proved challenging as the convergence properties can be signiﬁcantly sensitive to the learning rates.
To get around this problem, van Seijen et al. (2019) decided to keep β reg,t at a ﬁxed value in their deep RL experiments, contrary to the theory.
Our new condition, however, formally allows β reg,t to be set to a constant value as long as β f,t properly decays to zero. A somehow hidden step in the previous proof of Log Q-Learning is also that the TD error in the regular space, the second term in (9), must always remain bounded; we will make this condition explicit.
In practice, with bounds of the reward being known, one can easily ﬁnd bounds of return in regular as well as f j spaces, and ensure boundness of U ( j ) t − f − 1 j ( (cid:101) Q ( j ) t ) by proper clipping.
Lastly, the scenarios covered by Assumption 2, with the new convergence proof due to Theorem 1, may be favorable in many practical cases.
Moreover, several prior algorithms such as Q-Learning ,
Log Q-Learning , and Q-Decomposition can be derived by proper construction from Algorithm 1.
4. R EMARKS T IME - DEPENDENT C HANNELS
The proof of Theorem 1 does not directly involve the channel weights λ j .
However, changing them will impact Q t , which changes the action ˜ a t +1 in line 2 of Algorithm 1.
In the case that λ j ’s vary with time, if they all converge to their ﬁnal ﬁxed value soon enough before the learning rates become too small, and if additionally all state-action pairs are still visited frequently enough after λ j ’s are settled to their ﬁnal value, then the algorithm should still converge to optimality.
Of course this analysis is far from formal; nevertheless, we can still strongly conjecture that an adaptive case where the channel weights vary with time should be possible to design.
S LOPE OF M APPINGS
Assumption 2 asserts that the derivative of f j must be bounded from both below and above.
While this condition is sufﬁcient for the proof of Theorem 1, we can probe its impact further.
The proof basically demonstrates a bounded error term, which ultimately converges to zero under the conditions of Theorem 1.
However, the bound on this error term (see Lemma 2 in Appendix A) is scaled by δ max = max j δ ( j ) , with δ ( j ) being deﬁned as
[Equation]
where δ ( j )and δ ( j )are deﬁned according to Assumption 2 ( 0 < δ ( j )≤ | f (cid:48)
j ( x ) | ≤ δ ( j ) 2 ).
In the case of f j being a straight line δ ( j ) = 0 ; hence, no error is incurred and the algorithm shrinks to Q-Learning .
An important extreme case is when δ ( j ) 1 is too small while δ ( j ) 2 is not close to δ ( j ) 1 .
It then follows from (15) that the error can be signiﬁcantly large and the algorithm may need a long time to converge.
This can also be examined by observing that if the return is near the areas where f (cid:48)
j is very small, the return may be too compressed when mapped.
Consequently, the agent becomes insensitive to the change of return in such areas.
This problem can be even more signiﬁcant in deep RL due to more complex optimization processes and non-linear approximations.
The bottom-line is that the mapping functions should be carefully selected in light of equation (15) to avoid extremely large errors while still having desired slopes to magnify or suppress the returns when needed.
This analysis also explains why logarithmic mappings of the form f ( x ) = c · log( x + d ) (as investigated in the context of Q-Learning by van Seijen et al.
present unfavorable results in dense reward scenarios; e.g. in the Atari 2600 game of Skiing where there is a reward at every step.
In this expression c is a mapping hyperparameter that scales values in the logarithmic space and d is a small positive scalar to ensure bounded derivatives, where the functional form of the derivative is given by f (cid:48)
( x ) = cx + d .
Hence, δ 2 = cd , whereas δ 1 can be very close to zero depending on the maximum return.
As a result, when learning on a task which often faces large returns, Log Q-Learning operates mostly on areas of f where the slope is small and, as such, it can incur signiﬁcant error compared to standard Q-Learning .
See Appendix B for a detailed illustration of this issue.
E XPERIMENTAL R ESULTS
In this section, we illustrate the simplicity and utility of instantiating new learning methods based on our theory.
Since our framework provides a very broad algorithm class with numerous possibilities, deep and meaningful investigations of speciﬁc instances go far beyond the scope of this paper (or any single conference paper).
Nevertheless, as an authentic illustration, we consider the LogDQN algorithm (van Seijen et al., 2019) and propose an altered mapping function.
As discussed above, the logarithmic mapping in LogDQN suffers from a too-small slope when encountering large returns.
We lift this undesirable property while keeping the desired magniﬁcation property around zero.
Speciﬁcally, we substitute the logarithmic mapping with a piecewise function that at the break-point x = 1 − d switches from a logarithmic mapping to a straight line with slope c (i.e. the same slope as c · log( x + d ) at x = 1 − d ):
[Equation]
We call the resulting method LogLinDQN , or Logarithmic-Linear DQN .
Remark that choosing x = 1 − d as the break-point has the beneﬁt of using only a single hyperparameter c to determine both the scaling of the logarithmic function and the slope of the linear function, which otherwise would require an additional hyperparameter.
Also note that the new mapping satisﬁes Assumptions 1 and 2.
We then use two reward channels for non-negative and negative rewards, as discussed in the ﬁrst example of Section 3 (see (13) for the details), and use the same mapping function for both channels.
Our implementation of LogLinDQN is based on Dopamine (Castro et al., 2018) and closely matches that of LogDQN , with the only difference being in the mapping function speciﬁcation. Notably, our
LogLin mapping hyperparameters are realized using the same values as those of LogDQN ; i.e. c = 0 . 5 and d ≈ 0 . We test this method in the Atari 2600 games of the Arcade Learning Environment ( ALE ) (Bellemare et al., 2013) and compare its performance primarily against LogDQN and DQN (Mnih et al., 2015), denoted by “ Lin ” or “( Lin ) DQN ” to highlight that it corresponds to a linear mapping function with slope one.
We also include two other major baselines for reference: C51 (Bellemare et al., 2017) and Rainbow (Hessel et al., 2018). Our tests are conducted on a stochastic version of Atari 2600 using sticky actions (Machado et al., 2018) and follow a uniﬁed evaluation protocol and codebase via the Dopamine framework (Castro et al., 2018).
Figure 1 shows the relative human-normalized score of LogLinDQN w.r.t. the worst and best of LogDQN and DQN for each game. These results suggest that LogLinDQN reasonably uniﬁes the good properties of linear and logarithmic mappings (i.e. handling dense or sparse reward distributions respectively), thereby enabling it to improve upon the per-game worst of LogDQN and DQN
[Figure]
a set of games.
Figure 2 shows median and mean human-normalized scores across the suite of Atari 2600 games. Our LogLinDQN agent demonstrates a signiﬁcant improvement over most baselines and is competitive with Rainbow in terms of mean performance.
This is somewhat remarkable provided the relative simplicity of LogLinDQN , especially, w.r.t. Rainbow which combines several advances including distributional learning, prioritized experience replay, and n-step learning.
D ISCUSSION
In this paper we introduced a convergent class of algorithms based on the composition of two orthogonal foundations: (1) mapping the value function into a different space via arbitrary functions from a broad class, and (2) linearly decomposing the reward signal into multiple channels.
Together, this new family of algorithms enables learning the value function in a collection of different spaces where the learning process can potentially be easier or more efﬁcient than the original return space.
Additionally, the introduced methodology incorporates various versions of ensemble learning in terms of linear decomposition of the reward.
We presented a generic proof, which also relaxes certain limitations of previous proofs.
We also remark that several known algorithms in classic and recent literature can be seen as special cases of the present algorithm class.
Finally, we contemplate the research on numerous special instantiations as future work, following our theoretical foundation.
R EPRODUCIBILITY S TATEMENT
We release a generic codebase — based on the Dopamine framework (Castro et al., 2018) — with the option of using arbitrary compositions of mapping functions and reward decomposition schemes as easy-to-code modules.
This would enable the community to easily explore the design space that our theory opens up and investigate new convergent families of algorithms.
This also enables to reproduce the results of this paper through an accompanying conﬁguration script.
The codebase will be made publicly available after the review phase
A P ROOF OF T HEOREM
We use a basic convergence result from stochastic approximation theory.
In particular, we invoke the following lemma, which has appeared and proved in various classic texts; see for example, Theorem 1 in Jaakkola et al.
(1994) or Lemma 1 in Singh et al.
Lemma 1 Consider an algorithm of the following form:
[Equation]
with x being the state variable (or vector of variables), and α t and U t denoting respectively the learning rate and the update at time t .
Then, ∆ t converges to zero with probability one as t −→ ∞ under the following assumptions:
The state space is ﬁnite; 2.
(cid:80) t α t = ∞ and (cid:80) t
α 2 t < ∞ ; 3.
|| E { U t ( x ) | F t }|| W ≤ ξ || ∆ t
x ) || W , with ξ ∈ (0 , 1) and || · || W denoting a weighted max norm; 4.
Var { U t ( x ) | F t } ≤ C (1 + || ∆ t ( x ) || W ) 2 , for some constant C .
where F t is the history of the algorithm until time t .
Remark that in applying Lemma 1, the ∆ t process generally represents the difference between a stochastic process of interest and its optimal value (that is Q t and Q ∗ t ), and x represents a proper concatenation of states and actions.
In particular, it has been shown that Lemma 1 applies to Q-Learning as the TD update of Q-Learning satisﬁes assumptions 3 and 4 (Jaakkola et al., 1994).
We deﬁne Q ( j ) t ( s, a ) :
= f − 1 (cid:16) (cid:101) Q ( j ) t ( s, a ) (cid:17) , for j = 1 . . .
L .
Hence,
Q t ( s, a ) =
L (cid:88)
j
[Equation]
We next establish the following key result, which is core to the proof of Theorem 1.
The proof is given in the next section.
Lemma
Following Algorithm 1, for each channel j ∈ { 1 , . . .
, L } we have
[Equation]
with the error term satisfying the followings:
Bounded by TD error in the regular space with decaying coefﬁcient:
[Equation]
where δ ( j ) =
δ ( j ) 2 δ ( j ) 1 − 1 is a positive constant.
For a given f j , e ( j ) t does not change sign for all t (it is either always non-positive or always non-negative).
e ( j ) t is fully measurable given the variables deﬁned at time t .
From Lemma 2, it follows that for each channel:
[Equation]
with e ( j ) t converging to zero w.p.1 under condition 4 of the theorem, and U ( j ) t deﬁned as:
[Equation]
Multiplying both sides of equation 20 by λ j and taking the summation, we write L
[Equation]
Hence, using equation 17 we have
[Equation]
Deﬁnition of ˜ a t +1 deduces that
[Equation]
Deﬁne e t = (cid:80)
Lj =
λ j e ( j ) t ; we rewire equation 21 as the following:
[Equation]
This is a noisy Q-Learning algorithm with the noise term decaying to zero with a quadratic rate with respect to the learning rate’s decay; more precisely, in the form of ( β reg,t · β f,t ) 2 .
Lemma 1 requires the entire update to be properly bounded (as stated in its assumptions 3 and 4).
It has been known from the proof of Q-Learning (Jaakkola et al., 1994) that TD error satisﬁes these conditions, meaning that r t + γ max a (cid:48)
Q t ( s t +1 , a (cid:48) )
− Q t ( s t , a t ) satisﬁes assumptions 3 and 4 of Lemma 1.
To prove convergence of mapped Q-Learning , we therefore require to show that | e t | also satisﬁes similar property; hence, not only in the limit it disappears, but also in the course of training, it does not interfere intractably with the learning process.
To this end, we next show that as the learning continues, | e t | is indeed bounded by a value that can be arbitrary smaller than the TD error.
Consequently, as TD error satisﬁes assumptions 3 and 4 of Lemma 1, so does | e t | , and so does sum of the two.
Let δ max = max j δ ( j ) , with δ ( j ) deﬁned in Lemma 2.
Multiplying both sides of (19) by λ j and taking the summation over j , it yields:
[Equation]
The second line follows from Lemma 2.
If TD error in the regular space is bounded, then (cid:12)(cid:12)(cid:12) r ( j ) t + γ Q ( j ) t
( s t +1 , ˜ a t +1 ) − Q ( j ) t ( s t , a t ) (cid:12)(cid:12)(cid:12) ≤ K ( j ) for some K ( j ) ≥ 0 .
Hence, (23) induces:
[Equation]
with K = (cid:80)
Lj =
λ j K ( j ) ≥ 0 .
As (by assumption) β reg,t · β f,t converges to zero, we conclude that there exists T ≥ 0 such that for all t ≥ T we have
[Equation]
for any given ξ ∈ (0 , 1] .
Hence, not only | e t | goes to zero w.p.1 as t −→ ∞ , but also its magnitude always remains upperbounded below the size of TD update with any arbitrary margin ξ .
Since TD update already satisﬁes assumptions 3 and 4 of Lemma 1, we conclude that with the presence of e t those assumptions remain satisﬁed, at least after reaching some time T where (25) with a sufﬁciently small ξ (cid:48) is guaranteed.
Finally, Lemma 2 also asserts that e t is measurable given information at time t , as required by
Lemma 1.
Invoking Lemma 1, we can now conclude that the iterative process deﬁned by Algorithmconverges to Q ∗ t with probability one.
P ROOF OF L EMMAP ART
Our proof partially builds upon the proof presented in (van Seijen et al., 2019).
By deﬁnition we have (cid:101) Q ( j ) t ( s, a ) =
f
(cid:16) Q ( j ) t ( s, a ) (cid:17) .
Hence, we rewrite equations 3, 4, and 5 of Algorithm 1 in terms of Q ( j ) t :
[Equation]
The ﬁrst two equations yield
[Equation]
By applying f − 1 to both sides of (28)
[Equation]
we get:
[Equation]
which can be rewritten as:
[Equation]
where e ( j ) t is the error due to averaging in the mapping space instead of in the regular space:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
We next analyze the behavior of e ( j ) t under the Theorem 1’s assumptions.
To simplify, let us introduce the following substitutions:
[Equation]
The error e ( j ) t can be written as
[Equation]
We remark that both v and w lie between a and b .
Notably e ( j ) t has a particular structure, which we can use to bound w − v .
See Table 1 for the ordering of v and w for different possibilities of f .
We deﬁne three lines g 0 ( x ) , g 1 ( x ) , and g 2 ( x ) such that they all pass through the point ( a, f ( a )) .
As for their slopes, g 0 ( x ) has the derivative f (cid:48) ( a ) , and g 2 ( x ) has the derivative f (cid:48)
( b ) .
The function g 1 ( x ) passes through point ( b, f ( b )) as well, giving it derivative ( f ( a ) − f ( b )) /
( a − b ) .
See Fig.
3 for all the possible cases.
We can see that no matter if f is semi-convex, or semi-concave, and if it is increasing or decreasing these three lines will sandwich f over the interval [ a, b ] if b ≥ a , or similarly over [ b, a ] if a ≥ b .
Additionally, it is easy to prove that for all x in the interval of a and b , either of the following holds:
[Equation]
or
[Equation]
The ﬁrst one is equivalent with
[Equation]
35) while the second one is equivalent with
[Equation]
From the deﬁnition of g 1 it follow that in all the mentioned possibilities of f combined with either of a ≥ b or b ≥ a , we always have g 1 ( v ) =
˜ w and g − 1 1 ( ˜ w ) = v .
Hence, plugging ˜ w in equationand equation 36 (and noting that f − 1 ( ˜ w ) = w ) deduces
[Equation]
37) or
[Equation]
Either way, regardless of various possibilities for f as well as a and b , we conclude that
[Equation]
[Figure]
From deﬁnition of the lines g
and g 2 , we write the line equations as follows
[Equation]
Applying these equations on the points ( g − 1 0 ( ˜ w ) , ˜ w ) and ( g − 1 2 ( ˜ w ) , ˜ w ) [Equation] respectively
[Equation]
it yields
[Equation]
which deduce
[Equation]
Plugging the above in equation
[Equation]
it follows
[Equation]
We next invoke the mean value theorem , which states that if f is a continuous function on the closed interval [ a, b ] and differentiable on the open interval ( a, b ) , then there exists a point c ∈ ( a, b ) such that f ( b ) − f
( a ) =
f (cid:48) ( c )( b − a ) .
Remark that based on Assumption 2, c would satisfy f (cid:48)
j ( c ) ≤ δ ( j ) 2 , also that 1 f (cid:48) ( b ) − 1 f (cid:48) ( a ) ≤ 1 δ ( j ) 1 − 1 δ ( j ) 2 .
Hence,
[Equation]
From equation
[Equation]
it follows that
[Equation]
We therefore can write
[Equation]
where δ ( j ) =
δ ( j ) 2 ( 1 δ ( j )− 1 δ ( j ) 2 ) is a positive constant. This completes the proof of part 1 of the Lemma. P ART
For part 2, it can be directly seen from Fig.
3 that for a given f j , the order of w (cid:48) , w , v , and v (cid:48) is ﬁxed, regardless of whether a ≥ b or b ≥ a (compare each ﬁgure A, B, C, and D, with their counterparts at the bottom).
Hence, the sign of e ( j ) t = w − v will not change for a ﬁxed mapping. P ART
Finally, we note that by its deﬁnition, e ( j ) t comprises quantities that are all deﬁned at time t .
Hence it is fully measurable at time t , and this completes the proof.
B D ISCUSSION ON L OG VERSUS L OG L IN
As discussed in Sec. 4.3 (Slope of Mappings), the logarithmic ( Log ) function suffers from a too-low slope when the return is even moderately large. Figure 4 visualizes the impact more vividly. The logarithmic-linear ( LogLin ) function lifts this disadvantage by switching to a linear ( Lin ) function for such returns.
For example, if the return changes by a unit of reward from 19 to 20 , then the change will be seen as 0 .
05 in the Log space (i.e. log(20) − log(19) ) versus 1 . 0 in the LogLin space; that is, Log compresses the change by 95% for a return of around 20 .
As, in general, learning subtle changes is more difﬁcult and requires more training iterations, in such scenarios normal DQN (i.e. Lin function) is expected to outperform LogDQN .
On the other hand, when the return is small (such as in sparse reward tasks), LogDQN is expected to outperform DQN .
Since LogLin exposes the best of the two worlds of logarithmic and linear spaces (when the return lies in the respective regions), we should expect it to work best if it is to be used as a generic mapping for various games.
C E XPERIMENTAL D ETAILS
The human-normalized scores reported in our Atari 2600 experiments are given by the formula (similarly to van Hasselt et al.
[Equation]
where score agent , score human , and score random are the per-game scores (undiscounted returns) for the given agent, a reference human player, and random agent baseline.
We use Table 2 from Wang et al.
(2016) to retrieve the human player and random agent scores.
The relative human-normalized score of LogLinDQN versus a baseline in each game is given by the formula (similarly to Wang et al.
[Equation]
[Figure] where score LogLinDQN and score baseline are computed by averaging over the last 10% of each learning curve (i.e. last 20 iterations).
The reported results are based on three independent trials for LogLinDQN and LogDQN , and ﬁve independent trials for DQN .
D A DDITIONAL R ESULTS
Figure 5 shows the relative human-normalized score of LogLinDQN versus LogDQN (top panel) and versus ( Lin ) DQN (bottom panel) for each game, across the suite of 55 Atari 2600 games.
LogLinDQN outperforms both LogDQN and ( Lin ) DQN on several games signiﬁcantly, and is otherwise on par with them (i.e. when LogLinDQN is outperformed by either of LogDQN or
( Lin ) DQN , the difference is not by a large margin).
Figure 6 shows the raw (i.e. without human-normalization) learning curves across the suite of Atari 2600 games.