Adaptive Interest for Emphatic Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Emphatic algorithms have shown great promise in stabilizing and improving reinforcement learning by selectively emphasizing the update rule.
Although theemphasis fundamentally depends on an interest function which deﬁnes the intrinsicimportance of each state, most approaches simply adopt a uniform interest overall states (except where a hand-designed interest is possible based on domainknowledge).
In this paper, we investigate adaptive methods that allow the interestfunction to dynamically vary over states and iterations.
In particular, we leveragemeta-gradients to automatically discover online an interest function that wouldaccelerate the agent’s learning process.
Empirical evaluations on a wide range ofenvironments show that adapting the interest is key to provide signiﬁcant gains.
Qualitative analysis indicates that the learned interest function emphasizes statesof particular importance, such as bottlenecks, which can be especially useful in atransfer learning setting
Introduction
A fundamental challenge in reinforcement learning (RL) is to approximate key quantities such asvalue functions and optimal policies.
Under the assumption that the world in which an RL agentinteracts is large and the computational capacity is limited, a natural trade-off emerges in whichcertain quantities are more accurately predicted than others over the course of learning.
Standard
RL algorithms, such as temporal differences (TD) [39], perform updates at every state, therebyspending more resources on such frequent states at the expense of other potentially more usefulones.
A possible solution could be to selectively emphasize certain updates, for example through astate-dependent interest function .
However, when combined with standard bootstrapping as in TD(  ),such update rules are known to be unstable [25].
Emphatic algorithms propose a solution in which state-dependent selective updating can be appliedwhile maintaining stability under linear function approximation [44].
At their core, emphaticalgorithms determine the emphasis to be applied at each update by accounting for how much thecurrent state is being bootstrapped from as well as an intrinsic measure of its importance relative toother states.
This intrinsic measure is encoded through an arbitrary state-dependent interest functionwhich the practitioner can set to any desired positive value.
However, with this added ﬂexibilitycomes a problematic question: how should one select the interest function?
Previous works propose hand-crafted solutions for the interest function that showcase the usefulness ofselective updating [27, 3, 28].
However, when applying emphatic algorithms to complex environmentswhere external domain knowledge may be too hard to encode, other than rare and speciﬁc exceptions[59], most practitioners use a uniform interest over states [44, 53, 20, 60, 21, 17].
That is, they simplyset the interest to 1 for all states.
Building on the intuition that it can be beneﬁcial to learn morefrom certain states than others, we argue that different emphases may be useful at various stages of the learning process.
Indeed, as the RL learning process is inherently non-stationary, the relativeimportance of a particular state in the agent’s updates should likely vary over training iterates as well.
In this work, we study how to adaptively learn the interest function in complex environments wherehand-crafting an effective interest function is impractical.
A good approach should allow for fastand ﬂexible adaptation based on the agent’s interactions with its environment.
Considering theprevious success of meta-gradient framework in discovering hyperparameters [55, 58], objectivefunctions [54], intrinsic rewards [62], and temporal abstractions [51], we here propose to learn andadapt the interest function based on meta-gradients in an online fashion.
The interest function in ourmethod is parameterized by meta-parameters , which are updated by gradient descent along with theparameters of the policy and value function.
We empirically investigate the merits of adapting the interest function on a wide variety of environments and settings, ranging from prediction with linear function approximation to control onvision-based tasks.
Our contributions are the following.
In the off-policy setting, we see substantial gains in performance and sample efﬁciency when adapting the interest function.
(2) We extend thetraditional application of emphatic algorithms from the off-policy setting to on-policy control, wherewe ﬁnd it is crucial to adapt the interest function in order to observe consistent gains.
(3) Qualitatively,our learned interest function appears to naturally discover states of importance, such as bottlenecks[38].
Such discovery is demonstrated to be very useful in transfer learning experiments.
Our resultshighlight the general applicability of emphatic algorithms beyond the off-policy single-task settingconsidered in most previous studies of emphatic RL.
Background
We assume a Markov Decision Process M , deﬁned as a tuple h S , A , r, P i with a ﬁnite state space S , a 58 ﬁnite action space A , a transition probability distribution P : S ⇥
A ⇥ S ! [0 , 1] , and a scalar rewardfunction r ( s, a ) depending on action a 2 A in state s 2 S .
The policy ⇡ : A ⇥ S ! [0 , 1] speciﬁes 60 the agent’s behaviour and its expected discounted return starting from any state is represented asthe value function: V ⇡ ( s ) = E ⇡ ⇥P 1 i =
t  i  t R i +
S t =
s ⇤ , where  2 [0 , 1) is the discount factorand R t +1 is the sampled reward after performing action A t in state S t .
Under linear functionapproximation, the value function is deﬁned with parameters ✓ 2 R n and features  ( s ) 2 R n , thatis ˆ V ⇡ ( s ; ✓ ) =
✓ >  ( s ) 1 .
An efﬁcient family of algorithms for learning such functions builds on the
Temporal Difference (TD) algorithm [39] where the value parameters, ✓ , are updated as follows:
[Equation]
with ↵ denoting the step size.
In the control setting, the policy gradient theorem [45] for the episodiccase provides the gradient of the expected discounted return from an initial state distribution d ( s 0 )with respect to a stochastic policy ⇡ ( · | s ; ⌫ ) now parameterized by ⌫ :
[Equation]
where d ⇡ ( s ) =
P s 0 d ( [Equation]
P 1 t =
t P ⇡ ( S t = s | S
= s 0 ) is the discounted state occupancy measureof the target policy ⇡ and Q ⇡ ( s, a ) =
E ⇡ ⇥ P [Equation] i [Equation]
i  t R i +
S t = s, A t = a ⇤ is the state-action valuefunction.
For a more detailed presentation of the notation please refer to App.
C.
2.1 Emphatic Algorithms
Emphatic algorithms [44, 27] provide a way to emphasize and de-emphasize the updates made ateach iteration while preserving convergence.
Their development was motivated by the challengesthat arise under off-policy learning when using function approximation and bootstrapping [50, 41].
In the off-policy setting, a behavior policy b ( a | s ) generates the data to learn value functions orpolicies evaluated under the target policy ⇡ ( a | s ) .
Emphatic algorithms generalize TD in variousways, however of particular interest to our work is the added ﬂexibility of arbitrarily deﬁning theintrinsic importance of each state through the interest function.
In the following we present emphaticalgorithms in the general off-policy setting, as learning on-policy is a special case.
Policy Evaluation:
In its simplest one-step bootstrapping form, the Emphatic Temporal Difference(ETD) update rule for the value parameters ✓ takes the following form
[Equation]
where ⇢ t = ⇡
( A t | S t ) b ( A t | S t ) is the importance sampling ratio at time t and F t , the followon trace, is deﬁnedas,
[Equation]
where i ( · ) : S ! R + is the arbitrary user-deﬁned interest function.
The speciﬁc form of this trace 86 depends in part on the interest function, but also on how much a state is bootstrapped from by previousstates, discounted over time.
This speciﬁc form is what confers stability and convergence to ETD[57], without introducing the full product of importance ratios used for prior correction [30].
Control:
In the actor-critic setting, [10] proposed to maximize the excursions objective J b ( ⌫ ) =
P s d b ( s ) V ⇡ ( s ) where d b ( s ) is the stationary distribution of the policy b .
We explain in App.
C thereason why the stationary distribution appears instead of the discounted state occupancy measure of(2).
They proposed a way to approximate the policy gradient, where such approximation is only validin the tabular case.
[20] later derived the correct gradient for the more general objective that nowincludes the state dependent interest function,
[Equation]
where the correct stochastic gradient update for the policy parameters ⌫ takes the following form,
[Equation]
Interestingly, the same trace F t from the off-policy policy evaluation setting appears in the off-policycontrol setting.
Note that ⇢ t would be equal to 1 in the on-policy setting where target policy andbehavior policy are the same.
Adaptive Interest
[Figure]
The interest function was designed as a way to emphasize some states more than others and as suchcan be an efﬁcient way to encode useful inductive bias.
Although it may be possible to ﬁnd an interestfunction that is effective for a speciﬁc and simple case (e.g. when additional knowledge about thetask is readily available), it is not convenient to hand-design interest functions that effectively workfor complex domains.
For this reason, most previous works on emphatic algorithms consider a simpleuniform interest over states [44, 21].
Furthermore, we hypothesize that the usefulness of a particular interest function can vary through thelearning process itself.
This is obvious in the case of a changing environment, for example in continuallearning, but is also relevant in the single task setting where the agent’s policy or bootstrapping targetsvary in a non-stationary manner.
In the next section, we further motivate the advantage of an adaptiveinterest function through the example of a simple chain MDP.
3.1 Motivating Example
In Fig.
1, we consider the case of off-policy control in a simple chain MDP made of four non-terminalstates.
Here our agent uses one-step SARSA [32] to learn the action-value function (Q-value).
Theagent starts in state S
and can reach either the terminal state on the right with reward of 1 or themore distant terminal state on the left with reward of 100 .
In our example, suppose the behaviorpolicy is biased towards going right in the three rightmost states.
The resulting target policy learnedvia SARSA without any emphatic weighting is misguided toward a suboptimal solution (Fig. 1).
App.
B describes additional details of this experiment.
When designing a ﬁxed interest function for SARSA with emphatic weighting, it would be advantageous to emphasize the states on the left and de-emphasize the states on the right, as a way to try andavoid the sub-optimal solution.
However, Fig. 1 shows that although a well-designed ﬁxed interestfunction can improve upon the this baseline, the resulting emphatic SARSA is still unable to convergeto a good policy within 500 updates.
Finally we consider using an adaptive interest function.
Here we leverage the same pattern asin the previous ﬁxed interest function, but we only activate the interest in some states at certain times.
Particularly, at the start of training, only the left-most state is emphasized, and allother states have interest set to near-zero.
As credit is propagated from the terminal left statetowards the rest of the chain, the interest of the second left-most state is increased.
This continues until credit assignment reaches the starting state and the optimal action is selected.
Emphatic
SARSA with such a dynamic interest function is able to quickly converge to a good policy (Fig. 1).
Fig.
1 empirically shows that, in our example MDP,a standard SARSA agent (without emphatic weighting) is outperformed by emphatic SARSA with aﬁxed interest (supporting the general utility of emphatic algorithms), which is in turn outperformedby emphatic SARSA with an adaptive interest (supporting the additional utility of adaptive interest).
This example highlights that even in tabular settingswithout function approximation, the additional ﬂexibility of an adaptive interest function can be quitebeneﬁcial.
It is important to note that the ﬁxed interest agent eventually ﬁnds the right solution in ourexample, but it has much worse sample complexitycompared to our adaptive interest agent.
3.2 Meta-Gradient Interest (MINT)
As discussed in the previous section, we seek anadaptive interest function that would improve thelearning process during training.
However, automatically discovering such a function is not straightforward, in part as it can take any arbitrary value, andbecause we have to evaluate the effect of a particularchange in the interest function with regards to theagent’s parameters.
We explore adapting the interestfunction with a wide variety of heuristics such asthe prediction error and ﬁnd that such approaches do not generally provide improvements.
Alternatively, meta-gradients [55, 63] are a natural candidate as they can automatically discover suchinterest functions at each stage of learning through the interaction of an inner loop and an outerloop of optimization.
We therefore propose to learn and adapt the interest function parameterized bymeta-parameters ⌘ in an online manner, within a single lifetime and within a single environment.
During the inner loop, the meta-parameters ⌘ , together with the agent’s policy and value parameters{ ⌫ , ✓ } , appear in the base objective, J B ( ✓ , ⌫ , ⌘ ) .
Only the parameters are updated through thisobjective while the meta-parameters ⌘ remain ﬁxed and inﬂuence the gradients.
To illustrate theinﬂuence of the meta-parameters on the resulting parameters, we can write them as functions of ⌘ , i.e.{ ⌫ 0 ( ⌘ ) , ✓ 0 ( ⌘ ) } .
During the outer loop, the updated parameters are evaluated with respect to a meta-objective,
J M ( ⌫ 0 ( ⌘ ) , ✓ 0 ( ⌘ )) , from which we derive the gradients with respect to ⌘ .
This is referred to asthe meta-gradient , which evaluates how the values of the meta-parameters affected the performanceof the updated parameters.
By repeating this process, meta-gradients will adapt the meta-parametersin order to more efﬁciently improve the parameters themselves.
We now describe the speciﬁc choices behind applying meta-gradients to emphatic algorithms whenupdating the policy.
For simplicity, the derivation for the value function is relegated to App.
H.3.
In the inner loop, the agent maximizes the following inner objective,
[Equation]
where the interest function i is parameterized by the meta-parameters ⌘ and the policy ⇡ is parameterized by ⌫ .
This inner objective is based on the excursions objective [20], that is, the future reward achieved by following the target policy ⇡ starting from the distribution of states generated by thebehavior b .
Another possibility would have been to consider the counterfactual objective [59] or thealternative life objective [29], however these choices imply additional complexities which we leavefor future work.
We provide a more detailed discussion on the choice of the objective function in the
App.
H.1.
The inner loop can be written by using (5), obtaining:
[Equation]
where F t, ⌘ = i ( S t ; ⌘ ) + ⇢ t  1 F t  1 , ⌘ emphasizes the current state according to the current metainterest and the followon trace at the previous timestep.
When considering the meta-objective, practitioners usually employ the same form as the innerobjective.
In our case this would be written as
[Equation]
where ⇡ is deﬁned by the updated parameters ⌫ 0 ( ⌘ ) .
Recently, [14] argue that such an approachmay lead to a poor meta-optimisation landscape, as both objectives share the same curvature.
In
App.
D, we verify different meta-objectives, such as the variance of the reward-to-go [47], andreport no increase in performance when compared to (8) in our setting.
It is likely that obtainingtheir improvements also relies tackling myopia in meta-gradients.
From (8) we obtain the followingmeta-gradient
[Equation]
(9) where r ⌘ ⌫ 0 encodes how the meta-parameters affected the new parameters.
A stochastic sample ofthis quantity at time t can be expanded as (see also App. H.2),
[Equation]
where ⇢ i : t is a product of importance sampling ratios.
Pseudocode for our approach, which we call
MINT ( M eta-gradient Int erest), is presented in Algorithm 1 of App. A.
Performing the update rule in (9) would require a new set of samples.
In practice, the same samplesare re-used for both loops [63, 62, 51] through a sliding window of experience.
In our work we opt touse the importance sampling ratio method of [63].
Finally in the present derivation we only consider1-step meta-gradient [52] as it greatly simpliﬁes the exposition ( see App.
H.4 for derivations).
Wealso show in App.
H.4 why the sampling correction term [2] does not appear in the off-policy setting.
Experiments
We now validate our method on a wide range of scenarios to assess the following questions: 1) Canwe automatically learn an interest function on complex environment in order to improve performance?2) How robust is the meta interest with respect to the agent’s hyperparameters?
Is it possible toleverage the information encoded by the learned interest function for downstream tasks?
We ﬁrst conduct experiments in the off-policy policy evaluation setting under linear function approximation.
Next, we extend the usual ﬁeld of study of emphatic algorithms and verify their generalutility.
In particular, we study how they can improve the performance of on-policy algorithms underthe control setting, where considerable gains are witnessed only under an adaptive interest.
Finally,we further extend our investigation to the transfer learning setting by leveraging the interest functionlearned in a previous task in order to greatly speed up the learning process in a second task.
Allhyperparameters are available in the App.
E.
4.1 Linear Function Approximation
Setup.
Our ﬁrst experiments are done in the off-policy policy evaluation setting with linear functionapproximation using tile coding [41].
We adopt the setup from [16], who considered two variationsof the classical Four Rooms domain, shown in Fig.
We name these variations 4Rooms-8Tasks(4R8T) and HighVariance-4Rooms-8Tasks (H4R8T) to
[Figure]
Quantitative Results.
Fig.
2 shows the results for the 4R8T (top) and H4R8T (bottom row).
The yaxis indicates the mean squared value error (MSVE) averaged across all policies.
In Fig.
2c and 2f, wevary the learning rate on the x-axis and for each value report each algorithm’s best ﬁnal performancechosen across all values of the bootstrapping coefﬁcient  (and other possible hyperparameters).
Fig. 2b and 2e show the best learning curves for each algorithm.
In these two domains, we compare learning the interest through meta gradients, MINT, to a baselinethat adapts the interest with respect to the absolute value of the TD error (ETDLB - TD Error Interest,where ETDLB refers to the generalized version of emphatic TD [19]).
We compare these adaptivemethods to the standard emphatic baseline ETDLB.
Finally, we also compare to the recent TD with
Regularized Corrections (TDRC) algorithm [15], which follows the line of work on Gradient TD [43],as well as the V-trace [11], which is representative of the performance of methods that use truncatedimportance sampling ratios [26].
We notice that across both tasks as shown in Fig.
2e, leveragingan adaptive interest leads to a better ﬁnal value error and especially on H4R8T, our method almosthalves the error when compared to non-emphatic methods.
In App.
D we present additional ﬁgures that take into account a different metric: the area under thecurve (instead of the ﬁnal performance) and notice a similar pattern.
When looking at the learningcurves on the left, we notice that ETDLB pays a price in terms of slower convergence in order toachieve a better ﬁnal performance.
However, when using an adaptive interest, the difference withnon-emphatic methods is greatly reduced, especially in the 4R8T domain.
Examining closely Fig.
2c and 2f, we notice that leveraging an adaptive interest moves the bottom ofthe U-shaped curve to the right.
By selectively emphasizing some states, emphatic algorithms usingan adaptive interest are able to learn on a higher learning rate.
However, the shape of the U curvetends to cut drastically after a certain threshold, at which point the updates become unstable.
When we compare adapting the interest through meta gradients to the one deﬁned as the absolutevalue of the TD error, we notice that their performance is almost equal.
It can perhaps seem surprisingthat this would be case, as the meta gradients method can in theory learn any function.
For ourparticular choice of objectives, we show in App.
H.3 the form of the meta gradients under linearfunction approximation.
For simplicity, if we further assume that features are tabular and we areunder the on-policy setting, we get that the stochastic sample at time t of the gradient is,
[Equation]
(10) where e t indicates the one-hot vector represent the current state at time t .
(10) shows that themeta gradient updates in the direction of the squared TD error.
Although the convergence of the meta-parameters will not be to the sampled squared TD error, at each iteration the meta-parametersare affected in a similar way when compared to the absolute value of the TD error heuristic.
Inpractice, the features are not exactly tabular and therefore the updates made on one state may affectanother, which would also explain why MINT is slower to converge.
[Figure]
Qualitative Results.
In Fig.
3, we inspect the learnedinterest functions obtained by MINT on four of the eighttasks (one per room).
The top row shows the learning process for 4R8T, while the bottom row shows the process for
H4R8T.
In both domains, the ﬁrst state to be highlighted isthe one next to the goal in the hallway.
Indeed this state ishighly inﬂuential since all states bootstrap from it, directlyor indirectly.
As training progresses, the interest in the 4R8T diffuses toneighbouring states (this bears a close resemblance to theexample presented in Fig. 1 ).
In the H4R8T, a differentpattern emerges where the state with high variance is beinghighlighted early on in training.
As this state is visited bymany trajectories that the target policy would take, it inﬂuences the values of many other states that need to bootstrapfrom it.
However, since it exhibits higher variance thanneighbouring states, it requires more computation to becorrectly estimated.
At the end of training, we observe that some states areparticularly less important, like state near the oppositecorners of the hallways.
As the target policy does not visit them often and not many states bootstrapfrom them, it is less important for them to be accurate.
Moreover, it is interesting to note that thediffusion of interest observed in the 4R8T domain is not perfectly uniform.
Since tile coding [41]is used as the function approximator, a speciﬁc pattern of the interest function may be needed atdifferent states to lower the overall prediction error.
4.2 Experiments at Scale
Emphatic algorithms were initially derived for the off-policy setting.
However, the ﬂexibility given bythe interest function is generally applicable, even in the on-policy case.
To showcase this ﬂexibility,in this section we investigate the performance of emphatic algorithms in the on-policy setting usingnon-linear function approximation.
4.2.1 MinAtar
Setup.
We verify the generality of the proposed method by considering the MinAtar domain [56],which is a miniaturized version of some of the games from the classic Atari 2600 testbed [6].
The environment provides 10 ⇥ 10 ⇥ n state representations, where n varies for each game.
Theenvironments are implemented using sticky actions and randomization [24].
For all games we use 10random seeds and report the mean and standard deviation after 10M timesteps.
Results.
Fig.
4 shows that MINT provides good gains when compared to the two baselines.
Astandard PPO agent [37] and a meta-gradient approach [54] that meta-learns the target function whichappears in RL update rules.
We explain in detail this baseline in the App.
F and theoretically showthat learning the interest function is not simply special case of their approach.
Results in Fig 4 clearlydemonstrate that our method outperforms both baselines.
Our ﬁndings also agree with the experimentin [54] (Fig 3a in their paper) that meta-learning the target function requires many more samplesbefore it can match the baseline’s performance.
[54] also compares to an approach that meta-learnsthe complete loss function (which could recover our update rule) and ﬁnd that the agent is not able tolearn in the online, single-lifetime setting (which is our setting).
These results highlight the difﬁculties of more general meta-learning formulations and their impacton sample efﬁciency.
This suggests that meta-learning the interest-function may be a good trade-offbetween generality and the amount of inductive bias.
We additionally present results in Fig.
10 where [Figure] we vary the learning rate and present at the U shaped curves of performance, which seem to behavesimilarly to the linear function approximation case.
4.2.2 Continuous Control
Setup.
We perform experiments on the MuJoCo domain [49, 7], where states and actions arecontinuous.
We report the mean and standard error averaged across 10 random seeds.
We includeseveral emphatic baselines where we explore using a ﬁxed interest and various heuristics for adaptingthe interest function.
Additionally, we investigate the usefulness of adapting the learning rate itselfusing hypergradient descent [5].
We provide a description of all the baselines in App.
E.
Results.
As Fig.
5 shows, utilizing adaptive interest function is the key to get consistent improvementover PPO (across almost all environments).
Interestingly, the adaptive heuristic based on the TD errorthat worked well in the prediction setting does not generalize to this one.
One way to understand thisis to consider that a low TD error may not be indicative of a high performing policy in control.
We also compare our method to hypergradient descent (HD) [5] which dynamically updates thelearning rate during training.
We notice that HD does not seem to provide gains, except in Humanoidv3 where it reaches the performance of MINT at the cost of a slower learning process.
This is incontrast to our method which generally does not suffer from increased sample complexity to achievebetter performance.
We highlight that an important difference between HD and MINT is that theinterest function is a state-dependent quantity, which can provide additional ﬂexibility.
In the App.
Ewe compare to additional baselines, such as meta-learning the reward function [63] and variousadaptive heuristics for the interest function.
4.3 Transfer Learning across
RL Tasks
Setup.
As the interest function is automatically learned, we observe that it likely encodes knowledgeand information that can be useful later.
In particular, this knowledge may serve to speed uplearning in new environment over learning from scratch.
We investigate this hypothesis in the
FourRoomsTransfer environment [9], shown in Fig.
6a where we show mean and standard deviationacross 30 seeds.
The agent starts in the top left corner and has to get to the goal location in green.
The state highlighted in orange is a distractor state that provides a random reward, R ⇠ N (0 , 1) .
For the transfer setting, we change the location of each of these entities (see App. G).
In this setting,the agent learns in the ﬁrst environment and only transfers the interest function to the second one,which remains ﬁxed thereafter.
We also compare to an agent that uses meta gradients to learn anintrinsic reward function before transferring it to the second task, similarly to [62].
[Figure]
Results.
Fig.
6 shows that the learned interest function provides a signiﬁcant speed-up whencompared to the actor-critic baseline that learns from scratch without an interest function.
[Figure]
Here, transferring the intrinsic rewarddoes not seem to help.
Note we do notclaim that using meta-gradients to learnan intrinsic function is always a betterchoice than to learn an intrinsic rewardfunction, as it likely depends on the exact transfer learning setup 2 .
To understand how the interest functionhelps in transfer, we present in Fig.
6band Fig.
6c the learned interest functionand the learned reward function.
We notice that the interest function highlightsstates that are near the goal, but also thehallways of the starting room (top left),which are usually referred to as bottleneck states [38].
Such states are of particular importance as they inﬂuence the trajectory an agenttakes as well as many of the predictions it makes during such trajectory.
As we notice, the interestfunction highlights the hallway leading to the goal, but also the one leading to the distractor.
On theopposite, the reward function naturally highlights the path to the goal and de-emphasizes the oneleading to the distractor state.
This illustrates one useful property of the interest function: it highlightsthe location of rewards, whether they are positive or negative.
This kind of invariance is key for abetter transfer performance in our setting, and could be used more generally in continual learning[31].
This experiment also points to an interesting future direction where a universal interest functioncould be deﬁned similarly to universal value functions [33].
Related Work
Emphatic Methods .
Initially derived by [44, 27] as a stable and simple one-time-step solution to theproblem of off-policy prediction.
Its convergence is shown in [57] when employing the full traceand later [61] show convergence for the Truncated ETD algorithm.
The ideas in prediction wereextended to control by [20, 18, 60].
Emphatic algorithms have been shown to be a strong baseline inmany benchmarks under linear function approximation [16, 17], even in the on-policy case [53, 3].
Recently, emphatic algorithms have been extended to the deep RL by building on a variant of the IMPALA agent [11] with auxiliary heads and have shown superior performance on Atari [21, 22].
Selective Updating . Emphatic methods can be seen more generally as performing selective updating,whereby through a scalar we emphasize or de-emphasize the updates to each state. This has previouslybeen investigated in the context of model free learning [34] and model based planning [1]. Recently,[8] provide a unifying view of various algorithms as a form of selective credit assignment.
Inhierarchical reinforcement learning, selective updating in an integral part of the options frameworkeither through initiation sets [46] or interest functions [23].
Meta Gradients.
Meta learning [35, 48, 13] is a class of methods that have better capability inadapting to new tasks by learning a better prior from previously seen related tasks in the past [36, 4].
While these methods mainly focus on multi-task learning [12], meta-gradients [62, 63] based methodsinstead focus on learning the meta parameters online within a single task, based on online crossvalidation [40].
Conclusion
We propose to learn and adapt the interest function based on meta-gradients in an online fashion incomplex environments where hand-coded solutions are not feasible.
Comprehensive experimentson various settings suggest that automatically adapting the interest function from a stream of dataleads to improved performance.
Although certain heuristics for adapting the interest function areoccasionally beneﬁcial, our experiments point that consistency and general usefulness are achievedthrough meta-gradients.
, we do not consider using recurrent neural networks or a lifetime value function here, which were found critical for meta-learned intrinsic rewards to work well in transfer [62].
Also, the interest/reward functions are trained on a single environment at ﬁrst, before being transferred to a new one, without further training.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[No] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please
do
not
modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We provide discussion in Sec-
tion 3.2 and App.
H
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material [No] We are currently working to
clean up the code and release it.
We will make it public before the rebuttal.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] The code also will be released upon publication.
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
b) Did you mention the license of the assets?
[N/A]
) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]