Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
Abstract
Designing agents that acquire knowledge autonomously and use it to solve new tasks efﬁciently is an important challenge in reinforcement learning.
Knowledge acquired during an unsupervised pre-training phase is often transferred by ﬁne-tuning neural network weights once rewards are exposed, as is common practice in supervised domains.
Given the nature of the reinforcement learning problem, we argue that standard ﬁne-tuning strategies alone are not enough for efﬁcient transfer in challenging domains.
We introduce Behavior Transfer (BT), a technique that leverages pre-trained policies for exploration and that is complementary to transferring neural network weights.
Our experiments show that, when combined with large-scale pre-training in the absence of rewards, existing intrinsic motivation objectives can lead to the emergence of complex behaviors.
These pre-trained policies can then be leveraged by BT to discover better solutions than without pre-training, and combining BT with standard ﬁne-tuning strategies results in additional beneﬁts.
The largest gains are generally observed in domains requiring structured exploration, including settings where the behavior of the pre-trained policies is misaligned with the downstream task.
1. Introduction
Transfer in deep learning is often performed through parameter initialization followed by ﬁne-tuning, a technique that allows to leverage the power of deep networks in domains where labelled data is scarce (Yosinski et al., 2014; Donahue , London. Reinforcement Learning Workshop 38 th International Conference on Machine Learning , PMLR 139, et al., 2014; Zeiler & Fergus, 2014; Girshick et al., 2014;
Devlin et al., 2019).
This builds on the intuition that the pretrained model will map inputs to a feature space where the downstream task is easy to perform.
When combined with methods that can leverage massive amounts of unlabelled data for pre-training, this transfer strategy has led to unprecedented results in domains like computer vision (H ´ enaff et al., 2019;
He et al., 2019) and natural language processing (Devlin et al., 2019;
Radford et al., 2019).
The success of these approaches has led to an ever-growing interest in developing techniques for pre-training large scale models on unlabelled data (Brown et al., 2020; Chen et al., 2020; Grill et al., 2020).
In the reinforcement learning (RL) context, unsupervised methods that learn in the absence of reward have also garnered much research attention (Gregor et al., 2016; Florensa et al., 2017; Pathak et al., 2017; Eysenbach et al., 2019; Hazan et al., 2019).
The beneﬁts of unsupervised pre-training are typically evaluated by their ability to enable efﬁcient transfer to previously unseen reward functions (Hansen et al., 2020).
In spite of their different approaches to unsupervised RL, most of the top-performing methods in this setting transfer knowledge through neural network weights.
Such approaches deal with the data inefﬁciency associated to training neural networks with gradient descent, similarly to what is done in supervised learning, e.g. by pre-training encoders that extract representations from observations (Yarats et al., 2021).
However, RL introduces a challenge that is not present in supervised learning: the agent is responsible for collecting the right data to learn from.
This introduces a second source of inefﬁciency from which transfer approaches can also suffer if they rely on unstructured exploration strategies after pre-training, as these can lead to exponentially larger data requirements in complex downstream environments (Osband et al., 2016b;a).
To address this problem, one could consider ﬁne-tuning policies that produce meaningful behavior (Mutti et al., 2021;
Schwarzer et al., 2021), but this approach quickly disregards the pre-trained behavior when learning in the downstream task due to catastrophic forgetting.
In this work, we explicitly separate the transfer of behaviour
[Figure]
and weights. We propose to make use of the pre-trained behaviour itself (i.e., the pre-trained policy mapping from observations to actions) in contrast to pre-trained neural network weights for further ﬁne-tuning.
While pre-trained behavior has been used before for exploitation (Barto &
Mahadevan, 2003; Sutton et al., 1999; Barreto et al., 2017; 2018), our approach employs pre-trained policies to aid with exploration as well to collect experience that can be leveraged via off-policy learning.
This strategy accelerates learning, as the agent is exposed to potentially useful experience earlier in training, without compromising the quality of the discovered solution when the pre-trained behavior is not aligned with the downstream task.
We expose the pre-trained behaviour to the downstream agent in two ways: ﬁrstly, as an extra exploratory strategy that, when randomly activated, persists for a number of steps, and secondly as an additional pseudo-action for the learned value function where the agent may elect to defer action selection to the pre-trained policy instead of choosing itself.
We call this approach Behavior Transfer (BT).
Deﬁning unsupervised RL objectives remains an open problem, and solutions are generally inﬂuenced by how the acquired knowledge will be used for solving downstream tasks.
Instead of proposing yet another objective for unsupervised pre-training, we turn to existing techniques for training policies in the absence of reward and make our choice based on two general requirements.
First, the objective should scale gracefully with increased compute and data.
This has been key for the success of self-supervised approaches in other domains (Brown et al., 2020; Kaplan et al., 2020), and we argue that it is an important property for unsupervised RL as well.
Second, the pre-training stage should return a policy that produces complex behavior that may be leveraged in a subsequent transfer stage.
The Never Give
Up (NGU) (Puigdom`enech Badia et al., 2020b) intrinsic reward meets both requirements, and our experiments show that large-scale pre-training with this objective leads to state of the art scores in the reward-free Atari benchmark.
Figure 1 exempliﬁes our main ﬁndings.
We pre-train behaviour using the intrinsic NGU reward during a long unsupervised phase without rewards.
This gives rise to exploratory behaviors that seek to visit many different states throughout an episode, and we then compare different strategies for leveraging the acquired knowledge once rewards are reinstated.
While ﬁne-tuning the pre-trained weights enables faster learning, the exploratory behavior of the pre-trained policy is quickly disregarded as it is exposed to rewards.
On the other hand, Behavior Transfer (BT) does not modify the pre-trained policy while learning in the new task and is able to achieve higher end scores thanks to better exploration.
These two strategies are not mutually exclusive, and BT also beneﬁts from the faster convergence provided by initializing neural networks with pre-trained weights when these encode useful information for solving the downstream task.
Our contributions can be summarized as follows.
(1) We propose Behavior Transfer (BT), a technique that leverages pre-trained policies for exploration by treating them as black boxes that are not modiﬁed during learning on the downstream task.
BT uses the pre-trained policy to collect experience in two ways, namely randomly-triggered temporally-extended exploration and one-step calls based on value estimates.
(2) Our experiments show that large-scale unsupervised pre-training with existing intrinsic rewards can produce meaningful behavior, achieving state of the art results in the reward-free Atari benchmark.
These results suggest that scale is key for unsupervised RL, akin to what has been observed in supervised settings.
(3) We provide extensive empirical evidence demonstrating the beneﬁts of leveraging pre-trained behavior via BT.
Our approach obtains the largest gains in hard exploration games, where it almost doubles the median human normalized score achieved by our strongest baseline.
Furthermore, we show that BT is able to leverage a single task-agnostic policy to solve multiple tasks in the same environment and to achieve high performance even when the pre-trained policies are misaligned with the task being solved.
(4) BT brings beneﬁts to the table that are complementary to those provided by reusing pre-trained neural network weights, and we empirically show that combining these two strategies can result in larger gains.
2. Preliminaries
The interaction between the agent and the environment is modelled as a Markov Decission Process (MDP) (Puterman,
An MDP is deﬁned by the tuple ( S , A , P, d 0 , R, γ ) where S and A are the state and action spaces, P ( s (cid:48) | s, a ) is the probability of transitioning from state s to s (cid:48) after taking action a , d 0 ( s ) is the probability distribution over initial states, R : S
×
A × S −→ R is the reward function, and γ ∈ [0 , 1) is the discount factor.
The goal is to ﬁnd a policy π ( a | s ) that maximizes the expected return, G t = (cid:80) ∞ t =
γ t R t , where R t = r
( S t , A t , S t +1 ) .
A principled way to address this problem is to use methods that compute action-value functions, Q π ( s, a ) =
E π
[ G t | S t = s, A t = a ] , where E π [ · ] denotes expectation over transitions induced by π (Puterman, 1994).
We consider a setting where the agent is allowed to ﬁrst learn within an MDP without rewards, M R =
( S , A , P, d 0 ) , for a long period of time.
The knowledge acquired during the reward-free stage is later leveraged when maximizing reward in new MDPs that share the same underlying dynamics but have different reward functions, M i =
( S , A , P, d 0 , R i , γ i ) .
Interactions between the agent and the environment are often assumed to incur a cost, but we will consider this cost to be relevant only for transitions with reward (Hansen et al., 2020).
Even if the cost of unsupervised pre-training becomes non-negligible, it can be amortized when the acquired task-agnostic knowledge is leveraged to solve multiple tasks efﬁciently (Devlin et al., 2019; Brown et al., 2020).
Indeed, we would expect this transfer setting to become more relevant as the community moves towards more complex environments, where one may want to train agents to maximize multiple reward functions under constant dynamics.
In the limit, one could consider the real world: it has constant or slowly changing dynamics, and humans are able to leverage previously acquired skills to quickly master new tasks.
3. Behavior Transfer
Transfer in supervised domains often exploits the fact that related tasks might be solved using similar representations.
This practice deals with the data inefﬁciency of training large neural networks with stochastic gradient descent.
However, there is an additional source of data inefﬁciency when training RL agents: unstructured exploration.
Fine-tuning a pre-trained exploratory policy arises as a potential strategy for overcoming this problem, as the agent will observe rich experience much earlier in training than when initializing the policy randomly, but this approach suffers from important limitations.
Learning in the downstream task can lead to catastrophically forgetting the pre-trained policy, thus prematurely disregarding its exploratory behavior.
Moreover, the same neural network architecture needs to be used for both the pre-trained and the downstream policies, which in practice also imposes a limitation on the type of
RL methods that can be employed in the adaptation stage (for instance, if the pre-trained policy was trained using a policy-based method, it might not be possible to ﬁne-tune it using a value-based approach).
Let us assume that we have access to a pre-trained policy that exhibits exploratory behavior, and defer the discussion on how to train this policy to Section 4.
Following such a policy might bring the agent to states that are unlikely to be visited with unstructured exploration techniques such as (cid:15) -greedy (Sutton & Barto, 2018).
This property has the potential of accelerating learning even when the behavior of the pre-trained policy is not aligned with the downstream task, as it will effectively shorten the path between otherwise distant states (Liu & Brunskill, 2018).
Leveraging pre-trained policies for exploration differs from other approaches in the literature that use such policies directly for exploitation, e.g. via zero-shot transfer (Eysenbach et al., 2019), methods that deﬁne a higher-level policy that alternates between the given policies (Barto & Mahadevan, 2003; Sutton et al., 1999), or within the framework of generalized policy updates (Barreto et al., 2020).
Exploring with pre-trained policies can accelerate convergence by providing useful experience to the agent, which is possible even when the pre-training and downstream tasks are misaligned.
However, strategies that directly use the pre-trained policies for exploitation may result in sub-optimal solutions in such scenario (Barreto et al., 2017).
We propose to leverage the behavior of pre-trained policies during transfer to aid with exploration.
An explicit distinction between behavior and representation is made by considering pre-trained policies as black boxes that take observations and return actions.
This strategy is agnostic to how the pre-trained behavior is encoded and is not restricted to learned policies.
We rely on off-policy learning methods during transfer to leverage the behavior of a pretrained policy π p ( a | s ) .
We keep π p ﬁxed during transfer, which prevents catastrophic forgetting of the original behavior when it is parameterized by a neural network (i.e., we instantiate and train a new policy with its own set of parameters).
We propose Behavior Transfer (BT), which leverages two complementary strategies to achieve this.
Since BT is agnostic to the method used to pre-train policies, BT ( π p ) refers to behavior being transferred from policy π p .
We formalize BT in the context of value-based Q-learning agents, although similar derivations are in principle possible for alternative off-policy learning methods.
Pseudo-code for
BT is provided in Algorithm 1.
Temporally-extended exploration.
We draw inspiration from L´evy ﬂights (Viswanathan et al., 1996), a class of ecological models for animal foraging, where a ﬁxed direction is followed for a duration sampled from a heavy-tailed distribution.
This principle was implemented in the context of exploration in RL by (cid:15)z -greedy (Dabney et al., 2021), which encodes the notion of direction in the environment via exploration options that repeat the same action throughout the entire ﬂight.
Since π p is more likely to encode a meaningful notion of direction in complex environments than action repeats, we propose a variant of (cid:15)z -greedy where π p is used as the exploration option.
An exploratory ﬂight might be started at any step with some probability.
The duration for the ﬂight is sampled from a heavy-tailed distribution (Zeta with µ = 2 in all our experiments), and control is handed over to π p during the complete ﬂight.
When not in a ﬂight, actions are sampled from the behavior policy obtained while maximizing the task reward (e.g. an (cid:15) -greedy derived from the estimated Q values).
Extra action.
The previous approach switches to π p during experience collection blindly, and we now consider an alternative strategy for triggering these switches based on value.
This can be easily implemented through an extra action which samples an action from π p , which also allows the agent to use the pre-trained policy at test time if deemed beneﬁcial.
More formally, this amounts to training a policy over an expanded action set A
+ =
A ∪ { a + } , where a + is resolved by sampling an action from π p , a (cid:48) ∼ π p ( s ) (with a (cid:48) ∈ A ).
The additional action can be seen as an option that can be initiated from any state and always terminates after a single step.
Note that selecting the option will lead to the same outcome as if the agent had selected a (cid:48) as a primitive action, and we take advantage of this observation by using the return of following the option as target to ﬁt both Q ( s, π p ( s )) and Q ( s, a (cid:48) ) .
Intuitively, this approach induces a bias that favours actions selected by π p , accelerating the collection of rewarding transitions when the pre-trained policy is somewhat aligned with the downstream task.
Otherwise, the agent can learn to ignore π p as training progresses by selecting other actions.
4. Reward-free pre-training
It is a common practice to derive objectives for proxy tasks in order to drive learning in the absence of reward functions, and there exists a plethora of different approaches in the literature.
Model-based approaches can learn world models from unsupervised interaction (Ha & Schmidhuber, 2018).
However, the diversity of the training data will impact the accuracy of the model (Sekar et al., 2020) and deploying this type of approach in visually complex domains like Atari remains an open problem (Hafner et al., 2019).
Unsupervised RL has also been explored through the lens of empowerment (Salge et al., 2014; Mohamed & Rezende, 2015), which studies agents that aim to discover intrinsic options (Gregor et al., 2016;
Eysenbach et al., 2019).
While these options can be leveraged by hierarchical agents (Florensa et al., 2017) or integrated within the universal successor features framework (Barreto et al., 2017; Algorithm Input: Action set, A ; additional action, a + ; extended action set, A + =
A ∪ { a + } ; pre-trained policy, π p ; Q-value estimate for the current policy, Q π ( s, a ) ∀ a ∈ A
+ ; probability of taking an exploratory action, (cid:15) ; probability of starting a ﬂight, (cid:15) levy ; ﬂight length distribution, D ( N ) while True do n ←−// flight length while episode not ended do
Observe state s if n == 0 and random() ≤ (cid:15) levy then n ∼ D ( N ) // sample length if n > 0 then n ←− n −a ∼ π p ( s ) elseif random() ≤ (cid:15) then a ∼ Uniform ( A + ) else a ←− arg max a (cid:48) ∈A + [ Q π ( s, a (cid:48) )
] if a == a + then a ∼ π p ( s ) end
Take action a end end 2018; Borsa et al., 2019; Hansen et al., 2020), their potential lack of coverage generally limits their applicability to complex downstream tasks (Campos et al., 2020).
An alternative objective is that of exploring the environment by ﬁnding policies that induce maximally entropic state distributions (Hazan et al., 2019; Lee et al., 2019), although this might become extremely inefﬁcient in high-dimensional state spaces without proper priors (Liu & Abbeel, 2021;
Yarats et al., 2021).
Recall that our goal is to devise a pre-training objective that can help reduce the amount of interaction needed by the agent to collect relevant experience when learning in a downstream task.
We argue that such objective needs to meet two requirements.
First, as suggested by results in other domains (Brown et al., 2020; Kaplan et al., 2020), it should scale gracefully as the amount of compute and experience used for pre-training are increased.
This contrasts with the training regimes used in most unsupervised
RL approaches, which use a relatively small amount of experience (Hansen et al., 2020; Liu & Abbeel, 2021; Yarats et al., 2021) when compared to distributed agents that do make use of rewards (Horgan et al., 2018; Espeholt et al., 2018; Kapturowski et al., 2019).
Second, it must encourage the emergence of complex behaviors such as navigation or manipulation skills.
It has been argued that exploring the environment efﬁciently will serve as a proxy for developing such behaviors (Kearns & Singh, 2002), and exploration bonuses have been shown to produce meaningful behavior in the absence of reward (Pathak et al., 2017; Burda et al., 2018a).
However, many exploration bonuses vanish over the course of training and thus may not be well-suited for a long unsupervised pre-training phase.
It can be shown that many intrinsic rewards aim at maximizing the entropy of all states visited during training, and so the ﬁnal policy does not necessarily exhibit exploratory behavior (Lee et al., 2019).
We propose to use Never Give Up (NGU) (Puigdom`enech Badia et al., 2020b) as a means for training exploratory policies in an unsupervised setting.
The NGU intrinsic reward proposes a curiosity-driven approach for training persistent exploratory policies which combines perepisode and life-long novelty.
The per-episode novelty, r episodic t , rapidly vanishes over the course of an episode, and it is designed to encourage self-avoiding trajectories.
It is computed by comparing a representation of the current observation, f ( s t ) , to those of all the observations visited in the current episode, M = { f ( s 0 ) , f ( s 1 ) , . . .
, f ( s t − 1 ) } , where f : S → R p is an embedding function trained using a self-supervised inverse dynamics model (Pathak et al., 2017).
Such a mapping concentrates on the controllable aspects of the environment, ignoring all the variability present in the observation that is not affected by the action taken by the agent.
The life-long novelty, α t , slowly vanishes throughout training, and it is computed by using Random
Network Distillation (RND) (Burda et al., 2018b).
With this, the intrinsic reward r NGU t is deﬁned as follows:
[Equation]
where L is a ﬁxed maximum reward scaling, N k is the set containing the k -nearest neighbors of f ( s t ) in M , c is a constant and K : R p × R p
→ R + is a kernel function satisfying
K ( x, x ) = 1 (which can be thought of as approximating pseudo-counts (Puigdom`enech Badia et al., 2020b)).
The episodic component of the reward in Equation 1 is reset by emptying M with each episode, thus the NGU reward does not vanish throughout the training process.
This makes it suitable for driving learning in task-agnostic settings.
5. Experiments
Agents are evaluated in the Atari suite (Bellemare et al., 2013), a benchmark that presents a variety of challenges and that is a common test ground for RL agents with unsupervised pre-training (Hansen et al., 2020; Liu & Abbeel, 2021;
Schwarzer et al., 2021).
Experiments are run using the distributed R2D2 agent (Kapturowski et al., 2019) with
CPU actors and a single GPU learner.
Policies use the same
Q-Network architecture as Agent57 (Puigdom`enech Badia et al., 2020a), which is composed by a convolutional torso followed by an LSTM (Hochreiter & Schmidhuber, 1997) and a dueling head (Wang et al., 2016).
Hyperparameters and a detailed description of the full distributed setting are provided in the supplementary material.
All reported results are the average over three random seeds.
Reward-free learning.
The amount of task reward collected by unsupervised policies is often used as a proxy to measure their quality (Eysenbach et al., 2019).
While the actual utility of these policies will not be revealed until they are leveraged for transfer, this proxy lets us evaluate whether the discovered behavior changes as longer pre-training budgets are allowed.
We compare unsupervised NGU policies against VISR (Hansen et al., 2020) and APT (Liu & Abbeel, 2021), which utilize a small amount of supervised interaction to adapt the pre-trained policies.
We also consider two additional unsupervised baselines: (i) a constant positive reward at each timestep that favours long episodes, which correlate with high scores in some games (Burda et al., 2018a), and (ii) RND (Burda et al., 2018b), which rewards life-long novelty.
Note that the RND reward vanishes, but we include it in our analysis because it was previously used by Burda et al.
(2018a) in this setting and implementation choices such as reward normalization may prevent it from fading in practice.
Figure 2 (left) shows how the zero-shot transfer performance of unsupervised policies evolves during a long pre-training phase.
NGU reaches the highest scores, but both NGU and RND eventually outperform VISR and
APT even though these used supervised interaction.
In
Table 2 of Appendix C we show that unsupervised NGU policies largely outperform several other baselines using the standard pre-training and adaptation setting.
These results highlight the importance of large-scale unsupervised pretraining in RL, similarly to the trend observed in supervised domains (Brown et al., 2020).
Transfer setting.
Transfer approaches are typically evaluated in the Atari benchmark with a budget of 100k RL interactions with reward (400k frames), but we propose to allow a longer adaptation phase.
Randomly initialized networks tend to overﬁt in these very low data regimes without strong regularization (Kostrikov et al., 2021), and we are interested in studying the impact of leveraging behavior both in isolation and combined with transfer via pre-trained weights.
Moreover, since the pre-trained policies are already competent in the downstream tasks, 100k interactions are exhausted after few episodes and may be insufﬁcient for improving performance.
For these reasons, we provide results with up to 1.25B RL steps of supervised interaction (5B frames).
This allows evaluating both convergence speed and asymptotic performance, while still being a relatively small budget for these distributed agents with hundreds of [Figure]
Transfer via behavior.
We start by studying the impact of leveraging behavior in isolation, i e. without transferring pre-trained weights, when learning in downstream tasks.
We compare BT against two baselines that do not use pre-trained behavior, namely the standard R2D agent (Kapturowski et al., 2019) that uses (cid:15) -greedy policies for exploration (Sutton & Barto, 2018), as well as a variant of R2D2 with (cid:15)z -greedy exploration (Dabney et al., 2021).
Figure 3 shows that BT is superior to both baselines for any amount of environment interaction with rewards, converging faster early in training and also obtaining higher asymptotic performance.
These results also demonstrate the generality of the proposed approach, as it is able to beneﬁt from both
RND and NGU policies.
Note that BT performs particularly well in the set of six hard exploration games 1 deﬁned by
Bellemare et al.
(2016), which is aligned with our intuition
that reusing behavior helps overcoming the inefﬁciency associated to unstructured exploration.
Figure 2 (right) conﬁrms that a long pre-training phase is especially important in hard exploration games such as Montezuma’s Revenge, even it they do not translate into higher zero-shot transfer scores, as it produces more exploratory behavior.
On the other hand, the performance after transfer is independent of the amount of pre-training in dense reward games like Pong, where unstructured exploration is enough to reach optimal scores.
Ablation studies.
In order to gain insight on each of the components in BT, we run experiments on a subset of 12 games 2 requiring different amounts of exploration and featuring both dense and sparse rewards.
BT( π NGU ) achieves a median score of 368 in this subset, which compares favorably to the 196 median score of R2D2 with (cid:15) - greedy exploration.
Removing either the extra action or the temporally-extended exploration reduces the median score of BT( π NGU ) to 224.
These results suggest that the gains provided by both strategies are complementary, and both are responsible for the strong performance of BT.
To provide further insight about the beneﬁts of BT, Figure 4 reports the fraction of steps per episode in which the extra action is selected by the greedy policy.
It hints at the emergence of a schedule over the usage of the pre-trained policy, which increases early in training and decays afterwards.
We hypothesize that this is due to the fact that the unsupervised policies obtain large episodic returns, but their behavior is suboptimal when maximizing discounted rewards.
These policies take many exploratory actions in between rewards, and so the agent eventually ﬁgures out more efﬁcient strategies for reaching rewarding states by using primitive actions.
Transfer to multiple tasks.
An appealing property of taskagnostic knowledge is that it can be leveraged to solve multiple tasks.
In the RL setting, this can be evaluated by leveraging a single task-agnostic policy for solving multiple tasks (i.e. reward functions) in the same environment.
We evaluate whether the unsupervised NGU policies can be useful beyond the standard Atari tasks by creating two alternative versions of Ms Pacman and Hero with different levels of difﬁculty.
The goal in the modiﬁed version of Ms Pacman is to eat vulnerable ghosts, with pac-dots giving 0 (easy version) or − 10 (hard version) points.
In the modiﬁed version of Hero, saving miners gives a ﬁxed return of 1000 points and dynamiting walls gives either 0 (easy version) or − 300 (hard version) points.
The rest of rewards are removed, e.g. eating fruit in Ms Pacman or the bonus for unused power units in Hero.
Note that even in the easy version of the games exploration is harder than in their original counterparts, as there are no small rewards guiding the agent towards its goals.
Exploration is even more challenging in the hard version of the games, as the intermediate rewards work as a deceptive signal that takes the agent away from its actual goal.
In this case, ﬁnding rewarding behaviors requires a stronger commitment to an exploration strategy.
Unsupervised NGU policies often achieve very low or even negative rewards in this setting, which contrasts with the strong performance they showed when evaluated under the [Figure] [Figure] standard game reward.
Figure 5 shows that leveraging the behavior of pre-trained exploration policies provides important gains even in this adversarial scenario.
These results suggest that the strong performance observed under the standard game rewards is not due to an alignment between the
NGU reward and the game goals, but due to an efﬁcient usage of pre-trained exploration policies.
Combining pre-trained behavior and weights.
Our last batch of experiments focuses on studying transfer via pretrained weights and its compatibility with BT.
Policies are composed of a convolutional torso, an LSTM, and a dueling head.
We consider two initialization strategies: a partial initialization approach that loads the torso and the LSTM, but initializes the head randomly; and a full initialization scheme where all weights are loaded.
The former can be understood as transferring learned representations (Yarats et al., 2021), but deferring exploration to a random policy.
On the other hand, the full initialization approach can be seen as directly transferring the policy and is usually referred to as ﬁne-tuning the pre-trained policy (Mutti et al., 2021; Liu & Abbeel, 2021; Schwarzer et al., 2021).
Note that these approaches only change how weights are initialized before training.
As in previous experiments, all parameters in the new policy are trained and π p is kept ﬁxed when using BT.
Figure 6 (top) compares agents with and without BT for different amounts of transfer via weights on the Atari benchmark.
Loading pre-trained weights results in faster learning early in training, both with and without BT.
The largest gains are observed in dense reward games, which translates into higher median scores across the full suite because most games belong to this category.
Weights alone are not enough in hard exploration games, where leveraging the pre-trained policy via BT provides clear beneﬁts.
Perhaps surprisingly, we observe that transferring representations outperforms ﬁne-tuning the pre-trained policy, and we hypothesize that the former is more robust to misalignments between the pre-trained policy and the downstream task.
This intuition is further supported by the experiments on games with modiﬁed reward functions reported in Figure 6 (middle & bottom), where the faster learning provided by pre-trained weights often comes at the cost of lower end scores.
On the other hand, BT is crucial in tasks with sparse and deceptive rewards and also beneﬁts from pre-trained weights in tasks where positive transfer is observed.
6. Related work
Our work uses the experimental methodology presented by
Hansen et al.
Whereas that work only considered a fast, simpliﬁed adaptation process that limited the ﬁnal performance on the downstream task, we focus on the more general case of using a previously trained policy to aid in solving the full RL problem.
Hansen et al.
(2020) use successor features to identify which of the pre-trained tasks best matches the true reward structure, which has previously been shown to work well for multi-task transfer (Barreto et al., 2018).
Bagot et al.
(2020) augments an agent with the ability to utilize another policy, which is learned in tandem based on an intrinsic reward function.
This promising direction is complementary to our work, as it handles the case wherein there is no unsupervised pre-training phase.
Gupta et al.
(2018) provides an alternative method to metalearn a solver for reinforcement learning problems from unsupervised reward functions.
This method utilizes gradientbased meta-learning
(Finn et al., 2017), which makes the adaptation process standard reinforcement learning updates.
This means that even if the downstream reward is far outside of the training distribution, ﬁnal performance would
[Figure]
[Figure]
[Figure]
not necessarily be affected. However, these methods are hard to scale to the larger networks considered here, and followup work (Jabri et al., 2019) changed to memory-based meta-learning (Duan et al., 2016) which relies on information about rewards staying in the recurrent state.
This makes it unsuitable to the sort of hard exploration problem our method excels at.
Recent work has shown success in transferring representations learned in an unsupervised setting to reinforcement learning tasks (Stooke et al., 2020).
Our representation transfer experiments suggest that this might handicap ﬁnal performance, but the possibility also exists that different unsupervised objectives should be used for representation transfer and policy transfer.
7. Discussion
We studied the problem of transferring pre-trained behavior for exploration in reinforcement learning, an approach that is complementary to the common practice of transferring neural network weights.
Our proposed approach,
Behavior Transfer (BT), relies on the pre-trained policy for collecting experience in two different ways: ( i ) through temporally-extended exploration, which can be triggered with some probability at any step, and ( ii ) via one-step calls to the pre-trained policy based on value estimates.
BT results in strong transfer performance when combined with exploratory policies pre-trained in the absence of reward, with the most important gains being observed in hard exploration tasks.
These beneﬁts are not due to an alignment between our pre-training and downstream tasks, as we also observed positive transfer in games where the pre-trained policy obtained low scores.
In order to provide further evidence for this claim, we designed alternative tasks for Atari games involving hard exploration and deceptive rewards.
Our transfer strategy outperformed all considered baselines in these settings, even when the pre-trained policy obtained very low or even negative scores, demonstrating the generality of the method.
Besides disambiguating the role of the alignment between pre-training and downstream tasks, these experiments demonstrate the utility of a single task-agnostic policy for solving multiple tasks in the same environment.
Finally, we also demonstrated that BT can be combined with transfer via neural network weights to provide further gains.
Our experimental results highlight the importance of scale when training RL agents in reward-free settings, which is one of the key factors behind the recent success of unsupervised approaches in other domains.
This contrasts with the small budgets considered for reward-free RL in previous works and motivates further research in unsupervised RL approaches that scale with increased data and compute.
We argue that scale is one of the missing components in rewardfree RL, and it will be a necessary condition to unfold its full potential.
A. Pseudo-code
Algorithm 2 provides pseudo-code for the ﬂight logic that controls how the pre-trained policy is used for temporally-extended exploration.
At each step, a ﬂight is started with probability (cid:15) levy .
The duration of the ﬂight is sampled from a heavy-tailed distribution, D ( N ) , similarly to (cid:15)z -greedy (c.f. Appendix B for more details).
When not in a ﬂight, the exploitative policy that maximizes the extrinsic reward is derived from the estimated Q-values using the (cid:15) -greedy operator.
This ensures that all state-action pairs will be visited given enough time, as exploring only with π p does not guarantee such property.
Algorithm 3 provides pseudo-code for the actor logic when using the augmented action set, A + =
A ∪ { π p ( s ) } .
It derives an (cid:15) -greedy policy over |A| + 1 actions, where the ( |A| + 1) -th action is resolved by sampling from π p ( s ) .
do n ←−// flight length while episode not ended do
Observe state s if n == 0 and random() ≤ (cid:15) levy then n ∼ D ( N ) // sample from distribution over lengths endif n > 0 then n ←− n −a ∼ π p ( s ) else a ∼ (cid:15) -greedy
[ Q π ( s, a )] end
Take action a end end
Observe state s if random() ≤ (cid:15) then a ∼ Uniform ( A + ) else a ←− arg max a (cid:48) ∈A + [ Q π ( s, a (cid:48) )] endif a =
= a + then a ∼ π p ( s ) end end
B. Hyperparameters
All policies use the same Q-Network architecture as Agent57 (Puigdom`enech Badia et al., 2020a), which is composed by a convolutional torso followed by an LSTM (Hochreiter & Schmidhuber, 1997) and a dueling head (Wang et al., 2016).
When leveraging the behavior of the pre-trained policy to solve new tasks, we instantiate a new network with independent weights (c.f. Figure 7).
One can initialize some of the components of the new network using pre-trained weights without tying their values (as in common ﬁne-tuning approaches).
[Figure]
Table 1 summarizes the main hyperparameters of our method.
The pre-trained policies were optimized using Retrace (Munos et al., 2016).
Learning with rewards was performed with Peng’s Q( λ ) (Peng & Williams, 1994) instead, which we found to be much more data efﬁcient in our experiments.
The reason for this difference is that the beneﬁts of Q( λ ) were observed once unsupervised policies had been trained on all Atari games.
It should be noted that our (cid:15)z -greedy baseline under-performs relative to Dabney et al.
This is due to our hyper-parameters and setting being derived from Puigdom`enech Badia et al.
(2020b), which adopts the standard Atari pre-processing (e.g. gray scale images and frame stacking).
In contrast, Dabney et al.
(2021) use color images, no frame stacking, a larger neural network and different hyper-parameters (e.g. smaller replay buffer).
Studying if the performance of
NGU, RND and BT is preserved in this setting is an important direction for future work.
We suspect that improving the performance of our (cid:15)z -greedy ablation will also improve our method, since exploration ﬂights are central to both.
[Table]
C. Extended reward-free RL results
We compare the results of our unsupervised pre-training stage against other unsupervised approaches, standard RL algorithms in the low-data regime and methods that perform unsupervised pre-training followed by an adaptation stage.
Since the considered intrinsic rewards are non-negative, we consider a baseline where the agent obtains a constant positive reward at each step in order to measure the performance of policies that seek to stay alive for as long as possible.
Results for this baseline were already considered by Hansen et al.
( Pos Reward NSQ ), but we run our own version of this baseline using the distributed setting and longer pre-training of 16B frames considered in our experiments ( Pos Reward
R2D2 ).
Table 2 shows that unsupervised RND and NGU outperform all baselines by a large margin, conﬁrming the intuition that exploration is a good pre-training objective for the Atari benchmark.
These results suggest that there is a strong correlation between exploration and the goals established by game designers (Burda et al., 2018a).
In spite of the strong results, it is worth noting that unsupervised RND and NGU achieve lower scores than random policies in some games, and can be quite inefﬁcient at collecting rewards in some environments (e.g. they needs long episodes to obtain high scores).
These observations motivate the development of techniques to leverage these pre-trained policies without compromising performance even when there exists a misalignment between objectives.
26 Game Subset 47 Game Subset
[Table]
Games
[Table]
Algorithm
[Table]
H
[Table]
M >
[Table] >H
Mdn
M > 0 >H
IDF Curiosity @ – – – – 8 .24 . 51 – – – –
RF Curiosity @ – – – – 7 .29 .
Pos Reward NSQ @ 2 .50 .
Pos Reward R2D2 @ 9 .59 .
Q-DIAYN-5 @ 0 .
17 − 3 .
33 − 1 . 23 0 .
34 − 2 .
Q-DIAYN-50 @ − 1 .
65 − 21 .
− 1 .
69 − 16 .
0 − 3 .
16 − 20 .
VISR @ 5 .81 .
RND@ 48 .
NGU @ 80 .
SimPLe @100 k 9 .36 . 20 – – – – – – – –
DQN @10 M 27 .52 .
DQN @200 M 100 .
Rainbow @100 k 2 .10 .
PPO @500 k 20 .43 .
NSQ @10 M 8 .33 .
SPR @100 k
CURL @100 k 17 .38 .
DrQ @100 k 28 .35 .
Q-DIAYN-5 @100 k
Q-DIAYN-50 @100 k − 1 .
64 − 27 .
0 − 1 . 66 − 16 .
0 − 2 .
53 − 24 .
RF VISR @100 k 7 .58 .
VISR @100 k 9 .
GPI RF VISR @100 k 5 .58 .
GPI VISR @100 k 6 . 59 111 . 237 11 .
MEPOL @100 k 0 .17 . APT @100 k 47 .69 . D. Extended Atari-57 results
[Table]
[Table]
[Table]
+ BT( π NGU )
[Table]
+ BT( π RND )
[Table]
R2D2, partial π NGU init
[Table] + BT( π NGU ), partial π NGU [Table] [Figure]
π
[Figure]
init
[Table]
[Figure]
[Table]
[Figure]
E. Alternative reward functions
MsPacman: eating ghosts
• Pac-dots: 0 points (easy) or -10 points (hard) •
Eating vulnerable ghosts: – #1 in succession: 200 points – #2 in succession: 400 points – #3 in succession: 800 points – #4 in succession: 1600 points • Other actions: 0 points
Hero: rescuing miners
• Dynamiting walls: 0 points (easy) or -300 points (hard) • Rescuing a miner: 1000 points • Other actions: 0 points
F. Distributed setting
All experiments are run using a distributed setting.
The evaluation we do is also identical to the one done in R2D (Kapturowski et al., 2019): parallel evaluation workers, which share weights with actors and learners, run the Q-network against the environment.
This worker and all the actor workers are the two types of workers that draw samples from the environment.
For Atari, we apply the standard DQN pre-processing, as used in R2D2.
The next subsections describe how actors, evaluators, and learner are run in each stage.
F.1. Unsupervised stage
The computation of the intrinsic NGU reward, r NGU t , follows the method described in Puigdom`enech Badia et al.
(2020b,
Appendix A.1).
In particular, we use the version that combines episodic intrinsic rewards with the intrinsic reward from
Random Network Distillation (RND) (Burda et al., 2018b).
We now describe the distributed setup used for NGU, which is largely the same as the one used for RND.
Note that RND can be recovered by removing the components needed for the episodic reward.
L EARNER
• Sample from the replay buffer a sequence of intrinsic rewards r NGU t , observations x and actions a .
• Use Q-network to learn from ( r NGU t , x, a ) with Retrace (Munos et al., 2016) using the same procedure as in R2D2.
• Use last 5 frames of the sampled sequences to train the action prediction network in NGU.
This means that, for every batch of sequences, all time steps are used to train the RL loss, whereas only 5 time steps per sequence are used to optimize the action prediction loss.
• Use last 5 frames of the sampled sequences to train the predictor of RND.
CTOR • Obtain x t and r NGU t − 1 .
• With these inputs, compute forward pass of R2D2 to obtain a t .
• With x t , compute r NGU t using the embedding network in NGU.
• Insert x t , a t and r NGU t in the replay buffer.
• Step on the environment with a t .
E VALUATOR
• Obtain x t and r NGU t − 1 .
• With these inputs, compute forward pass of R2D2 to obtain a t .
• With x t , compute r NGU t using the embedding network in NGU.
• Step on the environment with a t .
D ISTRIBUTED TRAINING
As in R2D2, we train the agent with a single GPU-based learner and a ﬁxed discount factor γ .
All actors collect experience using the same policy, but with a different value of (cid:15) .
This differs from the original NGU agent, where each actor runs a policy with a different degree of exploratory behavior and discount factor.
In the replay buffer, we store ﬁxed-length sequences of ( x, a, r ) tuples.
These sequences never cross episode boundaries.
Given a single batch of trajectories we unroll both online and target networks on the same sequence of states to generate value estimates.
We use prioritized experience replay with the same prioritization scheme proposed in (Kapturowski et al., 2019).
F.2. Transfer with BT L EARNER
• Sample from the replay buffer a sequence of extrinsic rewards r t , observations x and actions a . • (expanded action set)
Duplicate transitions collected with π p and relabel the duplicates with the primitive action taken by π p when acting.
• Use Q-network to learn from ( r t , x, a ) with Peng’s Q( λ ) (Peng & Williams, 1994) using the same procedure as in R2D2.
A
CTOR • (once per episode) Sample (cid:15) levy .
• Obtain x t .
• If not on a ﬂight, start one with probability (cid:15) levy .
• If on a ﬂight, compute forward pass with π p to obtain a t .
Otherwise, compute forward pass of R2D2 to obtain a t .
If a t = |A| + 1 , a t ← π p ( x ) .
• Insert x t , a t and r t in the replay buffer.
• Step on the environment with a t .
E VALUATOR
• Obtain x t .
• Compute forward pass of R2D2 to obtain a t .
If a t = |A| + 1 , a t ← π p ( x ) .
• Step on the environment with a t .
D ISTRIBUTED TRAINING
As in R2D2, we train the agent with a single GPU-based learner and a ﬁxed discount factor γ .
All actors collect experience using the same policy, but with a different value of (cid:15) .
In the replay buffer, we store ﬁxed-length sequences of ( x, a, r ) tuples.
These sequences never cross episode boundaries.
Given a single batch of trajectories we unroll both online and target networks on the same sequence of states to generate value estimates.
We use prioritized experience replay with the same prioritization scheme proposed in (Kapturowski et al., 2019).
G. Intrinsic rewards G.1. Random Network Distillation
The RND (Burda et al., 2018b) intrinsic reward is computed by introducing a random, untrained convolutional network g : S → R d , and training a network ˆ g : S → R d to predict the outputs of g on all the observations that are seen during training by minimizing the prediction error err RND ( s t ) =
||
ˆ g
( s t ; θ ) − g ( s t )
|| 2 with respect to θ .
The intuition is that the prediction error will be large on states that have been visited less frequently by the agent.
The dimensionality of the random embedding, d , is a hyperparameter of the algorithm.
The RND intrinsic reward is obtained by normalising the prediction error.
In this work, we use a slightly different normalization from that reported in (Burda et al., 2018b).
The RND reward at time t is given by
[Equation]
where σ e is the running standard deviation of err RND ( s t ) .
G.2. Never Give Up
The NGU intrinsic reward modulates an episodic intrinsic reward, r episodic t , with a life long signal α t : r NGU t
[Equation]
(3) where L is a ﬁxed maximum reward scaling.
The life-long novelty signal is computed using RND with the normalisation:
[Equation]
) where err RND ( x t ) is the prediction error described in Appendix G.1, and µ e and σ e are its running mean and standard deviation, respectively.
The episodic intrinsic reward at time t is computed according to formula:
[Equation]
where N k is the set containing the k -nearest neighbors of f ( s t ) in M , c is a constant and K : R p × R p
→ R + is a kernel function satisfying K ( x, x ) = 1 (which can be thought of as approximating pseudo-counts (Puigdom`enech Badia et al., 2020b)).
Algorithm 4 shows a detailed description of how the episodic intrinsic reward is computed.
Below we describe the different components used in Algorithm 4:
• M : episodic memory containing at time t the previous embeddings { f ( s 0 ) , f ( s 1 ) , . . .
, f ( s t − 1 ) } .
This memory starts empty at each episode • k : number of nearest neighbours • N k = { f ( s i ) } ki =1 : set of k -nearest neighbours of f ( s t ) in the memory M ; we call N k
[ i ] =
f
( s i ) ∈ N k for ease of notation • K : kernel deﬁned as K ( x, y )
= (cid:15)
d 2( x,y ) d 2 m + (cid:15) where (cid:15) is a small constant, d is the Euclidean distance and d 2 m is a running average of the squared Euclidean distance
of the k -nearest neighbors • c : pseudo-counts constant • ξ : cluster distance • s m : maximum similarity
Compute the k neighbours of f ( s t in M and store them in a list N k Create a list of ﬂoats d k of size k /*
The list d k will contain the distances between the embedding f ( s t ) and its neighbours N k .
*/ for i ∈ { 1 , . . .
, k } do d k
[ i ] ← d 2 ( f ( s t ) , N k
[ i ]) end
Update the moving average d 2 m with the list of distances d k /* Normalize the distances d k with the updated moving average d 2 m .
*/ d
n ← d k d 2 m /* Cluster the normalized distances d n i.e. they become 0 if too small and 0 k is a list of k zeros. */ d n ← max( d
[Equation]
− ξ, 0 k ) /
* Compute the Kernel values between the embedding f ( s t ) and its neighbours
N k .
K v ← (cid:15) d n + (cid:15) /
* Compute the similarity between the embedding f ( s t ) and its neighbours N k .
*/ s ← (cid:113)(cid:80) ki
=1 K v [ i ] + c /* Compute the episodic intrinsic reward at time t : r it .
*/ if s > s m then r episodic t ←else r episodic t ← 1 /s
H. Scores per game
[Table]
[Table]
[Table]
[Table]
[Table]
I. Learning curves