On The Transferability of Deep-Q Networks
Abstract
Transfer Learning (TL) is an efﬁcient machine learning paradigm that allows overcoming some of the hurdles that characterize the successful training of deep neural networks, ranging from long training times to the needs of large datasets.
While exploiting TL is a well established and successful training practice in Supervised
Learning (SL), its applicability in Deep Reinforcement Learning (DRL) is rarer.
In this paper, we study the level of transferability of three different variants of Deep-Q
Networks on popular DRL benchmarks as well as on a set of novel, carefully designed control tasks.
Our results show that transferring neural networks in a DRL context can be particularly challenging and is a process which in most cases results in negative transfer.
In the attempt of understanding why Deep-Q Networks transfer so poorly, we gain novel insights into the training dynamics that characterizes this family of algorithms.
Over the last years the marriage between Reinforcement Learning (RL) algorithms and deep neural networks, commonly denoted as Deep Reinforcement Learning (DRL) [François-Lavet et al., 2018] has gained tremendous attention [Henderson et al.
Neural networks have, in fact, proven to be extremely successful both in a model-free RL setting as in a model-based one.
Large part of their success can be attributed to their ability of serving as feature extractors as well as function approximators, a property that allows them to successfully learn optimal value functions [Mnih et al., 2013, 2015, Van Hasselt et al., 2016, Zhao et al., 2016, Wang et al., 2016b, Sabatelli et al., 2020], stochastic policies [Lillicrap et al., 2015, Schulman et al., 2015b,a, Wang et al., 2016a
Mnih et al., 2016, Schulman et al., 2017, Haarnoja et al., 2018, Fujimoto et al., 2018], and models of an environment [Ha and Schmidhuber, 2018, Kaiser et al., 2019, Hafner et al., 2019a,b, 2020] that is usually formalized as a Markov Decision Process (MDP) Puterman [1990].
Despite the many remarkable achievements, training a DRL agent is a process that can be very time-consuming.
The task of solving an optimal decision making problem is, in fact, a challenging problem of its own, which is sometimes made even more difﬁcult by the DRL community itself, which requires DRL practitioners to test the performance of their algorithms on benchmarks that are computationally very expensive (for a position paper about this topic see [Obando-Ceron and Castro, 2020]).
One way of overcoming the need of individually training a DRL agent from scratch each time a new RL problem is encountered is based on Transfer Learning (TL).
TL focuses on designing training strategies that allow machine learning models to retain and reuse previously learned knowledge when getting trained on new, unseen problems [Pan and Yang, 2009, Zhuang et al., 2020].
Within deep learning, TL is largely adopted by the Supervised Learning (SL) community [Huh et al., 2016, Mormont et al., 2018,
Sabatelli et al., 2018a, Domínguez Sánchez et al., 2019, Vandaele et al., 2021, Ho and Kim, 2021], as it allows to train neural networks on problems that are characterized by a lack of appropriate training data or sufﬁcient computational resources; however, typical TL approaches such as off the shelf feature extraction, or ﬁne-tuning [Sharif Razavian et al., 2014] have rarely been thoroughly studied from a DRL perspective.
Therefore, the degree of transferability of DRL algorithms is not yet known.
In this work, we focus on value-based, model-free algorithms, a family of techniques which focuses on training neural networks with the intent of learning an approximation of an optimal value function.
While several of such algorithms, commonly denoted as Deep-Q Networks, exist, research studying their TL properties is, on the contrary scarce, and a clear answer to the question "How transferable are Deep-Q Networks?"
has yet to be given.
In the attempt to clearly answering this question, we present the following three contributions:
• We present a ﬁrst large scale empirical study that analyses the TL properties of popular model-free DRL algorithms on the Atari Arcade Learning Environment (ALE), where we show that transferring pre-trained networks in a DRL context can be a very challenging task.
• We design a set of novel, control experiments which allows us to thoroughly characterize the TL dynamics of Deep-Q Networks.
• While studying Deep-Q Networks from a TL perspective, we discover novel learning dynamics that provide a better understanding of how this family of algorithms deals with RL tasks.
A large-scale Empirical Study
In this section, we carry out a large-scale TL experiment on several games from the Atari Environment (Sec. 1.1).
The experimental protocol is detailed in Sec.
1.2 and results are discussed in Sec.
1.1 The Atari Environment
In this study, we use the Atari Arcade Learning Environment (ALE)
[Bellemare et al., 2013].
Next to being one of the most popular benchmarks in DRL, the ALE is particularly well suited for TL research as it allows to choose among a set of 57 Atari games that can be used as source M S and target M T MDPs within a deep transfer learning setting.
Since training a model-free agent on the games of the ALE is a process which can be computationally very expensive, we have carefully selected a subset of 10 different environments.
Numerous reasons guided the game selection process.
First, we have selected games for which we guarantee that a model-free DRL agent can learn a good policy for.
Since, as discussed by Lazaric [2012], one of the key requirements of TL is that of correctly identifying and transferring knowledge across source and target tasks, we naturally ensured that some knowledge coming in the form of neural network parameters representing a near-optimal value function was available for transfer.
Second, while it is true that all of the selected games result in an agent that can improve its policy over time, some games were chosen because the learned policy resulted in a ﬁnal performance that was not on par with that of a human expert player.
This is, for example, the case of the Frostbite game, where the gap in performance between an agent trained with the DQV-Learning algorithm [Sabatelli et al., 2018b] ( ≈ 270 ) and a human expert player ( ≈ 4300 ) is particularly signiﬁcant.
It follows that Frostbite is an interesting target task for transfer, as the agent’s performance can potentially be improved through TL.
Furthermore, we have also ensured that among the selected games, some environments are more similar to each other than others.
This is, for example, the case for the Ms. Pacman and Bank Heist games which, as can be seen in
Fig. 1, are two games where the state space is represented as a maze, and where the end goal of an agent is that of learning how to navigate it.
In like manner, we have also included games that are very different from each other as is, e.g., the case for the Crazy Climber and Pong games, where it is clear from Fig.
2 that no visual similarities are shared among the two environments.
Including visually similar and dissimilar games allows us to investigate whether, as is the case for supervised learning, a source task is particularly well suited for transfer if it is similar to its respective target task [Mensink et al., 2021].
1.2 Experimental Setup
We investigate the TL performance of agents that get trained with the DQV-Learning algorithm [Sabatelli et al., 2018a], and with the DDQN algorithm [Van Hasselt et al., 2016].
We take models which come as pre-trained on 10 Atari games and transfer them to all remaining environments.
We mostly consider the same games for both algorithms ( Bank Heist, Boxing, Crazy Climber,
Fishing Derby, Frostbite, James Bond, Ms.
Pacman, Pong and Zaxxon ) with the only exception that for DDQN Frostbite is replaced by Gopher , and Zaxxon is replaced by Ice Hockey as the Frostbite and Zaxxon DDQN agents failed to improve their policy whilst training.
It is worth noting that TL is particularly easy to perform as both algorithms learn an approximation of the optimal [Figure] state-action value function Q ( s, a ; θ ) by training a convolutional neural network (see Appendix A for its exact architecture and all hyper-parameter settings) directly on the images representing the state of the game.
Since the state space across Atari games is always represented as an 84 × 84 × 4 tensor, it is straightforward to transfer the same neural architecture among various Atari environments without needing special modiﬁcations.
However, the only modiﬁcation that we apply to a pre-trained network concerns its last layer responsible for estimating the different Q values, which we always replace and randomly re-initialize.
Following the typical deep transfer learning literature, we investigate whether in DRL it is as beneﬁcial as it is in supervised learning to transfer a network that comes as pre-trained on M S and ﬁne-tune it on M T .
We do this by quantitatively assessing the transfer learning beneﬁts on each M S / M T pair by computing the area ratio metric r [Taylor and Stone, 2009].
Speciﬁcally, given a learning curve representing the performance of an agent pre-trained on M S , and that of an agent that is instead trained from scratch, we compute r as follows:
[Equation]
1.3 Results
The results on each M S / M T pair for both the DQV and DDQN algorithms are presented in Table 1.
In each cell of the tables, we report the area ratio metric deﬁned in Eq.
1: the lower (resp.
higher) this score, the less (resp.
more) beneﬁcial it is to transfer and ﬁne-tune a pre-trained agent.
When it comes to the DQV algorithm, we can see that, out of nine target environments, there is only one Atari game for which it is always beneﬁcial to transfer and ﬁne-tune a pre-trained model: Fishing
Derby .
In fact, a positive area ratio score is obtained no matter which source environment is used for pre-training, although the best results have been obtained when starting from an Enduro or Pong pre-trained network, which both resulted in an area ratio score of ≈ 0 .
Positive transfer can also be observed on the Frostbite and James Bond games, but only for a limited number of source games.
For example, a Bank Heist pre-trained agent transfers well to both target games as it obtains an area ratio score of 0 .
729 and 0 .
973 respectively, but the same cannot be said for an Enduro pre-trained network, which on Frostbite results in absent transfer (the area ratio score is, in fact, − 0 .
017 ), and yields negative transfer on James Bond ( r = − 0 .
We can also observe that there are environments where it is surprisingly never beneﬁcial to transfer and ﬁne-tune a pre-trained agent.
This is, for example, the case for the Bank Heist and Pong games, where independently from which source game M S is used for pre-training, a negative area ratio score is always obtained.
Furthermore, it can also be observed that transfer learning across environments is not symmetric, as one source game M S can result in positive transfer when it gets transferred to a certain target game M T , but the same outcome is not obtained when transfer is performed in the opposite direction.
As an example we can consider the Boxing/Fishing Derby games: positive transfer is obtained when transferring from Boxing → Fishing Derby ( r = 0 . 552 ), but negative transfer is obtained when transferring from Fishing Derby → Boxing ( r = − 0 . 893 ).
When it comes to the DDQN algorithm, similar conclusions can be drawn: we can again observe that there are only very few cases for which it is beneﬁcial to transfer and ﬁne-tune a pre-trained DRL agent.
Examples of such cases are networks that are pre-trained on Ice Hockey and James Bond which get transferred to Boxing ( r = 0 .and
r = 0 .
232 respectively), or Boxing and Enduro models that get transferred to Pong ( r = 0 .and
r = 0 . 248 ).
Bank Heist and Pong are again the two target environments for which most of the transferred source models resulted in negative transfer, while differently from the experiments performed with the DQV algorithm, this time no positive transfer can be observed on Fishing Derby .
[Table]
Overall, the process of ﬁne-tuning a pre-trained DDQN agent mostly results in absent transfer, as can be observed by the area ratio scores obtained on Enduro, Fishing Derby, Gopher and Ice
Hockey which are all ≈ 0 on average.
Control Experiments
The results presented in the previous section seem to be questioning the level of transferability of DRL agents.
In fact, the training strategy of ﬁne-tuning a pre-trained model on M T does not result in the same type of performance gains that have been extensively observed in a supervised learning context.
To better characterize their TL properties, we have designed a set of simple control experiments that allow us to examine their transfer learning behavior in training conditions that do not require extraordinarily long training times for learning an optimal policy.
2.1 The Catch Environments
To this end, we have implemented four different versions of the Catch game, a simple RL task that was ﬁrst presented by Mnih et al.
[2014], and that has been widely used within the literature for investigating the performance of DRL algorithms in a fast, and computationally less expensive manner than the one required by the Atari games [van de Wolfshaar et al., 2018, Aittahar et al., 2020].
In the game of Catch , an agent controls a paddle at the bottom of the environment, represented by a 21 × 21 grid, and has to catch a ball falling from top to bottom, which can potentially bounce off walls.
At each time step, the agent can choose between three actions: move the paddle one pixel to the right, move it to the left, or keep it in the same position in the grid.
An RL episode ends either when the agent manages to catch the ball, in which case it receives a reward of 1 , or when it misses the ball, which naturally results in a reward of 0 .
Following the design choices presented in [van de
Wolfshaar, 2017], we model the ball to have vertical speed of v y = − 1 cell/s and horizontal speed of v x ∈ {− 2 , − 1 , 0 , 1 , 2 } .
From now on, we will refer to this version of the game as Catch-v0 , as it is the most basic and simplest form of the game that will be used throughout our experiments.
Next to Catch-v0 we have implemented three slightly different and arguably more complex versions of the game as well: Catch-v1 , where we increased the complexity of the game by reducing the size of the paddle that the agent controls.
While for Catch-v0 its size is of ﬁve pixels, in Catch-v1 it is of two pixels, therefore requiring the agent to be more precise if it wants to successfully catch the falling ball.
The second alternative version of Catch is Catch-v2 .
In this case, the dynamics of the game are identical to the ones that deﬁne Catch-v0 ; however, the way the 21 × 21 grid is represented changes.
While in Catch-v0 as well as in Catch-v1 the state is represented by a binary grid where all pixels, but the ones representing the paddle and the ball have a value of 0 , in Catch-v2 the cells around the paddle and the ball can have a random value between 0 and 255 .
This design choice makes it
[Figure]
much harder for a convolutional network to correctly locate and identify the position of the paddle and of the falling ball and makes Catch-v2 the arguably most complex version among the different Catch environments.
Lastly, we have implemented Catch-v3 , a version of Catch which is identical to the one that is modeled by Catch-v0 with the main difference that the representation of the state is now mirrored, therefore requiring the agent to look at different parts of the grid if it wants to locate the paddle, and understand that the ball is unnaturally moving from the bottom to the top.
For an impression of all four Catch versions see the left image of Fig. 3.
Given the overall simplicity of the different Catch environments, we now train a DQN agent instead of the arguably more complex DQV and DDQN agents that we considered in Sec.
As we can see from the results reported in the right plot of Fig. 3, averaged over ﬁve different runs, the agent is able to successfully learn a near optimal policy for all Catch versions.
When it comes to Catch-v0 , Catch-v2 and Catch-v3 we can observe that by the end of training, the agent is able to catch ≈ 100% of the falling balls, whereas its performance is slightly worse ( ≈ 90% ) when it comes to Catch-v1 1 .
We can also observe that among the different Catch versions, Catch-v0 and Catch-v3 appear to be the easiest ones, as the agent requires signiﬁcantly less training episodes to converge when compared to Catch-v1 and Catch-v2 .
Furthermore, in line with the explanation presented beforehand, our results also conﬁrm the hypothesis that Catch-v2 is the overall most complicated Catch version, as learning requires signiﬁcantly more, and potentially unstable, training epochs.
2.2 From one Catch to Another
We now replicate the TL study presented in Sec.
1 on the aforementioned Catch environments, with the hope of identifying why the process of ﬁne-tuning a pre-trained convolutional neural network in a DRL context, seems to not be as beneﬁcial as it is in the supervised learning one.
Our goal is to ﬁnd at least one pair of Catch environments which results in positive transfer, and to then potentially identify some properties within the different Catch versions that could also hold for the pairs of Atari games which have yielded positive transfer in Table 1.
We formulate two hypothesis, based on which, we expect to experimentally observe positive transfer.
First, we foresee that positive transfer will happen for all possible Catch combinations, as in the end the source MDP M S and the target
MDP M T do not signiﬁcantly differ from each other: in fact, the main task across different Catch versions remains that of catching a falling ball; the action space is identical; and so is the reward function that always returns a value of 1 when the agent succeeds in catching the falling ball.
This hypothesis is also motivated by supervised learning research which shows that the higher the similarity between the source domain D S and the target domain D T , the better the performance of a transferred pre-trained network [Sabatelli et al., 2018a].
Second, in the case the previous hypothesis will not be empirically supported, we expect to at least observe positive transfer when using a model that comes as pre-trained either on Catch-v1 or on Catch-v2 .
In fact, as described above and also shown by the performance reported in Fig. 3, these are the two most complicated versions of the Catch environment.
In supervised learning, one of the main factors that makes a certain source task T S good for transferring is its complexity
[Mensink et al., 2021], which is usually deﬁned in terms of dataset size and number of classes to classify, therefore we expect the complexity of the source game to play an important role within DRL as well.
Similarly to what we did for DQV and DDQN in Sec. 1, we take the four different models that have been trained from scratch on their respective Catch version, randomly re-initialize their last layer (responsible for estimating the different state-action values Q ( s, a ) ) and fully ﬁne-tune the pre-trained network on the three remaining Catch environments.
[Table]
The results of this study are reported in Fig.
4, where, from left to right, we show the performance that is obtained when considering
Catch-v0 , Catch-v1 , Catch-v2 and
Catch-v3 as target MDP M T .
The performance of each transferred network is compared against the performance that is obtained after training a DQN agent from scratch which matches with the results reported in
Fig.
Surprisingly we found that ﬁne-tuning a pre-trained DQN agent never resulted in positive transfer learning.
This can clearly be seen in all plots represented in Fig.
4 and by the results reporting the area ratio metric in Table 2.
The only case where starting from a pre-trained network appeared to be at least in part beneﬁcial is represented by the ﬁrst plot of Fig.when Catch-v1 is considered as source MDP M S .
In this case we can in fact observe some learning speed improvements within the ﬁrst 25 learning epochs.
This is not surprising as an agent which is able to catch a ball with a small paddle (as deﬁned by the game Catch-v1 ) should in principle also be able to do this when the size of its paddle is larger (which is the case for Catch-v0 ).
What is more surprising, however, is that while training progresses we see that the performance of a Catch-v pre-trained model starts deteriorating and that this model barely converges to the same performance that is obtained by a model trained from scratch.
When it comes to all other M S / M T pairs we see that pre-trained networks always perform signiﬁcantly worse than randomly initialized models trained from scratch, with some extreme cases, as the one reported in the last plot of Fig.
4, where a Catch-v0 pre-trained agent is barely able to improve its policy over time at all.
These results invalidate our two hypotheses mentioned above as they clearly show that positive transfer in DRL does not arise when M S and M T are similar, nor when M S is more complex than M T .
[Figure]
2.3 Self-Transfer
The surprisingly poor transfer learning performance observed in the previous experiment made us question the level of transferability of Deep-Q Networks even more.
To further characterize their TL properties, we decided to investigate whether pre-trained DRL agents are at least able to transfer to themselves.
To this end, we studied what happens when a DQN agent gets transferred to a version of Catch that matches with the version of the game that was also used during the pre-training stage.
This experiment is in large part identical to the one presented in Sec. 2.2, with the only difference being that now M S = M T .
Moreover, differently from the previous study, we now also investigate what happens if instead of ﬁne-tuning the network completely, we just use the pre-trained DQN agent as a simple feature extractor, therefore only training its last layer (the head) responsible for estimating the different state-action values.
[Table]
Our results are presented in Fig.
5, where for each Catch version, the mance from scratch, and port the is tained when the network is either used as simple feature extractor or entirely ﬁne-tuned.
We can see that if a pre-trained Deep-Q Network is used as a simple feature extractor, the agent can converge to the optimal policy almost immediately.
In fact, as can consistently be observed in all plots of Fig.
5 and from the results presented in Table 3, training only the last of a pre-trained network yields positive transfer for all different Catch versions a ﬁne-tuning training strategy is adopted, much more surprising results have been obtained.
First, and more importantly, we can see that despite all models showing some learning speed improvements at early training iterations, their ﬁnal performance is never on par with the one that is obtained the same kind of model is either used as a feature extractor or trained from scratch (the dotted lines are consistently below the dashed and full lines).
While when it comes to Catch-v0 the policy learned by a ﬁne-tuned model still allows the agent to successfully catch ≈ 95% of the falling balls, the same cannot be said when the models are tested on Catch-v1, Catch-v2 and Catch-v3 , where the difference in terms of performance between a model trained from scratch and a ﬁne-tuned one is much more signiﬁcant.
Please also note that special attention should be given to Catch-v2 , which is an environment where the area ratio score reported in Table 3 can be misleading as it does not entirely reﬂect the quality of the ﬁnal policy learned by the agent.
In fact, while it is true that an r value of 0 .
393 is obtained, it is worth noting that a ﬁne-tuned network converges to a policy that is signiﬁcantly worse than the one of a network trained from scratch, as the agent is only able of catching ≈ 80% of the falling balls.
Second, as highlighted by the large variance across different training runs, ﬁne-tuning on Catch-v1 and Catch-v3 resulted in highly unstable learning as well.
[Figure]
-v3 after either training only linear a pre-trained Deep-Q Network lines), or after wholly -tuning model lines We can see that the former transfer learning strategy yields signiﬁcantly better results, and that a ﬁne-tuning approach results in networks that in three cases out of four are not even able to transfer to themselves.  The Two Learning Phases of Deep-Q Networks
A prototypical Deep-Q Network takes as input an image representing the state of the environment and processes it through a series of convolutions and a fully connected layer.
When this is done, it outputs as many Q ( s, a ) values as there are actions available to the agent, a process that corresponds to learning a linear policy in the latent feature space of the network.
It follows that by the end of training, such a model has to serve two purposes: it has to perform as a feature extractor as well as an optimal value function approximator.
Extracting relevant features from high dimensional inputs and learning an optimal value function can arguably be seen as two separate tasks; yet, despite their dissimilarity, we believe that they are more interconnected than one might expect.
Speciﬁcally, we hypothesize that while learning, a Deep-Q Network has to carefully ﬁnd a balance between training the parts that serve as feature extractors and the components that are responsible for estimating a policy.
The poor TL performance observed throughout this work could therefore be the result of using models where the feature extractor component of an agent, as it comes as pre-trained, is too detached from the respective ﬁnal layer of the network, which is randomly initialized instead.
To show that a Deep-Q Network has to carefully coordinate training its feature extractor components and its ﬁnal layer, let us consider the left image of Fig.
The ﬁgure depicts how the weights of an agent, whose feature extractor layers are represented by a square and the linear layer is represented through circles, change according to the self-TL experiments presented in Sec. 2.3. Each experiment is represented through two networks, one on the left side of the arrow representing the source model, and a second one, on the right part of the arrow, obtained by the end of training. From top to bottom, and from left to right, the ﬁrst three network pairs represent the training process of: a randomly initialized model trained from scratch, a pre-trained model whose only last linear layer is trained after random initialization, and a pre-trained agent who gets fully ﬁne-tuned.
Following the results presented in Sec. 2.3, we know that positive transfer is only obtained when the last linear layer is trained in isolation after being randomly re-initialized, whereas negative transfer is obtained if a ﬁne-tuning training strategy is adopted. Pre-training
<latexit sha1_base64="DPAl+5FDIlo1aNqhDjSdlH9OSW [Table] +hpVhhNNpqZdoGmMyxkPatVRiQbWfzY+eojOrhGgQKVvSoLn6eyLDQutUBLZTYDPSy95M/M/rJmZw42dMxomhkiwWDRKOTIRmCaCQKUoMTy3BRDF7KyIjrDAxNqeSDcFbfnmVtC6q3mX16v6yUrvN4yjCCZzCOXhwDTWoQwOaQOARnuEV3pyJ8+K8Ox+L1oKTzxzDHzifP0ZjknA=</latexit>
[Figure]
Catch- [Figure]
Catch-v
We now investigate the TL performance of a model that is a combination of the Only-Head and Fine-Tuning settings (see bottom right image of Fig. 6 for a visualization).
Speciﬁcally, we ﬁne-tune a Deep-Q Network whose last linear layer is initialized with the parameters that yielded positive transfer in Sec.
2.3 (Only-Head in Fig. 6), whereas its convolutional and fully connected layers are initialized with the parameters that yielded negative transfer (Fine-Tuning in Fig. 6).
We visualize the self-transfer performance of these models, denoted as ‘Hybrid", as they are a hybrid combination of two differently pre-trained networks, in Fig. 7 with a cyan dashed dotted line.
We can observe that learning is characterized by a very atypical behavior: the network starts by improving its performance (thanks to the already trained ﬁnal layer); it then goes through a second stage where it starts to perform more poorly (due to the poor feature extractor part), and then ﬁnally starts learning stably (when the feature extractor and the head of the model are synchronized).
We believe that the poor TL performance observed throughout this work is, therefore, the result of models which could not ﬁnd a balance between a randomly initialized head and their respective pre-trained layers which are too biased towards the source task.
Based on these results, one critical question still remains to be answered: how come positive transfer for some Atari games was observed in Sec.
We believe that the answer to this question does not lie within the feature representations that a Deep-Q Network learns, but rather in some inner properties of the environment that is used as target task and that favors a Deep-Q Network to correctly synchronize its components.
As a proof of concept, we have created one ﬁnal Catch environment, called Catch-v4 , which is identical to Catch-v0 with the only difference being that a positive reward is returned to the agent only if it manages to catch ﬁve falling balls in a row.
We can see from the right image of Fig.
6 that a model trained from scratch (represented by the purple line) is not able to improve its policy over time at all, as the reward signal is probably too sparse for learning, whereas a Catch-v0 model is now able to yield positive transfer.
We hence believe that some environments, if combined with certain learning algorithms, are more prone to positive transfer than others, as was, e.g., the case for Fishing Derby and DQV-Learning where positive transfer was observed no matter what source task was used for pre-training.
This is not due to the representations that are learned by a pre-trained network but rather because of some speciﬁc dynamics within the target MDP M T .
[Figure]
Related Work & Conclusion
The closest research to the one presented in this work is certainly that of Farebrother et al.
[2018] and Tyo and Lipton [2020], who also studied the generalization properties of Deep-Q Networks in a model-free DRL context.
Our extensive experiments conﬁrm some of the preliminary claims that were made by the former about the potentially poor TL properties of Deep-Q Networks, but contradict the study of the latter, who suggested that ﬁne-tuning DRL agents results in positive transfer when moving from a simpler task to a harder.
While both works are certainly valuable, we also believe that their experimental results are not as thorough and on par with the ones of this study as they both considered a very limited number of RL problems (four and three respectively), therefore leaving the question "How transferable are Deep-Q Networks?" unanswered.
We believe that the answer to it is "barely" , but we also believe that their poor TL properties, as well as the learning dynamics identiﬁed in Sec. 3, are inherent to this family of algorithms only.
In fact, it is worth noting that several works describing the beneﬁts of TL in RL do exist (but they all differ from the study presented in this work):
Tirinzoni et al.
[2018] show that it possible to successfully transfer value functions across tasks, yet their work does not consider deep networks as function approximators but rather Gaussian mixtures.
Parisotto et al.
[2015] show that it can be beneﬁcial to ﬁne-tune a pre-trained DRL agent, but they consider multi-task learning and policy gradient algorithms as a way of pre-training.
Rusu et al.
[2016] also show that ﬁne-tuning can be beneﬁcial, but in the context of progressive networks and again of policy gradient techniques.
Similar conclusions for Actor-Critic algorithms can also be found in the works of Zhu et al.
[2017] and Chen et al.
Furthermore, Landolﬁ et al.
[2019] and Sasso et al.
[2021] show that ﬁne-tuning a pre-trained network can be beneﬁcial for DRL tasks, but for model-based RL approaches, which are again part of a family of techniques that is different from the ones analyzed in this work.
To conclude, we would like to stress out that despite the overall poor TL performance observed throughout this paper, positive transfer can nevertheless be obtained in a model-free DRL setup, and hope that this paper can serve as a solid starting point for the DRL community which is interested in designing general and transferable agents.
Acknowledgements
Matthia Sabatelli acknowledges the ﬁnancial support of BELSPO, Federal Public Planning Service Science Policy, Belgium, in the context of the BRAIN-be project.
He also wishes to thank Gilles Louppe and Antoine Wehenkel for their valuable feedback on preliminary versions of this work.
Lastly, one ﬁnal thought goes to Marco A. Wiering, his teacher; supervisor; colleague; but mostly beloved friend: ‘I will miss you deeply” . A Appendix
We hereafter provide additional information about how the model-free DRL algorithms considered throughout this paper were trained.
In Table 4 and 5, we provide the hyper-parameters that were used for the large scale empirical study presented in Sec. 1, while we then describe the DQN architecture used for the control experiments presented in Sec. 2.
[Table] [Table]
a two-hidden layer convolutional neural network is units the layer estimating the network