Paper: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
I have the same question about transfer results, how important is the factor of (1+\gamma)/2. Why does it make the bound substantially looser? Even then we are comparing upper bounds which are not tight, so what is the correct way to make sense of this result?

Author response:
*(**Q2**). I have the same question about transfer results, how important is the factor of (1+\gamma)/2. Why does it make the bound substantially looser? Even then we are comparing upper bounds which are not tight, so what is the correct way to make sense of this result?*

(**A2**.) Thank you very much for your insightful questions. To offer a comprehensive understanding of Theorems 3 and 4, we provide the intuition to understand the theorems, an illustration of the theoretical framework used to derive the results in this response.

**Firstly, we would like to clarify the intuition in understanding the bound with the dependence on $\gamma$.**  Intuitively, the performance of transfer learning is influenced by the knowledge gap between previous tasks and the current task. Recall that the goal is to maximize $E_{\pi_i^\star} \sum_{t}\gamma^t r_i(s_t, a_t)$ for a task $\mathcal{T}_i$. The knowledge gap can be attributed to two main factors: (i) the difference in reward functions between the current task and previous tasks, and (ii) the difference in the optimal policy (and thus the corresponding data distribution) between the current task and previous tasks. In the context of SF-DQN, factor (i) has minimal influence because $\phi$ can be considered as the reward when learning the successor feature, and the function $\phi$ remains fixed across all tasks. The parameter $\gamma$ affects the impact of factor (ii). When $\gamma$ is small or close to zero, the emphasis is primarily on immediate rewards, meaning that the influence of data distribution on the knowledge gap between previous and current tasks is minimal. However, as $\gamma$ increases, prioritizing future rewards, the data distribution becomes a significant factor affecting the final knowledge gap. Therefore, factor (ii) does not exhibit a significant difference when using SF-DQN or DQN, as the knowledge gap always exists. When $\gamma$ is small, the influence of factor (i) dominates the knowledge gap, resulting in a significant improvement in the transfer learning performance using SF-DQN compared to DQN. Conversely, when $\gamma$ is large, the knowledge gap derived from factor (ii) dominates, leading to no significant improvement when using SF-DQN over DQN in transfer learning settings.

**Secondly, this paper provides the first theoretical justification that SF-DQN outperforms DQN in transfer reinforcement learning settings.** Notably, since $\gamma$ is always smaller than 1, the factor $(1+\gamma)/2$ is consistently less than 1. As a result, the upper bound of transfer learning using SF-DQN is consistently superior to DQN. As far as we know, there are no comparable results in existing literature for transfer learning using DQN, let alone for SF-DQN, or the direct comparison between SF-DQN and DQN. While our analysis focuses only on the upper bound, no comparable theoretical results are observed in the existing literature.

**Thirdly, the improvement of SF-DQN arises from the decomposition of the Q-function into the successor feature function and the reward mapping function, which are absent in DQN.** Both proofs follow similar steps, except for bounding the difference between the generalized policy and individual policies, as illustrated in (67) for SF-DQN and (75) for DQN. Compared with (75), (67) in SF-DQN lacks the factor of $\frac{1}{1-\gamma}\cdot \phi_{\max}\cdot||w_{n+1}^\star-w_j^\star||_2$ due to our ability to learn the reward mapping and successor feature separately, mitigating the influence of changes in data distribution. Furthermore, all other constants remain consistent, as we follow the exact proof steps for Theorems 3 and 4. The proofs for Theorems 3 and 4 are included in Appendix C. 

**Finally, our theoretical results justified the observation of SF-DQN over DQN in transfer reinforcement learning in real applications.** Our manuscript (Appendix E.2 in the supplementary) presents enlightening empirical findings that are obtained based on real-data experiments using the SF-DQN and DQN. We showed that SF-DQN achieves a better performance than DQN when transferring learning from the source tasks to the target task. In addition, similar observations are presented in (Barreto et.al, 2017)  for SF-DQN and DQN.

Reviewer comment (Novelty):
The experiments seem to present results that seem obvious. If weights are initialized closer to optimal weights convergence is fast and if the weight of transfer task is similar to prior task the transfer is fast. This is also similar to bounds we have seen for successor features in previous works.

Author response:
*(**Q3**) The experiments seem to present results that seem obvious. If weights are initialized closer to optimal weights convergence is fast and if the weight of transfer task is similar to prior task the transfer is fast. This is also similar to bounds we have seen for successor features in previous works.*

(**A3**.) we would like to clarify that there are some fundamental differences between our work and (Barreto et al., 2017) although we share some similar observations. Here, we suspect that the previous works are referring to the representative work of successor features (Barreto et al., 2017). If the reviewers suggest comparing our work with other studies, we are open to providing further clarification.

**Firstly, previous work (Barreto et al., 2017) did not provide the theoretical justification or empirical justification for the case that the weights are initialized closer to optimal weights.** (Barreto et al. 2017) assume that mapping weights $w^star_i$ are known, and therefore they did not provide any conclusion related to the initialization of the weights. While in our manuscript, we assume $w^\star_i$ is unknown and we need to learn it. Therefore, there are no similar results that “weights are initialized closer to optimal weights convergence is fast” presented in (Barreto et al., 2017).

**Secondly, this paper provides both convergence analysis and sample complexity for successor feature learning, while previous work by Barreto et al. (2017) did not include such analyses.** The previous work by Barreto et al. (2017) did not include a convergence analysis. Additionally, in their analysis of transfer learning, Barreto et al. (2017) did not provide a comparison with DQN. Furthermore, the convergence analysis for SF-DQN is challenging due to the non-convex nature of the objective function involving deep neural networks, and existing learning theory primarily focuses on shallow neural networks.

**Finally, although these observations seem to be intuitive, our paper provides the first theoretical justification for these numerical observations.** To the best of our knowledge, our manuscript provides (i) the first convergence analysis of SF-DQN to the optimal Q-function with generalization guarantee, (ii) the first theoretical characterization of enhanced performance using GPI, and (iii) the first theoretical characterization of the superior performance with SF-DQN over DQNs.  As elaborated in Section 4.4, we encountered several challenges in deriving these results. These challenges involved extending the deep learning theory from the supervised learning setting to the successor feature Q-learning setting, as well as adapting the GPI theorem to the context of successor feature learning.

Reviewer comment (Theory):
On page 5, Equation 15, should the gradient for theta be wrt. the best action across all (tasks) successor features based Q function or just the gradient with best action across the successor feature based Q function of current task.

Author response:
*(**Q4**.) On page 5, Equation 15, should the gradient for theta be wrt. the best action across all (tasks) successor features based Q function or just the gradient with best action across the successor feature based Q function of the current task.*

(**A4**.) Thanks for pointing out our typo. The gradient of theta should be w.r.t. The best action across all successor feature-based Q-functions, and we have revised it accordingly in the paper. 

$$\Theta_i^{(t+1)}= \Theta_i^{(t)} - \eta_t\cdot\sum_{m\in\mathcal{D}_t} \Big(\psi(\Theta_i^{(t)};s_m,a_m) -\boldsymbol{\phi}(s_m,a_m,s_m^\prime) -\gamma \cdot \psi(\Theta_i^{(t)};s_m',a_m') \Big)$$

where $a_m^\prime = \pi_\beta(\max_{c\in[i]}\psi(\hat{\Theta}_c;s, a)^\top w_i^{(t)}|s_m^\prime)$.

Reviewer comment (Writing):
In equation 17, I dont believe the notations are correct or have been explained previously in the paper. What is P(s_tau\in .)? What does v_\tau stand for?

Author response:
*(**Q5**.) In equation 17, I don't believe the notations are correct or have been explained previously in the paper. What is P(s_tau\in .)? What does v_\tau stand for?*


**A5**. We apologize for any confusion caused by the notations. Equation 17 is derived from a standard assumption in TD learning with function approximation, and thus, we did not provide further clarification for it. The term  $P(s_\tau \in .)$ represents the probability measure of the state after $\tau$ steps following a Markovian chain. While there is no specific notation for $v_\tau$, we presume that the reviewer is referring to the notation $v^\tau$, which denotes $v$ raised to the power of $\tau$. Please inform us if you need further clarification. 

Equation 17 is introduced to control the statistical bias in the gradient updates when the samples are collected from a Markov observation model. Additionally, Equation 17 is a standard assumption in analyzing TD learning with function approximation, see e.g.,  Assumption 1 in (Zou et al., 2019) and Assumption 5.2 in (Xu & Gu et al., 2020) for Q-learning.

Reviewer comment (Reproducibility):
How is \phi obtained in all the experiments? I dont think \phi is trained in this work based on the exposition?

Author response:
*(**Q6**.) How is \phi obtained in all the experiments? I don't think \phi is trained in this work based on the exposition.*

(**A6**.) We apologize for omitting the description of obtaining $\phi$.

In the synthetic experiments detailed in Section 5, we derive $\phi$ based on task 1. Subsequently, for the remaining tasks, we maintain the fixed $\phi$ while generating the corresponding reward $r_i$ for the $i$-th task using equation (8) with a (randomly or specifically) generated reward mapping $w_i$.

In the Reacher environment experiments presented in Appendix E, we can theoretically compute the optimal $\phi$, which is the concatenation of the rewards for all the tasks considered in the environment. This aligns with the standard setup outlined in (Barreto et al., 2017).

Reviewer comment (Theory):
The Theorems do not seem to be correct. For instance, it does not seem possible that the size of a replay memory impacts the bound as it does given that it could all be filled with bad tuples.

Author response:
(**A1**.) Thanks for your insightful questions! The replay memory size is just one condition for ensuring convergence. **As you rightly pointed out, if all tuples are bad, then $1-C^\star$ will become smaller than $1$, leading to divergence.** 

**Firstly, $C^\star$ is corresponding to the fraction of bad tuples. $C^\star$ is proportional to the constant $C$ defined in Assumption 3.** $C$ captures that the data distribution generated by the behavior policy and optimal policy should be bounded, and this is proportional to the fraction of unfavorable tuples. Therefore, $C^\star$ is corresponding to the fraction of bad tuples.

**Secondly, $C^\star$ has an implicit upper bound.** Without delving into the mathematical details, this can be inferred from our summarized takeaway. As discussed in our first takeaway (T1) in Section 4.1, we concluded that "the generalization error of the learned Q-function scales on the order of $\frac{||w^{(0)}-w^\star||_2}{1-\gamma - N^{-1/2}-C^\star}$". It is evident that $C^\star$ has an upper bound of $1-\gamma-N^{-1/2}$ (otherwise the error bound is zero ). Without this upper bound, the estimation bound of the Q-function becomes meaningless. 

Finally, for more details, we recommend the reviewer check the [General Response (GR)](https://openreview.net/forum?id=s6bKLlF4Pe&noteId=5sCZCZJLVj).

Reviewer comment (Experiments):
The experiments do not follow all good practice (see questions below).

Author response:
(**A2**.) Thank you very much for pointing out our bad practice in presenting the numerical experiments. Following your suggestions, we have included the error bar (denotes the variance) in the numerical results, including Figures 1&2, and Table 2 in Section 5, and Figure 4 in Appendix E.2 in the supplementary materials. We provide the average and standard deviation over 5 seeds. Note that, due to the tuning of the experiment setup, some numerical values have changed, but the observations are consistent with the prior results. The codes have also been updated in the supplementary materials.

Regarding the normalized average reward decreasing in Figure (2) (for both methods), we found out this is due to our choice of learning rate and epsilon annealing method when implementing respective algorithms. We fine-tuned the learning rates and epsilon annealing for all the methods and rerun all the experiments. Specifically, we tuned down the learning rates from 1 to 0.1 for all methods, which improved the stability in the convergence. Furthermore, we changed the epsilon annealing to start with $\epsilon=0.5$ and rapidly changed it to $\epsilon=0$ by iteration 200, which improved the fast and stable convergence of the methods. Since the choice of epsilon was not the main focus of our analysis, this was not carefully chosen in the previous experiments. However, it was observed this careful choice was needed for the stability of the convergence of all the methods.

Reviewer comment (Writing):
There are some typos, e.g. in Assumption 1 "such that minimizes (12) for".

Author response:
(**A4**.) Thanks for pointing out our typo! The corresponding sentence has been revised as “There exists a deep neural network with weights $\Theta_i^\star$ such that it minimizes (12) for the $i$-th task, i.e, $f(\Theta_i^\star) = 0$.” Following your suggestions, we have proofread the manuscript afterward and corrected several other typos.

Reviewer comment (Evaluation):
Why is there no variance in the results reported for the experiments?

Author response:
(**A2**.) Thank you very much for pointing out our bad practice in presenting the numerical experiments. Following your suggestions, we have included the error bar (denotes the variance) in the numerical results, including Figures 1&2, and Table 2 in Section 5, and Figure 4 in Appendix E.2 in the supplementary materials. We provide the average and standard deviation over 5 seeds. Note that, due to the tuning of the experiment setup, some numerical values have changed, but the observations are consistent with the prior results. The codes have also been updated in the supplementary materials.

Regarding the normalized average reward decreasing in Figure (2) (for both methods), we found out this is due to our choice of learning rate and epsilon annealing method when implementing respective algorithms. We fine-tuned the learning rates and epsilon annealing for all the methods and rerun all the experiments. Specifically, we tuned down the learning rates from 1 to 0.1 for all methods, which improved the stability in the convergence. Furthermore, we changed the epsilon annealing to start with $\epsilon=0.5$ and rapidly changed it to $\epsilon=0$ by iteration 200, which improved the fast and stable convergence of the methods. Since the choice of epsilon was not the main focus of our analysis, this was not carefully chosen in the previous experiments. However, it was observed this careful choice was needed for the stability of the convergence of all the methods.

Reviewer comment (Evaluation):
Why does SFDQN (GPI) have a normalized average reward that decreases with the number of episodes? Given the theory that the paper describes, this should slowly converge towards the optimum.

Author response:
(**A2**.) Thank you very much for pointing out our bad practice in presenting the numerical experiments. Following your suggestions, we have included the error bar (denotes the variance) in the numerical results, including Figures 1&2, and Table 2 in Section 5, and Figure 4 in Appendix E.2 in the supplementary materials. We provide the average and standard deviation over 5 seeds. Note that, due to the tuning of the experiment setup, some numerical values have changed, but the observations are consistent with the prior results. The codes have also been updated in the supplementary materials.

Regarding the normalized average reward decreasing in Figure (2) (for both methods), we found out this is due to our choice of learning rate and epsilon annealing method when implementing respective algorithms. We fine-tuned the learning rates and epsilon annealing for all the methods and rerun all the experiments. Specifically, we tuned down the learning rates from 1 to 0.1 for all methods, which improved the stability in the convergence. Furthermore, we changed the epsilon annealing to start with $\epsilon=0.5$ and rapidly changed it to $\epsilon=0$ by iteration 200, which improved the fast and stable convergence of the methods. Since the choice of epsilon was not the main focus of our analysis, this was not carefully chosen in the previous experiments. However, it was observed this careful choice was needed for the stability of the convergence of all the methods.

Reviewer comment (Writing):
The first half of the paper is confusing as the authors seem to conflate optimal value/policy at the task level with some generic $Q^*$ and $\pi^*$. It was hard to follow and figure out what they mean by the notations and/or possible notational mistake.

Author response:
Thank you for your valuable suggestions. We have included a brief introduction to our problem formulation before introducing the preliminaries of $Q$-learning. Specifically, we add a paragraph at the beginning of Section 2 as “in this paper, we address the learning problem involving multiple tasks $\\{ T_i \\}_{i=1}^n$ and aim to find the optimal policy $\pi_i^\star$ for each task $\mathcal{T}_i$. We begin by presenting the preliminaries for a single task and then elaborate on our algorithm for learning with multiple tasks in the following section.” All notations have been modified to follow the same conventions as the subsequent sections to avoid confusion.

Specifically, we include this table to clarify some of the important notations in this paper.
| Notation | Description |
| ----------- | ----------- |
| $Q_i^\star$ | The optimal Q-function for the $i$-th task. |
| $\pi^\star_i$ | The optimal policy for the $i$-th task. |
| $\phi$ | Universal transition features. |
|$w_i$|the reward mapping for $i$-th task.|
|$\psi^\star_i$|The optimal successor feature for the $i$-th task.|
|$\psi_i(\Theta_i)$|The function parametered by $\Theta_i$ for learning the successor feature of the $i$-th task.|
|$\Theta_i^\star$|The ground truth for the successor feature, namely, $\psi_i(\Theta_i^\star)=\psi_i^\star.$|
|  |  |

Reviewer comment (Theory):
As an important point, note that the GPI theorem induces improvement for the case of several policies but **the same** reward (i.e., only one task). If there are multiple rewards, then maximum of their corresponding value functions is simply meaningless and does not represent any physical concept regarding a new reward function (see the comments bellow). [This is OK when using SF as the $w_i$ corresponds to the current task and only $\psi_j$ comes from some previous task.]

Author response:
We appreciate your valuable suggestions, and we apologize for not clearly describing our motivations for using GPI in our settings.

Firstly, as you mentioned, the GPI theorem brings about improvements in cases where multiple policies share the same reward. In our setting, the learning process of the optimal $Q$-function is linked with learning the optimal successor features. When learning these successor features $\psi_i$, we treat $\phi$ as the reward. Given that $\phi$ remains consistent across all tasks, applying GPI across the policies learned from different tasks becomes a viable strategy in learning the new successor feature, thereby enhancing the overall performance in learning the new Q-function. Motivated by the success of GPI, we have introduced the GPI operator within the framework of successor features. 

Additionally, existing work (Barreto et al., 2017) empirically verified the efficiency of GPI in the context of successor features and deep reinforcement learning. Furthermore, we have provided theoretical justification (Theorem 2) for the efficiency of GPI in our proposed algorithm SF-DQN. Regarding your examples, if the reward is arbitrarily designed, we agree with you that $\max_{i=1,2}Q_i$ may yield suboptimal actions in all states. However, since the rewards across different policies are connected with $\phi$, $\max_{i=1,2}Q_i$ still performs well in our setting.

Finally, to enhance the clarity of our motivation for incorporating GPI, we have revised the last three lines of page 4.

“When the reward is consistent across different policies, such as ${\pi_i}{i=1}^n$, and considering that the optimal Q-function represents the maximum across the entire policy space, the maximum of multiple Q-functions corresponding to different policies, $\max{1\le i \le n} Q^{\pi_n}$, is expected to be closer to $Q^\star$ than any individual Q-function, $Q^{\pi_i}$. In this paper, the parameter $\phi$ in learning the successor feature is analogous to the reward in learning the Q-function. Since $\phi$ remains constant across different tasks, this analogy has motivated the incorporation of GPI in our setting, even when the rewards change.”

Reviewer comment (Writing):
There were a number of other typos (a proof read aside from the points mentioned above is strongly suggested).

Author response:
Thank you for bringing the typos to our attention. We have proofread the paper and corrected several errors.

Reviewer comment (Theory):
If you have multiple tasks, then what do you mean by $Q^*$ and $\pi^*$? In fact, both $\pi^*$ and $Q^*$ are not well-defined unless the reference for optimality is specified. For instance, you may define a total reward as the summation of all the task-level rewards, whose $\pi^*$ and $Q^*$ would be different from the case where a total reward is a *weighted* sum of task-based rewards or some non-linear function of them.

Author response:
We apologize for any confusion arising from the notations, which we suspect may be partially caused by the inappropriate use of the term 'multi-task learning' in the abstract. When employing the phrase 'multi-task learning,' our goal is to execute Q-learning for multiple tasks. We later realize that ‘multi-task learning’ is not a proper word because the conventional understanding of 'multi-task learning' is to learn a single policy for multiple tasks. In this paper, instead of learning a single Q-function or policy for all tasks, we will learn a separate $Q$-function for each individual task, which is closer to a continual learning setup. Our algorithm aims to provide a quick adaptation from one task to another new task utilizing the learned knowledge from previous tasks (transfer reinforcement learning).

For the $i$-th sub-task, we use $Q_i^{\star}$ to denote the optimal Q-function for $i$-th subtask.  We use $\pi_i^{\star}$ to denote the optimal policy for $i$-th sub-task. For the $i$-th sub-task, our algorithm returns a Q-function, denoted as $Q_i$.

Reviewer comment (Theory):
Eq (13) is not a policy improvement operator (PI is to take an argmax of the current value to induce a new policy).

Author response:
Thank you for pointing out our improper definition, we have updated (13) as 

$$\pi(a|s) = \arg\max_{a} \max_\{1\le i \le n+1\} \hat{\psi}_i(s,a)^\top {w}\_{n+1}^\star.$$

Reviewer comment (Writing):
Table 1 --> what is $\phi_{i}(\Theta^{*}_{i})$ ? $\phi$ is supposed to be the feature vector.

Author response:
We apologize for the typo. $\phi(\Theta^\star_i)$ should be $\psi(\Theta^\star_i)$ in Table 1. $\psi(\Theta^\star_i)$ represents the optimal success feature for the $i$-th task.

Reviewer comment (Theory):
Assumption 3 is unclear. Note that $\pi(a|s) \in [0,1]$, Hence, for sufficiently large $C$ this assumption is always true! Am I missing something?

Author response:
Thanks for your reflective questions! $C$ has an implicit upper bound that cannot exceed a value less than $1$. As discussed in our first takeaway (T1) in Section 4.1, we conclude that "the generalization error of the learned Q-function scales on the order of $\frac{||w^{(0)}-w^\star||_2}{1-\gamma - N^{-1/2}-C^\star}$," where $C^\star$ is, in fact, a linear function of $C$. It's evident that $C^\star$ has an upper bound in the order of $1-\gamma-N^{-1/2}$. Otherwise, the estimation bound of the Q-function becomes meaningless. Consequently, $C$ also has an upper bound. We apologize for not clearly explaining the relationship between $C$ and $C^\star$ and for not specifying the upper bound of $C^\star$. For more details, we recommend the reviewer check the [General Response (GR)](https://openreview.net/forum?id=s6bKLlF4Pe&noteId=5sCZCZJLVj).

Reviewer comment (Theory):
Is (20) a s fair assumption? For most cases, $K$ would be a large number, so it looks like that the initial approximation of $\psi_1$ must be already very good! Perhaps a better presentation could be that there has to be some pre-training for $\psi_1$ before using this algorithm?

Author response:
Thanks for your insightful suggestion! A presentation of the known pre-trained model for $\phi_1$ is indeed a good suggestion, and we will incorporate it into the revised paper. In addition, it is noteworthy that within the NTK framework for supervised learning using neural networks [Zhu et al. 2019 & Arora et al. 2019], a good convergence point—almost as good as the ground truth—is consistently found near random initializations. We believe this observation underscores the practical validation of equation (20). Most importantly, we would like to further clarify that **our initialization assumption in (20) is the state-of-the-art practice in the theoretical analysis of DQNs.**

Specifically, due to the significant convexity of the DQN learning problem, existing DQN results with non-linear activations require various initialization or objective function landscape assumptions to prove convergence, as seen in (i) two-layer assumptions for characterizable Eluder dimension, (ii) the achievability assumption of global optimal in Besov space framework, and (iii) the assumption of a good initialization near the ground truth in analyzing the convergence [Yang et al., 2020; Cai et al., 2019; Xu&Gu, 2020; Du et al., 2020; Nguyen-Tangetal., 2022]. This paper concentrates on deep neural networks, aligning with the conditions outlined in (ii) and (iii). The initialization assumption, as described in equation (20), shares similarities with (iii) and is less stringent than (ii).

Following your suggestion, we have added a paragraph to deliver a better understanding of (20). Specifically, it says “Note that (20) requires a good initialization. Firstly, it is still a state-of-the-art practice in analyzing Q-learning via deep neural network approximation. Secondly, according to the NTK theory, there always exists some good local minima, which is almost as good as the global minima, near some random initialization. Finally, such a good initialization can also be adapted from some pre-trained models. ” 


[Zhu et al. 2019] Allen-Zhu, et al. "Learning and generalization in overparameterized neural networks, going beyond two layers." NeurIPS, 2019.

[Arora et al. 2019] Arora, Sanjeev, et al. "Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks." In ICML, 2019