From: ICLR.cc/2023/Conference/Paper5091/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper5091/Reviewer_XRX1
title: Re: Authors' response
comment: I would first like to thank the authors for a thorough response and a hard work on paper improvements. It happens that while there are definitive improvements which made the submission stronger, my recommendation, after reading the response and the comments by other reviewers, still remains the same, as there are concerns still  to be addressed:

- from the discussion with Reviewer Yp6v: 

**"This can be extrapolated by Figure 9 where the NTK for a = 1 and b = 100 is clearly sparser than the one for a = 100 and b = 1. We should note that the performance of the proposed method does not vary as much for different coefficient ratios on the rest of the environments of the DMC (except for the Quadruped environments). "**

The results at Figure 9 demonstrate that only for the Quadruped environment; perhaps it could be solved by adding the detailed empirical justification in the appendix for multiple environments

-(still) limited experimental analysis: the inclusion of the pixel-based tasks evaluation is definitely a step in the right direction. However, to show experimentally that the pretraining method helps improve the performance of reinforcement learning methods, there should be analysis with more environments; that would help show the scenarios where it is and is not advantageous. This might also help further illustrate (and justify) the effects described in your rebuttal: "The second point has to do with the fact that, sometimes, the improvement in the pretrained models “kicks in” after the mid-phase of training". I find that the explanation could become a very important aspect of the narrative, but then it needs more empirical justification.

- I also find that the following comment from R Yp6v needs to be addressed: "There are also several works showing RL attaining performance much higher than 130 on Humanoid run (e.g., [5]) even working with pixel observations [1]. Hence, I would suggest removing the claim that the current state-of-the-art performance in this environment is 130 and that the Tanh baseline performs reasonably well."

From: ICLR.cc/2023/Conference/Paper5091/Reviewer_Yp6v
title: Further comments
comment: I would like to thank the authors for acknowledging the provided feedback. While I think that the current version of the work is a step in the right direction, I believe there is still significant room for improvement and I will try to provide additional feedback addressing some of the main unsatisfactory points in the authors' rebuttal:

I appreciate the inclusion of some additional comparisons. However, I found the quality of the new empirical analysis to be quite poor and unable to provide considerable additional insights. 

I believe it would be important to compare with the Fourier features approach on all proprioceptive and pixel-based environments, effectively treating it as an additional main baseline other than the SAC algorithm. Moreover, even in the currently considered environments, the performance of the Fourier approach appears to be considerably lower than the original performance reported by Yang et al. 2022 [3] (especially for Hopper hop). Are those results collected with the original code shared in [3]? Regardless, I feel this difference in results should be noted and directly addressed in the paper. 

I also found the newly added claims about the limitations of the Fourier feature approach to lack solid motivation. For instance, the paper appears to argue for the superiority of pre-training, by claiming that in the Fourier features approach "small deviations from the optimal variance can result in reward collapse due to noisy value estimates." I feel that such a statement about prior work should be backed by either thorough empirical evidence or a formal argument, both of which are currently missing. 

Also, the FF + pretraining baseline on Hopper hop (Figure 7) appears to underperform as compared to only using pre-training. This could suggest that the two approaches might not be compatible, contradicting the claim that "the Fourier preprocessing and the proposed pretraining can work in conjunction with one another". To provide more meaningful and interpretable insights about the compatibility of these approaches, I would suggest evaluating this baseline on more than a single environment and also including the original 'pretraining' performance curves in the Figure. 

The newly-added pixel-based results also appear to be very limited, considering only a subset of the easier DMC environments and only evaluating against a SAC baseline for a low number of episodes. I would suggest the authors incorporate their pre-training with modern pixel-based RL algorithms, such as DrQv2 [1] (as suggested in my original review and done in the related prior work from Yang et. al 2022 [3]). This would allow for a direct comparison of the results with the alternative Fourier features approach. Furthermore, I would also suggest providing results on the same environments used in the proprioceptive experiments to improve their interpretability.

I also believe the statement "We should note that the performance of the proposed method does not vary as much for different coefficient ratios on the rest of the environments of the DMC (except for the Quadruped environments)" is not properly substantiated, since the authors only provide results systematically varying the coefficient ratios on Quadruped.

There are also several works showing RL attaining performance much higher than 130 on Humanoid run (e.g., [5]) even working with pixel observations [1]. Hence, I would suggest removing the claim that the current state-of-the-art performance in this environment is 130 and that the Tanh baseline performs reasonably well.

As of this comment's date, there is no access to the author's implementation. I believe it would have been important for the authors to provide access to their code early in the review process for enabling the reviewers to assess the implementation's validity and reproducibility.


Additional references

[5] Hansen, Nicklas, Xiaolong Wang, and Hao Su. "Temporal Difference Learning for Model Predictive Control." arXiv preprint arXiv:2203.04955 (2022).

From: ICLR.cc/2023/Conference/Paper5091/Authors
title: Answers to Comments by Reviewer XRX1 (0)
comment: We would like to thank reviewer XRX1 for the thorough and detailed review of our work. The points raised by the reviewer prompt us to reassess certain parts of the contribution and to dig deeper and find new aspects that we have included in our latest uploaded version. 
The additions in comparison to the initial version are highlighted in the latest version in magenta. We provide our answers to the reviewer’s questions below, along with references to the additions in our newly uploaded version. 

“While the idea is incrementally developed … it would be good to show its advantage/synergy in relation to other strategies”

We found that this is an important point and we tried to address it in the following way. We strongly believe that our work is tightly related to the works in [1,2], since both methods (ours and the Fourier preprocessing) strive to impose a certain structure on the underlying NTK of the critic. We added a new subsection 4.5 in our new updated version of the paper. The subsection is entitled “Comparing with the Fourier Features”. In the new subsection we provide the comparison of our proposed pretraining scheme with the Fourier features preprocessing proposed by [1], on 4 environments of the DMC. The comparison is illustrated in Figure 6 of the revised paper. It is prevalent that the proposed pretraining performs on par with the Fourier preprocessing when it comes to the “Walker-run” and the “Quadruped-walk” environments. When it comes to the “Quadruped-run”, the Fourier preprocessing performs noticeably better than our proposed preprocessing. On the other hand, the proposed pretraining performs better than the Fourier feature preprocessing when it comes to the “Hopper-hop” environment of the DMC. The comparison with the Fourier features approach enables us to clarify the effect of hyperparameters to some further extent.  In particular, the performance of the proposed pretraining, which is illustrated in Figure 6, corresponds to the same set of hyperparameter values for all environments. This is testament to the fact that even though there is variability between the performances of the proposed approach for different hyperparameter values, there is a set of values that generalizes gracefully across the tasks of the DMC domain. This is important for two reasons. The first reason is that the same principle holds true for the SAC baseline itself. In particular, the SAC baseline introduces a number of hyperparameters (e.g., the temperature coefficient of the policy) that can be tuned per environment, but there is a set of values for all hyperparameters that generalizes for all DMC environments. The second reason is that this is not the case for the Fourier features approach [1] where the variance of the kernel elements at initialization should be tuned separately for each environment. Nonetheless, we acknowledge the power of the Fourier preprocessing which, when tuned carefully, provides significant speed of convergence and stability. Another interesting finding that we include in the new version of our paper (again in subsection 4.5), is that the Fourier preprocessing and the proposed pretraining are not orthogonal contributions and can work in conjunction with one another (as pointed out correctly by the reviewer). In particular, the performance of the Fourier preprocessing on the “Hopper-hop” environment is unstable (even after being tuned carefully in terms of the variance of the kernel elements at initialization). When we combine the Fourier features with our proposed preprocessing, the resulting SAC algorithm performs stably on the “Hopper-hop”. This is illustrated in Figure 7 of our newly uploaded version of the paper.

From: ICLR.cc/2023/Conference/Paper5091/Authors
title: Answers to Comments by Reviewer XRX1 (1)
comment: “The experimental results are limited…. so the question about hyperparameter sensitivity arise (see comment 4 below)”.

In Figure 6 of the latest upload we provide the comparison of our proposed approach with the Fourier features approach in [1]. The results of our unsupervised pretraining method, when it comes to Figure 6, correspond to the same set of hyperparameters (=0.0001, a = 1, b=1) for all environments. As can be seen by Figure 6 , the proposed pretraining with fixed hyperparameter values performs well across the depicted environments and compares well against the Fourier features approach. This is attestation for the fact that, even though the proposed pretraining exhibits variability when it comes to the performance for different values of the hyperparameters (subsection 4.6), there is a set of hyperparameters that generalizes across the DMC environments. This is not the case for the Fourier features approach, where the variance of the Fourier Kernel elements at initialization should be tuned separately for each environment and the values are significantly different for different tasks [1]. Additionally, we provide the performance of our approach on image-based tasks of the DMC in subsection 4.3 of the latest version. In particular, we apply our approach on the pixel SAC provided by [3]. The results are depicted in Figure 3 of the updated version. As can be seen by Figure 3, our proposed approach extends well on image-based tasks and critics with Convolutional layers in the sense that it provides noticeable improvements in comparison to an unpretrained baseline. The new subsection 4.3 touches on the effect of the hyperparameters. In particular, the performance of our pretraining, depicted in Figure 3, corresponds to the same hyperparameter values for all environments. This also points in the direction that, even though there is a certain degree of hyperparameter sensitivity (see subsection 4.6 of the updated version), one can find hyperparameter values that result in good performance for multiple environments.

From: ICLR.cc/2023/Conference/Paper5091/Authors
title: Answers to Comments by Reviewer XRX1 (2)
comment: 
“There is only limited improvement in Figure 1 despite … do the authors see any reasons behind it?”

This is a great question that raises two distinct points. We would like to address the two points separately.
The first point made by the reviewer is that there is limited improvement in Figure 1 (comparison of our proposed pretraining with an unpretrained SAC baseline). We agree that there is small improvement in terms of achieved per-episode reward at the end of training for most of the environments. On the other hand, our incentive was to primarily improve sample efficiency and stability during training (which is a major issue that inhibits the real-world deployment of value-based deep RL systems). On that front, we feel that the proposed pretraining achieves a considerable improvement in stability and convergence speed. We provide the plot that corresponds to the “Quadruped-walk” environment in Figure 1 as a representative example. The unpretrained SAC baseline reaches (on average) the mark of 800 per-episode reward at about 600 training episodes. On the other hand, the SAC with the proposed pretraining reaches the 800 per-episode reward mark at about 300 training episodes. We also refer to the works by [4] and [1] that also target stability and speed of convergence and provide similar results (similar in the sense that the respective proposed approaches provide improvements primarily in the context of stability and sample efficiency and not very significant increase in per-episode reward at the end of training).

The second point has to do with the fact that, sometimes, the improvement in the pretrained models “kicks in” after the mid-phase of training. This is an excellent remark that revolves around the trade-off between generalization and bias. Our explanation is as follows. By decreasing the off-diagonal elements of the NTK we impede generalization (how the update using a set of state-action pairs propagates to the value estimation of other state-action pairs). Therefore, by starting with small off-diagonal elements we incentivize the critic to not generalize aggressively at the beginning of training. This results in slowing down the effect of updates at the beginning (at least in some cases) across unseen state-action pairs. The modest generalization that might “hurt us” at the beginning is what enables stability down the road and allows the learning of high frequency components of the value function (this is also hinted at in [1]).  The effect of this phenomenon is more prevalent when we take a look at the performance of the SAC baseline without the pretraining both for the Tanh MLP and the ReLU MLP. We refer to Figure 11 of the new updated version (Appendix A.2) where we provide the comparison between the Tanh MLP and the ReLU MLP critic both with and without the pretraining. We encourage the reviewer to focus on the unpretrained versions. It is known that the Tanh MLP generalizes more aggressively than the ReLU MLP. It is visible, in most cases, that the Tanh MLP critic “kicks in'' faster than the ReLU MLP critic, but as the training progresses, the Tanh MLP gets “stuck” and behaves more unstably, in contrast to the ReLU MLP that finally gets to higher rewards (We refer only to unpretrained versions). The same effect makes the pretrained versions show their effect, sometimes, later during training. In general, we feel that approaches that provide tools that make explicit or implicit trade-offs between generalization and bias are very important for Value-based RL and the aforementioned concept should be taken into consideration when designing RL algorithms. This is one of the overarching messages of the proposed work.

From: ICLR.cc/2023/Conference/Paper5091/Authors
title: Answers to Comments by Reviewer XRX1 (3)
comment: 

“Section 4.2: the reviewer is not sure .. the setting of supervised training of deep learning architectures”

We agree on the fact that the Tanh MLP is an overall suboptimal choice for the environments of the DMC Domain. Our effort to showcase the improvement that is induced by the proposed pretraining on the Tanh MLP critic is not to claim the suitability of the Tanh MLP as a parameterization of the critic. Our aim is to claim the efficacy and generality, to some extent, of our proposed pretraining. In essence, we want to showcase that if one is able to impose structure on the underlying NTK, they can even supersede, to some extent, the general properties of the function approximator class. This is prevalent in Figure 11 of our newly uploaded version (Appendix A.2), where the Tanh MLP with the pretraining can even surpass the ReLU MLP critic for the “Quadruped-run” (which is a generally difficult environment). In fact, we feel that the Tanh MLP experimentation of the paper has the potential to constitute motivation for future works on critic initialization schemes, since it clearly demonstrates the importance of initialization, which is somewhat neglected in the existing literature.

“Section 4.4: the evidence .. e.g., with some sort of grid search over hyperparameter space”

We refer to Figure 6 of the new version of the paper where we compare our proposed pretraining with the Fourier features preprocessing. The performance of our proposed pretraining, which is illustrated in Figure 6, corresponds to the same set of hyperparameter values for all environments. This is testament that there is such a set of hyperparameters that performs well across all tasks. The same principle applies in Figure 3 of the new version where the proposed approach is compared with an unpretrained baseline on pixel-based DMC tasks. We employ the same set of hyperparameter values for all depicted environments and the pretraining provides improvements across all depicted tasks in comparison to the unpretrained baseline.

From: ICLR.cc/2023/Conference/Paper5091/Authors
title: Answers to Comments by Reviewer XRX1 (4)
comment: Answers to comments on clarity and quality:
 
1) We tried to address the comment about the references. We checked to make sure that we use the correct bib reference for all cited works. Unfortunately,  we feel that it is the template itself that presents certain papers in the format that the reviewer mentions. For example the Tassa et al (2018) reference is formatted in the same fashion in [1] as is in our paper.

2) We agree with the reviewers remark on the distinction between the value function and the Q function. We contemplated the title and formalism of the paper for a long time before submission. In the end, we decided to use the term value function (therefore value NTK, neural value approximation, value learning etc) due to the fact that we feel that there is an intrinsic connection between our work and the work in [1]. In [1], the authors use the term value function and value approximation (the title of the paper refers to neural value approximation) extensively. In that sense, we chose  the term value (instead of Q) to avoid confusing the interested reader that is likely to read both papers on the topic. We would like to thank you for bringing this up into our attention during the rebuttal. To address this, we have added the following phrase in the last paragraph of the Background section to clarify the chosen formalism to the interested reader: “In addition, we use the term "value function" to refer to the function that depends both on the action and the state (Equation 1) and not the function where the actions are marginalized out Schaul et al, 2015”

3) 

4) We refer to Figure 6 of the new version of the paper where we compare our proposed pretraining with the Fourier features approach in [1]. The performance of our proposed pretraining corresponds to the same hyperparameter values for all environments. This is a testament to the fact that there exists a set of hyperparameter values for our proposed pretraining that generalizes across the DMC domain. Same principle applies for Figure 3 of the updated version where we apply the proposed pretraining on pixel-based tasks. We use the same hyperparameter values for all depicted environments.

5) Even though we do not have a principled way to guide the process of finding the best hyperparameter values for the task at hand, we feel that the tuning of the hyperparameters should be executed under the intuition that pertains to the generalization-bias trade-off that is described in section 4.6 of the updated version (section 4.4 of the initial version). We also refer to Figure 6 of the updated version where we provide the performance of the proposed pretraining for the same set of hyperparameter values for all environments (also Figure 3 for the pixel-based tasks.

6) We agree with the reviewer on the fact that since we closely follow the analysis by [4], it seems natural to compare with their approach.On the other hand, even though we deeply appreciate the theoretical results by [4] and rely on them quite a bit for the motivation of the work, we feel that the best approach to compare with and train in conjunction with, is the proposed approach by [1] (Fourier features). Both proposed approaches ([1] and [4]) are grounded in the same analysis ([4]) and operate under the same incentive (to impose a certain structure on the underline NTK that adheres to sufficient conditions for convergence). The proposed approach by [1] is more recent and performs better, to the best of our knowledge, than the approach by [4]. In addition, the approach proposed by [4] is significantly more computationally intensive than the approach by [1] (solves a minimization problem at every update step). Motivated by the aforementioned thought process we compare with [1] (subsection 4.5) and not [4]

[1]  Yang, Ge, Anurag Ajay, and Pulkit Agrawal. "Overcoming the spectral bias of neural value approximation." arXiv preprint arXiv:2206.04672 (2022).

[2] Brellmann, David, Goran Frehse, and David Filliat. "Fourier features in reinforcement learning with neural networks." (2021).

[3] Yarats, Denis, et al. "Improving sample efficiency in model-free reinforcement learning from images." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.

[4] Achiam, Joshua, Ethan Knight, and Pieter Abbeel. "Towards characterizing divergence in deep q-learning." arXiv preprint arXiv:1903.08894 (2019).

From: ICLR.cc/2023/Conference/Paper5091/Authors
title: Answers to Comments by Reviewer iUkV (0)
comment: We would like to thank reviewer iUkV for the thorough and detailed review of our work. We tried to address the points made by reviewer iUkV on the new uploaded version of our paper. In that process, we feel that we have strengthened the quality and clarity of our contribution significantly. The additions in comparison to the initial version are highlighted in the latest version in magenta. We present our answers to the points made below, along with references to new additions in the latest version.

1) We agree with the reviewer that, by not comparing our method with the Fourier preprocessing, we are on the verge of grave omission. In our new version, we have added section 4.5, entitled “Comparing with the Fourier features”.  In the aforementioned subsection we provide Figure 6 that demonstrates the comparison between our proposed pretraining and the Fourier preprocessing approach of [1], for 4 environments of the DMC. It is prevalent that the proposed pretraining performs similarly to the Fourier preprocessing when it comes to the “Walker-run” and the “Quadruped-walk” environments. When it comes to the “Quadruped-run”, the Fourier preprocessing results in noticeably better performance than our proposed pretraining method. On the other hand, the proposed pretraining results in better performance than the Fourier feature preprocessing on the “Hopper-hop” environment of the DMC. The comparison with the Fourier features approach enables us to clarify the effect of the hyperparameter values, to some further extent.  In particular, the performance of the proposed pretraining that is illustrated in Figure 6 corresponds to the same set of hyperparameter values for all environments. This is testament to the fact that, even though there is some discrepancy between the performances of the proposed approach for different hyperparameter values (subsection 4.6 of the new version), there is a set of values that generalizes well across the tasks of the DMC domain. This is important for two reasons. The first one is that the same principle holds true for the SAC baseline itself. In particular, the SAC baseline introduces a number of hyperparameters (e.g., the temperature coefficient of the policy updates) that can be tuned per environment, but there is a set of values for all the hyperparameters that generalizes well across all DMC tasks. The second reason is that this is not the case for the Fourier feature approach [1] where the variance of the kernel elements at initialization should be tuned separately for each environment. Nonetheless, we acknowledge the power of the Fourier preprocessing which, when tuned carefully, provides significant speed of convergence and stability. Another interesting finding that we include in the new version of our paper (again in subsection 4.5), is that the Fourier preprocessing and the proposed pretraining can work in conjunction with one another. In particular, the performance of the Fourier preprocessing on the “Hopper-hop” environment is unstable (even after being tuned carefully in terms of the variance of the kernel elements at initialization). When we combine the Fourier features with our proposed preprocessing, the approach performs stably on the “Hopper-hop”. This is a testament for the fact that our approach and the Fourier preprocessing [1,2] are not orthogonal contributions, but they can be jointly employed to improve neural value approximation. This is illustrated in Figure 7 of our newly uploaded version of the paper.

From: ICLR.cc/2023/Conference/Paper5091/Authors
title: Answers to Comments by Reviewer iUkV (1)
comment: 2) We understand the reviewer’s point on the motivation behind the Tanh MLP experiments. We agree on the fact that the Tanh MLP is an overall suboptimal choice for the environments of the DMC Domain. Our efforts to showcase the improvement that is induced by the proposed pretraining on the Tanh MLP critic is not to claim the suitability of the Tanh MLP as a parameterization choice of the critic. Our aim is to demonstrate the efficacy and generality, to some extent, of our proposed pretraining. In essence, we want to showcase that if one is able to impose structure on the underlying NTK, they can even supersede, to a certain degree, the general properties of the function approximator class. This is prevalent in Figure 11 of our newly uploaded version (Appendix A.2), where the Tanh MLP with the pretraining can even surpass the ReLU MLP critic for the “Quadruped-run” (which is a generally difficult environment). In fact, we feel that the Tanh MLP experiment subsection of the paper can potentially incentivize future works on initialization schemes. 

3) We are in complete agreement with the reviewer’s opinion that the proposed pretraining should be tested on image-based tasks in order to validate its general efficacy. In this direction, we added subsection 4.3, which is entitled “Pixel-based DMC tasks”, in our latest uploaded version of the paper. In the aforementioned subsection, we provide the performance of our proposed pretraining on a number of DMC tasks where the state is not proprioceptive, but consists of a concatenation of multiple raw image frames. We combine our approach with the pixel SAC implementation by [3] that employs a critic which consists of a Convolutional Neural Network (CNN) backbone feature extractor and a ReLU MLP, both connected sequentially. We provide Figure 3 that compares the performance of the proposed pretraining with an unpretrained baseline. The proposed approach provides consistent improvements in terms of stability and control performance for all depicted environments. Another nuance that we want to point out is that the performance of the proposed pretraining in Figure 3 (of the new uploaded version) corresponds to the same hyperparameter values for all depicted environments. 

4) We would like to thank the reviewer for pointing out the nuance about the pretraining part in the Figures. The pretraining phase is not included in the Figures that demonstrate the performance of the pretraining approach. We included this clarification in the Figure captions (we explicitly mention that the pretraining phase is not illustrated in the plots).We want to stress that the overall pretraining time is significantly less that the time consumed by the main training process. This was the motivation behind the zeroth order approximation of the gradient in the introduction of the surrogate objective.

[1] Yang, Ge, Anurag Ajay, and Pulkit Agrawal. "Overcoming the spectral bias of neural value approximation." arXiv preprint arXiv:2206.04672 (2022).

[2] Brellmann, David, Goran Frehse, and David Filliat. "Fourier features in reinforcement learning with neural networks." (2021).

[3] Yarats, Denis, et al. "Improving sample efficiency in model-free reinforcement learning from images." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021

From: ICLR.cc/2023/Conference/Paper5091/Authors
title: Answers to Comments by Reviewer 1sio (1)
comment: We would, again, like to thank reviewer 1sio for the encouraging comments and for pointing out the comparisons with model-based RL and self-supervised learning approaches. We acknowledge their merits and we will follow their suggestions in our future research.

Finally, we would like to inform the reviewer on the new updates that we have added in the new uploaded version of our paper. The additions in comparison to the initial version are highlighted in the latest version in magenta.:

a) We added subsection 4.5 entitled “Comparing with the Fourier Features”. In the aforementioned subsection we provide Figure 6 that demonstrates the comparison between our proposed pretraining and the Fourier preprocessing approach in [2], for 4 environments of the DMC. It is prevalent that the proposed pretraining performs approximately the same as the Fourier preprocessing for the “Walker-run” and the “Quadruped-walk” environments. When it comes to the “Quadruped-run”, the Fourier preprocessing performs noticeably better than our proposed preprocessing. On the other hand, the proposed pretraining performs better than the Fourier feature preprocessing on the “Hopper-hop” environment. There is another nuance regarding Figure 6 that we stress on the paper. The performance of the proposed pretraining in Figure 6 corresponds to the same hyperparameter values (μ, a, b) for all environments. On the other hand, the Fourier preprocessing approach is tuned per environment ( variance of the normal distribution where the elements of the Fourier Kernel are sampled from, at initialization). This is attestation that there is a set of pretraining hyperparameter values that generalizes well across the DMC environments. This is important since the same concept applies for the SAC baseline. The SAC algorithm invokes many hyperparameters (e.g., the temperature coefficient of the policy), but there is a value set for all of them that corresponds to good performance across all domain environments.

b) Some of the experiments of the main body have been moved to the Appendix  (some of the environments of Figure 1 and Figure 2 of the initial version along with the Figure that summarizes the performance of ReLU MLP and Tanh MLP critic, with and without pretraining, for the 4 largest environments of the DMC). 

c) We erased the statement “We observe that the pretraining is generally robust with respect to the range of the hyperparameter values, therefore the values do not change significantly between environments” since we feel it can be misleading and does not paint the full picture regarding the role of the introduced hyperparameters. We added the following phrase that, we feel, summarizes more accurately the underlying effect of the hyperparameters: “We investigate the effect of different hyperparameter values in Section 4.5. In general, we found that choosing 𝞵=0.0001, a = 1, b=1 generalizes well across the DMC environments (see Figure 6)”. The aforementioned phrase can be found in the third paragraph of subsection 4.1 of the new version of the paper.

d) We added subsection 4.3, entitled “Pixel-based DMC tasks”, in which we apply our proposed pretraining method on DMC tasks where the state is not proprioceptive but consists of concatenated raw image frames of the environment. We combined our approach with the pixel SAC algorithm by [3] and compared it with an unpretrained baseline. The critic consists of a Convolutional Neural Network (CNN) backbone along with a ReLU MLP, both connected sequentially. Figure 3 illustrates the performance of the pretraining on 4 pixel-based DMC tasks. It is prevalent that the pretraining approach provides considerable improvement in all the demonstrated tasks. Another nuance is that we employed the same pretraining hyperparameter values for all pixel-based tasks (Figure 3). We consider this to be testament for the general efficacy of the proposed approach.

[1] Achiam, Joshua, Ethan Knight, and Pieter Abbeel. "Towards characterizing divergence in deep q-learning." arXiv preprint arXiv:1903.08894 (2019).

[2] Yang, Ge, Anurag Ajay, and Pulkit Agrawal. "Overcoming the spectral bias of neural value approximation." arXiv preprint arXiv:2206.04672 (2022).

[3] Yarats, Denis, et al. "Improving sample efficiency in model-free reinforcement learning from images." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.

From: ICLR.cc/2023/Conference/Paper5091/Authors
title: Answers to Comments by Reviewer Yp6v (0)
comment: We would like to thank reviewer Yp6v for the detailed and thorough review of our work. The reviewer’s comments, in terms of the paper’s weaknesses/areas of improvement, gave us the incentive to reassess certain parts of the current work and to discover new aspects. We tried to include these new aspects in the new version of the manuscript that we uploaded. The additions in comparison to the initial version are highlighted in the latest version in magenta. We provide our answers to the points made by the reviewer, along with references to the new aspects of the current upload, below:

1)  We are in complete agreement with reviewer Ypv6 on the fact that the initial version of the submission required an extension on the experimental part. In our newly submitted version, we have added subsection 4.5, entitled “Comparing with the Fourier Features”. In the aforementioned subsection we provide the comparison of our proposed pretraining with the Fourier features preprocessing approach by [1] in 4 environments of the DMC (Figure 6 of the new upload).  The approach in [1] (which is similar to the one in [2]) is closely related to our proposed work since the underlying motivation of both papers is to impose a particular structure on the NTK of the critic. Our findings consist of the following extractions. The Fourier preprocessing provides noticeably better performance than our proposed pretraining on the “Quadruped run” environment of the DMC. On the other hand, the proposed pretraining performs noticeably better than the Fourier preprocessing when it comes to the “Hopper-hop” environment. Other than the two aforementioned tasks, the proposed unsupervised pretraining and the Fourier features approach perform similarly for the rest of the domain environments. What should be noted is that when it comes to the curves of Figure 6 (latest upload) that correspond to the Fourier preprocessing, we tuned the variance of the elements of the Fourier Kernel, at initialization,  per environment (according to the values provided in the Appendix of [1]). On the other hand, the pretraining approach is not tuned per environment (this is for Figure 6 and not  for Figures 1 and 2). We employ the same set of hyperparameters ( = 1e-4, a = 1, b=1) for all demonstrated environments. It is true that the proposed pretraining introduces 3 new hyperparameters in contrast to the sole one by [1]. Nonetheless we believe that it is generally safer to tune our proposed approach in comparison to the Fourier preprocessing. The performance of the proposed pretraining is affected by the choice of the hyperparameters, but it is very difficult to find a set of hyperparameters where the proposed pretraining causes the SAC to diverge completely and its reward to collapse. This is sometimes the case for the Fourier preprocessing. The approach in [1] is admittedly very powerful, but can result in catastrophic divergence of the SAC, when the variance of the kernel at initialization deviates , sometimes even slightly, from the per -environment optimal. In the case of the “Hopper-hop”, the Fourier preprocessing diverges frequently (for several seeds) even when the variance of the kernel elements is carefully tuned (the performance plummets for certain seeds).  Another interesting finding that we included in the new subsection 4.5 is that the Fourier preprocessing and the proposed pretraining can work in conjunction with one another. The Fourier features approach behaves unstably for the “Hopper-hop” environment. When combined with the proposed pretraining the method behaves stably. This is demonstrated in Figure 7 (subsection 4.5) of the new upload, where we illustrate the performance of the Fourier preprocessing, with and without our proposed pretraining, on the “Hopper-hop” environment.  Furthermore, we applied our pretraining method on DMC tasks where the state is not proprioceptive, but consists of the concatenation of multiple raw image frames of the environment. We combined our pretraining with the pixel SAC by [3]. In the aforementioned implementation the critic consists of a Convolutional Neural Network backbone feature extractor and a ReLU MLP, both connected sequentially. We provide our findings in the subsection 4.3, entitled “Pixel-based DMC tasks”, of our newly uploaded paper. The new Figure 3 summarizes the performance of our pretraining on 4 pixel-based environments of the DMC. Our method results in considerable improvements in reward accumulation and sample efficiency for all depicted environments in comparison to an unpretrained baseline.

From: ICLR.cc/2023/Conference/Paper5091/Authors
title: Answers to Comments by Reviewer Yp6v (1)
comment: 2) We have tried to address the hyperparameter sensitivity topic on the latest uploaded version of the paper. We feel that, in the previous version, we did not succeed in providing a holistic view in terms of the hyperparameters. We introduce the new subsection 4.5 entitled “Comparing with the Fourier Features”. Figure 6 of the new subsection provides an overview of the performance of the proposed pretraining with the same set of hyperparameters for all environments. As can be seen, the same set of hyperparameters provides performance improvement in comparison to the SAC baseline and, also, compares well with the per-environment tuned Fourier feature approach [1]. This is a testament for the general efficacy of the proposed approach. On the other hand, subsection 4.6 (subsection 4.4 of the initial version) provides the performance of the proposed pretraining for 3 different values of the coefficient ratio for the two largest environments of the DMC. Admittedly, the performance of the proposed approach varies noticeably with the coefficient ratio. We purposefully chose 3  ratio values that are significantly different from one another (0.01, 1, 100). The underlying motivation for this is to talk about the trade-off between generalization and bias that is being imposed by the pretraining method.  The performance discrepancy is not as prevalent for different hyperparameter values for the rest of the environments of the domain. We added a phrase at the end of the third paragraph of subsection 4.6 that states the following:”We should note that the performance of the proposed method does not vary as much for different coefficient ratios on the rest of the environments of the DMC (except for the Quadruped environments)”. Furthermore, we include subsection  4.3 in our latest uploaded version where we apply our pretraining on pixel-based tasks with a CNN critic. Figure 3 provides the performance of the pretraining method where the hyperparameters remain the same for all depicted environments.  

3) We are in complete agreement with the reviewer on the fact that some visualization that pertains to the evolution of the NTK during training would provide additional intuition in terms of the intricacies of the proposed method. Due to the restricted time interval we were not able to perform such experiments, but we are more than willing to provide such visualizations during the review period to assist the reviewers in the process of evaluating the work.

From: ICLR.cc/2023/Conference/Paper5091/Authors
title: Answers to Comments by Reviewer Yp6v (2)
comment: 4)  We agree with the reviewer’s position that the statement “We observe that the pretraining is generally robust with respect to the range of the hyperparameter values, therefore the values do not change significantly between environments” can be misleading. Therefore it has been deleted from the latest uploaded version. We replace the aforementioned phrase with the following: “We investigate the effect of different hyperparameter values in Section 4.6. In general, we found that choosing =0.0001, a = 1, b=1 generalizes well across the DMC environments (see Figure 6).” The aforementioned phrase can be found at the end of paragraph 3 of subsection 4.1 in the latest uploaded version of the paper. In essence, our overall claim consists of the following propositions. First, there can be a set of hyperparameters that generalizes well across the DMC environments (Figures 3 and 6). This is important since the SAC baseline itself has multiple hyperparameters, but there is a set of values for all of them that yields good performance for all environments of the domain. Second, there can be explicit tuning of the pretraining hyperparameters per environment that yields even better results (Figures 1,2). Finally, we would like to address the comment regarding the performance of the proposed approaches on the “Humanoid-run” environment. Even though the proposed approach (both for ReLU MLP and Tanh MLP) achieves a performance that is approximately 100, this is quite close to the state-of-the-art performance for this environment. We refer to the Appendix of [1] that, to the best of our knowledge, constitutes the current state-of-the-art for the “Humanoid-run” from proprioceptive states. The authors claim a value that is close to 120/130 (as can be extracted from the corresponding graph). We understand that the reader can be confused since the rest of the environments are considered to be solved when the reward per episode approaches values close to 1000. That is why we have added the following phrase in the second paragraph of subsection 4.2: “Note that the current state-of-the-art performance for the “Humanoid-run” from proprioceptive states is close to 130 per episode (Yang et al, 2022).” 

5)  We are currently working on providing a version of the repository to the reviewers during the review period.

[1] Yang, Ge, Anurag Ajay, and Pulkit Agrawal. "Overcoming the spectral bias of neural value approximation." arXiv preprint arXiv:2206.04672 (2022).

[2] Brellmann, David, Goran Frehse, and David Filliat. "Fourier features in reinforcement learning with neural networks." (2021).

[3] Yarats, Denis, et al. "Improving sample efficiency in model-free reinforcement learning from images." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.

From: ICLR.cc/2023/Conference/Paper5091/Reviewer_Yp6v
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes a new pre-training technique for neural networks tasked to approximate the value function in off-policy actor-critic RL algorithms. In particular, given a set of priorly collected environment transitions, the derived method attempts to maximize the norm of the gradients w.r.t. the value function outputs; while minimizing the norm of the dot product of the gradients of samples between consecutive state-action pairs. To do this tractably, it uses 0th-order methods to approximate the gradient. The authors motivate their method with recent theoretical results suggesting that having a value function with very different gradients for different inputs facilitates convergence, using results from the NTK literature. The authors show gains by applying their method to SAC on some environments from the DeepMind Control suite from proprioception, using both ReLU and Tanh activations.
strength_and_weaknesses: **Strengths**

1. The paper is very clearly written and does a good job of explaining the motivation and related literature.
2. The proposed pre-training technique is relatively novel, simple, and well-explained.
3. Stabilizing value learning with function approximators is a problem of key relevance in the off-policy RL literature.

**Weaknesses/Areas of improvement**

1. I believe the experimental setting is very limited. The authors evaluate their work in 8 environments in the DeepMind control suite only from proprioceptive states. They compare their method only against a SAC baseline with no pretraining, showing gains in learning speed in some environments. These choices make it hard to contextualize the results, as most of the recent literature considers this benchmark only for pixel-based experiments. Comparing/applying their method to more recent algorithms using pixel observations (e.g. [1]) and/or in more popular benchmark protocols (e.g. [2]) would allow evaluating its actual generality. Furthermore, I believe it would be highly relevant to empirically compare against some of the cited prior methods attempting to regularize the value function network for similar purposes (e.g., [3, 4]).

2. The proposed methodology introduces several additional hyperparameters (e.g., Appendix A) that are tuned per environment, even differing by whole orders of magnitude. Furthermore, it also introduces many additional variables into the optimization process that would likely influence its outcome (e.g., number of pre-training epochs, pre-training data, ...). The paper only shows results with two alternative hyperparameter configurations on two environments, showing performance is severely affected by these choices. Given these conditions and sensitivity, I am not entirely convinced that the proposed methodology yields significant performance advances over the SAC baseline (which instead does not have any task-specific tuned hyper-parameter or variable).

3. The qualitative analysis of how the proposed methodology affects learning has also room for improvement. The authors only provide a visualization showing that, after pretraining, the resulting NTK adheres to their optimization objectives. For instance, I would suggest additionally comparing measures of how the value of the utilized pre-training objective evolves *during* training and compare with the non-pretrained baseline. Does the inequality from Eq. 8 hold throughout training? Do the evolutions of the gradient magnitudes differ in different environments?

4. Some of the statements are not entirely justified and appear to contradict some of the results. For instance, the authors claim "We observe that the pretraining is generally robust with respect to the range of the hyperparameter values, therefore the values do not change significantly between environments." It appears that this contradicts Appendix A, where the reported hyper-parameters even differ by orders of magnitude in some environments. In another case, the author claim "We provide empirical evidence that the proposed pretraining method improves neural value learning for different variations of the value network architecture". Yet, I believe there is a single variation tested, simply swapping the non-linearities from ReLU to Tanh. Furthermore, the claim that "The version that employs a pretrained critic with Tanh activations performs reasonably well [in Humanoid run]" contradicts the displayed results, where the final performance is only 100/1000.

5. Given the contribution is a new pre-training phase for RL, I believe sharing a version of the implementation and the utilized pre-training data would be crucial to allow correct reproducibility.

[1] Yarats, Denis, et al. "Mastering visual continuous control: Improved data-augmented reinforcement learning." arXiv preprint arXiv:2107.09645 (2021).

[2] Chen, Xinyue, et al. "Randomized ensembled double q-learning: Learning fast without a model." arXiv preprint arXiv:2101.05982 (2021).

[3] Yang, Ge, Anurag Ajay, and Pulkit Agrawal. "Overcoming the spectral bias of neural value approximation." arXiv preprint arXiv:2206.04672 (2022).

[4] Kumar, Aviral, et al. "Implicit under-parameterization inhibits data-efficient deep reinforcement learning." arXiv preprint arXiv:2010.14498 (2020).
clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written, yet the quality of its analysis has a large room for improvement. The proposed methodology has novel aspects, however, given the current analysis, I am not convinced of its efficacy and generality. Unfortunately, I believe the experiments are not reproducible given the lack of source code, pre-trained data utilized, and detailed descriptions.
summary_of_the_review: While I appreciated the work's clarity and the spirit of the contribution, I found its analysis very limited. Hence, I believe this paper has considerable room for improvement and, thus, is not ready for publication in its current form.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper5091/Reviewer_1sio
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper studies unsupervised pre-training of the value function for more efficient downstream RL. 

The method builds on intuition from prior work Achiam et al. that the magnitudes of the neural tangent kernel (NTK) of the value function are correlated with how much value estimates of a transition changes based on the update from another transition. As a result, for stable convergence, the NTK magnitude along the diagonal should be higher than the off-diagonal elements. 

Motivated by this, they propose a pre-training scheme that trains on a dataset of transitions (without rewards), with the objective of directly maximizing the difference between the on-diagonal and off-diagonal elements (with some hyperparameters to balance the terms and prevent collapsing to the degenerate case). Additionally to reduce the computational overhead of computing the Hessian they use a 0th-order approximation. 

In experiments, they find that pre-training on unlabeled data improves the performance of SAC on a number of online RL tasks. They additionally show that their pre-training indeed biases the NTK towards being more diagonal. 
strength_and_weaknesses: *Strengths*

- This paper studies a really interesting problem in pre-training value functions from unlabeled data. This domain is interesting as it has the potential to enable large-scale pre-training for decision-making. 
- The paper is written very clearly and nicely motivates its approach. It does a great job explaining prior work Achiam et al. and following the background the proposed method is intuitive. 
- The experiments are convincing, and it does seem like this pre-training improves performance in a number of domains. The qualitative experiments also show that the pre-training is indeed doing what the authors are suggesting. 

*Weaknesses*

- My main criticism is that there seem to be other pre-training methods that would be good to compare to. Specifically, it is also self-supervised to train a dynamics model or world model, which could then be used for better policy optimization in the model or be used to generate synthetic data. I think (1) comparing to model-based baselines like DreamerV2 (Hafner et al), and (2) pre-training the world model with the same pre-training data used by their method would be important baselines to include to really show that their approach is the best pre-training to use. 
- Another relevant baseline is pre-training state representations on the pre-training data using some self-supervised representation learning algorithm like BYOL or Barlow Twins.
clarity,_quality,_novelty_and_reproducibility: The work seems novel and is very clearly written. 
summary_of_the_review: Overall this is a good paper, it is written well and proposes a novel (to the best of my knowledge) approach for pre-training value functions on unlabeled data. The experiments suggest it works well. I think the main improvement I would like to see is more comprehensive comparisons to model-based RL approaches which can even more easily incorporate unsupervised pre-training. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5091/Reviewer_iUkV
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: **Summary:** The paper tries to improve the stability and convergence speed of off-policy TD learning-based reinforcement learning algorithms (like TD3, DDPG, SAC). It argues that the aggressive generalization (i.e. NTK of the network having large off-diagonal elements compared to the main diagonal) in critic (Q function) often causes these instabilities and divergences (Achiam et al., 2019). 

To prevent these instabilities, the paper proposes to properly condition the NTK of the critic with a pretraining method. The proposed pretraining method involves collecting data using a random policy and then using the data to incentivize the NTK of the critic to have smaller off-diagonal elements compared to the main diagonal (Equation 9). Once the pretraining of the critic is done, the paper uses Soft Actor-Critic (SAC) to train the policy. The paper empirically shows that the pretraining of the critic improves sample efficiency and speeds up the convergence of the policy on DeepMind Control (DMC) tasks. Furthermore, the paper notes that pretraining allows for training critics with tanH activations.
strength_and_weaknesses: **Pros:**
1. The proposed pretraining scheme improves the performance and convergence speed of policy on DMC tasks
2. The paper clearly shows that pretraining improves the conditioning of NTK of Q function (Figure 3 and 4)
3. The paper provides a detailed analysis on how the choice of hyperparameters a,b, and mu affects the performance of the policy

**Cons:**
1. Important baselines/comparisons are missing: 
    (a) Prior works (Li et al., 2021, Brellman et al., 2021, Yang et al., 2022) attempt to solve a similar problem by passing the state through Fourier transformation, but the proposed method isn’t compared against these prior works. On a closer look,  Li et al. (2021) and Yang et al. (2022) have experimented with DMC tasks, and seems like they outperform/match the proposed approach on most tasks except Acrobat-Swingup. Hence, a thorough comparison with these baselines would be important.
    (b) The proposed method introduces 3 new hyperparameters which are tuned per environment. In contrast, Yang et al. (2022) introduce 1 hyperparameter, tuned per environment, and Li et al. (2021) introduce 1 hyperparameter, which stays constant across environments.
    (c) Overall, it would be nice to argue why the proposed method is better than prior works such as Li et al. (2021),  Yang et al. (2022) that attempt to solve a similar problem.
2. I am not sure why the paper shows that pretraining allows for the training of critics with TanH activation. The motivation for this experiment is lacking. Are there environments where TanH activation would help more than ReLU activation? If ReLU works better than TanH (for critic) across most continuous control tasks, then I don’t see the point of improving the training of critic with TanH activation.
3. I am not sure if the pretraining scheme would help with image-based tasks and hence it would be nice to see results on image DMC tasks.
4. The paper says that the pretraining phase collects 100 episodes with a random policy. Do the graphs (i.e. blue lines corresponding to the method utilizing the pretraining) in Figure 1 include those 100 episodes used in pretraining?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the proposed pretraining scheme is novel. Furthermore, the paper contains enough details to be able to reproduce the work.
summary_of_the_review: Given the pros and cons as listed above, I am currently leaning toward rejection. I am happy to change my score if authors can address my concerns.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper5091/Reviewer_XRX1
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper builds upon the idea from Achiam et al. (2019) to  use neural tangent kernel (NTK) based criteria to characterise the divergence of deep Q-learning.  In contrast to  Achiam et al. (2019), the idea is not to diagnose the divergence but take advantage of it to enable pretraining to improve convergence.
strength_and_weaknesses: Strengths:
- It is definitely an interesting and sound idea to use NTKs for pretraining the Q-function; 
 - the paper is clearly written

Weaknesses:
- While the idea is incrementally developed from Achiam et al, it still has merit of novelty of application to the setting of pretraining; to improve the paper it would be good to show its advantage/synergy in relation to other strategies of pretraining
- The experimental results are limited (adding more tasks could help illustrate the use case of the proposed method), the experimental settings use different hyperparameters so the questions about hyperparameter sensitivity arise (see comment 4 below)
- There is only limited improvement in Figure 1 despite such fine-tuning; what is even more interesting that the improvement for the pretrained models kicks in after the mid-phase of training while usually with pretraining it makes most of its impact in the beginning; do the authors see any reasons behind it?
- Section 4.2 : the reviewer is not sure that changing the activations gives a testimony that the model is better suited for challenging training conditions. This is because this scenario might seem unpractical. Why would choosing Tanh nonlinearity, given the worse empirical outcomes (and higher computational demands) be a realistic use case for using the proposed method (see Krizhevsky et al, 2017 on the advantages of ReLU for the setting of supervised training of deep learning architectures)?
- Section 4.4: the evidence of hyperparameter sensitivity is only based on one task and a few deliberate hyperparameter choices; is it possible to run such experiments on multiple tasks, e.g. with some sort of grid search over hyperparameter space 

clarity,_quality,_novelty_and_reproducibility: Novelty: the idea of using NTKs in the setting of controlling stability of reinforcement learning is known before (the authors cite Achiam et al); however it was not applied to the critic pretraining as in this paper, but to stabilisation of the training process.

Quality: the previous stated questions about the experiments in Section 4.2, also there is a concern about Section 4.4, where different hyperparameters are chosen for different tasks, which puts into doubt whether all tasks will still work with the same choice of hyperparameters (or if there is a systematic procedure for choosing these hyperparameters)

Comments on clarity and quality:

1. “Tassa et al (2018)” Something went wrong with the citations of the paper (low-case names of some of the authors, full stop in between the authors’ names…), could the authors correct this reference?
2. Terminology: given that there is a separate notion of the value function in reinforcement learning (see Schaul et al, 2015), which is essentially the Q-function with actions marginalised out, it looks like the use of the term ‘value NTK’ might be confusing and contrary to the existing terminology; Achiam et al (2019), for example, call it Q-value. Therefore, even the very name of the paper might sound confusing (as I would expect the paper to do with the value function, not Q-function)
3. Eq. 9: it is important to change the notation so that $\mathbf{a}$ is not confused with $a$ (e.g. by calling the coefficients $\alpha$ and $\beta$
4. In Appendix A, it is stated that different tasks in Figure 1 are solved with different hyperparameters; would it be possible to show the results with fixed hyperparameters?
5. Given that there is a notion of trade-off between generalisation and stability (see p.4), do the authors think it is possible to showcase this scenario and show empirically how this method could control it, perhaps with choosing hyperparameters? In Section 4.4, the impact of hyperparameters is shown, but how do the authors propose to select them?
6. Given that the method is based on Achiam et al (2019), is it compatible  with their method? Is it beneficial to use the proposed pretraining in conjunction with Achiam et al’s stabilised RL method?

References:

Schaul, Tom, Daniel Horgan, Karol Gregor, and David Silver. "Universal value function approximators." In International conference on machine learning, pp. 1312-1320. PMLR, 2015
Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Communications of the ACM 60, no. 6 (2017): 84-90.
summary_of_the_review: The paper needs multiple improvements, including in experimental procedure (showing the results with fixed hyperparameters as opposed to fine-tuned for different tasks, clarifying on choice of hyperparameters to balance between stability and generalisation, clarify on limited improvements in Figure 1)
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold