From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: This paper presents a new approach for representation learning in reinforcement learning by designing a new family of auxiliary tasks based on the successor measure. All reviewers found this idea interesting. The AC agrees and recommends acceptance. 
justification_for_why_not_higher_score: While the ideas and results are interesting, the (perceivable) impact is not high enough to reach the bar of the spotlight.
justification_for_why_not_lower_score: The new idea in this paper is interesting and novel.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper5901/Reviewer_SdmH
title: Response
comment: I thank the authors for their response above, as well as the additional results they have incorporated into their revision.
The additional results in the tabular domain (Appendix C) are especially appreciated.

From: ICLR.cc/2023/Conference/Paper5901/Reviewer_PVaX
title: Response
comment: I appreciate the author's detailed response. After reading the rebuttal and the additional experiments, I believe most of my major concerns are addressed. a) The experiments now are more complete. b) Literature review is also more complete. c) Many previous confusions are also addressed. d) The usage of state-only representation is better justified for the atari setting. Thus I am flipping my evaluation after the rebuttal.

From: ICLR.cc/2023/Conference/Paper5901/Authors
title: Thanks to Reviewers and Summary of Changes
comment: We thank the reviewers $R1$ (`tzdq`), $R2$ (`PVaX`), $R3$ (`SdmH`), and $R4$ (`ZQM9`) for their time, and kindly request them to reconsider their scores in light of our changes and clarifications. Since the author-reviewer discussion period ends today, we’d be happy to engage in any further discussion until then. In summary, we have:
1. Added Behavior Cloning (BC) and Self-Predictive Representations (SPR) baselines to **Section 5.2** and **Figure 3**. PVN still performs favorably against these baselines [$R2$].
2. Added training curves to **Appendix G**, showing PVN outperforms baselines with fewer online interactions [$R4$].
3. Increased the number of offline pre-training seeds from 1 -> 3 resulting aggregate results over 414 runs (up from 138) for each method in **Figure 3** [$R4$].
4. Added a proof in **Appendix C** that under some assumptions PVN would learn the same representation as PVFs in the tabular case [$R3$].
5. Added further discussion on the relationship between the successor representation and proto-value functions in **Appendix A** [$R1$, $R2$, $R3$].
6. Added additional related work referenced by reviewers, with clarifications on the novelty of our work in **Section 2** [$R2$].
7. Added additional discussion on MDS plots in **Appendix E** [$R2$].

From: ICLR.cc/2023/Conference/Paper5901/Authors
title: Author Response
comment: We thank the reviewer for the constructive feedback regarding our empirical methodology. In regards to your questions not already answered in the general response:

> The number of seeds is low.

We agree that more seeds/trials is always better. Running Atari experiments is inherently expensive; each offline run takes 3-5 TPU-days to perform. We chose to run with more games rather than fewer pre-training seeds per game, as this minimizes the risk of overfitting to specific games. Following the best practices outlined in [4], we aggregate our results using stratified bootstrapping, and report multiple point estimates.

However, in our updated manuscript, we have improved our main experiments to now use 3 seeds during the pre-training phase (up from 1), which now results in 3 (offline seeds) x 3 (online seeds) x 46 (games) = 414 runs (up from 138) per method. Unfortunately, the SPR baseline results take longer than other methods, so the additional 2 seeds for that method are not reflected in the manuscript yet, but will be by the end of the discussion period.

> [Why] 3.75M [samples]? Are the conclusions stable for other number of samples? Could you provide results for other number of samples?

We apologize for the seeming arbitrariness. The real number is 15M Atari frames (3.75M steps with a frame-skip of 4). The choice was made as a practical compromise between 100k (“low-data regime”) and 200M (“standard DQN”), because the former has been shown to be an excessively high-variance setting [4] and the full setting would require substantially longer training periods (multiple weeks) with our larger Impala x8 network.


This said, the conclusions are stable for other numbers of samples. We added training curves for our method and the baselines in the Appendix which we hope can alleviate concerns around the number of samples we employed.

> I would also appreciate some discussion about the games, for which random trajectories do not explore well the state space.

Unfortunately, these datasets were not published as part of the RL Unplugged benchmark. This is presumably due to the inability to learn in hard exploration games with sparse rewards. This isn’t to say that PVNs wouldn’t be useful in such a setting. The line of work on random cumulants stems from the observation that this class of auxiliary tasks seems to disproportionately help in sparse-reward environments in online settings (see [1, 2]). It would be interesting as follow-up work to look at how PVNs can help augment online agents in sparse-reward environments. Additionally, PVNs could also be employed with complementary work to incentivize agents to more efficiently explore in hard exploration environments [3].

> Why measure the quality of representation by online RL training? Does it make sense to measure the quality of representation using some other metric?

We chose to use online RL training for evaluation due to its prevalence in the RL literature. Via online learning, we are able to ascertain whether our features are suitable for control, which encapsulates other properties of representation learning such as approximation error, stability, and generalization. In particular, it demonstrates that our representation is capable of approximating multiple value functions along the value improvement path [1].

It may be possible to use other metrics to measure the quality of our representations. However, these are often difficult or impossible to calculate directly for large state spaces. We supplemented our online experiments with a qualitative evaluation (MDS plots) to illustrate the properties of our learnt representations that online learning may not capture. We included a brief extension of our discussion on the MDS plots in Appendix E.

### References:

[1] Will Dabney, Andre Barreto, Mark Rowland, Robert Dadashi, John Quan, Marc G. Bellemare, David Silver. The Value-Improvement Path: Towards Better Representations for Reinforcement Learning. AAAI Conference on Artificial Intelligence. 2021.

[2] Clare Lyle, Mark Rowland, Georg Ostrovski, Will Dabney. On the Effect of Auxiliary Tasks on Representation Dynamics. International Conference on Artificial Intelligence and Statistics (AISTATS). 2021.

[3] Marlos C. Machado, Marc G. Bellemare, Michael Bowling. Count-Based Exploration with the Successor Representation. AAAI Conference on Artificial Intelligence. 2020.

[4] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, Marc G. Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice. Advances in Neural Information Processing Systems (NeurIPS). 2021.

From: ICLR.cc/2023/Conference/Paper5901/Authors
title: Author Response
comment: We thank you for your review. Thank you for raising minor issues and typos, these have been resolved in our updated paper. In regards to your questions not already answered in the general response:

> How does the set inclusion reward compare to the state equality reward?

In the Appendix, we now include a theoretical result in the tabular setting under symmetry conditions where we show that when using set-based indicators (i.e., choosing subsets of a fixed size) you learn the same representation as the state-equality indicator. With this result, we can view the PVN algorithm parameterized by subsets of a fixed size as converging to the space spanned by the PVFs. We hope this result helps provide motivation for our method.

> Is there an assumption that the dataset used to learn these representations has coverage? 

Yes, our method will perform better when we have better coverage as we can more accurately model the temporal structure of the environment. We presented PVNs as a pre-training method for Offline RL but our work is complementary to work on exploration using PVFs as a reward bonus that can incentivize the agent towards these bottlenecks to increase state-coverage [1].

> How does the scaling of the Impala network's penultimate layer affect the number of parameters in the network? If the scaling does not change the number of parameters substantially, it might be a possible explanation of the saturation of the number of tasks that can be learned.

Scaling changes the number of parameters substantially. See our general response for more details.

> Can these features be used for purely offline RL? Or would that introduce some bias which required online learning to learn the task specific policy?

Yes, although the online learning setting is somewhat simpler because divergence/overfitting is usually less likely (this is true for other deep RL algorithms too).

> Is the reason that the greedy policy fails because of the max? Or because the representations no longer capture the environment dynamics alone (and also represent the policy)?

In some cases we find that the representations learned with the max diverge, in the sense that increasing feature norms leads to instability in learning. We don’t believe this is entirely surprising; it’s known that in Offline RL performing Bellman backups with out-of-distribution actions can lead to instability and divergence [2]. In our case the max-action for a given RNI reward function could exacerbate this problem.



> What if the policy being evaluated here was the Boltzmann softmax policy?

This is an intriguing idea, there is indeed a whole range of policies we could try. As discussed in the general response, the uniform policy has appealing properties. The Boltzman softmax policy is also not a non-expansion and the optimal value function isn’t well defined, this may lead to additional divergence that may be hard to characterize.

### References:

[1] Marlos C. Machado, Marc G. Bellemare, Michael Bowling. Count-Based Exploration with the Successor Representation. AAAI Conference on Artificial Intelligence. 2020.

[2] Sergey Levine, Aviral Kumar, George Tucker, Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. arXiv CORR abs/2005.01643. 2020.

From: ICLR.cc/2023/Conference/Paper5901/Authors
title: References
comment: [1] Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R. Devon Hjelm, Philip Bachman, Aaron C. Courville. Pretraining Representations for Data-Efficient Reinforcement Learning. Advances in Neural Information Processing (NeurIPS) 2021.

[2] Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, Jeff Clune. Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos. Advances in Neural Information Processing (NeurIPS) 2022.

[3] Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron C. Courville, Philip Bachman. Data-Efficient Reinforcement Learning with Self-Predictive Representations. International Conference on Learning Representations (ICLR). 2021.

[4] Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R. Devon Hjelm, Philip Bachman, Aaron C. Courville. Pretraining Representations for Data-Efficient Reinforcement Learning. Advances in Neural Information Processing Systems (NeurIPS). 2021.

[5] Walter Rudin. Functional Analysis. International Series in Pure and Applied Mathematics. 1974.

[6] Marek Petrik. An Analysis of Laplacian Methods for Value Function Approximation in MDPs. International Joint Conference on Artificial Intelligence (IJCAI). 2007.

[7] Charline Le Lan, Stephen Tu, Adam Oberman, Rishabh Agarwal, Marc G. Bellemare. On the Generalization of Representations in Reinforcement Learning. International Conference on Artificial Intelligence and Statistics (AISTATS). 2022.

[8] John N. Tsitsiklis, Benjamin Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control. 1997.

From: ICLR.cc/2023/Conference/Paper5901/Authors
title: Author Response
comment: We thank you for your review. In regards to your questions not already answered in the general response:

> The paper should consider adding some baselines and try to at least show the performance is comparable.

Based on the reviewer’s feedback, we added behavior cloning (BC) and an SPR-loss-based algorithm as baselines (Figure 3). BC has been shown to outperform contrastive pre-training methods in Offline RL and scales favorably with the amount of data [1, 2]. SPR [3] is similar to our method in that it also attempts to capture transition dynamics by directly optimizing for temporal consistency. We find that PVN 8x outperforms both BC and the representation learned with the SPR loss.

> Introducing another function to learn a representation that obeys the bellman consistency on that function is actually not new even in representation learning in RL.

Thank you for pointing out these relevant and interesting works and the potential connection between our method and the min-max formulation of representation learning in RL. We agree that there are a number of works, including the ones the reviewer cites, that consider learning a representation that encodes the transition structure of the environment (almost always implying Bellman consistency). We discuss some of these in our related work section, and have added the works you mention along with a clarification of what is novel in our paper.

For contrast, our approach is to begin with auxiliary tasks, which are known to provide a stable, well-understood driver of representation learning in deep RL. Doing so requires very little hyperparameter tuning – we just ask the network to predict those tasks. Because of this, the PVN approach doesn’t assume or require any particular structure such as the existence of a low-rank description of the MDP – it’s always well-defined and will always result in a meaningful representation, whose approximation error can be characterized (e.g. [6, 7]).

>  " These eigenvectors are of special importance because they encode the spatial structure of the MDP in terms of a diffusion process". I am very confused about what this claim, and it would be great if further explanation could be provided and show why this claim is relevant.

We apologize if this wasn’t clear. This sentence is specifically pointing out to the theoretical result given by Mahadevan and others showing that the top eigenvectors of the successor representation (when $P^\pi$ is symmetric) encode the transition matrix structure at multiple scales, beginning with features that represent the space coarsely down to finer details (see also the general response above). This is relevant because it implies that in principle, PVNs should also encode this spatial structure (see revised Appendix A). The MDS plots we supplied in the appendix also support this finding - temporally successive states have more similar representations with PVN than the other methods.

> Second to last paragraph in page 4: " Second, set inclusion allows us to incorporate a notion of closeness to ψ, ..." Here the notion of closeness seems ambiguous, especially in the context of set notations.

Here we mean similarity in a topological sense (e.g., see [5]): two states are neighbours when they are in the same set (and close if they are in many sets together).

> On the top of page 5: ξ here is a distribution, then the definition of the diagonal matrix seems weird. Is ξ just a n-dimensional simplex?

Here we treat xi as a n-dimensional vector whose entries are the probabilities assigned to different states. We improved the notation in the paper to clarify.

> Should we learn a feature mapping ϕ that only operates on states only?

Most Atari-playing agents learn this mapping and so our choice follows standard usage (this is also computationally simpler for |A| ~ 18 actions). However, it is certainly possible to learn a state-action encoding by the same method, in which case the weight vector would not depend on a.

> Why are human interpretable features useful?

We apologize for the lack of clarity here. We do not argue that human interpretable features are useful, but rather that features that capture dynamics of the environment are useful. We added a further discussion in Appendix E. 

> Reproducibility concerns.

Based on your request, we have now provided code as supplementary material to aid with reproducibility of our method.

### References

(Over character limit, will be posted in next comment)

From: ICLR.cc/2023/Conference/Paper5901/Authors
title: Author Response
comment: We thank you for your review. In regards to your questions not already answered in the general response:

> Is there a more efficient target policy that incorporates the prior knowledge of the tasks?

See our general response for the motivation behind the random policy. In theory, we can use other policies, but it is hard to know what this target policy would be a priori. Attempting to learn this policy in conjunction with the representation might result in instability. In practice, people often use the max operator (e.g. [1]), which we find isn’t as effective.

> The empirical results do not provide the comparison with the widely used policy gradient methods such as PPO. Is it possible for PVN to achieve comparable performance to these policy gradient algorithms?

We think that there might be two separate questions here. a) Is it worthwhile to compare the performance of our algorithm (PVN + Q-learning) to PPO, and b) Can we use the learned representation and apply PPO to it (with linear approximation).

To answer a), we took the PPO scores from [2] and aggregated them similar to Figure 3. We find that PPO achieves a human-normalized median score of 0.5 after 3.75M interactions which is directly comparable to PVN 8x.

For b), yes, it is certainly possible. We chose to focus on value-based methods as these are more commonly evaluated on the ALE. However, it should be possible to reuse the learned PVN representation with most RL methods, including PPO.

### References:

[1] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, DavidSilver, and Koray Kavukcuoglu.  Reinforcement Learning with Unsupervised Auxiliary Tasks In International Conference on Learning Representations (ICLR), 2017.

[2] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, Peter Zhokhov. OpenAI Baselines. GitHub repository: https://github.com/openai/baselines. 2017.

From: ICLR.cc/2023/Conference/Paper5901/Authors
title: General Response to Reviewers
comment: We thank the reviewers for their time and their detailed reviews and valuable feedback. Here we respond to common points.

> Why use a uniform random policy in the pre-training stage?

Our choice is justified by the original proto-value functions (PVFs) work by Mahadevan et al. (2005, 2007), which considered a random walk that corresponds to the uniform policy in our setting. They showed that under a symmetric assumption, PVFs encode the structure of the MDP at different spatial scales and have appealing generalization properties. The relationship between PVFs and SRs is made by Stachenfeld et al. (2014) and Machado et al. (2017); Madjiheurem et al. (2019) consider fancier random walks. 

Although we could consider other, non-uniform policies, the uniform policy is simple to implement (np.mean) and applies to any discrete-action domain (a continuous extension is also possible). We will clarify these points in the main text and give a more complete PVF <-> successor representation connection in the appendix.

> How large are the feature layers of our networks? How do parameters scale with the number of features?
> Is it possible to improve the performance with larger feature dimensions (e.g. 16x)?

The default Impala encoder (1x) has 512 features in the penultimate layer. Our 8x network therefore has 4096 features. We believe that performance will continue to increase as we increase the dimensionality, but naively scaling the width incurs a large memory footprint. For example, the weight matrix that maps to the 4096 features is of size: [12 x 12 x 256, 4096] which is approximately 4Gb using 32-bit floats. Models larger than 8x would not fit on our hardware without custom implementations or e.g., sharding the model across multiple accelerators. We will clarify this in the text.

This said, we do believe there might be benefits to scaling the width beyond 8x. An interesting direction for future work is to further analyze how to efficiently scale width while retaining the performance improvement we demonstrate in the paper. We will discuss these challenges in greater detail in the revision.

### References:

[1] Sridhar Mahadevan. Proto-Value Functions: Developmental Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML). 2005.

[2] Sridhar Mahadevan and Mauro Maggioni. Proto-value functions: A laplacian framework for learning representation and control in markov decision processes. Journal of Machine Learning Research (JMLR), 2007.

[3] Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, Murray Campbell. Eigenoption Discovery Through the Deep Successor Representation. In the International Conference on Learning Representations (ICLR). 2018.

[4] Kimberly L Stachenfeld, Matthew Botvinick, Samuel J Gershman. Design Principles of the Hippocampal Cognitive Map. In Advances in Neural Information Processing Systems (NeurIPS). 2014.

[5] Sephora Madjiheurem, Laura Toni. Representation Learning on Graphs: A Reinforcement Learning Application. International Conference on Artificial Intelligence and Statistics (AISTATS). 2019.

From: ICLR.cc/2023/Conference/Paper5901/Reviewer_tzdq
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper studies the power of representation learning via auxiliary tasks in deep reinforcement learning. To be concrete, it focuses on find a good representation of the states that is capable of predicting the optimal value functions if followed by a linear predictor. Correspondingly, it proposes a method called the proto-value networks. The proto-value networks take advantage of the success measure over subsets of states to learn a good representation. Empirical studies show that these learned features combined with linear functions capture the optimal value functions in many Atari games. The evaluation rewards are comparable to that of the original version of DQN.
strength_and_weaknesses: The paper has multi-level contributions: 

1. The problem studied is interesting and important in the field of deep RL. Most of the works in RL are based on the representation learning, so it is vital to understand the properties of these representations. 
2. To design such a representation, this work proposes to use the success measure as the learning objective of state representations. Further, they find that it is more effective to learn the success measure over subsets of states, instead of using the original success measure defined on a single state. Finally, they study how to choose such subsets of states as auxiliary tasks. They propose universal hashing functions and random network indicators to effectively help design the state representations.
3. With the learned representations, they conduct systematic experiments on all the Atari games to verify the effectiveness of the representations. The results show that linear functions with regards to the representations are sufficient to learn optimal policies on many tasks, with comparable performance to the DQN algorithm. The experimental results also verify the scaling properties of the number of auxiliary tasks, and the importance of the indicator functions.

On the other hand, I also have several questions regarding to the paper:

1. Could you provide some intuitions on why choosing the uniform random policy as the target policy in the pre-training stage? The performance degradation of the greedy policy (using the max operator instead of averaging) perhaps comes from the correlation of the target policy and current feature estimator. However, is there a more efficient target policy that incorporates the prior knowledge of the tasks?
2. It seems that the learned features of the states at the end of the pre-training stage is the output of the last but one layer of the PVN, since the last layer is a linear prediction layer. How large is the dimension of such features? Is it possible to improve the performance with larger feature dimensions?
3. The empirical results do not provide the comparison with the widely used policy gradient methods such as PPO. Is it possible for PVN to achieve comparable performance to these policy gradient algorithms?

clarity,_quality,_novelty_and_reproducibility: The paper is written in a very clear and organized way. The reproducibility should be good since the descriptions of the algorithms and hyper parameters are clear clear and complete.
summary_of_the_review: Overall, I think this is a good paper, which is suitable for the conference.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper5901/Reviewer_PVaX
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper introduces an extension of learning successor feature that ensures that the learned feature could obey the bellman consistency on a set of the states. To empirically define which sets of states that the feature should operates on, the paper introduces two empirical surrogates: using random hash functions or random neural networks. Finally the experiments visualize that the learn feature may indeed show some meaningful semantics over the states.
strength_and_weaknesses: # Strength
- The paper proposes an easy extension of successor features, and the two proposes state aggregation methods seem to provide meaningful results in practice. Also the state aggregation methods are interesting.
- The paper is well-organized and easy to follow.

# Weakness
- One would naturally wonder why when we want to learn the value function $Q$, we should only learn a feature mapping $\phi$ that only operates on states only (specifically, section 3). One example one could think of is where we have finitely many actions, then $\phi(s)$ could be the $Q$ function for each action and $w_a$ is a one-hot encoder. But intuitively this formulation seems have its limitation on the generalizability against, for example, learning the state-action representation.

- The paper does a good job showing that the learned representation are interpretable to human, but there seems to lack of a section that shows that learning representation this way is indeed useful. The paper should consider adding some baselines and try to at least show the performance is comparable, because the method also enjoys the advantage of having the offline data which may provide good coverage.

- Introducing another function to learn a representation that obeys the bellman consistency on that function is actually not new even in representation in RL. For example, in MOFFLE [1] and BRIEE [2], the representation learning objective is to train a feature map that ensures the bellman consistency on the sequence of the most adversarial discriminator function. The idea of using many random network is the same: it ensures that it could cover the whole discriminator class and learn a feature map that works on all of them.

- The usage of random policy still sounds heuristic to me. If you want to do distribution shift to arbitrary policy, you may still need to pay for exponential on $|\mathcal{A}|$.

- There are several confusions in the paper:
1. Right above section 4.1: " These eigenvectors are of special importance because they encode the spatial structure of the MDP in terms of a diffusion process". I am very confused about what this claim, and it would be great if further explanation could be provided and show why this claim is relevant.

2. Second to last paragraph in page 4: " Second, set inclusion allows us to incorporate a notion of closeness to $\psi$, ..." Here the notion of closeness seems ambiguous, especially in the context of set notations. 

3. On the top of page 5: $\xi$ here is a distribution, then the definition of the diagonal matrix seems weird. Is $\xi$ just a $n$-dimensional simplex?

[1] Modi, Aditya, et al. "Model-free representation learning and exploration in low-rank mdps." arXiv preprint arXiv:2102.07035 (2021).

[2] Zhang, Xuezhou, et al. "Efficient reinforcement learning in block mdps: A model-free representation learning approach." International Conference on Machine Learning. PMLR, 2022.

clarity,_quality,_novelty_and_reproducibility: ## Clarity and Quality
The overall writing is good, but several details could need some polishing.

## Novelty
The contribution is quite limited given that previous representation learning methods in RL already consider similar ideas.

## Reproducibility
The paper provides detailed experiments details, but the reproducibility statement and code are missing in the submission.
summary_of_the_review: Overall this is a well-written paper and the paper provides some good insights, some specific aspect of the paper is interesting, but I am not convinced that the paper actually makes enough contribution and seems to overlook some previous methods. I would recommend a borderline reject. 

### Post-rebuttal
The authors make the paper more ready for publishing after the rebuttal thus I raised my score. See the discussion thread for more details.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5901/Reviewer_SdmH
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper studies the question of representation learning in deep RL by means of learning successor features for auxiliary tasks. It presents an approach to learn successor features with reward agnostic tasks which will learn features that can then be used to learn a linear action-value function for downstream tasks.
The features learned via this approach have connections to a previous approach for representation learning, proto-value functions.

These reward agnostic tasks are based on a set inclusion reward, where the paper proposes to generate sets using universal hash functions or random network indicators (RNIs) to ensure that the sets will cover the entire state space. The main technique used in the paper then utilizes this idea of learning successor features with sets generated using RNIs and tuned with quantile regression to learn a representation from offline Atari datasets. The representation is tested with some online interaction.
strength_and_weaknesses: ### Strengths:
* This paper presents a method to approximate proto-value functions using the idea of auxiliary tasks.
* It presents an interesting workaround to the state equality problem by instead formulating it as a set inclusion reward

### Shortcomings:
* The paper states that the set inclusion reward is a relaxation of the state equality, and this relaxation allows the learning of features that relate to the proto-value functions. A toy experiment on a grid world to show how the set inclusion reward compares to the state equality reward would have been illuminating.
* Some questions are raised (See below) that are not entirely answered by the paper.

### Questions:
* Is there an assumption that the dataset used to learn these representations have coverage? Since the learned representations be dependent on the transition structure of the environment, if the MDP has bottleneck states that a random policy that starts from a restricted set of start states might find hard to get through, will the learned representations be suboptimal?
* How does the scaling of the Impala network's penultimate layer affect the number of parameters in the network? If the scaling does not change the number of parameters substantially, it might be a possible explanation of the saturation of the number of tasks that can be learned.
* Can these features be used for purely offline RL? Or would that introduce some bias which required online learning to learn the task specific policy?
* In the ablations, PVN is compared to a version that learns the successor features for the optimal policy for the auxiliary tasks. The reduced performance of this approach is attributed to the max operator needed for the Q-learning update. This conclusion seems preliminary. Is the reason that the greedy policy fails because of the max? Or because the representations no longer capture the environment dynamics alone (and also represent the policy)? What if the policy being evaluated here was the Boltzmann softmax policy with respect to the task, where the expected sarsa-type update for the successor features could still be performed, but the policy is not random?

### Minor issues and typos:
* In Section 3, it is unclear which policy the successor representation is being learned for. If it is implicitly the random policy, clarify.
* In Section 3, for the indicator function formulation, is previous work that lays out this formulation different from the Dayan paper? If so, cite it.
* Section 4.1 starts off by discussing $\psi^\pi$ but then proceeds to discuss $\psi$. It is unclear why $\psi$ has become policy agnostic.
clarity,_quality,_novelty_and_reproducibility: * Clarity and quality: The paper is clear and straightforward to follow, with high quality and readability.
* Novelty: The paper bridges several ideas through some novel and interesting techniques.
* Reproducibility: The results seem reproducible given the details in the paper. The authors are planning to release learned representations as well.
summary_of_the_review: The paper presents an analysis of a particular class of auxiliary tasks to learn successor features which can be connected to the proto-value functions and thus encode the spatial structure of the MDP. They present practical methods to generate these auxiliary tasks and show effectively that features learned via this method are useful for the downstream task.
The idea is straightforward, the proposed technique is interesting, and the results bear out the approach.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper5901/Reviewer_ZQM9
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper studies representations obtained due to using auxiliary tasks and larger networks. The main claim is that tasks derived from randomly initialized networks are sufficient to train strong representations.

strength_and_weaknesses: The major strength of the work is the fact that it addresses an important problem of augmenting the RL training with unsupervised data. Moreover, the execution and clarity of the paper are very good.

The weakness, in my opinion, is that the presented evaluation protocol is unsatisfactory. 

clarity,_quality,_novelty_and_reproducibility: The paper is clear and nicely executed. The quality of the experiments is high; ablations are satisfactory. In some cases, the number of seeds is low, I would, however treat this as a minor issue as the overall number of experiments seems to be enough to get tangible claims.

summary_of_the_review: The paper studies relevant research questions: how the quality of representations depends on the number of auxiliary tasks and the size of the network. It suggests that it is beneficial to substantially increase the model size (e.g. 8x), while the number of auxiliary tasks needed is around $\approx 10$ (depending on the model size).

Importantly, the auxiliary tasks used in the paper are defined by a randomly initialised neural network, meaning that they are very cheap to obtain. The main finding is that these are enough to induce useful representations even when using random trajectories. 

My main concern is about the evaluation protocol. The authors measure the quality of the representations by the performance of $3.75$M samples training. This seems quite arbitrary for at least two reasons. The first one, very down-to-earth, is why $3.75$M. Are the conclusions stable for other number of samples? Secondly, more philosophically, why to measure the quality of representation by online RL training? Clearly, I can see the practical motivation; however, I can also see some problems due to the fact that RL setup is complex. I can imagine, for example, that representations are good for performance but poor for exploration or vice versa. I do not know a canonical answer to these problems, and I am ready to be convinced in the discussion period.

I would also appreciate some discussion about the games, for which random trajectories do not explore well the state space. 

Questions:
- Could you provide results for other number of samples?
- Does it make sense to measure the quality of representation using some other metric. Simple and perhaps too crude would be if (how fast) the reward can be learned using the representation.
- What would be the benefits for $16x$, i.e. when the improvements are saturated.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold