From: ~Ezgi_Korkmaz2
title: Adversarial Training and Generalization
comment: The relationship between adversarial training and generalization in deep reinforcement learning has already been discussed in prior work [1,3]. It could also be reasonable to mention that certified adversarial training in deep reinforcement learning has been shown to be non-robust to many different adversarial and non-adversarial settings [1,2,4] from perturbations that can transfer [2] to natural directions [1]. 
It would be reasonable for this paper to refer to these studies.

[1] Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustness. AAAI Conference on Artificial Intelligence, 2023.

[2] Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPs. AAAI Conference on Artificial Intelligence, 2022.

[3] Adversarial Training Blocks Generalization in Neural Policies. International Conference on Learning Representation (ICLR) Robust and Reliable Machine Learning in the Real World Workshop, 2021.

[4] Investigating Vulnerabilities of Deep Neural Policies. Conference on Uncertainty in Artificial Intelligence (UAI), Proceedings of Machine Learning Research (PMLR), 2021.

From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: The paper presents a new measure for generalization in reinforcement learning - the performance of a policy when starting from states generated by another trained policy. The scenario is motivated by autonomous driving where the agent has to take over from a human. The paper shows that even if both policies perform well in isolation, the newly learned policy might be poor at this generalization as it focuses on a different state space. Including states that have occured during training as intial states is shown to solve this issue.

Strength and weaknesses: The paper presents a new and very intuitive measure of generalization for reinforcement learning. In addition, the presented work-around, i.e., including initial states that occurred during early training is intuitive and simple (in a good way) and seems to be effective. In the downside, the experiments could be more exhaustive as only 4 mujuco environments are tested in an "agent takes over from an agent" setting instead of testing the "taking over from a human" setup. Yet, all reviewers agreed that the presented idea is intruiging and could lead to many follow up work.
justification_for_why_not_higher_score: Experiments are too limited.
justification_for_why_not_lower_score: Paper motivates a new way to measure generalization in RL which is very well motivated. Moreover, the solution strategy to get a better generalization regarding this measure is intuitive and effective. 
summary_of_AC-reviewer_meeting: N/A
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper6011/Authors
title: Thank you for pointing out many future directions. 
comment: Thank you for recognizing the importance of our paper and pointing out many future directions. We believe these are very interesting directions to look further. As the first paper introducing the problem of relay generalization, we hope that our work can raise the attention of relay generalization by thoroughly studying these settings on Mujoco and by conducting a few experiments in **Procgen**. We also hope that our STA can serve as a baseline solution for this problem. In the following, we briefly discuss the future directions you suggested. 

> there may be less of an issue with overfitting of the policy to on-policy states and more of an issue with a lack of sufficient exploration from off-policy states. Relay generalization failure may occur because the replay buffer lacks the trajectories needed to identify a path out of the unusual failure states.

We agree that relay generalization is not about overfitting its own policy since the agents trained with STA also only explore **a small part of state space** (Appendix H, Fig 3). The only difference is that STA agents know how to adjust others’ **unseen** postures to their own (Section 3.3, last paragraph). We consider this as good behavior. For example, if a test agent is faster than the stranger agent (e.g., train to be comfortable.), we will want the test agent to run in its own way, which is faster, after taking over the control.

> One question is whether the issue could be at least partially resolved by a more sophisticated exploration mechanism, such as a novelty-based intrinsic reward signal.

We agree that sophisticated exploration mechanism methods may increase the probability of passing relay-evaluation. For example, even though both SAC and PPO have a high return in Humanoid on ordinary evaluation, SAC is better at relay-evaluation since it adds an exploration bonus term in its goal function. However, the failure rate of SAC is still unacceptable. 


> There is also the inherent trade-off between supporting relay generalization, and maximizing on-policy performance. Generating roll-outs that provide the additional data needed for relay generalization means less time spent rolling-out from the true initial distribution, which may incur a performance penalty when under standard evaluation. 

Although there might be a trade-off between relay generalization and performance, we do not observe it in our experiment. Moreover, STA has a better performance than SAC (Our paper Table 5). 

In addition, we even believe the part of STA that skips the beginning states and start from the middle states may increase the efficiency of RL since we don’t need to waste time on the beginning states that the agents are familiar with.  It is just like how humans learn a skill. Humans will focus on the part that they are not familiar with instead of training on all the parts every time. 

> The rate of STA rollouts is currently just a tunable hyperparameter, but it would be interesting to see if this value could be connected to (or computed from) the desired "robustness" of the final policy to adversarial perturbations of the trajectory (i.e., how much the initial state distribution can vary relative to the initial distribution seen during training).

Thank you for your suggestion. We will add this experiment in the revision. Currently, we just let the number of states starting from initial state distribution $d_0$ almost equal to the number of states starting from  $S_{STA}$ in the replay buffer since it is the simplest way to balance the trade-off. But we agree it’s possible to better control the tradeoff based on the desired properties. 

> It would also be interesting to see a version of STA that does not require the ability to reset the environment but instead plans to reach the states from which it wishes to learn to recover.

Thank you for your suggestion. We think this is an interesting direction to pursue.

From: ICLR.cc/2023/Conference/Paper6011/Authors
title: Thank you for your response. We have the relay evaluation results of Procgen. We also discuss your concerns.
comment: As reviewer 2 (KKAE) suggested, we have conducted experiments on 7 of the games in the Procgen. Procgen is like Atari, but each game has numerous variants. [Table 1](https://i.imgur.com/hFirQwk.png) shows the failure rate of playing with the same agent and with different agents. We show that the failure rate of relay-evaluation is likely (5/7) to be twice higher than continuing to play with the same player. More details are [here] (https://openreview.net/forum?id=ipflrGaf7ry&noteId=-MxO3eFb4Am).

 [Link to Table 1](https://i.imgur.com/hFirQwk.png)

We also want to clarify that the components we chose for our STA framework are only to demonstrate that it is possible to improve the relay-generalization with the STA framework. We agree the other kinds of components that you suggested can be better if they are fine-tuned or maybe in a different environment.  More discussions are in the following. 

> I suggest the authors add $\eta$ to the definition even if the choice of $\eta$ doesn't seem to impact the empirical results.

Thank you for the suggestion. We will change it in our revision. 

> I still don't see how adding uncontrollable states can harm performance. 

We agree that it is not necessary that the uncontrollable states will harm performance if we have enough resources to fine-tune. We do not include the uncontrollable states since, under the same amount of training time, we want to train our agent with more trajectories that start from unfamiliar but controllable states as possible.


> I find it weird that the average reward performs worse than using the parameter, as the latter is really domain specific and therefore needs to be fine-tuned.

We agree it will be interesting to find a hyper-parameter-free way to set up the criterion. However, we found that the average reward is noisier since the RL algorithm is conducting exploration during training and may not always follow its best policy in the next hundreds of steps. Therefore using the reward within shorter $\lambda$ steps performs better in practice (as shown in our experiments). Another reason is that some states might have a high average reward in the next 10 steps and then fail. We will not consider these states as controllable.  


> It could also be due to the value of that particular state being low in general.

When we add states to $S_{STA}$, the procedure ensures that the state is controllable (with at least one high enough return). Therefore, if the Q value is low, it means the current agent is bad at this state while there exists a policy to obtain a high return. Hence, we will force the agent to start from this state, hoping that it can “remember” how to play starting from this state.


> I still think that using the Q estimates is a more sensible choice here, as the MC returns can be quite noisy in most environments.

We agree that using the Q estimates can be another valid approach, but since Q may be noisy in the initial training phase, we are not sure whether using Q is better than using the MC returns. In this work, we just want to show that it is possible to improve the relay-generalization with our STA framework, and there are definitely several potential ways to improve our algorithm. In addition, we can not use the new Q to evaluate the state generated by old policies since the new Q is trained to predict the latest policy. (SAC is an off-policy algorithm).

From: ICLR.cc/2023/Conference/Paper6011/Authors
title: We show that relay-generalization can also be an issue in discrete action games like Procgen. 
comment: As you suggested, we conducted an experiment in Procgen and demonstrated that the problem of relay-generalization is not only in Mujuco but also in Procgen. For the experimental setting, we randomly selected 10 of the games in ProcGen. For each game, we train 4 agents with PPO that have the same performance reported in the ProcGen paper [1]. We remove three of the environments where their agents are not able to run 500 steps for more than 70% trajectories. 

To test the relay generalization of each pair of agents, we let the first agent start playing 400 steps and see if the second agent can continue playing for additional 100 steps. If the second agent can keep playing without being terminated by the environment, then we consider it a success. According to the results shown in [Table 1] (https://i.imgur.com/hFirQwk.png), even trained with the same PPO algorithm, agents in most of the environments **(5/7)** have relay-generalization issues. For example, in the game of heist, the agents only have a **2%** failure rate on their trajectories but have a **9.6%** failure rate on relay-evaluation.  This is surprising to us since Procgen’s action space (some only 15 actions to select) is way smaller than Mujoco. 

We also thank you for your suggestion for future directions. We will respond ASAP.

 [Link to Table 1](https://i.imgur.com/hFirQwk.png)

[1] Cobbe, Karl, et al. "Leveraging procedural generation to benchmark reinforcement learning." International conference on machine learning. PMLR, 2020.

From: ICLR.cc/2023/Conference/Paper6011/Reviewer_kkAe
title: Future Directions
comment: So after thinking about relay generalization and STA a bit more, my interpretation is that there may be less of an issue with overfitting of the policy to on-policy states, and more of an issue with a lack of sufficient exploration from off policy states.  Relay generalization failure may occur because the replay buffer lacks the trajectories needed to identify a path out of the unusual failure states.  Even with a large buffer, the fact that we have seen a failure state before does not mean the we have seen a trajectory from the failure state to the desired part of the state space (nor enough data to predict what this trajectory would look like).

This suggests looking at relay generalization from a broader exploration perspective.  One question is whether the issue could be at least partially resolved by a more sophisticated exploration mechanism, such as a novelty-based intrinsic reward signal.  Such reward signals could also be used to guide STA itself in selecting which initial states to begin roll-outs from.  

There is also the inherent trade-off between supporting relay generalization, and maximizing on-policy performance.  Generating roll-outs that provide the additional data needed for relay generalization means less time spent rolling-out from the true initial distribution, which may incur a performance penalty when under standard evaluation.  The rate of STA rollouts is currently just a tunable hyperparameter, but it would be interesting to see if this value could be connected to (or computed from) the desired "robustness" of the final policy to adversarial perturbations of the trajectory (i.e., how much the initial state distribution can vary relative to the initial distribution seen during training).

Though this is a minor issue, it would also be interesting to see a version of STA that does not require the ability to reset the environment, but instead plans to reach the states from which it wishes to learn to recover.

From: ICLR.cc/2023/Conference/Paper6011/Reviewer_7NUq
title: Response to authors
comment: I want to thank the authors for addressing my concerns and for the additional experiments. I've read the other reviews and authors' responses.

First, my concern about the environments being all of the same nature was not addressed by the authors. This was also pointed out by reviewer 'kkAe'.

Although most of my other concerns have been addressed, I find that most of the authors' arguments are based on empirical evidence and lack the appropriate theoretical support.

Given the above, I decided to keep my original score for now.

Please find some follow-up discussions below.

> In definition 2.1, we do not explicitly define “high reward” since it depends on the reward structure of a particular environment, and we want the definition to be generic. In our experiments, we defined “high return” as if the return of a trajectory is higher than the top ratio of all the trajectories. According to our internal experiments, the result is insensitive to this definition, and the observations are almost the same when $\eta \in [0.25, 0.75]$. Hence, we just simply select $\eta = 0.5$.

If a general definition that applies to all environments is not possible, I would probably go with an environment-specific definition instead of using the term "high return", which I honestly find quite vague. I suggest the authors add $\eta$ to the definition even if the choice of $\eta$ doesn't seem to impact the empirical results.

> Our goal is to train the agent such that it can play well starting from controllable states instead of any state; if a state is uncontrollable, then there’s no way to expect the agent to play well starting from that state. Therefore we only add controllable states to the initial set.

I still don't see how adding uncontrollable states can harm performance. Moreover, given that your definition of 'uncontrollable' is rather weak I would argue that going with all states is more sensible.

> Using the Q value may also miss some of the controllable states since the Q value may also be wrong. Moreover, it might even add some uncontrollable states into $S_{\text{STA}}$. Using a high return as the criterion ensures the added states are controllable.

and

> Most of the state’s score is just single runs. Only the states that have been sampled from $S_{\text{STA}}$  as the initial state will have multiple runs. For each of those states, the score is the maximum score of all the scores it has gotten in all the runs. The score served as evidence showing that it is possible to get a high reward at the next $\lambda$ steps from the state. Although we might miss some of the states, the states we added are able to increase the relay-generalization of the STA agent.

I still think that using the $Q$ estimates is a more sensible choice here as the MC returns can be quite noisy in most environments. It is true that the approximate $Q$ value might over(under)-estimate the true $Q$ value. However, this might not necessarily be a problem if the $Q$ estimate is higher than the true value. In that case, the overestimated states may be initially included in $S_{\text{STA}}$  but they will eventually be removed once the estimates become more accurate. If, on the contrary, the estimate is lower than the true value then the state will not be included in $S_{\text{STA}}$. There is little one can do in this case aside from increasing $\eta$. Perhaps one could start with a high value for $\eta$ that is progressively decreased during training as the estimates become more and more accurate.

> If the $Q$ value is low, it is either 1) the agent is bad at this state or 2) the agent is good at this state, but its Q function is wrong. For both scenarios, more training is needed for the candidate state. Moreover, according to the ablation study in Table 2, given the agent is bad at this state, the is normally pretty low.

It could also be due to the value of that particular state being low in general.

> Thank you for the suggestion. We agree that using the average reward (score divided by the number of time steps) is another option. To test this alternative, we conduct an experiment using the average reward of the remaining steps in Table 9, Appendix E. According to the results, using the average reward can also reduce the failure rate compared to the original SAC. However, its failure rate is still higher than using our score function (column STA), which uses the sum of the reward of the next steps.

I find it weird that the average reward performs worse than using the $\lambda$ parameter as the latter is really domain specific and therefore needs to be fine-tuned.

From: ICLR.cc/2023/Conference/Paper6011/Authors
title: Glad to know our main concerns are addressed, and please kindly let us know if you have any further questions
comment: Thank you for reading our rebuttal and responding to us promptly. We are glad that your main concerns have been addressed. We also thank you for reading the thread of other reviewers. Currently, we are still working on clarifying some of the potential misunderstandings with other reviewers. Please kindly let us know if you have any additional questions or concerns regarding our paper. Thank you!

From: ICLR.cc/2023/Conference/Paper6011/Authors
title: We thank you for your suggestions, we want to clarify some misunderstandings and we are running ProcGen experiments
comment: We are glad that most of your concerns have been addressed. Thank you for suggesting the ProcGen benchmark, and we are currently training PPO agents on a few ProcGen environments to see if the agent can pass the relay evaluation, although this benchmark was developed for a different type of generalization. However, before the results are ready, we want to clarify some of the potential misunderstandings. 

First of all, although all the types (e.g., unseen state space,  unseen transition function,  unseen reward function, or unseen initial states (ours)) of generalization of RL identify and solve the RL agent performance on “unseen data,” there is **no** direct connection between different types [1]. That is, even if an agent is good at one of the types, it can still be bad at another type that changes different parts of MDP in the testing phase. Hence, we do not expect STA can improve the performance on all types of RL generalization, such as playing in a different state space like the ProcGen environments. However, as you say, it will be interesting if STA can improve out-of-distribution performance in all the other kinds of types. 

[1] Kirk, Robert, et al. "A survey of generalization in deep reinforcement learning." arXiv preprint arXiv:2111.09794 (2021).

Second, our relay-evaluation is a method to check RL generalization performance under the **same** environment (same state space, transition function, and reward function in MDP), which seems to be a **simpler**  generalization problem for the RL agent than the generalization problems that have  **different**  transition function and state/action space during training and testing. However, in Mujoco, one of the most used environments, many famous RL algorithms fail the relay-evaluation. We believe these results are important to be noticed since Mujoco environments represent an easy version of robot control under real-world physic, and in harder and real-world robot control problems, the lack of relay generalization can be a more severe issue. 

Besides robot control, we believe that there are many (but not all) other environments where RL agents may fail the relay-evaluation, since they will overfit to the states generated by their own trajectory but have a high failure rate on other "out-of-distribution" controllable states in the **same state space**. This will easily be a severe problem for those environments that have a higher degree of freedom since there are multiple solutions. For example, in a game environment where one can kill a monster with either an arrow or a sword, an agent may only learn to pull the sword at the beginning of the game instead of learning that he needs to pull the sword when he doesn’t have the sword in hand.  Therefore, if the agent starts from the middle of the game with a bow in its hand, it will try to kill the monster by swinging the bow since it does not know that it needs to pull out the sword before swinging. Hence, it is important that the RL agents need to be general enough to pass the relay-evaluation.

From: ICLR.cc/2023/Conference/Paper6011/Reviewer_9fm8
title: Response to Rebuttal
comment: Thank you for the response  based on my feedback. This response addresses my main concerns. I have also read the other reviewers' feedback and the corresponding authors' responses. I maintain my score and look forward to discussing more in detail with reviewers and AC.

From: ICLR.cc/2023/Conference/Paper6011/Reviewer_kkAe
title: Reviewer Response
comment: So I believe the authors addressed the questions I raised about the MuJoCo termination criteria, and the effects of replay-buffer size.  I still believe that failure modes that STA addresses may be somewhat specific to the MuJoCo environment.  It would be helpful to explore whether such "relay" failures occur in other domains, and whether similar failures can occur even without the somewhat unusual "relay" setup (simply due to variation in the environment itself).  It would be interesting to see whether STA can improve out of distribution performance in environments specifically designed to asses generalization, such as the ProcGen environments.

From: ICLR.cc/2023/Conference/Paper6011/Authors
title: Thank you for reevaluating our paper!
comment: Dear Reviewer vBmJ,

We really appreciate your prompt reply.
We are sorry about all the confusion. We agree that there is a gap between "agent takes control of humans" and "agent takes control of agents," even though they all belong to “agents take control from controllable states.” 
We are glad to know that the self-driving example is interesting to you. 
In the future, we hope that we can solve the human-agent relay setting based on this work on environments that can be evaluated with human studies. Thank you again for bringing this question up. We will revise our paper to avoid any confusion. We really enjoy discussing our paper with you.

Best regards,
Paper 6011 Authors

From: ICLR.cc/2023/Conference/Paper6011/Reviewer_vBmJ
title: Response to the authors
comment: Thank the authors for addressing my concerns. However, I consider the authors may have some misconceptions. I am not saying that the work "should not be rejected just by a motivating example in the introduction". When I first read your paper, I found the motivation in the introduction of this work very interesting, but later I found the claimed motivation deviated from the proposed methods, and that's why I have some confusion about this work. After the discussion with the authors, I think most of my concerns are released, and the only concern that remains is the human-agent part. Admittedly, I think there remains a gap between "agent takes control of humans" and "agent takes control of agent", and I consider supposing the human behavior as the agent with different RL algorithms is debatable. 

Above all, after the discussion, I think this work is valuable but with a little concern mentioned before. Therefore, I decide to raise the score from 5 to 6 but with less confidence.

From: ICLR.cc/2023/Conference/Paper6011/Authors
title: Thank you for the response!  
comment: Dear Reviewer vBmJ,

Thank you for your suggestion. We will revise our paper by changing the first example from human-agent to agent-agent to avoid confusion. The example of self-driving systems taking over control from humans was a motivating example in the introduction, and it is **not the main claim** of our paper. We used it as an example only because we wanted to emphasize the importance of taking over control from **any controllable state**. Sorry for the confusion.

We will add more discussion regarding the human example to our paper. In short, although we do not conduct human-agent experiments in our paper, we believe that we did make progress in this human-agent example by using agent-to-agent evaluation as a cheaper proxy. First, we demonstrate that the current RL agents are bad at taking over from the agents that are trained even with **the same algorithm**. Hence,  **current RL agents are likely also bad at taking over from humans** since human behaviors are much different from agents.  Second, by observing how our STA agents take over the control directly, we observe that they do learn how to adjust other agents’ states to states that they are familiar with so that they can keep running without failing.  We believe this is an essential skill that an agent needs to learn to take over control from humans, and our study can serve as a preliminary step for future studies involving human evaluation.

Please let us know if you have any other concerns or suggestions. **We sincerely hope that you can re-evaluate our work**, especially since it should not be rejected just by a motivating example in the introduction.

Sincerely,  
Paper 6011 Authors

From: ICLR.cc/2023/Conference/Paper6011/Reviewer_vBmJ
title: Response to the authors
comment: Thank the authors for addressing my concerns and I consider most of my concerns are released. Now only the applications of this work remain uncertain to me. Still, about the self-driving system part, I admit that if one agent takes control of another agent, the proposed framework can come in handy, however, if we need the agent to take control of the humans, I think more discussion is required. I understand "The modeling of human behavior is much harder and beyond the scope of this work", but since this application is the main claimed application in the submitted paper, I think the authors need a more rigorous discussion. To me, "agent takes control of humans" is more important than "agent takes control of agent". 

Moreover, regarding the reproducibility part, I want to clarify that my evaluation of reproducibility is only to state the status of this paper, and not require the authors to open-source their code ASAP. If I have transferred such a message, please accept my apologies.

From: ICLR.cc/2023/Conference/Paper6011/Authors
title: Thank you for the very helpful review. We answer your questions on STA training, relay-evaluation and Tandem RL below (2/2)
comment: > (Weakness bullet 2) Is the score of a state averaged over multiple runs?

Most of the state’s score is just single runs. Only the states that have been sampled from $\mathcal{S}_{STA}$ as the initial state will have multiple runs. For each of those states, the score is the maximum score of all the scores it has gotten in all the runs. 
The score served as evidence showing that it is possible to get a high reward at the next $\lambda$ steps from the state. Although we might miss some of the states, the states we added are able to increase the relay-generalization of the STA agent. 

> (Weakness bullet 6) How is $\eta_{\text{STA}}$ computed?

It is selected by experiments of using different $\eta_{\text{STA}} = 0.5, 0.75, 0.9, 0.95$.  

> (Weakness paragraph 5) Simulator can be reset to any state

Although resetting to any valid state is possible for most environments (e.g., by setting physical states of objects or game memory and program states), if simulators do have restrictions, we can include only those states that are both controllable and "reset-able."

### Questions of Relay-evaluation

> (Weakness paragraph 1) Evaluation is computationally heavy it requires training multiple other agents to ensure most of the test and stranger agents are not very similar

In our experiments, **failure is very common**. It even happens on the agents trained with the same algorithms as stranger agents. According to our results, all agents in Humanoid have at least a 10% failure rate on any other agents' trajectories, so one can observe the problem of relay generalization even when sampling only one agent. We evaluate each agent by multiple (4*10) other agents **just for a thorough and comprehensive comparison** and to reduce the variance in reported results, and users can choose the number of agents according to their budgets. In addition, we will ​publicize the **states of stranger agents as a dataset** so that people can reproduce our experiment easily without training a large number of agents

> (Weakness paragraph 2) Definition of controllable state: what does “high return” mean

In definition 2.1, we do not explicitly define “high reward” since **it depends on the reward structure of a particular environment**, and we want the definition to be generic. In our experiments, we defined “high return”  as if the return of a trajectory is higher than the top $\eta$ ratio of all the trajectories. According to our internal experiments,  the result is insensitive to this definition, and the observations are almost the same when $\eta \in [0.25, 0.75]$. Hence, we just simply select $\eta=0.5$. 

> (Weakness paragraph 2) how large does L need to be

In our setting, the maximum steps of the environments are “$L_{max}$=1000”.  If L is too small, the agent might pass the relay-evaluation by luck. If L is too large, we will have fewer states to sample since we will only sample $s_t$ that $t<L_{max}-L$. Hence, we select $L=500$. We also conduct experiments with different $L$ in [Table 11](https://i.imgur.com/c4rt8Op.png) in Appendix I. The results show that most of the failures happened in the first 100 steps after the test agent took over the control. The results also show that for some of the cases, even if the agent does not fail in the next 100 steps, it still has a chance to fail in the next 400 steps. 

[Link to Table 11](https://i.imgur.com/c4rt8Op.png)

### Tandem RL 

> “The Difficulty of Passive Learning in Deep Reinforcement Learning”

Thank you for sharing this interesting paper. We have cited this paper in our revision.
Tandem RL shows that it is hard to **train** an offline RL agent even if the agent can observe the historical trajectory of another agent and follow all the training processes. On the other hand, relay-evaluation shows that during **evaluation**, it is difficult for a well-trained agent to start from a state of another agent’s trajectory. 

### Typos

Thank you so much for carefully reading our paper. We have fixed those typos. Thanks!

From: ICLR.cc/2023/Conference/Paper6011/Authors
title: Thank you for the very helpful review. We improved paper clarity and answer your questions on STA training, relay-evaluation and Tandem RL below (1/2)
comment: We thank you for all the valuable comments and suggestions. We have **revised Section 3.2** according to your suggestions to improve clarity and added pseudo code to present our algorithm clearly. As the pseudo-code shows, STA is a very simple framework. In the following, we first conduct **several ablation experiments** to justify our choice of the components in STA.  We then address concerns on relay-evaluations. Finally, we compare our work to Tandem RL.



### Questions about our STA training method

> (Weakness paragraph 4) Can the STA training scheme negatively impact the agent’s performance? Is it possible that the optimal policy for this augmented MDP is different from that of the original MDP?

Although the optimal policy for our augmented MDP may be different from the original MDP, the results in Table 4 show that our methods can usually improve the return compared to standard training. In Table 4,  we show the results of the agent’s performance in the ‘ordinary’ rows. As the results show, the ‘Return’ column of STA and the naive method is greater or equal to the ‘Return’ column of SAC in all three environments. Hence, we believe that our method is **similar to data augmentation in computer vision, which can lead to a solution with better generalization and testing performance**.

> (Weakness bullet 1) Some states may not be included just because these bad policies can't perform well when starting from them.

We agree that some controllable states may not be included in $\mathcal{S}_{STA}$. During training, we need a quick estimation for the "controllability," so we only added states that can get a high reward in the next $\lambda$ steps.  **It is unavoidable to miss some controllable states because it can be costly to enumerate all of them during training**; similarly, in data augmentation, we cannot add all possible ways of augmentation and rather just sample some augmented images. By adding these states (which may be a subset of controllable states), we hope the agent can be more generalizable and work well with other controllable states. According to our experiments, even missing some of the controllable states, STA can greatly improve the relay-evaluation results.

> (Weakness bullet 3) Why do the states need to be controllable to be added to the initial set

Our goal is to train the agent such that it can play well starting from controllable states instead of any state; **if a state is uncontrollable, then there’s no way to expect the agent to play well** starting from that state. Therefore we only add controllable states to the initial set.  

> (Weakness bullet 3) What happens if you just add all states to the starting set regardless of whether they are controllable?

We agree that adding some uncontrollable states to $S_{STA}$ might be helpful. However, including too many uncontrollable states is not always a good idea. In [Table 7](https://i.imgur.com/oKArMUY.png),  we show that if we further increase $\gamma$ (adding more uncontrollable states to $S_{STA}$ ), the Humanoid performance will drop, indicating adding too many uncontrollable states is not beneficial.

[Link to Table 7](https://i.imgur.com/oKArMUY.png)


> (Weakness bullet 4) Why is low $Q(s,\pi(s))$  a good metric to determine if the agent is not familiar with a particular state?

If the $Q(s,\pi(s))$ is low, it is either 1) the agent is bad at this state or  2) the agent is good at this state, but its Q function is wrong. **For both scenarios, more training is needed for the candidate state $s$**. Moreover, according to the ablation study in Table 2, given the agent is bad at this state, the  $Q(s,\pi(s))$ is normally pretty low.

> (Weakness bullet 5) Parameter $\lambda$: Can't you just divide the score by the number of remaining timesteps?

Thank you for the suggestion. We agree that using the average reward (score divided by the number of time steps) is another option. To test this alternative, we conduct an experiment using the average reward of the remaining steps in [Table 9](https://i.imgur.com/9cSUeNK.png), Appendix E. According to the results, using the average reward can also reduce the failure rate compared to the original SAC. However, its failure rate is still higher than using our score function (column STA), which uses the sum of the reward of the next $\lambda$ steps. 

[Link to Table 9](https://i.imgur.com/9cSUeNK.png)


> (Weakness bullet 2) Why don’t you use the Q network to estimate the score?

Using the Q value may also miss some of the controllable states since the Q value may also be wrong. Moreover, it might even add some uncontrollable states into $\mathcal{S}_{STA}$. Using a high return as the criterion ensures the added states are controllable.

From: ICLR.cc/2023/Conference/Paper6011/Authors
title: Thank you for the encouraging comments! We answered your questions about our weaknesses below.
comment: Thank you for the positive and encouraging review! We address your concerns below.

>  Reproducibility

To increase the reproducibility of our STA algorithm, we have rewritten section 3.2 with the pseudo codes provided. We also upload our code anonymously ( https://anonymous.4open.science/r/spinningup-E8B1). The implementation of STA is in ./spinup/algos/pytorch/gsac/gsac2.py. We will open-source our project in the future.

> Authors sampled states from trajectories generated by stranger agents, which are trained with only different random seeds.

We also sampled states from trajectories generated by stranger agents, which are trained with different **algorithms**  and different seeds. 

> PBT[1] might be of help in finding more "controllable" states

Thank you for sharing this interesting paper. We have cited it in our revision. 
In our work, we have generated different policies using a diverse set of training algorithms, but we agree that it’s interesting to generate diverse policies using different hyperparameters. To study whether the diversity of hyperparameters helps, we conduct the following experiments. We train 40 SAC agents by choosing hyperparameters randomly and then use the top 10 agents to generate controllable states. The relay evaluation results on these controllable states are shown in [Table 12](https://i.imgur.com/DJIKINx.png) in Appendix J. According to Table 12, we observe that the states generated by random hyperparameters are easier for other agents to pass the relay-evaluation, especially when evaluating SAC and TD3 agents. A potential reason is that some of these hyperparameters lead to worse policies and generate slower trajectories, and it’s easier for the target agent to take over from those slower states. 

[Link to Table 12](https://i.imgur.com/DJIKINx.png)

> Many components applied in STA are more likely intuitive.

As the first paper that tries to define and solve the relay-generalization, we hope our method can serve as a simple and intuitive baseline and could be widely used in future developments of potentially more complicated algorithms.

> a provable improvement might not be guaranteed 

We agree a formal theoretical guarantee could be an interesting future work. 
To experimentally show the consistency of our improvement, we not only test on different environments but also on different stranger algorithms. The results in Table 4 show that our method can consistently drop the failure rate dramatically.

From: ICLR.cc/2023/Conference/Paper6011/Authors
title: Thank you for the insightful review, we have added additional experiments and clarified questions (2/2)
comment: > (Q4) Definition 2.1 is not entirely clear.

Thank you for bringing up the question. We do require valid actions to move to the next states (no jumps, it’s not an arbitrary state sequence), although we focus on states and not actions. We have revised the definition to make it more clear in the paper by including the actions in our definition of a trajectory. 

> (Q3) The "Failure" criteria should be described for each environment

Thank you for the reminder. The criteria defined by OpenAI Gym Mujoco environments are now listed in Appendix F.

> (Q5) it is not explicitly stated that the "stranger" and "test" policies are trained using the same configuration for the same number of steps.

Different algorithms use different configurations. For off-policy algorithms like SAC, TD3, the naive method, and STA, we use the same configuration. For example, they all trained for three million interactions.  For on-policy like PPO, SA-PPO, and ALTA-PPO, they are all trained for one million interactions. For other parameters of the on-policy algorithm, we follow the fine-tuned parameter of those papers’ GitHub. More details are in Appendix A.

> (Q2) Table 1 should show returns

As you suggested, we list the returns of the agents in [Table 13](https://i.imgur.com/EbiECNJ.png) in Appendix K.

[Link to Table 13](https://i.imgur.com/EbiECNJ.png)

> (Q1, Q6) Typos, related work section is a long block

We have fixed the typos and divided the related work section into a few points. Thank you for the suggestion.

From: ICLR.cc/2023/Conference/Paper6011/Authors
title: Thank you for the insightful review, we have added additional experiments and clarified questions (1/2)
comment: We thank the reviewer for the clear summarization and all the constructive suggestions. To address your concerns, we conduct additional experiments to show that the lack of generalization in relay-generalization is persistent under different failure criterions and infinite replay buffer size. We also demonstrate that our STA agents indeed generalize better to states they did not find during training. 

> (Q3) It is possible the termination criteria in the MuJoCo environments are too strict and terminate policies that might eventually succeed.

Thank you for pointing this out this possibility. To understand the severity of this problem, we conduct the following experiment. Instead of terminating the agent when it is flagged as "terminated" by the environment, we disabled termination in environments and let the agent runs to the maximum number of steps anyway and redefines "failure" by the return and a threshold. If the agent gets a return lower than the threshold, then we define  the agent fails. Since the expected return of agents in standard evaluation are all greater than 2800, we use three different thresholds (500, 1000, 1500) to define failure in our experiment. 

The results are shown in [Table 10](https://i.imgur.com/AipVY1U.png) in Appendix G and Table 10. According to the results, there did exist trajectories that eventually succeeded. However, the failure rates are still unacceptably high. For example, even if we define having a return that is higher than 500 as success, the fail rate on Humanoid SA-PPO is still 81.83, which is not much different from Mujoco’s definition. In general, we are only trying to provide the general evaluation framework for relay-generalization in this paper, and the definition of "failure" may depend on the use cases and the environments.

[Link to Table 10](https://i.imgur.com/AipVY1U.png)
> (Weaknesses) How much of an issue would this be with fully off-policy methods with large replay buffers?

Both SAC and TD3 are off-policy methods with large replay buffers. Their replay buffer sizes are one million, while the total number of environment interactions is only three million. To understand if increasing the replay buffer will help, we train the SAC agent with an infinite replay buffer. That is, all the trajectories will always be in the replay buffer. The results are in [Table 8](https://i.imgur.com/9R4AJqR.png) in Appendix D. 


According to the results, SAC with an infinite replay buffer still has poor performance on relay evaluation compared to our method. Moreover, the average return under ordinary evaluation is lower than the vanilla SAC, while our method is greater or equal to the vanilla SAC.

> (Weakness) it is unclear how much it can improve generalization performance overall, for example, from states that no policy would be likely to find during training

During STA training we do not use the agents for relay-evaluation (no leaking of test data); only 1 STA agent is trained using SAC.The trajectories used in relay-evaluation are generated by many stranger agents, and it's very unlikely that they appear during STA training. The purpose of this paper is to evaluate agent performance on a diverse set of valid and controllable states (much larger than standard evaluation) that are unlikely to be seen during training, and the good performance of STA shows that it can improve generalization performance in this setting. Indeed, we cannot evaluate generalization to states that no stranger policy can generate. However, to our best knowledge, no previous works can evaluate this type of generalization in complex environments because of the hardness of finding valid and controllable states for evaluation. Our paper has made a major step towards better generalization evaluation via relay evaluation.

For empirical evidence, to show that the states of the stranger agents are indeed unseen to our STA agent during training, we use t-SNE to visualize the historical states of an STA agent during training along with the states from 6 stranger agents. The figure is shown in [Figure 3](https://i.imgur.com/ZY0lENZ.png) in Appendix H. According to Fig. 3, although the distribution of historical states of the STA agent is more diverse than the distribution of other agents’ states, there are still many areas that it does not cover. This suggests that the STA agents have a better performance on relay evaluation due to being more general, not just memorizing its training states

[Link to Figure 3](https://i.imgur.com/ZY0lENZ.png)

> (Weakness) the solution STA is relatively simple

As the first paper that tries to define and solve the relay-generalization, we hope our method can serve as a simple and intuitive basic method and could be widely used in future developments of potentially more complicated algorithms, just like data augmentation in image classification.

From: ICLR.cc/2023/Conference/Paper6011/Authors
title: Thank you for your insightful questions, we addressed your concerns on motivation of our work below 
comment: Dear Reviewer vBmJ,
We really appreciate your constructive comments. We believe the “gap” between the experiments and the claimed application scenarios, as you mentioned, is partially caused by a misunderstanding. We will address each of your questions and concerns below:

> "a self-driving system may need to take over the control from humans in the middle of driving and must continue to drive the car safely.” 

In this example, the self-driving system is the test agent, and the humans are the stranger agent. Humans are strangers to the self-driving system since it is hard to fully present the "driving style" of a particular user when training the agent. However, the self-driving system still needs to be general enough to safely take over control from a state produced by an unknown human driver. Our relay-evaluation is one way to evaluate agent performance starting from the states that it may not see during training.

> Agents using *different* RL algorithms are considered the main part of the experiments…there exists a gap between the experiments and the claimed application scenarios

We consider stranger agents trained using **both the same and different** RL agents because a wider range of agents will cover more controllable states and leads to better evaluation of generalization, hoping to **reduce the gap** to realistic application scenarios. In the human driver example, although it is difficult to exactly model human behavior, if we can cover as much space of controllable states as possible during evaluation (e.g., using states generated from multiple RL algorithms), we are more likely to cover the states that human drivers will produce.

> I wonder does there exist a gap between the experiments and the claimed application scenarios.

Because human behaviors are hard to model, we agree there may exist a gap between the experiments and *real-world* application scenarios since we only consider relay-revaluation of a diverse set of RL agents, not humans. The modeling of human behavior is much harder and beyond the scope of this work. Our main contribution is to provide a cheap and practical way to evaluate relay-generalization as the first step to solving the real application scenario. In addition, taking over the control from other RL agents (not just humans) also has real application scenarios. For example, assume we have driving agents trained for different goal functions such as speed, gas efficiency, or comfortability. A user may want to switch from the most comfortable agent to the fastest agent since he is running late.

> Can authors compare their works with several popular meta-RL frameworks?

We have added a discussion of meta-RL in Related Work. The goal of relay generalization is different from meta-RL. In our work, the test agent is only trained for **one** task, and we evaluate if the test agent can complete the task starting from a stranger agent’s trajectory. In Meta RL, people normally first train an agent for task A and use the prior knowledge learned in task A to train another agent for task B. On the one hand, our method does not require the agent to do another task. After the test agent takes over the control, its goal is still to complete its original task. On the other hand, we aim to study the generalization to a "stranger agent's trajectory." Thus the agent **cannot know** the stranger agent's trajectory beforehand, and **cannot be fine-tuned** according to the stranger agent's trajectory.

> So, I begin to question the performance of STA, maybe the simple way in the naive algorithm can already handle relay evaluation?

The biggest difference between STA and naive is the **training cost**. For the naive method, if we pretrain K agents, then it will increase the training time (and the number of environment interactions) by K+1 folds, which is usually impractical.  In contrast, STA barely increases the training time compared to vanilla SAC.  For example, the time for training SAC in Humanoid is about 130,000 seconds, while the training time of STA is about 140,000 seconds. Moreover, in the hardest environment – Humanoid, STA is consistently better than the naive method by a significant margin. For example, if the stranger algorithm is SA-PPO, the failure rate of STA is 25%, while the naive method is 58%.

> Reproducibility

To increase the reproducibility of our STA algorithm, we have rewritten section 3.2 with the pseudo codes provided. We also upload our code anonymously ( https://anonymous.4open.science/r/spinningup-E8B1). The implementation of STA is in ./spinup/algos/pytorch/gsac/gsac2.py. We will open-source our project after our paper is accepted.

We thank the reviewer again for your insightful comments, and we hope we have addressed your concerns on the motivation, connection to meta-RL, and the differences between native and STA agents.

Sincerely,
Paper 6011 Authors

From: ICLR.cc/2023/Conference/Paper6011/Reviewer_vBmJ
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: In this paper, the authors propose a new experimental setting for RL called relay evaluation, which tests the ability of the RL agent to handle the trajectory of other agents (strangers) in the mid-way. Such a situation may be helpful in the application of RL when the agent needs to take control from humans and is seldom considered in the literature. It turns out that current RL algorithms can easily fail in the relay evaluation. Therefore, the authors propose a naive framework and a self-trajectory augmentation framework to improve the generalization ability of the RL agent, with any RL algorithm in theory. They use the trajectories, either trajectories from other stranger agents or historical trajectories to enhance the RL agent. The results on multiple datasets with different RL algorithms show the performance of the proposed frameworks.
strength_and_weaknesses: Strength:

1. To the best of my knowledge, the authors are the first to consider the relay evaluation setting in the field of RL, which does have real-world usage such as in the self-driving system. I think this evaluation could be used as an important check before the application of RL to the real world.
2. The results of the relay evaluation show that existing RL algorithms can easily fail to handle this situation. Figure 1 illustrates the results well and shows a possible explanation for the failure.
3. The authors propose two frameworks to improve existing RL algorithms, both of which better utilize the trajectories, either trajectories from other stranger agents or historical trajectories. In theory, these frameworks can be combined with any RL algorithms and are easy to modify.

Weakness:

1. In my view, the motivation of this work is unquestionable but I have some concerns. First, the authors provide an example situation that "a self-driving system may need to take over the control from humans in the middle of driving and must continue to drive the car safely."  Under such a situation, the algorithm used in the RL agent is known by the users, right? However, in the experiments, agents using different RL algorithms are considered the main part of the experiments (A-B pairs in Table1). I wonder does there exist a gap between the experiments and the claimed application scenarios.
2. If we consider the adaptation of different trajectories, I think the works of meta-RL may be very relevant to this work. In my view, meta RL can handle different distributions of the environment, which could be seen as different trajectories as well. Therefore, could the authors compare their works with several popular meta-RL frameworks?
3. After looking at the results of Table 2 and Table 3, I think the performance of STA is kind of unsatisfactory compared to the naive algorithm. So, I begin to question the performance of STA, maybe the simple way in the naive algorithm can already handle relay evaluation?
clarity,_quality,_novelty_and_reproducibility: In my view, the novelty of this work is unquestionable since the relay evaluation is hardly considered in the literature. But the quality of the work is not so satisfactory due to the weakness I find. As for reproducibility, I can not make a judgment since the authors do not plan to open-source their code.

summary_of_the_review: This paper aims to solve the ray evaluation issue which is hardly considered in the literature. However, except for this point, the other parts of this paper look not good enough to me. As I mentioned in the weakness part, the authors need to clarify their motivation and enhance their experiments. Therefore, I consider this paper is not yet ready for ICLR. Admittedly, I am not an expert in this area, if there are some fatal mistakes, please feel free to tell me.

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6011/Reviewer_kkAe
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper discusses the challenge of "relay generalization", where a policy trained with RL is evaluated when starting from initial states observed under an independently trained policy.  Relay generalization is intended to measure the out-of-distribution generalization error, by testing how well the policy performs starting from the "reasonable" states found by another good policy.  They demonstrate in some of the MuJoCo environments that policies that perform well from from the environment's default initial state distribution can fail catastrophically when they are required to take over control from other, equally effective policies.

They determine that the issue stems at least in part from the fact that independently trained policies tend to settle into different regions of the state space, and so never learn how to take successfully control the agent from the states that other policies concentrate on.  The propose a simple method to mitigate this issue, in which the current policy is, with some tunable probability, rolled-out from a state encountered previously during training, rather than an initial state of the environment.  They demonstrate that this significantly improves relay generalization when applied to the soft actor-critic algorithm.
strength_and_weaknesses: The main contribution of the paper is the idea of "relay generalization", and empirical demonstration that this issue actually exists.  While it is not entirely surprising that on-policy methods would fail when asked to take control from a state they would not typically encounter during training, this work provides a means of directly measuring this type of out-of-distribution error.  They also provide some qualitative insight into exactly why these failures occur in the MuJoCo environments.

The main weakness of the paper is that the empirical results are limited to four MuJoCo environments.  The failure mode, at least in the humanoid environment, seems to be somewhat specific to this setting, and so it is unclear how large a problem this would be in general.  It is also unclear how much of an issue this would be with fully off-policy methods with large replay buffers, which might have a similar mitigating effect to the initial state sampling method they propose here.  Additionally, the solution they present (STA) is relatively simple, and it is unclear how much it can improve generalization performance overall, for example, from states that no policy would be likely to find during training.
clarity,_quality,_novelty_and_reproducibility: The novelty of the work comes mainly from the explication of relay generalization as a specific form of generalization error in RL.  The article appears to be technically correct, and is clearly for the most part.  A few suggestions for improvements, however:
1. The related works section could be more organized, as it is currently just a long block of text.
2. Table 1 (or an additional table) could show actual returns in addition to failure rates.  Table two provides this, but divided out between "failure" and "success" trajectories, so the actual impact of relay generalization of quantitative performance is not immediately clear.
3. "Failure" in this case is determined by each environments termination criteria, these should be described for each environment, as it is possible the termination criteria in the MuJoCo environments are too strict, and terminate policies that might eventually succeed.
4. Definition 2.1 is not entirely clear, as it does not seem to require that a trajectory be "feasible" given the action space available to the agent (seems to admit trajectories that arbitrarily jump between states).
5. While we assume this is the case from the context, it is not explicitly stated that the "stranger" and "test" policies are trained using the same configuration, for the same number of steps.
5. There are a number of typos, for example "very slow" rather than "very low" in the first paragraph of page 4.
summary_of_the_review: Overall this work empirically demonstrates an interesting phenomenon that is important for RL researchers to be aware of.  It provides a heuristic solution to this specific issue, but does not tackle the problem of out-of-distribution error more generally.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6011/Reviewer_9fm8
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors propose "relay-evaluation", a method for evaluating the generalization performance of RL agents. Specifically, it forces the agent under test to start from the state s\_0, which is sampled from the trajectory generated by stranger agents. A sharp degradation of performance is observed under relay-evaluation, which proves the limitations of existing methods. Correspondingly, the authors proposes STA, a novel method that can be applied in any modern RL algorithm, to improve the generalization ability.

Furthermore, there is no code or any other supporting material for this paper, and the reproducibility of the paper is unconvincing.

strength_and_weaknesses: #### Strength

Overall, generalization is a popular and essential topic in RL, and many methods have been explored to evaluate or improve the generalization of RL agents from different aspects. It is quite interesting that the authors studied the generalization from a brand-new point of view and proposed a concise but effective method. The experiments are compelling and well-rounded. I think the paper did contribute some new ideas and open up a new direction worth being studied.

#### Weakness

Despite of the outstanding performance illustrated in the experiments, a provable improvement might can not be guaranteed as many components applied in STA are more likely intuitive.   

In relay-evaluation, authors sampled states from trajectories generated by stranger agents, which are trained with only different random seeds. PBT[1] might be of help to find more "controllable" states where test agent fails and thus evaluating the generalization more accurately.

Not really a weakness, more a suggestion. A pseudocode that describes the process of STA overall might help authors understand it more clearly. Would you consider adding it to the paper in the next version, please?

There is no code or any other supporting material for this paper, and the reproducibility of the paper is unconvincing.

clarity,_quality,_novelty_and_reproducibility: #### Clarity

The paper is mainly well-written and is generally easy to follow. The phenomenon that existing methods fail in relay-evaluation is motivating, and the analysis is persuasive. 

#### Quality

The quality of the paper is high, and the experiments are compelling, as far as I can see.

#### Novelty

To my knowledge, the paper is the first to propose a new method for evaluating the generalization of RL agents on "controllable" states and improve it via STA, a new training paradigm.

#### Reproducibility

A/N

summary_of_the_review: Overall, the paper is very interesting as it studies the generalization of RL agents from a novel view, despite some minor defects like the lack of provable improvement. The paper is well organized, and the analysis of the experiments is convincing. The proposed method seems concise but effective enough without introducing extra sample complexity.  

[1] Derek, K., & Isola, P. (2021). Adaptable Agent Populations via a Generative Model of Policies. *Advances in Neural Information Processing Systems*, *34*, 3902-3913.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6011/Reviewer_7NUq
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes “relay-evaluation” a method to asses the generalization capabilities of RL agents when being forced to start from a state that is out of the distribution defined by the agent’s policy. The paper also introduces “Self-Trajectory Augmentation” (STA) a training method to improve generalization in which the agent is randomly set to start from a state that is sampled from one of the agent’s old policies.
strength_and_weaknesses: ### Strengths
This paper addresses an important problem in RL that I believe hasn’t been explored sufficiently. Generalization in RL is normally regarded as the ability of an agent to perform well on similar tasks/environments.  However, this paper evaluates generalization to out-of-distribution trajectories, showing that, even when the environment stays the same agents may fail to perform well if states are sampled from a different policy. The paper is clearly written, for the most part, and the experiments are thorough.

### Weaknesses
The relay-evaluation metric is computationally heavy it requires training multiple other agents to ensure that there is sufficient variety in the policies of the strangers. If the number of stranger policies is small, the method may incorrectly determine that an agent is able to generalize well just because the algorithms used for training the test and stranger agents are very similar.

The definition of “controllable state” is somewhat loose (i.e. what does “high return” mean and how large does L need to be?).

I would have liked to see experiments on other types of environments. The environments used are all of the same nature. Agents need to learn to walk/move and keep their balance. These require a lot of coordination and thus it is natural for the agents to overfit to a particular trajectory.

Can the STA training scheme negatively impact the agent’s performance? As explained in Section 3.1 STA is equivalent to training in an augmented MDP with a larger initial state set. Is it possible that the optimal policy for this augmented MDP is different from that of the original MDP? 

As far as I can see, a simulator that can be reset to any state is needed for STA.

I found Section 3.2 particularly unclear: 

  * The score used by STA to evaluate if a state is controllable depends on the policy being followed. Hence at the beginning of training when policies perform badly, the inclusion of states in the initial set seems arbitrary. That is, some states may not be included just because these bad policies can't perform well when starting from them.

 * Is the score of a state averaged over multiple runs? Or is it just a single sample estimate? The latter is a poor estimate of the true score if the policy and/or the environment are stochastic. Why don’t you use the Q network to estimate the score?

 * I am not sure why the states need to be controllable to be added to the initial set. What happens if you just add all states to the starting set regardless of whether they are controllable? Wouldn’t that also improve generalization?

 * Why is $Q(s, \pi(s))$ a good metric to determine if the agent is familiar (not familiar) with a particular state? A high (low) Q value may be due to the agent being overconfident (underconfident) about that particular state.

 * The use of a parameter $\lambda$ to truncate the return is a bit hacky. Can't you just divide the score by the number of remaining timesteps?

 * How is $\eta_{STA}$ computed?
clarity,_quality,_novelty_and_reproducibility: ### Clarity
The paper is generally clear although it could benefit from proofreading. The second half of the paper where STA is introduced is hard to follow. In particular, Section 3.2 is quite convoluted. It is difficult to parse how the method works especially given that STA uses so many new hyperparameters.  Perhaps adding bullet points or pseudocode explaining step-by-step how states are added to the initial set may improve clarity.

Typos/grammar mistakes:

Abstract “generalizatin failure” -> “generalization failure”

“pretraining additional models are time-consuming” -> “pretraining additional models is time-consuming”

“based the experiment of Table 2” -> “based ON the experiment IN table 2”? 

“which are very slow” -> “which are very low”?


### Novelty 
I found the paper novel, although I must admit I am not very familiar with the literature on generalization in RL.

The idea of “relay-evaluation” seems similar in nature to that of “Tandem RL” (Ostrovski et al., 2021). Could the authors comment on how these two papers are related?

Ostrovski et al., 2021: The Difficulty of Passive Learning in Deep Reinforcement Learning.

### Reproducibility

Source code is not provided. However, the method should be relatively easy to implement.

summary_of_the_review: I am overall in favor of accepting this paper. I really liked the first part of the paper where the problem of generalization to out-of-distribution trajectories is evaluated using the relay method. I am a bit less excited about STA since the method makes lots of choices that seem rather arbitrary and very domain dependent. I am tentatively giving the paper a score of 6 but I am willing to increase it if the authors can clarify some of my concerns in their rebuttal.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold