Paper: Assessing the Impact of Distribution Shift on Reinforcement Learning Performance (ICLR-2024)

Reviewer comment (Presentation):
The authors don't list potential weaknesses of their current method in the main text.

Author response:
Thank you for pointing that out. The second paragraph in section 6 of the improved version now mentions some possible weaknesses of our methodology.

We also invite reviewer okG3 to read the improved version of the main article and appendix. There are some significant changes that we believe enhance the quality of the paper.

Reviewer comment (Writing):
It did not come across to me what is the precise problem the paper is trying to solve. The problem formulation is somewhat vague and relies on artificial examples that make it difficult to connect it with the application of RL algorithms.

Author response:
Sorry for not making this clearer. Accounting for distribution shift has not been previously investigated in reliable RL. This is a more realistic setting for agents that will be deployed in the real world. Hence, we wanted to provide a methodology for this under-researched problem that measures RL performance under distribution shift using time series. Past RL reliability papers rely on point estimates alone. Time series analysis is also needed check if a well-trained agent continues to perform well over a period of time. For example, you would want a methodology in controlled settings that shows how a RL-trained self-driving car performs in the presence of many distribution shifts over a long-enough period of time before releasing it out to the public. The new protocol at the end of section 4 should clarify our proposed methodology. The new analyses in section 5 show that our methodology does not rely on artificial examples.

Reviewer comment (Reproducibility):
The methodology proposed is somewhat scattered. It is unclear how this evaluation methodology would be applied. Perhaps the paper could include a pseudo-code or a flowchart to ground all the steps of the proposed methodology.

Author response:
Thank you for pointing this out. To fix this, we included a clear step-by-step protocol at the end of section 4.

Reviewer comment (Novelty):
The related work section only lists the contribution of related papers but does not provide a description of how this paper distinguishes from those.

Author response:
Thank you for pointing this out. We included a few more sentences at the end of the last paragraph of section 2 of the improved version to distinguish ourselves from past RL reliability work.

Reviewer comment (Theory):
Could you describe more formally the assumption that "the trained agents achieved a clear trend in performance"? In particular, how is this related to the convergence of an RL agent? Does it mean the agent has reached an optimal performance? Furthermore, as the assumption does not consider a distribution shift, does it mean the agent continues to update its behavior?

Author response:
Thank you for catching that. What we should have said was "a flat (slope = 0) trend in raw performance." We updated the article to fix this issue. Here, we just assume some baseline performance (not necessarily optimal) for the causal impact plots. We do not assume the agent updates its behavior. That might be interesting for future research to see how agents with adaptive learning can fare under this methodology.

Reviewer comment (Theory):
Could you comment on the connections from this methodology with non-stationary MDPs?

Author response:
Sure! With some modifications, our methodology should still hold if the parallel trends assumption holds for the treatment and control groups. This would require verification that both groups show parallel trends in the current environment, which I don't expect to hold in all cases. If this line of thinking were to continue, I expect we would need to use other sophisticated methods of causal inference to draw conclusions about agents in non-stationary environments. One drawback of going outside of stationary MDPs is the possible need to use environment-specific knowledge (or covariates besides rewards), which we deliberately avoid in this paper to keep it simple and general. This is out of the scope of this paper, but interesting nonetheless.

Reviewer comment (Evaluation):
Considering the change in the dynamics of the environment, I think we cannot always conclude that the change in performance is due to the agent's behavior. For instance, in some cases, although the dynamics change, the optimal policy may remain the same. Could this methodology help identify if the agent is underperforming?

Author response:
We do not claim our methodology necessarily assigns blame between agent and environment for the decrease in agent performance. Yes, it could be that the agent was able to reach optimal behavior in the training environment, but that does not necessarily mean it will still perform optimally when the test environment is different in some way. Our framework can identify if the agent is underperforming if you know what distribution shift to use during the causal impact experiments. Hopefully, things will become clearer if you read the protocol in section 4.

Reviewer comment (Novelty):
Lack of novelty: The paper does not have a novel contribution. The idea of using simulators to perform interventional analysis is not new [1-5]. Though they talk about the focus in the paper being on "adversarial attacks on images (Atari game observations) and agent switching in multi-agent environments", there is nothing that is specific to the adversarial nature of the distribution shifts. The paper as is will be applicable if distribution shifts happen due to any other factor.

Author response:
We apologize for the confusion. Our paper is a RL reliability/robustness paper. We are following past RL reliability papers like [Measuring the Reliability of Reinforcement Learning Algorithms](https://openreview.net/forum?id=SJlpYJBKvH) and [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://openreview.net/forum?id=uqv8-U4lKBe). The strength and originality of these papers is not in the metrics themselves, but the proposed protocols that ensure reproducible and reliable RL. We contribute a methodology that includes counterfactual time series analysis, using the impact visualization strategy from Brodersen et al. (2015), as a way to measure the impact of distribution shift after training.  For clarity, we explicitly state in the first paragraph of section 2 of our improved version of the article that it is not a causal machine learning paper and does not claim to be the first to use simulators for interventions.

Reviewer comment (Presentation):
Verbose description: The setup is unnecessary and made complex to justify the simple idea of using a simulator to perform interventions. Simple things like an average of sampled data is dedicated a definition and equation (eq. 1), which I think is not needed. Only Section 4.2 is something that talks about a new approach, everything before that is motivation or background.

Author response:
Thank you for the suggestion. In the updated article, we moved eq. 1 and section 4.1 to the appendix.

Reviewer comment (Evaluation):
Insufficient experimental analysis: Experimental evaluation is not sufficient. Figures 3 and 4 are not analyzed in detail, and the inferences from these experiments are not explained properly. Even in the appendix, only the plots are added without any analysis.

Author response:
Thank you for pointing this out. In section 5, we included more in-depth analyses in section 5. In the Atari games, we conclude that RL Baselines3 Zoo PPO agents tend to perform better but the impact plots show clear signs of overfitting in some environments. On the other hand, A2C agents tend to be relatively more robust against adversarial attacks in Atari games.

Reviewer comment (Novelty):
How would you comment on the related literature showing that simulators can be used for interventions? How is your idea new compared to them? My failure to understand this is the biggest reason for my score. Maybe I am missing something, and I would appreciate it if you could comment on it.

Author response:
We apologize for the confusion. Our paper is a RL reliability/robustness paper. We are following past RL reliability papers like [Measuring the Reliability of Reinforcement Learning Algorithms](https://openreview.net/forum?id=SJlpYJBKvH) and [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://openreview.net/forum?id=uqv8-U4lKBe). The strength and originality of these papers is not in the metrics themselves, but the proposed protocols that ensure reproducible and reliable RL. We contribute a methodology that includes counterfactual time series analysis, using the impact visualization strategy from Brodersen et al. (2015), as a way to measure the impact of distribution shift after training.  For clarity, we explicitly state in the first paragraph of section 2 of our improved version of the article that it is not a causal machine learning paper and does not claim to be the first to use simulators for interventions.