Paper: Tackling Non-Stationarity in Reinforcement Learning via Causal-Origin Representation (ICLR-2024)

Reviewer comment (Experiments):
I am not convinced that this approach is better than the assumption of strong prior knowledge required to model NS in the system.

Author response:
Our paper does not assert performance superiority over methods that utilize prior knowledge about the environments. Rather, we concentrate on addressing the impracticality of these methods in broader scenarios, where acquiring detailed prior knowledge is often challenging or unfeasible. Indeed, our approach introduces additional computational demands, but this trade-off is accepted to broaden our capability in addressing NS challenges in RL. In more complicated scenarios, the necessary prior knowledge for NS is often unavailable or unknown, limiting the applicability of methods that heavily rely on such information. By developing a methodology that effectively addresses NS without extensive prior knowledge, we believe our contribution to the field is meaningful.

Reviewer comment (Evaluation):
This appears to be far more computationally burdensome.

Author response:
Our paper does not assert performance superiority over methods that utilize prior knowledge about the environments. Rather, we concentrate on addressing the impracticality of these methods in broader scenarios, where acquiring detailed prior knowledge is often challenging or unfeasible. Indeed, our approach introduces additional computational demands, but this trade-off is accepted to broaden our capability in addressing NS challenges in RL. In more complicated scenarios, the necessary prior knowledge for NS is often unavailable or unknown, limiting the applicability of methods that heavily rely on such information. By developing a methodology that effectively addresses NS without extensive prior knowledge, we believe our contribution to the field is meaningful.

Reviewer comment (Theory):
Theoretically, the authors need to justify why such a more computationally complex solution is justified given the existing solutions.

Author response:
We thank the reviewer's feedback and hope to precisely address your confusions. 

In the settings used in our work, non-stationary changes are **continuously varying**, meaning that the environment cannot be decomposed into separate mixtures of pre-defined stationary environments. As you mentioned, mixture-based MMAC methods necessitate pre-defined stationary environments, making them unsuitable for the continuously varying non-stationary environment in our work. In such an environment, where **predefined prior knowledge is absent**, mixture-based MMAC methods become unworkable. Therefore, such a setting is **not a solved problem**, and our COREP method is designed to handle these scenarios.

We again remind the reviewer that our theoretical attempt to decompose the environment into K distinct environments serves merely as **an inspiration or theoretical guide for our algorithm design**, rather than reflecting the actual environment used, where K could theoretically be infinitely large, approximating continuous non-stationarity.

If you still have any questions, we would greatly appreciate your further feedback.

Reviewer comment (Novelty):
In control theory, there are well-known solutions to systems under such an assumption, for example, multiple model adaptive control (MMAC) is a known solution. Given a collection of pre-defined stationary environments, if we generate a controller for each of these environments, then we have a guaranteed controller for any mixture of these stationary environments.

Author response:
We thank the reviewer's feedback and hope to precisely address your confusions. 

In the settings used in our work, non-stationary changes are **continuously varying**, meaning that the environment cannot be decomposed into separate mixtures of pre-defined stationary environments. As you mentioned, mixture-based MMAC methods necessitate pre-defined stationary environments, making them unsuitable for the continuously varying non-stationary environment in our work. In such an environment, where **predefined prior knowledge is absent**, mixture-based MMAC methods become unworkable. Therefore, such a setting is **not a solved problem**, and our COREP method is designed to handle these scenarios.

We again remind the reviewer that our theoretical attempt to decompose the environment into K distinct environments serves merely as **an inspiration or theoretical guide for our algorithm design**, rather than reflecting the actual environment used, where K could theoretically be infinitely large, approximating continuous non-stationarity.

If you still have any questions, we would greatly appreciate your further feedback.

Reviewer comment (Writing):
Equation 3.1 is hard to understand.

Author response:
> Eq. 3.1 is hard to understand Why not use standard dynamical systems state-space representation? Or probabilistic representation? if you invent notation, it should be better than what exists. This is not.

Section 3.1 introduces the key concept of our work: learning the causal-origin graph structure in RL to address non-stationarity, with a focus on causality and stability. To this end, we add binary masks in Eq. 3.1 to denote causal relationships between variables. Specifically, when all binary mask elements are set to 1, indicating causal relationships exist between all variables, Eq. 3.1 then reverts to a standard dynamics function. These masks are crucial for capturing and analyzing causality-driven non-stationarity, and helps causally explain variable relationships and roles in non-stationarity, aligning with our core concept. We argue that this addition does not significantly complicate the understanding. Furthermore, we have strived to ensured that the notation and representation remain accessible and interpretable to readers familiar with standard dynamics in RL.

Reviewer comment (Writing):
The paper does not use standard dynamical systems state-space representation or probabilistic representation. If notation is invented, it should be better than what exists, but this is not the case here.

Author response:
> Eq. 3.1 is hard to understand Why not use standard dynamical systems state-space representation? Or probabilistic representation? if you invent notation, it should be better than what exists. This is not.

Section 3.1 introduces the key concept of our work: learning the causal-origin graph structure in RL to address non-stationarity, with a focus on causality and stability. To this end, we add binary masks in Eq. 3.1 to denote causal relationships between variables. Specifically, when all binary mask elements are set to 1, indicating causal relationships exist between all variables, Eq. 3.1 then reverts to a standard dynamics function. These masks are crucial for capturing and analyzing causality-driven non-stationarity, and helps causally explain variable relationships and roles in non-stationarity, aligning with our core concept. We argue that this addition does not significantly complicate the understanding. Furthermore, we have strived to ensured that the notation and representation remain accessible and interpretable to readers familiar with standard dynamics in RL.

Reviewer comment (Experiments):
The experiments compare COREP with FNVAE, VariBAD, and PPO. Unless COREP is compared with a mixture-based MMAC solution, it is difficult to understand how well it performs, since the baselines used are only RL-internal methods.

Author response:
> unless you compare COREP with a mixture-based MMAC solution then I cannot see how to understand how well it does. These are only RL-internal methods.

We reiterate that our experimental setting is not mixtures of pre-defined stationary environments. As previously discussed, COREP's primary goal is to tackle more complicated non-stationarity in RL. This broader focus on non-stationarity, extending beyond the usual scope of the mentioned mixture-based MMAC solutions, guided our decision to compare COREP with methods such as FN-VAE and VariBAD which are designed for more generalized non-stationary environments. These methods align more closely with the environments and challenges COREP is designed to address.

Reviewer comment (Novelty):
In the Related Work section, the authors completely miss known solutions referenced earlier.

Author response:
We thank the reviewer for thoughtful feedbacks and valuable suggestions for our work. To address the reviewer's concerns, we provide detailed responses below. If the concerns can be resolved, we hope the reviewer are willing to raise the score.

> The primary issue with the paper is that there are known solutions to this problem that use far simpler approaches than those proposed. The authors are clearly unaware that this is a solved problem. 

We emphasize that the 'non-stationary mixture of various stationary environments' is an idealized theoretical assumption, not a prerequisite for applying the COREP algorithm. As stated in our paper, 'We assume that there are K distinct stationary environments, where K is an unknown number,' which suggests K could be large enough to encompass a near-continuous spectrum of environmental changes. Informed by this insight, COREP aims to learn a stable, shared graph structure, aiding RL algorithms in tackling non-stationarity. Besides, incorporating causality into COREP offers a deeper understanding of environmental dynamics, leading to more robust policy learning, especially in complex and dynamically evolving environments where mixture-based MMAC methods may be less effective.

We reiterate that this assumption serves as a basis for theoretical discussion rather than a constraint on the COREP algorithm. In practice, COREP is effective in environments not strictly adhering to this assumption, as shown in our experiments (eq 4.1). This positions COREP as more versatile than MMAC methods, capable of adapting to a broader range of non-stationary environments. Additionally, in response to the reviewer's suggestion, we have enhanced our revision with a discussion on mixture-based MMAC methods, providing a more comprehensive perspective on existing works in this area.

Reviewer comment (Writing):
This paper presents some challenges in terms of clarity and coherence, particularly in explaining how latent state dimensions (denoted as "h") are identified. This identification is crucial for understanding how the MAG is learned over the joint variables {s, h}.

Author response:
> How are the latent state dimensions h identified?

In Section 3.3, we acknowledge the inherent challenges of theoretically guaranteeing the identifiability of the structure of $\mathcal{M}_{(k)}$ and the latent state dimensions h. This limitation is well-recognized in the study of complex systems like ours. The intricacies of these systems often preclude a straightforward theoretical framework for complete identifiability, especially given the current state of research in this area.

To facilitate intuitive understanding, we have already included a simplified example in Appendix C. This example demonstrates how h might be identified in a simpler scenario. While not a comprehensive solution to identifiability, it serves as a useful illustration of our thought process and methodology.

Reviewer comment (Theory):
While the authors use bidirected edges in the MAG to capture changes in the marginal distributions of s and h across various environments, it's worth noting that relying solely on bidirected edges may not suffice. It's important to consider that edges with circles might also have latent variable "e," resulting in distribution shifts.

Author response:
> Why only consider bidirected edges to indicate the changes?

As discussed in the paper, we adopt the idea of Saeed et al[1] for characterizing causal DAG mixtures, which models them as a family of maximal ancestral graphs (MAGs). This approach represents the observed variables, depicting latent confounders between variable pairs with bidirected edges. For each ancestral graph, a corresponding MAG with the same independence restrictions can be created by adding bidirected edges, as the Theorem 5.1 of [2]. Therefore, our focus will be on graphs with only directed and bidirected edges.

[1] Saeed et al. Causal structure discovery from distributions arising from mixtures of dags. ICML 2020.

[2] Richardson and Spirtes. Ancestral graph markov models. The Annals of Statistics, 2002.

Reviewer comment (Theory):
Why only consider bidirected edges to indicate the changes?

Author response:
> Why only consider bidirected edges to indicate the changes?

As discussed in the paper, we adopt the idea of Saeed et al[1] for characterizing causal DAG mixtures, which models them as a family of maximal ancestral graphs (MAGs). This approach represents the observed variables, depicting latent confounders between variable pairs with bidirected edges. For each ancestral graph, a corresponding MAG with the same independence restrictions can be created by adding bidirected edges, as the Theorem 5.1 of [2]. Therefore, our focus will be on graphs with only directed and bidirected edges.

[1] Saeed et al. Causal structure discovery from distributions arising from mixtures of dags. ICML 2020.

[2] Richardson and Spirtes. Ancestral graph markov models. The Annals of Statistics, 2002.

Reviewer comment (Theory):
From the standpoint of empirical design, I can grasp the concept of amalgamating two graphs to extract the authentic causal graph. However, there remains a lack of clarity regarding the assurance that this specific design is capable of successfully distilling the causal origin of the changes observed. Providing theoretical insights or references that elucidate this process would contribute greatly to the understanding of the methodology.

Author response:
We have already provided a directly computable, simple example in Appendix C of our paper, which clearly illustrates the process of tracing the causal origin of non-stationarity, highlighting its key aspects. To facilitate a better understanding, we have enhanced Appendix C with more detailed explanations in our revision. We hope this example can aid the reviewer in better understanding this process.

Reviewer comment (Experiments):
Alternatively, empirical validations, such as demonstrating a correspondence between the graphs generated by the model and the actual causal origins as defined within a simulator, would also serve to reinforce the credibility and effectiveness of the proposed design.

Author response:
We have already provided a directly computable, simple example in Appendix C of our paper, which clearly illustrates the process of tracing the causal origin of non-stationarity, highlighting its key aspects. To facilitate a better understanding, we have enhanced Appendix C with more detailed explanations in our revision. We hope this example can aid the reviewer in better understanding this process.

Reviewer comment (Theory):
The choice of GNNs (specifically GAT) for representing causal graphs necessitates a stronger justification, given that causality inherently relies on the structure of directed acyclic graphs.

Author response:
Our paper chooses GAT to represent causal graphs due to its efficient handling of DAGs. This aligns with causality's inherent nature, typically depicted in DAGs. GAT's ability to prioritize neighboring nodes through its masked attention mechanism enables the effective capture of complex causal relationships among state elements, which is crucial for our research's core challenge: tracing the causal origins of non-stationarity. The suitability of GAT for this purpose is further elaborated in our paper, particularly in the section discussing the transformation of observed states into node matrices and the generation of weighted adjacency matrices for core-GAT and general-GAT (reference: Section 3.1).

Reviewer comment (Theory):
The adoption of causal-origin representation within the reinforcement learning framework also needs more justifications.

Author response:
We emphasize that the causal-origin representation, a novel design proposed in this paper, is a significant contribution of our research, not an existing method adopted into our work. This design is detailed extensively in Sections 3.1, 3.2, and 3.3, with a thorough theoretical analysis in Appendix A. As outlined in Section 3.1, the causal-origin representation provides RL algorithms with a stable, causally-oriented state representation. This protects RL algorithms from the adverse effects of non-stationarity, enhancing their robustness and effectiveness in complex environments. Consequently, this design significantly enhances the RL algorithms' ability to tackle complex non-stationarity.

Reviewer comment (Theory):
One of my biggest concerns lies in the theoretical analysis. As the paper is concerned with the non-stationary RL, the performance guarantees, i.e., regret bound and convergence rate, are not studied at all.

Author response:
We agree with the importance of performance guarantees such as regret bounds and convergence rates in RL research. However, we clarify the focus and scope of our theoretical analysis in this context. Our primary goal is to establish a method for providing non-stationarity-informed state representations for RL algorithms. The core of our analysis validates our COREP algorithm's design and efficacy. This focus is crucial, providing theoretical support for COREP's rationality and offering key insights into our algorithm's design.

Additionally, our work, while theoretical in parts, is not purely theoretical. Consistent with much academic research, we substantiate our framework's performance through empirical experimentation. Our results demonstrate that COREP achieves considerable performance improvements across various environments. This approach aligns with field norms, where complex models are often validated through experimental evidence rather than purely theoretical guarantees. While regret bounds and convergence rates are important, they represent a different aspect of theoretical analysis that is not our primary focus. Our emphasis on causal-origin representation and its performance in non-stationary environments is substantiated through rigorous experimentation, which we believe provides compelling evidence of our algorithm's effectiveness.

Reviewer comment (Novelty):
Another biggest concern is that the novelty and contribution of this work are not significant. The GAT, causal discovery, and non-stationary RL are well studied in many existing literature. The paper does not provide significantly novel methodological or theoretical techniques in the current stage.

Author response:
We reiterate the key idea of our research, highlighting its originality and significance: the design of the causal-origin representation.
1. **Innovation in Causal-Origin Representation:** Contrary to the implication of merely adopting existing methods, we emphasize that the causal-origin representation is an original concept, conceived and developed within this work. Far from being a minor addition, it forms the core of our theoretical analysis and methodological design, representing a significant advancement in the field of non-stationary RL.
2. **Contribution to Non-Stationary RL:** The introduction of causal-origin representation addresses a critical gap in non-stationary RL. It provides a novel approach for examining and responding to the complexities of non-stationary environments, diverging from conventional methods focused on post-change detection or latent variable modeling of change factors. This innovation not only enhances current understanding but also paves the way for future advancements in handling dynamic environments in RL.

Reviewer comment (Experiments):
The mixing of varying components for policy optimization makes the algorithm difficult to train. I am concerned about the efficiency and complexity of the proposed method, especially compared to the existing baselines (in terms of model complexity, computation, and sample efficiency).

Author response:
We would like to address these concerns by highlighting key aspects of our approach detailed in our paper:

1. **Integration of VAE for Enhanced Learning Efficiency:** In Section 3.4, we discuss integrating a Variational AutoEncoder (VAE) into the COREP framework to improve learning efficiency. The VAE helps manage the complexities of the causal-origin representation, enhancing learning efficiency.
2. **Empirical Evidence of Sample Efficiency:** Figure 4.1 demonstrates COREP's sample efficiency advantage over baselines. These results confirm that COREP maintains, and often surpasses, the sample efficiency of existing methods, despite integrating multiple components.
3. **Computational Resource Utilization:** Appendix F provides detailed information about computational resources and time. The results demonstrate COREP's normal computational level in environments like the DeepMind Control Suite, evidencing its computational feasibility in standard RL settings.
4. **Limitations in High-Dimensional Environments:** As Section 6 outlines, we recognize COREP's potential complexity issues in high-dimensional environments. Addressing this in future work, we aim to optimize COREP's scalability and performance in such settings. However, this limitation is beyond the current work's scope and does not detract from COREP's contributions and effectiveness in tested environments.

Reviewer comment (Experiments):
The experimental design to emulate non-stationarity is somewhat naive and may not encapsulate the variety and complexity of non-stationarities encountered in practical settings. The algorithm's efficacy under more typical real-world conditions remains unverified, thus limiting the potential impact of the work.

Author response:
- In Section 4, under Experiment Settings, we mention our introduction of non-stationarity through periodic noise in the dynamics, aligning with the SOTA method FN-VAE. This representation of non-stationarity is common in similar field research [1, 2]. We adopted this setting to ensure a fair and comparable performance assessment across methodologies.
- While this approach may not capture all complexities of real-world non-stationarity, it provides a suitable benchmark for exploratory research like ours. The primary goal of our research is to validate new methodological designs in widely-used academic environments. We are confident that our chosen settings effectively demonstrate COREP's capability to surpass SOTA methods. We agree that testing our algorithm in more complex, real-world settings is a valuable direction for future research.

[1] Feng et al. Factored adaptation for non-stationary reinforcement learning. NeurIPS 2022.

[2] Xie et al. Deep reinforcement learning amidst lifelong nonstationarity. ICML 2021.

Reviewer comment (Writing):
The format of notations is not standard in RL, causal inference, or GNN literature, which puts additional burdens on readers to understand.

Author response:
- The main modification we made to the notation is simply adding binary masks to the variables of RL dynamics in Equation 3.1, which are used to define causal relationships between elements. This modification is necessary for the subsequent introduction and analysis of our method. Particularly, when all elements have a mask value of 1, Equation 3.1 degenerates into a standard RL dynamics function.
- In our discussion of the union graph in Sections 3.2 and 3.3, we adhere to standard graph theory notations.
- In Section 3.4, our description of GAT details strictly follows the notation system prevalent in GNN literature. 

We believe that our notation format is consistent with respective fields and should not impose additional burdens on the readers.

Reviewer comment (Writing):
The logical flow of the paper is not clear, and the central claim is vague.

Author response:
Our paper's logic begins in the introduction, where we discuss the motivation for using causal information in representations to help RL algorithms address non-stationarity. Following this, we define non-stationarity from a causal perspective and conduct a theoretical analysis. This leads to the innovative design of the union graph, proven to encapsulate the defined non-stationarity information. Building on these insights, we develop the COREP algorithm, which learns the union graph structure and ensures inclusion of non-stationarity information in the graph representation. This design equips RL algorithms with awareness of environmental non-stationarities, effectively addressing the core problem. We believe this progression from problem identification to methodological insights and design is logically coherent.

Reviewer comment (Writing):
There exist many typos and errors, like the repeated authors' names (redundant citations) in the section of related works.

Author response:
Regarding the redundant names in the citation format in related works, we appreciate the reviewer for pointing it out, and we have made adjustments in the revision. We also further thoroughly examine the paper to make it accurate and rigorous.