Paper: Causally Aligned Curriculum Learning (ICLR-2024)

Reviewer comment (Theory):
Example 1's motivation for the paper is very contrived. The only reason a curriculum designer would design such a source task (where the box color is fixed to yellow) is if the curriculum designer was completely unaware of the causal dependencies of the reward. While this is possible in the case of causally-unaware automated curriculum generation methods, it seems a bit unlikely unless the crucial confounder variable U_i was either excluded from the state space (which would violate the usual Markovian reward assumption) or it is included in the state space, but the agent cannot observe it (a partial observability assumption).

Author response:
Recall that in the definition of SCMs, each variable $V$’s value is determined by a function of $f_V(\operatorname{pa}(V), \boldsymbol{U}_V)$ where $\operatorname{pa}(V)$ are the values of parents of $V$ and $\boldsymbol{U}_V$ are exogenous variables. The inherent randomness of the system is due to the existence of exogenous variables. When an exogenous variable affects more than one endogenous variable, it is called a confounder\, which is unobservable to the agent and the curriculum designer. This holds for both automatic curriculum generation and manual curriculum designing. While it is a common practice to assume Markovian rewards or fully observable state space, as we stated in the introduction, our work investigates a more challenging scenario where the underlying environment contains unobserved confounders. For instance, in example 1 (the Colored Sokoban), one unobserved confounder is $U_i$, which is affecting both the box color $C_i$ and the reward $Y_i$. The randomness of both $C_i, Y_i$ are determined by $U_i$. 
  
1.1 Please see the following chart for the percentage of misaligned source tasks generated by the baseline methods we tested in the both environments during training (300K environment steps). We can see that the existing non-causal curriculum generators are indeed generating a large portion of misaligned source tasks during the training even though they have the option to choose aligned source tasks. This indicates that the existing non-causal curriculum generators cannot effectively avoid misaligned source tasks and thus they are detrimental to the agent training when applied to environments with unobserved confounders.


|Environments/Curriculum Generators |ALP-GMM|PLR|Currot|Goal-GAN
|--|--|--|--|--|
|**Colored Sokoban**|68.10%$\pm$0.71%|69.10%$\pm$2.77%|68.36%$\pm$1.06%|66.41%$\pm$1.84%|
|**Button Maze**    |88.41%$\pm$0.43%|87.33%$\pm$3.77%|88.62%$\pm$1.00%|90.30%$\pm$0.27%|

1.2 Even without confounders, if the state space is partially observable, the same situation as in the Colored Sokoban could happen. When the Markovian reward assumption holds, where the agent can observe all the parents of the reward variable, the reward cannot be confounded with any other variables. Under this stronger assumption, as our Theorem 1 indicates, all state variables are editable. On the other hand, we are dedicated to solving the curriculum generation problem in the presence of unobserved confounders. Thus, the setting of Markovian rewards actually falls into the traditional curriculum reinforcement learning problem studied in the literature, which our work is trying to relax.

Reviewer comment (Experiments):
Can the authors experimentally verify how often existing non-causal curriculum generation methods generate causally unaligned tasks?

Author response:
1.1 Please see the following chart for the percentage of misaligned source tasks generated by the baseline methods we tested in the both environments during training (300K environment steps). We can see that the existing non-causal curriculum generators are indeed generating a large portion of misaligned source tasks during the training even though they have the option to choose aligned source tasks. This indicates that the existing non-causal curriculum generators cannot effectively avoid misaligned source tasks and thus they are detrimental to the agent training when applied to environments with unobserved confounders.


|Environments/Curriculum Generators |ALP-GMM|PLR|Currot|Goal-GAN
|--|--|--|--|--|
|**Colored Sokoban**|68.10%$\pm$0.71%|69.10%$\pm$2.77%|68.36%$\pm$1.06%|66.41%$\pm$1.84%|
|**Button Maze**    |88.41%$\pm$0.43%|87.33%$\pm$3.77%|88.62%$\pm$1.00%|90.30%$\pm$0.27%|

Reviewer comment (Evaluation):
Can the authors comment on the expected benefit of their method in a domain with Markovian reward, or a fully observable domain with no unobserved confounders?

Author response:
1.2 Even without confounders, if the state space is partially observable, the same situation as in the Colored Sokoban could happen. When the Markovian reward assumption holds, where the agent can observe all the parents of the reward variable, the reward cannot be confounded with any other variables. Under this stronger assumption, as our Theorem 1 indicates, all state variables are editable. On the other hand, we are dedicated to solving the curriculum generation problem in the presence of unobserved confounders. Thus, the setting of Markovian rewards actually falls into the traditional curriculum reinforcement learning problem studied in the literature, which our work is trying to relax.

Reviewer comment (Writing):
The inputs/outputs of the authors' method are not presented in plain language. As understood, the authors' method assumes access to the SCM of the task, and a curriculum generator such as GoalGAN. Such details should be made much clearer, perhaps through a summary methods figure or by adding a discussion of limitations.

Author response:
2.2 The plain text descriptions of input/output of algorithms 1 and 3 can be found in the paragraph beside the algorithm pseudo-code, which is also shown in each pseudo-code box's input/output field. We would also like to clarify that SCMs are *not* assumed as inputs. Instead, we only require causal diagrams as inputs, a graphical representation of the dependency relationships among the variables without any detailed causal mechanisms or transition probabilities (the F and P(u) in the definition of SCM in the preliminary section).

Reviewer comment (Presentation):
Figures are not standalone. It is common practice to read a paper by skimming the figures and captions first, but it was not possible to get the main idea of the figure, method, and experimental results by doing so. Even after reading most of the accompanying text, the figures and captions alone remained ambiguous. It was necessary to find the specific part of the text referencing the figures — sometimes needing to jump to other parts of the paper — and read very carefully to understand what was happening. The authors should rewrite the figure captions and perhaps add a new "methods" figure summarizing the flow of their method.

Author response:
Thanks for your suggestion! We agree that the caption should be self-contained for ease of skimming. We have improved them and added a method flow chart to the main text.

Reviewer comment (Presentation):
Figure 5: It is unclear why the curves labeled "causal" performed differently across the four columns. Also, the names "causal" vs "original" were confusing. This may be because the fact that the proposed method augments existing curriculum generation methods was presented only in a single sentence towards the end of the introduction, and the paper assumed this knowledge thereafter. Since this is crucial to understanding the paper, the authors should add this important piece of information.

Author response:
3.1 We agree that the caption in Figure 5 doesn’t stand alone, and we have improved the wording in the revised version. Still, we would like to clarify that we also have provided detailed descriptions of the figure in the experiment section that we compare the performance between “the original curriculum generators” and “the causal version of these generators” using “four best-performing curriculum generators” (four columns).
In Figure 5, “causal” refers to the augmented version of the baseline curriculum generators, and “original” refers to the unaugmented, original version of the baseline curriculum generators. The first row is the performance in the Colored Sokoban, and the second row is the performance in the Button Maze. Each column is a different curriculum generator we tested.

Reviewer comment (Presentation):
Figure 3: It is confusing that figure (a) and (b) are identical except for the edit indicators. It was necessary to visually trace all arrows of both figures to verify this. The presence/absence of edit indicators should be explained in the caption, and the figure could be modified so that it is much quicker for the reader to notice that the only addition is the edit indicators (perhaps through the use of opacity/shading).

Author response:
3.2 Thanks for pointing this out, we have adjusted the figure.

Reviewer comment (Writing):
In Definition 1, the SCM M* is labeled with a *, yet this notation does not appear elsewhere in the paper.

Author response:
Thanks for noting this. We have removed the * notation in the updated version.

Reviewer comment (Experiments):
The authors mentioned that curriculum learning could overcome the curse of dimensionality. However, all the examples discussed in this paper are tabular. There seems to be a mismatch. It would be interesting to discuss and experiment on an example with exponentially large state and action space.

Author response:
We agree that the examples are in the grid world, but they are only for illustration purposes and don’t indicate any limitations of our proposed method. For the ease of understanding, we delibrately choose such simple domains to better demonstrate our ideas and the implications of unobserved counfounders in the environment. One of the initial observations in these settings is that applying standard non-causal curriculum generators could be detrimental. Given that the environment is simple, we were able to explain the subtle interplay between curriculum generation, decision-making and unobserved confounding. 

Having said, we note that our technical results (e.g., theorems, algorithms) are non-parametric, which means that they are immediately applicable to high-dimensional, continuous domains given merely qualitative causal knowledge about the domain. Once the proposed augmentation procedure is executed, we can leverage the computational power of existing curriculum generation methods to address those more complex domains. We also note that the observation spaces in our experiments are all in raw pixels instead of low-dimensional state vectors, which are high-dimensional. From the performance of agents trained directly in the target task (figure 5, horizontal green line), we can see that without causal curriculum learning, those agents cannot converge to the optimal in the target task at all, which suggests that our method helps with the learning. 

Finally, and based on the feedback provided, we further conducted an extra experiment with continuous actions and states to demonstrate the effectiveness of our methods in exponentially large state and action space empirically. Please see the common response to all the reviewers for the detailed results.

Reviewer comment (Theory):
What is the computation-complexity of FindCausalCurriculum? Does it scale with the size of state and action space? If that is the case, it seems to be against the motivation of applying curriculum learning.

Author response:
The worst-case time complexity of FindCausalCurriculum is $\mathcal{O}(Hdn^2)$ where $H$ is the length of the horizon, $d$ is the number of actions, $n$ is the total number of nodes in the causal diagram. In an improved version of FindCausalCurriculum, which we show in the appendix (Algo 6 Causal Curriculum Learning), the worst-case time complexity is reduced to $\mathcal{O}(Hn^2)$. Since there is no free lunch, given the performance boost and this relatively slow scaling overhead w.r.t the state/action size, applying causal curriculum learning is still preferable. More specifically, as shown in Figure 5, without curriculum learning, the agent trained directly in the target task cannot converge to the optimal at all. Even with curriculum learning, failing to utilize causal knowledge results in worse performance.

Reviewer comment (Experiments):
In general, I believe the environment tested in this paper are rather naive.

Author response:
We agree that these two environments may seem relatively naive from a human perspective. However, this fact actually strengthens the message of our paper. Namely, if deep reinforcement learning agents cannot solve such seemingly simple problems perfectly (figure 5, trained directly in target task) and if using curriculum learning without incorporating causal knowledge can sometimes be detrimental, how can we expect these agents to achieve similar  goals in more challenging environments? 

For the ALP-GMM experiments, we would like to clarify that we didn’t force any of the curriculum generators to fix the box color. In the context space, all curriculum generators have the option to “not fix the box color.” Thus, they are already running under your suggested scenario where “C is a proxy of the underlying U, and choosing C by running ALP-GMM (also true for other baseline curriculum generators) could be harmful.” You can also see the following chart summarizing the portion of misaligned source tasks against aligned source tasks during training.

|Environments/Curriculum Generators |ALP-GMM|PLR|Currot|Goal-GAN
|--|--|--|--|--|
|**Colored Sokoban**|68.10%$\pm$0.71%|69.10%$\pm$2.77%|68.36%$\pm$1.06%|66.41%$\pm$1.84%|
|**Button Maze**    |88.41%$\pm$0.43%|87.33%$\pm$3.77%|88.62%$\pm$1.00%|90.30%$\pm$0.27%|

The reason those curriculum generators fail to avoid such harmful source tasks adaptively is that there is still a non-trivial chance (in this setting, $P(U=1)=1/4$) that the agent can get rewarded for pushing an intervened blue color box (undesired behavior). There is also a high probability ($P(U=0)=3/4$) that the agent gets punished for pushing an intervened yellow color box (desired behavior).

Our theorems/algorithms are non-parametric, which means we don’t restrict the parametric form or the size of the scope of those state/action variables neither the distribution over exogenous conditions. They readily apply to high-dimensional (large) or continuous domains. From the experiments, all observations are in the pixel space, which is high-dimensional. Also, note that the environments we tested are not MDP due to the existence of confounders. 

Our method is an augmentation to the existing curriculum generators, preparing them for domains where unobserved confounding cannot be ruled out a priori. It does not contradict but rather complements these generators, and is applicable in the same environments as they can support. We agree that showing an experiment in a continuous domain would make the contributions of the paper clearer. Therefore, we have included an extra experiment featuring continuous state and action variables to empirically demonstrate the effectiveness of our method. For the results, please refer to the common response to all the reviewers.

Reviewer comment (Experiments):
The authors implement ALP-GMM by fixing the color. I don't think this is a good way to construct baseline. One can run ALP-GMM as a task sampler on a fixed set of tasks. ALP-GMM essentially assigns a probability for different tasks at the each round. If one task has the underlying U that is not aligned with the target task, ALP-GMM will adaptively reduce the weight for that task.

Author response:
We agree that these two environments may seem relatively naive from a human perspective. However, this fact actually strengthens the message of our paper. Namely, if deep reinforcement learning agents cannot solve such seemingly simple problems perfectly (figure 5, trained directly in target task) and if using curriculum learning without incorporating causal knowledge can sometimes be detrimental, how can we expect these agents to achieve similar  goals in more challenging environments? 

For the ALP-GMM experiments, we would like to clarify that we didn’t force any of the curriculum generators to fix the box color. In the context space, all curriculum generators have the option to “not fix the box color.” Thus, they are already running under your suggested scenario where “C is a proxy of the underlying U, and choosing C by running ALP-GMM (also true for other baseline curriculum generators) could be harmful.” You can also see the following chart summarizing the portion of misaligned source tasks against aligned source tasks during training.

|Environments/Curriculum Generators |ALP-GMM|PLR|Currot|Goal-GAN
|--|--|--|--|--|
|**Colored Sokoban**|68.10%$\pm$0.71%|69.10%$\pm$2.77%|68.36%$\pm$1.06%|66.41%$\pm$1.84%|
|**Button Maze**    |88.41%$\pm$0.43%|87.33%$\pm$3.77%|88.62%$\pm$1.00%|90.30%$\pm$0.27%|

The reason those curriculum generators fail to avoid such harmful source tasks adaptively is that there is still a non-trivial chance (in this setting, $P(U=1)=1/4$) that the agent can get rewarded for pushing an intervened blue color box (undesired behavior). There is also a high probability ($P(U=0)=3/4$) that the agent gets punished for pushing an intervened yellow color box (desired behavior).

Our theorems/algorithms are non-parametric, which means we don’t restrict the parametric form or the size of the scope of those state/action variables neither the distribution over exogenous conditions. They readily apply to high-dimensional (large) or continuous domains. From the experiments, all observations are in the pixel space, which is high-dimensional. Also, note that the environments we tested are not MDP due to the existence of confounders. 

Our method is an augmentation to the existing curriculum generators, preparing them for domains where unobserved confounding cannot be ruled out a priori. It does not contradict but rather complements these generators, and is applicable in the same environments as they can support. We agree that showing an experiment in a continuous domain would make the contributions of the paper clearer. Therefore, we have included an extra experiment featuring continuous state and action variables to empirically demonstrate the effectiveness of our method. For the results, please refer to the common response to all the reviewers.

Reviewer comment (Experiments):
Although fixing color matches the general story of the whole paper, I don't think this is a natural choice in examples where we have access to a simulator. It would be interesting to demonstrate a non-trivial example where C is a proxy of the underlying U and choosing C by running ALP-GMM could be harmful.

Author response:
We agree that these two environments may seem relatively naive from a human perspective. However, this fact actually strengthens the message of our paper. Namely, if deep reinforcement learning agents cannot solve such seemingly simple problems perfectly (figure 5, trained directly in target task) and if using curriculum learning without incorporating causal knowledge can sometimes be detrimental, how can we expect these agents to achieve similar  goals in more challenging environments? 

For the ALP-GMM experiments, we would like to clarify that we didn’t force any of the curriculum generators to fix the box color. In the context space, all curriculum generators have the option to “not fix the box color.” Thus, they are already running under your suggested scenario where “C is a proxy of the underlying U, and choosing C by running ALP-GMM (also true for other baseline curriculum generators) could be harmful.” You can also see the following chart summarizing the portion of misaligned source tasks against aligned source tasks during training.

|Environments/Curriculum Generators |ALP-GMM|PLR|Currot|Goal-GAN
|--|--|--|--|--|
|**Colored Sokoban**|68.10%$\pm$0.71%|69.10%$\pm$2.77%|68.36%$\pm$1.06%|66.41%$\pm$1.84%|
|**Button Maze**    |88.41%$\pm$0.43%|87.33%$\pm$3.77%|88.62%$\pm$1.00%|90.30%$\pm$0.27%|

The reason those curriculum generators fail to avoid such harmful source tasks adaptively is that there is still a non-trivial chance (in this setting, $P(U=1)=1/4$) that the agent can get rewarded for pushing an intervened blue color box (undesired behavior). There is also a high probability ($P(U=0)=3/4$) that the agent gets punished for pushing an intervened yellow color box (desired behavior).

Our theorems/algorithms are non-parametric, which means we don’t restrict the parametric form or the size of the scope of those state/action variables neither the distribution over exogenous conditions. They readily apply to high-dimensional (large) or continuous domains. From the experiments, all observations are in the pixel space, which is high-dimensional. Also, note that the environments we tested are not MDP due to the existence of confounders. 

Our method is an augmentation to the existing curriculum generators, preparing them for domains where unobserved confounding cannot be ruled out a priori. It does not contradict but rather complements these generators, and is applicable in the same environments as they can support. We agree that showing an experiment in a continuous domain would make the contributions of the paper clearer. Therefore, we have included an extra experiment featuring continuous state and action variables to empirically demonstrate the effectiveness of our method. For the results, please refer to the common response to all the reviewers.

Reviewer comment (Experiments):
An important way to improve the paper is to propose an algorithm for large (or continuous) state and action space. The current version seems to only work with tabular MDPs, which limits the complexity of environments the algorithm can be applied to.

Author response:
We agree that these two environments may seem relatively naive from a human perspective. However, this fact actually strengthens the message of our paper. Namely, if deep reinforcement learning agents cannot solve such seemingly simple problems perfectly (figure 5, trained directly in target task) and if using curriculum learning without incorporating causal knowledge can sometimes be detrimental, how can we expect these agents to achieve similar  goals in more challenging environments? 

For the ALP-GMM experiments, we would like to clarify that we didn’t force any of the curriculum generators to fix the box color. In the context space, all curriculum generators have the option to “not fix the box color.” Thus, they are already running under your suggested scenario where “C is a proxy of the underlying U, and choosing C by running ALP-GMM (also true for other baseline curriculum generators) could be harmful.” You can also see the following chart summarizing the portion of misaligned source tasks against aligned source tasks during training.

|Environments/Curriculum Generators |ALP-GMM|PLR|Currot|Goal-GAN
|--|--|--|--|--|
|**Colored Sokoban**|68.10%$\pm$0.71%|69.10%$\pm$2.77%|68.36%$\pm$1.06%|66.41%$\pm$1.84%|
|**Button Maze**    |88.41%$\pm$0.43%|87.33%$\pm$3.77%|88.62%$\pm$1.00%|90.30%$\pm$0.27%|

The reason those curriculum generators fail to avoid such harmful source tasks adaptively is that there is still a non-trivial chance (in this setting, $P(U=1)=1/4$) that the agent can get rewarded for pushing an intervened blue color box (undesired behavior). There is also a high probability ($P(U=0)=3/4$) that the agent gets punished for pushing an intervened yellow color box (desired behavior).

Our theorems/algorithms are non-parametric, which means we don’t restrict the parametric form or the size of the scope of those state/action variables neither the distribution over exogenous conditions. They readily apply to high-dimensional (large) or continuous domains. From the experiments, all observations are in the pixel space, which is high-dimensional. Also, note that the environments we tested are not MDP due to the existence of confounders. 

Our method is an augmentation to the existing curriculum generators, preparing them for domains where unobserved confounding cannot be ruled out a priori. It does not contradict but rather complements these generators, and is applicable in the same environments as they can support. We agree that showing an experiment in a continuous domain would make the contributions of the paper clearer. Therefore, we have included an extra experiment featuring continuous state and action variables to empirically demonstrate the effectiveness of our method. For the results, please refer to the common response to all the reviewers.

Reviewer comment (Writing):
I was a bit confused by the wordings in abstract. The authors first state that invariant optimal decision rules does not necessarily hold, then they propose condition characterizing causally aligned source tasks, i.e., the invariance of optimal decision rules holds. The flow does not seem right here.

Author response:
We appreciate the opportunity to clarify this issue. In the abstract, we say that the expectation of applying curriculum learning is for the optimal decision rules to remain invariant across source tasks and the target task. However, this expectation does not always hold, particularly when the underlying state variables are factorized and unobserved confounders exist. To address this, we study this challenge in depth and propose formal conditions of when this expectation is met or not. This analysis enables us to extend existing curriculum learning algorithms to a broader range of environments where unobserved confounding is prevalent (Recall that by the definition of SCMs, endogenous variables are confounded if they share the same exogenous variables as inputs in determining their values).

Reviewer comment (Experiments):
Need extra clarifications on Figure 1 and 2. What does it mean by fixing the color? When the different algorithms are trained, are they only allowed to choose tasks from misaligned source tasks? Should this example be too artificial? It would be interesting to see the results when ALP-GMM can generate tasks with unknown color, while the algorithm may reuse these tasks. In this way, it is possible for ALP-GMM to discover that certain tasks can be more helpful for learning the target task. This is in fact more aligned with the situations in the real-world.

Author response:
“Fixing the color” refers to the misaligned source tasks where the box color is intervened. The example in Figure 1 and 2 is simplified for illustration purposes, it does not indicate the limitations of our approach. However, this example is not too artificial at all because during the experiments, all curriculum generators are free to choose any context, including fixing the box color (misaligned) and not fixing the box color (aligned). This means that learners are trained in a combination of both aligned and misaligned source tasks. In the above chart (in response to Weakness 3&4), we counted the portion of misaligned source tasks generated by an unaugmented ALP-GMM and other curriculum generators we tested. 

We can see that they all fail to avoid those misaligned source tasks. As a result, agents trained by them fail to achieve the same convergence rate as the optimal agent trained only in the aligned curriculum. We have updated the captions in the revised version; thank you.

Overall, the challenge of misaligned source tasks cannot be addressed by changing the distribution of the box color. This means even for a source task induced by a stochastic intervention on the box color, and its distribution coincides with the box color distribution in the target task, the misalignment could still occur. For concreteness, consider Example 1, instead of using do-interventions, we apply a stochastic intervention $\sigma_{C_i}$  to the box color such that $P(C_i = 0|\sigma_{C_i}) = \frac{1}{4}$ and $P(C_i = 1|\sigma_{C_i}) = \frac{3}{4}$. Under this intervention, the observational distribution of box color is exactly the same as the one from the target task. However, the conditional reward distributions that the policy is optimized towards are drastically different. In the target task, we have,  
$$P\left(Y_i = 10 \big| B_i =  \text{next to goal}, C_i, \operatorname{do}\left(X_i = \text{push}\right)\right) = 1[C_i = 0]$$, and
$$P\left(Y_i = -10 \big| B_i =  \text{next to goal}, C_i, \operatorname{do}\left(X_i = \text{push}\right)\right) = 1[C_i=1]$$.
But in the intervened world, by sigma-calculus [1], we have,
$$P\left(Y_i = 10 \big| B_i =  \text{next to goal}, C_i, \operatorname{do}\left(X_i = \text{push}\right); \sigma_{C_i}\right) = \frac{1}{4}$$, and
$$P\left(Y_i = -10 \big| B_i =  \text{next to goal}, C_i, \operatorname{do}\left(X_i = \text{push}\right); \sigma_{C_i}\right) = \frac{3}{4}$$.
Thus, the optimal policy we learned in this source task still intends to not push the box at all while the optimal target task policy should be to push the yellow box.

[1] Correa, Juan, and Elias Bareinboim. "A calculus for stochastic interventions: Causal effect identification and surrogate experiments." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 06. 2020.

Reviewer comment (Experiments):
Toyish experiments. The experiments conducted are limited to grid-world environments, which significantly narrows the scope of application. For the conclusions to be generalized, the experiments should ideally be extended to a more diverse set of environments.

Author response:
This paper provides an augmentation procedure that allows existing curriculum generation algorithms to handle  environments with unobserved confounders. This augmentation procedure (Theorems 1-3) is non-parametric, relying on qualitative knowledge about the underlying environment. Once this augmentation procedure is executed, we can leverage the computational power of existing curriculum generation methods to address more complex domains. Consequently, our proposed method is generalizable to other, more diverse, high-dimensional/continuous environments.   

After all, we are aware that scaling to high-dimensional domains is an important challenge. However, this paper investigates an orthogonal challenge that arises due to the existence of unobserved confounding bias, which we believe is also important in the design of curriculum generators. The proposed approach does not contradict but complements existing curriculum learning methods, by generalizing them to a more diverse set of environments where confounding bias may be present.  (Example 1 illustrates how current curriculum generators could fail when applied to environments with unobserved confounding, which motivated this work.) 

Also, we would also like to note that our experiments use raw pixels as the observational space, which is high-dimensional, as stated in the experimental section and the conclusions. Still, following the reviewers’ concerns, we included an additional experiment for continuous state/action variables to illustrate the effectiveness of the proposed method empirically. Further details can be found in the common response to all the reviewers.

Reviewer comment (Experiments):
Scalability. The paper does not adequately address how the proposed method scales to continuous environment variables. This is a significant concern as many practical applications in continuous or high-dimensional environment variable space. By not tackling this challenge, the authors leave a gap in understanding the full potential and limitations of their method.

Author response:
This paper provides an augmentation procedure that allows existing curriculum generation algorithms to handle  environments with unobserved confounders. This augmentation procedure (Theorems 1-3) is non-parametric, relying on qualitative knowledge about the underlying environment. Once this augmentation procedure is executed, we can leverage the computational power of existing curriculum generation methods to address more complex domains. Consequently, our proposed method is generalizable to other, more diverse, high-dimensional/continuous environments.   

After all, we are aware that scaling to high-dimensional domains is an important challenge. However, this paper investigates an orthogonal challenge that arises due to the existence of unobserved confounding bias, which we believe is also important in the design of curriculum generators. The proposed approach does not contradict but complements existing curriculum learning methods, by generalizing them to a more diverse set of environments where confounding bias may be present.  (Example 1 illustrates how current curriculum generators could fail when applied to environments with unobserved confounding, which motivated this work.) 

Also, we would also like to note that our experiments use raw pixels as the observational space, which is high-dimensional, as stated in the experimental section and the conclusions. Still, following the reviewers’ concerns, we included an additional experiment for continuous state/action variables to illustrate the effectiveness of the proposed method empirically. Further details can be found in the common response to all the reviewers.

Reviewer comment (Novelty):
Missing related work:  
Hu, Xing, et al. "Causality-driven Hierarchical Structure Discovery for Reinforcement Learning." Advances in Neural Information Processing Systems 35 (2022): 20064-20076.  
Cho, Daesol, Seungjae Lee, and H. Jin Kim. "Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generation." arXiv preprint arXiv:2301.11741 (2023).  
Huang, Peide, et al. "Curriculum reinforcement learning using optimal transport via gradual domain adaptation." Advances in Neural Information Processing Systems 35 (2022): 10656-10670.

Author response:
Thank you for the suggested references; we have now incorporated them into the related work section. After reading them, we would like to contextualize their relevance to our work. In particular, Paper 1 focuses on causal discovery learning the underlying data generating mechanism directly. In the future, we may explore incorporating such methods to learn causal diagrams from data, rather than assuming them as an input. Papers 2 and 3, which are both curriculum generators like those we have tested, align with our theorem. They can be augmented to be robust to, and function in the presence of unobserved confounders. Overall, these references provide interesting perspectives that  complement our work, which develops the foundations for curriculum generation in more generalized settings where unobserved confounders are present.

Reviewer comment (Theory):
In Algorithm 3, how is the actions sequence $\{X_1, \ldots, X_H\}$ obtained in the first place? If we have this optimal action sequence before the learning starts, why do we need RL?

Author response:
Action sequence $X_1, \dots, X_H$ follows the topological ordering of these variables in the environment. This representation is common in RL literature. For instance, consider a finite-horizon POMDP with H steps; its actions are ordered as $X_1, X_2, \dots, X_H$, following the time steps that the system dynamics unroll. In other words, in POMDP models, the topological ordering of actions is fixed a priori. RL algorithms are used to find an optimal policy determining the values of these actions.

Reviewer comment (Theory):
In Theorem 3, one of the conditions is that: For every $j=1, \ldots, N-1$, actions $\boldsymbol{X}^{(j)} \subseteq \boldsymbol{X}^{(j+1)}$. Does this mean that the transition must be deterministic and there only exists one unique optimal solution?

Author response:
The set of actions $\boldsymbol{X}^{(j)}$ is a set of random variables. Each action in this set is still a random variable, not a specific value. More specifically, condition (i) in Theorem 3 requires that each source task must be causally aligned with a set of actions that is the set of actions whose optimal decision rules are the same as those in the target task. Then, to avoid forgetting (Fig. 4), condition (ii) further requires that the curriculum orders those source tasks in a way that the set of causally aligned actions is expanding along following the curriculum. Combined together, we have the conditions for causally aligned curricula that is implemented as an algorithm in Algo. 3. 

In general, Theorem 3 identifies an effective ordering of source tasks so that the learner can converge to the optimal target policy without forgetting useful skills (Fig. 4).  However, the transition distribution in each source task is not deterministic but stochastic, and it does not assume a unique optimal solution. Thus, this condition doesn’t indicate that the transition is deterministic or there only exists one unique optimal solution. We hope this clarifies the issue.

Reviewer comment (Experiments):
It remains unclear whether the proposed causal curriculum strategy can be extended to domains with continuous state and action spaces.

Author response:
There might be some misreadings on our results and we take this as an opportunity to clarify this point. We first note that the results in the paper (Theorems 1-3) are non-parametric, in the sense that they do not rely on any parametric assumptions of the underlying variable domains and structural causal model. This means that these results are readily applicable to state variables and action variables over continuous domains. To illustrate this point empirically, we conducted extra experiments in a continuous domain to demonstrate the effectiveness of our methods. Please see the common response for the experiment result.

Reviewer comment (Experiments):
In the Colored Sokoban example, in the baseline, the context/task space is restricted to tasks where the box color remains constant throughout the game. However, considering a more extensive task space that encompasses all possible combinations of box colors might lead the state-of-the-art curriculum strategies to automatically select relevant/aligned tasks during training. In contrast, the proposed causal curriculum limits the task space to initial agent and box positions, with the environment determining box colors based on intrinsic randomness. Formally, let us denote the initial positions of the agent and the box as $(a_0^x, a_0^y)$ and $(b_0^x, b_0^y)$, respectively. Furthermore, let $c_t$ represent the color of the box at time step $t$, and designate $H$ as the maximum number of steps allowable in the game. In the results presented, the baseline methodology confines the context/task space to instances of the form $[a_0^x, a_0^y, b_0^x, b_0^y, c_0 = c_1 = \cdots = c_H]$. The extensive task space contains all possible task configurations denoted as $[a_0^x, a_0^y, b_0^x, b_0^y, c_0, c_1, \dots, c_H]$. The proposed causal curriculum imposes a task space restriction, characterized by tasks of the form $[a_0^x, a_0^y, b_0^x, b_0^y]$, delegating the selection of $[c_1, c_2, \dots, c_H]$ to the environment, predicated upon intrinsic randomness. Please provide your insights on this aspect.

Author response:
For this more extensive task space that encompasses all possible combinations of box colors, those state-of-the-art curriculum generators will still fail to select causally aligned source tasks. Because in this Colored Sokoban, as long as the box color is intervened by the outside curriculum generator, it will be a misaligned source task (as indicated by our theorem), no matter how the color is intervened. Specifically, in example 1, comparing equation 3 against equation 4 (there was a typo here and we have fixed it in the revised version), we see that in target task, the optimal policy is maximizing the reward distribution of,
$$P\left(Y_i = 10 \big| B_i =  \text{next to goal}, C_i, \operatorname{do}\left(X_i = \text{push}\right)\right) = 1[C_i = 0]$$, and
$$P\left(Y_i = -10 \big| B_i =  \text{next to goal}, C_i, \operatorname{do}\left(X_i = \text{push}\right)\right) = 1[C_i=1]$$.

However, in the color-intervened source task (yellow box), the optimal policy is maximizing the reward distribution of, 
$$P\left(Y_i = 10 \big| B_i =  \text{next to goal}, \operatorname{do}\left(C_i = 0, X_i = \text{push}\right)\right) = \frac{1}{4}$$, and
$$P\left(Y_i = -10 \big| B_i =  \text{next to goal}, \operatorname{do}\left(C_i = 0, X_i = \text{push}\right)\right) = \frac{3}{4}$$.
If we intervene the color to be blue, similarly, we have,
$$P\left(Y_i = 10 \big| B_i =  \text{next to goal}, \operatorname{do}\left(C_i = 0, X_i = \text{push}\right)\right) = \frac{1}{4}$$, and
$$P\left(Y_i = -10 \big| B_i =  \text{next to goal}, \operatorname{do}\left(C_i = 0, X_i = \text{push}\right)\right) = \frac{3}{4}$$.
We see that for a curriculum composed of source tasks intervening on the box color, the optimal policy learned is maximizing the reward under the last two distributions instead of the target task distribution. Thus, this more extensive task space also won’t help. 
Intuitively, because the reward and the box color are confounded by an unobserved random variable (intrinsic randomness), the agent can only access this information by observing the box color. Once the box color is intervened externally, the agent will lose access to this “intrinsic randomness” and cannot know precisely the right time to push the box. In conclusion, even if the baseline methods change the box color at each time step, they still cannot train the agent successfully. From a theoretical perspective, and as characterized by our theory, there are some variables we should not intervene on during the curriculum generation at all; box color in this case falls exactly into this category. We appreciate the opportunity to clarify this sublet and fundamental matter. One of the primary goals of the theoretical framework proposed in this paper is to identify and characterize the variables that cannot be altered without negatively impacting the agent's convergence following training, which constitutes a causally aligned curriculum.