Paper: From Child's Play to AI: Insights into Automated Causal Curriculum Learning (ICLR-2024)

Reviewer comment (Writing):
Level progress is a confusing term for curriculum learning researchers. “Learning progress” or “competence progress” is a standard proposition of intrinsic motivation to guide curricula, it refers to the derivative of the competence measure with respect to time: how much the competence increases or decreases. Here you use the term “progress” but it refers to what people call “competence,” a measure of success, a score. For instance, if you play the same level 10 times and reach the same score each time, there is no progress at all across-episodes, the progress is only intra-episode. I know that level progress describes what you expect it to describe in the language of video games, but it’s confusing for researchers in this field: I thought you meant learning progress for the first 4-5 pages. Could you consider using another term? Competence is the standard term I think, score or performance could be good as well.

Author response:
**Regarding the terminology of ‘level progress’.**

We very much appreciate your  suggestion to better define level progress and other key terms for our paper. We have revised the terminology from ‘level progress’ to ‘level competence’, to reflect that this is not a derivative, but rather the current level of success the agent/ human achieves on the level. We have also written up more precisely defined definitions (see General Comments above). 

**Level Competence**: An indicator of success in an episode of game play on a specific level, reaching 100% when the specific level is successfully solved in that episode. (Note: This is the term formerly referred to as 'level progress.') 

**Global Competence**: A measure of success in an episode of game play with respect to the target level, reaching 100% if the target level is successfully solved in that episode.

**Level Advancement**: Difference between current level competence and initial level competence within a particular level in the curriculum; it is a measure of how much competence increases or decreases in the same level

**Global Advancement**:  Difference between current global competence and initial global competence across levels in the curriculum; it is a measure of how much competence increases or decreases with respect to achieving the target level
Auxiliary reward: A scaled level competence

Reviewer comment (Evaluation):
The hypothesis supported here is not very clearly stated. There are several alternative hypotheses that could explain the data:
* Children have some form of heuristic curriculum strategy that would look a bit like this: if i’m very good, i switch to a harder task, if i intermediate I stay there, if i’m bad i either stay or try something easier. 
* Children have a causal understanding of the world and maximize their causal learning
* Children have an intermediate difficulty bias (~Florensa’s paper): they keep doing the task for which they have intermediate competence / performance. 
* Children have a bias towards learning progress: they can estimate expected learning progress in a model-based way and select the task that maximizes it. When a game is solved there is no further learning progress so they switch up, when they perform poorly they might expect more progress soon and stay, or not expect progress and recalibrate their estimations, which leads them to switch down.

Author response:
**Regarding the alternative hypotheses that could explain the data.**

We very much appreciate the suggestion to make our hypothesis more explicit, and we appreciate the reviewer for diligently differentiating between the hypotheses. The reviewer is correct in their understanding of what our proposed hypothesis is: Children have a causal understanding of the world and maximize their causal learning for a given goal (i.e., solving the target level). We appreciate the reviewer’s suggestion “to include levels from games different from the target game level.” We are currently collecting data for a follow-up study that addresses this question. We have a version of this study where we add an irrelevant “non-causal” axis, e.g., changing the color shades, and easy levels from other games. We will mention this follow-up study in the work. As an additional response to the reviewer’s question about examples of when children almost never select the irrelevant game, it can also be seen with our addition of the extra difficult level 4 (with the target being level 3) in the game design. Solving level 4 can be argued as irrelevant because it takes more unnecessary effort and does not help children achieve the extrinsic goal and win a sticker. We find that the majority of children never select the irrelevant, overly-difficult level 4 for building their curriculum to solve the target level. 

We also ran additional exploratory analyses to address your third and fourth alternative hypotheses:
On intermediate difficulty bias: because the easier levels are direct subsets of harder levels in the games Leaper and Climber, we can calculate competence on easier levels as a percentage of competence in the target level to get global competence for each episode. We can then also determine which level children should objectively choose next by comparing their global competence with the percentage of the target level that each easier level represents. Here, we found that half of the time children were making objective choices, but another half of the time children chose harder levels than what they should objectively choose, suggesting that they have an intermediate to high difficulty bias that is likely also driven by the desire to causally solve the difficult target level
On bias towards learning progress: we ran further analyses and found that children do not have a bias towards learning progress or advancement: our child participants neither used global advancement nor level advancement (as defined in our definitions in the General Comments) to guide their choice of next level to play, suggesting that they are not driven by how much progress they have made, but more by how well they are currently performing to move towards the goal (as demonstrated by our statistics on local competence guiding next level change)
We will revise our paper to clearly state our hypothesis and include statistics of these additional findings.

Reviewer comment (Theory):
The paper seems to be arguing for the interpretation that children have a causal understanding of the world and maximize their causal learning, although it is not stated clearly. There is nothing in the paper that would allow the reader to argue for one hypothesis more than the others.

Author response:
We thank the reviewer for their time to follow up with additional comments for interpreting the child data.

We are interested in both kinds of causal understanding (rules and physics of the game + practicing on easier levels causes improvement on target levels) and have ongoing experiments probing both ideas.

We also informally looked at children’s open responses and found that they often referred to both the causal structure of the rules of the game and the causal structure of how solving an easier level could then enable them to solve a harder level of the same game. 

In our games, the causally relevant axis to help you achieve ultimate success at the goal level is difficulty. A child or an agent learns the game as a causal curriculum by choosing how to move along the difficulty axis. We consider changing the color shades to be a non-causal axis because there is no causal relationship between color shade and winning the game. Color does not have any significant causal power or insight in the game for achieving the target level.

In our experiment setup, each child is subject to only a maximum of 10 rounds of automated curriculum learning; in every round they have the choice to select any of the game levels. Many children even solved the target level and terminated in less than 10 rounds (the average total number of rounds played is 7 across participants, with the minimum being 3). Thus, we do not have enough statistical power at the moment to adopt existing learning progress approaches (e.g., comparison between last 3 rounds and previous 3 rounds within a level) in our measure of progress or advancement. To circumvent this, we calculated level competence as just the competence in the active episode, and level advancement as the difference between the most recent episode and the initial episode of the same level. 

While we only have 22 children and are underpowered to do model fitting and model predictions, we have built linear mixed effects models to compare various hypotheses, and found that our data can be best explained by the idea that “children have a causal understanding of the world and use their level competence to guide their causal curriculum learning for a given goal (i.e., solving the target level)” (this model has the lowest AIC and BIC). After further examination of the data, intermediate difficulty bias and level advancement (not computed as the classic learning progress due to limited data per individual participant) so far did not explain much of child behavior - a notable portion of children seemed to remain at the same level or move to higher levels even when they had very low level competence or negative level advancement because they were motivated to achieve the extrinsic goal of solving the target level, but we have yet to collect more data to confirm this idea. 

Once we have collected enough data, we will definitely try adopting your recommendations on fitting different models with some portion of child data and using them to predict other children!

Reviewer comment (Experiments):
An experiment that could be interesting to include is levels from games different from the target game level. If children optimize for learning progress only, then we should see children select easy levels of the non-target games as well. Instead, we would probably see children almost never select the irrelevant game, which would argue for a combination of intrinsic (LP-like) and extrinsic (going for the target level) curriculum.

Author response:
We very much appreciate the suggestion to make our hypothesis more explicit, and we appreciate the reviewer for diligently differentiating between the hypotheses. The reviewer is correct in their understanding of what our proposed hypothesis is: Children have a causal understanding of the world and maximize their causal learning for a given goal (i.e., solving the target level). We appreciate the reviewer’s suggestion “to include levels from games different from the target game level.” We are currently collecting data for a follow-up study that addresses this question. We have a version of this study where we add an irrelevant “non-causal” axis, e.g., changing the color shades, and easy levels from other games. We will mention this follow-up study in the work. As an additional response to the reviewer’s question about examples of when children almost never select the irrelevant game, it can also be seen with our addition of the extra difficult level 4 (with the target being level 3) in the game design. Solving level 4 can be argued as irrelevant because it takes more unnecessary effort and does not help children achieve the extrinsic goal and win a sticker. We find that the majority of children never select the irrelevant, overly-difficult level 4 for building their curriculum to solve the target level.

Reviewer comment (Experiments):
The switch threshold in the RL experiments appears to be an important hyperparameter, and it would be interesting to see how it affects the results. A threshold of 0.9 seems high given that children seem to switch up around 75%.

Author response:
We agree that the switch threshold is an important hyperparameter. We are also interested in seeing the results and will report back what we learn.

Reviewer comment (Experiments):
There is a lack of clarity in understanding the manual curriculum in the RL experiments.

Author response:
To clarify the manual curriculum, there are 16 parallel tasks so that the agent can collect data more efficiently by interacting in parallel across 16 tasks. When we increment the difficulty by 1/16, this means that, of the 16 parallel tasks, one of the tasks has the difficulty incremented (and the others do not change). So, the progression would be 1) 16 1-lanes, 2) 15 1-lanes and 1 2-lane, 3) 14 1-lanes and 2 2-lanes, etc. We did originally experiment with switching all 16 tasks by one difficulty, but we found that this posed a sizable distribution shift to the agent that caused training divergence at much easier difficulties than if the difficulty was instead incremented by 1/16. However, we did notice as in Section 5.3 that training divergence does eventually occur. In Section 5.4, the level progress used in the auxiliary reward is calculated at the termination of an episode and then added to the extrinsic reward. For cases where we present average episode level progress (e.g., Figure 6b), this is calculated by taking the mean level progress over all finished episodes between policy updates.

Reviewer comment (Experiments):
It is unclear why there are 16 parallel tasks in the manual curriculum.

Author response:
To clarify the manual curriculum, there are 16 parallel tasks so that the agent can collect data more efficiently by interacting in parallel across 16 tasks.

Reviewer comment (Evaluation):
It is unclear what it means to increment the difficulty by 1/16. If the difficulty is the number of lanes (1–5) and the agent passes 0.9 in the current level (e.g., 1 lane), it would be expected that all 16 tasks become 2-lane tasks and the agent trains on these. Here it sounds like one of the 16 tasks becomes a lane-2 task, raising the question of how the score metric is computed—whether it is the average level progress over the 16 tasks (15 at 1-lane and 1 at 2-lane).

Author response:
When we increment the difficulty by 1/16, this means that, of the 16 parallel tasks, one of the tasks has the difficulty incremented (and the others do not change). So, the progression would be 1) 16 1-lanes, 2) 15 1-lanes and 1 2-lane, 3) 14 1-lanes and 2 2-lanes, etc. We did originally experiment with switching all 16 tasks by one difficulty, but we found that this posed a sizable distribution shift to the agent that caused training divergence at much easier difficulties than if the difficulty was instead incremented by 1/16. However, we did notice as in Section 5.3 that training divergence does eventually occur. In Section 5.4, the level progress used in the auxiliary reward is calculated at the termination of an episode and then added to the extrinsic reward. For cases where we present average episode level progress (e.g., Figure 6b), this is calculated by taking the mean level progress over all finished episodes between policy updates.

Reviewer comment (Experiments):
The on-policy algorithm used does not leverage a replay buffer, meaning that old data from easier levels are discarded. Using an off-policy algorithm (e.g., DDPG) would enable reuse of past data, which may mitigate the catastrophic forgetting problem.

Author response:
We thank the reviewer for the suggestion for trying an off-policy algorithm that uses a replay buffer. In the baselines that we just ran, we found that stochastically sampling previously solved levels indeed mitigates the catastrophic forgetting issue, however, progress made within the curriculum / towards solving the goal level is less than what we found in Section 5.4.

Reviewer comment (Experiments):
Catastrophic forgetting for curriculum learning is a known problem, which is why all curriculum approaches perform stochastic selection: they do not switch from one level to the other deterministically but sample all levels with varying probabilities that are a function of the intrinsic motivation. For example, selecting the current best level with probability 0.7 and the rest with 0.3 / n_other_levels may fix the issue (see Jiang’s paper and Colas 2019 for examples).

Author response:
We thank the reviewer for the suggestion for trying an off-policy algorithm that uses a replay buffer. In the baselines that we just ran, we found that stochastically sampling previously solved levels indeed mitigates the catastrophic forgetting issue, however, progress made within the curriculum / towards solving the goal level is less than what we found in Section 5.4.

Reviewer comment (Experiments):
Studies of curriculum algorithms always include a presentation of the random baseline selecting levels uniformly. This baseline does not suffer from distribution shifts, and its absence here is a gap.

Author response:
**Regarding additional baselines**

We thank the reviewer for their suggestion on baselines to run. Please see the general comments for the baselines we are currently running and will report back on.

Reviewer comment (Theory):
The use of the intrinsic reward is unclear. As understood, the curriculum (level selection) is the same as before but PPO now uses what is called intrinsic reward in addition. This is problematic because intrinsic motivations must be agnostic of the goal. The proposed approach assigns a reward to a state, but this reward is extrinsic because it measures performance in the task. This essentially replaces a sparse reward with a dense reward, with no intrinsic motivation present.

Author response:
Regarding the auxiliary reward, at the end of each episode, the agent receives a reward that is the sum of 1) an extrinsic reward (which, in this case, only occurs if the level is solved) and 2) the current advancement of the agent within the level. The agent only receives these rewards at the end of an episode, not at each step. We agree that the consequence of providing such an auxiliary reward is that it “densifies” the rewards.

Although it is not directly curriculum learning, we see that it impacts curriculum learning results. Having such a reward helps prevent forgetting which arises due to the distribution shift induced by switching levels. It may not be the only way to prevent forgetting, but having the auxiliary reward appears promising to both prevent forgetting and advance the curriculum. In the baseline experiment we recently conducted using a stochastic curriculum with sampling previously solved levels, we see that forgetting was also avoided, but the agent made less progress within the curriculum than with auxiliary rewards.

The relationship between the auxiliary reward in the RL experiments and the human experiments is that we observed that the children’s competence within a level improved without necessarily solving the level. The analogy for the RL agent is that if there was an additional reward besides the sparse extrinsic reward that could help with learning, the agent could potentially learn more effectively than with extrinsic rewards alone.

At the present time, our RL experiments were conducted using a hand-specified curriculum. In future work, we are interested in experimenting with different level selection policies. However, our results do show that, at least for this task with sparse rewards, that other algorithmic capabilities besides level selection (e.g., capabilities for preventing forgetting that arises when switching levels) may be needed for effective curriculum learning.

Reviewer comment (Writing):
Minor comment: Vygotsky’s citation is broken (missing "y", no year).

Author response:
**Regarding the broken citation.**

We fixed the broken Vygotsky citation, thank you for letting us know.

Reviewer comment (Writing):
Level progress is not defined clearly at all through the whole paper leaving the other discussion built upon it shaky.

Author response:
We would like to thank the reviewer for their suggestion to better define level progress and other key terms for our paper. We have introduced a definition table, which contains clearer explanations. These definitions are presented below. Level Competence (previously referred to as ‘level progress’): an indicator of success in an episode of playing a level, reaching 100% when the specific level is successfully solved in that episode.

Reviewer comment (Novelty):
From my understanding after reading the texts, level progress seems to be a rough measure of how far you have gone in this level. Putting aside how we can access such information, if we do, this will vary from environment to environment and it seems to be simply a denser supervision signal for each move the agent makes towards accomplishing the task. So, the proposed method is both inaccessible and not generalizable.

Author response:
We would appreciate it if the reviewer could clarify their concerns regarding the inaccessibility and generalizability of level progress. We agree that, in our current formulation, the use of level progress as an auxiliary reward does have the effect of providing a denser supervision signal. To the question of how we plan to access level progress, we believe the work of Bruce 2023 “Learning About Progress From Experts” (as mentioned by reviewer Rafo) could be extended to our domain.

Reviewer comment (Experiments):
In the experiment section, the authors didn't show training results with multiple random seeds but only single runs.

Author response:
As for the RL experiments, each of the experiments in Section 5.3 and 5.4 are run over six trials each, where each trial has a unique random seed. Figure 4 and Figure 6 are from one representative trial within the six trials.
For Section 5.3, we consistently saw catastrophic forgetting across random seeds. The hyperparameters used were those from Jiang 2021 “Prioritized Level Replay”.

Reviewer comment (Writing):
What is the formal definition of level progress?

Author response:
We would like to thank the reviewer for their suggestion to better define level progress and other key terms for our paper. We have introduced a definition table, which contains clearer explanations. These definitions are presented below. Level Competence (previously referred to as ‘level progress’): an indicator of success in an episode of playing a level, reaching 100% when the specific level is successfully solved in that episode.

Reviewer comment (Evaluation):
How do you plan to access the level progress you defined?

Author response:
We would appreciate it if the reviewer could clarify their concerns regarding the inaccessibility and generalizability of level progress. We agree that, in our current formulation, the use of level progress as an auxiliary reward does have the effect of providing a denser supervision signal. To the question of how we plan to access level progress, we believe the work of Bruce 2023 “Learning About Progress From Experts” (as mentioned by reviewer Rafo) could be extended to our domain.

Reviewer comment (Evaluation):
The explanation of the experimental results is somewhat vague, and the experimental environment appears to be limited.

Author response:
**Regarding clarifications on the experimental results and environments.**

We appreciate the reviewer's suggestion regarding clarifications on our experimental results and the  experimental environments. Could we ask the reviewer to specify additional details or highlight specific areas of concern for us to address.  We are enthusiastic about enhancing the clarity of our explanations and offering a more comprehensive understanding of our experimental setup.

Reviewer comment (Experiments):
If the way 5-7-year-old children change their levels does not apply to RL agents, it suggests that the study has not fully integrated the observations from children's behavior into RL agents.

Author response:
We thank the reviewer for their time to follow up with additional questions regarding the experiments.
Correct, in the RL experiments, we use a hand-designed curriculum that switches to a more difficult level when the average reward exceeds 9. (In 5.4, this threshold only looks at the average extrinsic reward, to make a fair comparison to 5.3 when switching levels).
The insights from the children studies in how children can learn and increase their competence without necessarily solving the level inspired our study in 5.4 of seeing if, assuming a metric was available to assess level competence, using this metric as an auxiliary reward to improve learning. Indeed, we found this was the case: it prevented catastrophic forgetting. The effect of training using this auxiliary reward can indeed be seen as a way of “densifying” the reward signal. (Note that during the rebuttal, we changed the terminology of the “intrinsic reward” to “auxiliary reward” to better align with the literature in this area.)
We also believe it would be useful in future work to integrate how children change levels (Figure 3) into the curriculum learning policy for the RL agent (how it changes levels). Nonetheless, we believe the insight of how children learn without necessarily solving the levels is important in the design of a curriculum learning agent. This is because curriculum learning involves intentionally inducing distribution shifts, so capabilities that mitigate the consequences of that (such as using an auxiliary reward) would be fruitful for effective curriculum learning.

Reviewer comment (Experiments):
There is no comparison with baselines or prior work.

Author response:
**Regarding additional baselines.**

We thank the reviewer for their suggestion on baselines to run. To clarify the experiments in Sections 5.3 and 5.4, the agent advances to the next level of the curriculum when the average reward obtained exceeds 9. The auxiliary reward mentioned in Section 5.4 is inspired by human curriculum learning, but the advancement of level is still just done when the average reward obtained exceeds 9. We are interested in running a baseline where the curriculum also changes based on lambda.

Reviewer comment (Experiments):
In both Experiment 5.3 and 5.4, does the agent move to the next level when it achieve an average reward of 9 means (as explained in section 5.2)?

Author response:
**Regarding additional baselines.**

We thank the reviewer for their suggestion on baselines to run. To clarify the experiments in Sections 5.3 and 5.4, the agent advances to the next level of the curriculum when the average reward obtained exceeds 9. The auxiliary reward mentioned in Section 5.4 is inspired by human curriculum learning, but the advancement of level is still just done when the average reward obtained exceeds 9. We are interested in running a baseline where the curriculum also changes based on lambda.

Reviewer comment (Experiments):
Did the proposed method in 5.4 incorporate the way 5-7-year-old children change their levels?

Author response:
We thank the reviewer for their time to follow up with additional questions regarding the experiments.
Correct, in the RL experiments, we use a hand-designed curriculum that switches to a more difficult level when the average reward exceeds 9. (In 5.4, this threshold only looks at the average extrinsic reward, to make a fair comparison to 5.3 when switching levels).
The insights from the children studies in how children can learn and increase their competence without necessarily solving the level inspired our study in 5.4 of seeing if, assuming a metric was available to assess level competence, using this metric as an auxiliary reward to improve learning. Indeed, we found this was the case: it prevented catastrophic forgetting. The effect of training using this auxiliary reward can indeed be seen as a way of “densifying” the reward signal. (Note that during the rebuttal, we changed the terminology of the “intrinsic reward” to “auxiliary reward” to better align with the literature in this area.)
We also believe it would be useful in future work to integrate how children change levels (Figure 3) into the curriculum learning policy for the RL agent (how it changes levels). Nonetheless, we believe the insight of how children learn without necessarily solving the levels is important in the design of a curriculum learning agent. This is because curriculum learning involves intentionally inducing distribution shifts, so capabilities that mitigate the consequences of that (such as using an auxiliary reward) would be fruitful for effective curriculum learning.

Reviewer comment (Reproducibility):
While it's mentioned that 5-7-year-old children use level progress as an intrinsic reward signal to decide which level to play to solve the most challenging level, does the proposed method in section 5.4 actually use this intrinsic reward function (2* lambda / 100) for transitioning between levels? (i.e. do they use it when they determine whether they change their level or not?, or just determine the level transition based on if(mean episode reward>9)? )

Author response:
We thank the reviewer for their time to follow up with additional questions regarding the experiments.
Correct, in the RL experiments, we use a hand-designed curriculum that switches to a more difficult level when the average reward exceeds 9. (In 5.4, this threshold only looks at the average extrinsic reward, to make a fair comparison to 5.3 when switching levels).
The insights from the children studies in how children can learn and increase their competence without necessarily solving the level inspired our study in 5.4 of seeing if, assuming a metric was available to assess level competence, using this metric as an auxiliary reward to improve learning. Indeed, we found this was the case: it prevented catastrophic forgetting. The effect of training using this auxiliary reward can indeed be seen as a way of “densifying” the reward signal. (Note that during the rebuttal, we changed the terminology of the “intrinsic reward” to “auxiliary reward” to better align with the literature in this area.)
We also believe it would be useful in future work to integrate how children change levels (Figure 3) into the curriculum learning policy for the RL agent (how it changes levels). Nonetheless, we believe the insight of how children learn without necessarily solving the levels is important in the design of a curriculum learning agent. This is because curriculum learning involves intentionally inducing distribution shifts, so capabilities that mitigate the consequences of that (such as using an auxiliary reward) would be fruitful for effective curriculum learning.

Reviewer comment (Reproducibility):
How many seeds were used to obtain the experimental results?

Author response:
**Regarding the number of random seeds for our RL experiments.**

For the experiments in Sections 5.3 and 5.4, six trials were used for each experiment, where each trial has a unique random seed. The results shown in Figure 4 and Figure 6 are for one representative trial within the six trials.

Reviewer comment (Experiments):
I’m sorry that I’m not very familiar with the human experiments in cog-science. But I worry that the number of children (only 22 children) in this study is too small. How do you guarantee that the results obtained from such a small group of children is unbiased and trustable? Maybe please point us to some related works that also conducted human experiments and claimed that such a small experimental population would be enough.

Author response:
**Regarding the number of children in the study.**

We are collecting more data on children at the moment. There are developmental cognitive science papers that have published experiments with a sample size of about 20-30 each (e.g., see Experiment 1-3 in [3],  see Experiment 1 in [4]). (Children are much harder to recruit than adults, but they are important for understanding the bases of curriculum learning.)

But more importantly, even though 22 children looks like a small sample size, each child does not only contribute to one data point. We are interested in examining children’s steps (or rounds of the game played) throughout their curriculum; within each step, we have details about which level they select, how well they perform on the specific step, etc. We currently have data for 159 steps that children made in their curricula and even when we factor in individual differences in our regression model, we are still able to find a statistically robust pattern between their level competence and their choice of next level. 

[1] Kushnir, T., Gopnik, A., Chernyak, N., Seiver, E., & Wellman, H. M. (2015). Developing intuitions about free will between ages four and six. Cognition, 138, 79-101.
[2] Sobel, D. M., & Corriveau, K. H. (2010). Children monitor individuals’ expertise for word learning. Child development, 81(2), 669-679.

Reviewer comment (Novelty):
Another concern is about novelty. I admire the motivation from human experiments. However, utilizing task progress is not a novel idea in reinforcement learning [1], so I worry that the proposed method is more like a trick.

Author response:
**Regarding related work by Bruce 2023.**

We also thank the reviewer for bringing Bruce 2023 to our attention, as this work seems highly relevant. Indeed, the concept of task progress is not novel to our work. However, we note a difference between our setting and Bruce 2023 is that Bruce 2023 requires expert trajectories to learn their progress model. In our case, we do not have access to expert trajectories, so we can envision that our approach could introduce novelty by relaxing this assumption and thereby extend their method.

Reviewer comment (Experiments):
There are many previous works taking advantage of various heuristics to design curriculars. I think it is necessary to compare your method with them. Would you please show us some comparison results with existing techniques?

Author response:
**Regarding additional baseline experiments.**

We appreciate the reviewer for pointing this out.  thank the reviewer for their suggestion on baselines to run.  We have indeed conducted additional experiments, and we report the results here. First, we have conducted a baseline where environments are chosen randomly from 1-5 water lanes, without the curriculum. This is done for six trials/random seeds with a limit of 10M frames. We find that the agent makes very little learning progress over 10M frames. The return for the goal level (5 water lanes) is generally zero over the entire 10M frames. Moreover, the agent does not obtain much training return, either. We hypothesize that this is a difficult exploration problem for the agent without a curriculum, and although the agent may eventually make progress, the timescale would far exceed the 10M frame budget we used for our experiments.

Second, we conducted a baseline against Section 5.3 (curriculum without auxiliary reward) where, 30% of the time, previously solved levels are chosen based on where the agent currently is in the curriculum. For example, if the current curriculum progress is 3.5 water lanes, then levels would be chosen from [1, 2] water lanes 30% of the time. This is done for 6 trials/random seeds with a limit of 10M frames. We do find that this mitigates catastrophic forgetting, as training divergence did not happen in any of the trials. However, the agent makes less progress towards the curriculum than in Section 5.4 with the auxiliary reward. In Section 5.4, all six trials got to at least 4 water lanes, whereas in this baseline, only one trial got to 4 water lanes. The average number of water lanes across all trials was 2.958. Additionally, in this baseline, none of the trials were able to solve the goal level (unlike in Section 5.4, where 5 of the trials could solve the goal level). We find that, although stochastically selecting previously solved levels prevents catastrophic forgetting, it may not be an optimal approach for advancing the curriculum. We hypothesize that this occurs because 30% of the training budget is being spent to prevent agent forgetting, rather than learning how to solve the latest level in the curriculum. Therefore, it would be valuable to have an agent learn and utilize the auxiliary reward used in Section 5.4, as this both prevents catastrophic forgetting and advances the curriculum.

Third, we conducted a baseline that was similar to the second one, except instead of sampling previously solved levels, all possible levels (from 1 to 5 water lanes) were sampled. Six trials/random seeds were used with a limit of 10M frames. This baseline generally does poorly. In 4 trials, the training divergence still occurred. In only one trial was the training still active; this trial reached 3.0 water lanes. We hypothesize that this baseline performs worse than sampling previously solved levels, because sampling more difficult levels generally leads to more difficult levels that may not be solvable. This yields sparser returns and induces the same training divergence behavior seen in Section 5.3.

We hope this has addressed the reviewer’s concern, we would be happy to discuss/ conduct any additional experiments if the reviewer has additional experiments/ concerns in mind.

Reviewer comment (Experiments):
Although very inspirational work, there are many questions around the user study that is not clear. How reliable are children responses? How was this controlled for and simplified so children could understand and respond appropriately?

Author response:
**Regarding questions on the user study of children.**
We agree that it is important to have a well controlled and understandable setup for children. In our experiment setup, we included an extra difficult level beyond the target level to validate that children were not just making random choices: we found that so far only 14% of children selected to play this extra difficult level, suggesting that most children understood what their goal was and did not attempt what was beyond that goal. Furthermore, we observe statistically significant evidence that children use their game level competence to guide their decision of which level to play next, again reflecting that they understood their task and were not making random responses.
With regards to multiple choice questions, there are developmental studies studying children of similar age ranges that use multiple choice questions [1],[2]. We will include relevant citations in our paper. More importantly, we found that most children were able to respond correctly to the multiple choice question pertaining to the true causal rule of the game.
[1] Kominsky, J. F., Gerstenberg, T., Pelz, M., Sheskin, M., Singmann, H., Schulz, L., & Keil, F. C. (2021). The trajectory of counterfactual simulation in development. Developmental Psychology, 57(2), 253.
[2] Rafetseder, E., & Perner, J. (2018, April). Belief and counterfactuality: A teleological theory of belief attribution. Zeitschrift für Psychologie, 226 (2), 110-121. doi: 10.1027/ 2151-2604/a000327

Reviewer comment (Experiments):
Asking 5 year olds multiple choice questions pertaining to the true causal rules of games needs more supportive evidence.

Author response:
**Regarding questions on the user study of children.**
We agree that it is important to have a well controlled and understandable setup for children. In our experiment setup, we included an extra difficult level beyond the target level to validate that children were not just making random choices: we found that so far only 14% of children selected to play this extra difficult level, suggesting that most children understood what their goal was and did not attempt what was beyond that goal. Furthermore, we observe statistically significant evidence that children use their game level competence to guide their decision of which level to play next, again reflecting that they understood their task and were not making random responses.
With regards to multiple choice questions, there are developmental studies studying children of similar age ranges that use multiple choice questions [1],[2]. We will include relevant citations in our paper. More importantly, we found that most children were able to respond correctly to the multiple choice question pertaining to the true causal rule of the game.
[1] Kominsky, J. F., Gerstenberg, T., Pelz, M., Sheskin, M., Singmann, H., Schulz, L., & Keil, F. C. (2021). The trajectory of counterfactual simulation in development. Developmental Psychology, 57(2), 253.
[2] Rafetseder, E., & Perner, J. (2018, April). Belief and counterfactuality: A teleological theory of belief attribution. Zeitschrift für Psychologie, 226 (2), 110-121. doi: 10.1027/ 2151-2604/a000327