Paper: Rethinking Teacher-Student Curriculum Learning under the Cooperative Mechanics of Experience (ICLR-2024)

Reviewer comment (Presentation):
This is a minor problem with respect to the presentation of the paper. I personally found the extensive use of italic type text quite distracting when reading the paper.

Author response:
**Minor**: We apologize for our excessive use of italics to highlight concepts and ideas throughout the paper. We have significantly scaled back their use in the updated version of the manuscript.

Reviewer comment (Evaluation):
It is mentioned that the experimental result in classical games "contrasts with folk knowledge in population-based training". How to interpret this result? It should be briefly summarised in the main paper.

Author response:
**Our Work & Population-based Training**: As we explained in Appendix B, if one interprets TSCL as a meta-strategy solver over a population of opponents, our results of selecting TitForTat instead of AlwaysDefect contrast with other meta-strategy solver methods [1] that prescribe playing against the (meta) Nash (strongest strategy) of the population. While we consider this an exciting finding, we deferred it to the appendices due to a space constraint. 

[1] Lanctot, Marc, et al. “A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning.” Advances in Neural Information Processing Systems, 2017

Reviewer comment (Novelty):
Can you explain the practical use of analysing TSCL algorithms from the proposed game-theoretic and data-centric perspective?

Author response:
**Usefulness**: We would like to understand the reviewer’s point about the usefulness of our approach when contrasted to previous work. As mentioned in the manuscript, ourlly differs work fundamenta from Wu et al. [1] and Lee et al [2]. Both works are (computationally expensive) empirical investigations of the effect of curricula while ours introduces a systematic and formal approach, grounded in game-theoretic arguments, to investigate TSCL. We believe the main contribution of our work is foundational.  We hope it serves as a starting point to better understand the problem space and develop novel algorithmic contributions that address the issues we uncovered.

Reviewer comment (Experiments):
Lack of comparison: The paper does not compare the proposed data-centric perspective with existing approaches or frameworks in the field of TSCL. A comparative analysis could provide insights into the advantages and limitations of the proposed approach.

Author response:
**Comparison to Existing Algorithms**: The field of curriculum learning, and approaches that adopt the teacher-student framework, is vast [1, 2] and algorithms are evaluated on a wide range of dissimilar tasks. In many cases, the notion of curriculum among those tasks are not clear. In our experimental setting, we favored the evaluation across different machine learning paradigms to showcase the broad applicability of our framework. From an algorithmic perspective, we selected the multi-armed bandit approach as it is in foundational work on TSCL [3,4] and is commonly utilized through the literature [5,6,7,8]. If the reviewer has suggestions on what algorithm/family of algorithms within TSCL would be better suited for this comparison, we would gladly consider it.

Reviewer comment (Novelty):
Could you provide a comparative analysis of the proposed data-centric perspective with existing approaches or frameworks in the field of TSCL? This would help in understanding the advantages and limitations of the proposed approach.

Author response:
**Comparison to Existing Algorithms**: The field of curriculum learning, and approaches that adopt the teacher-student framework, is vast [1, 2] and algorithms are evaluated on a wide range of dissimilar tasks. In many cases, the notion of curriculum among those tasks are not clear. In our experimental setting, we favored the evaluation across different machine learning paradigms to showcase the broad applicability of our framework. From an algorithmic perspective, we selected the multi-armed bandit approach as it is in foundational work on TSCL [3,4] and is commonly utilized through the literature [5,6,7,8]. If the reviewer has suggestions on what algorithm/family of algorithms within TSCL would be better suited for this comparison, we would gladly consider it.

Reviewer comment (Theory):
While the paper laid down a solid framework for understanding curriculum learning, it lacks a theoretical understanding of the problem, and there are no affirmative theoretical results coming out of it. It provided some generic methodology, but the conclusions are drawn from empirical results, and I am not sure what theoretical questions can be further studied under this topic. This makes the paper somewhat weak and less satisfying.

Author response:
**Further Theoretical Questions**: To the reviewer concern of lack of future theoretical avenues that can be derived from our work, we can mention, for instance, the connection between notions of convex games [1] with sub/super modularity in discrete combinatorial optimization [2,3,4]. However, we considered this connection and further alternative theoretical investigations outside of the scope of the present work. There is a significant communication complexity in presenting ideas at the intersection of these three generally disconnected fields. As the reviewer mentioned, our work presents a foundational framework to further study TSCL and its relationship with other areas like continual learning, transfer learning, or multitask learning under cooperative game-theoretic ideas. That’s why we believe our results warrants further communication to the community.

The revised version of the manuscript now more clearly points to these more theoretical avenues that should be a continuation of the path we have outlined here.

Reviewer comment (Experiments):
The framework proposed in this paper is nice and elegant, but I am very concerned about the practical side of the methodology - it requires training a model multiple times and evaluating the value of each unit of experience. This is very time consuming if each unit of experience is a large dataset or an RL environment. Therefore, it limits the generalizability of the proposed approach.

Author response:
**Scalability & Computational Limitations**: We share the reviewers’ concerns about the scalability of our work. Undoubtedly, cooperative solution concepts are computationally-hard problems with well-known complexity classes [4]. However, recent advances [5, 6] make us hope to obtain better approximations of the cooperative game we designed. On the other hand, the central argument of our work is that, whether we can efficiently compute these solution concepts or not, the cooperative game among units of experience that we develop impacts TSCL's capacity to find curriculum in relatively simple settings. We believe our work should serve as a starting point to develop novel algorithmic contributions that address the problems this data-centric perspective uncovered.

Reviewer comment (Theory):
Please derive solid theoretical results to support the proposed method, or at least provide some theoretical insights.

Author response:
**Further Theoretical Questions**: To the reviewer concern of lack of future theoretical avenues that can be derived from our work, we can mention, for instance, the connection between notions of convex games [1] with sub/super modularity in discrete combinatorial optimization [2,3,4]. However, we considered this connection and further alternative theoretical investigations outside of the scope of the present work. There is a significant communication complexity in presenting ideas at the intersection of these three generally disconnected fields. As the reviewer mentioned, our work presents a foundational framework to further study TSCL and its relationship with other areas like continual learning, transfer learning, or multitask learning under cooperative game-theoretic ideas. That’s why we believe our results warrants further communication to the community.

The revised version of the manuscript now more clearly points to these more theoretical avenues that should be a continuation of the path we have outlined here.

Reviewer comment (Evaluation):
Please discuss the generalizability of the proposed method - see comment (2) in the weaknesses part.

Author response:
**Scalability & Computational Limitations**: We share the reviewers’ concerns about the scalability of our work. Undoubtedly, cooperative solution concepts are computationally-hard problems with well-known complexity classes [4]. However, recent advances [5, 6] make us hope to obtain better approximations of the cooperative game we designed. On the other hand, the central argument of our work is that, whether we can efficiently compute these solution concepts or not, the cooperative game among units of experience that we develop impacts TSCL's capacity to find curriculum in relatively simple settings. We believe our work should serve as a starting point to develop novel algorithmic contributions that address the problems this data-centric perspective uncovered.

Reviewer comment (Novelty):
The readers would be much interested in discussions on the differences and advantages of the proposed method from the existing algorithms.

Author response:
We thank the reviewer for the positive feedback on the novelty and broad applicability of our work.

**Comparison to Existing Algorithms**: The field of curriculum learning, and approaches that adopt the teacher-student framework, is vast [1, 2] and algorithms are evaluated on a wide range of dissimilar tasks. In many cases, the notion of curriculum among those tasks is not clear. In our experimental setting, we selected the multi-armed bandit approach as it is the foundational work on TSCL [3,4] and it is commonly utilized through the literature [5,6,7,8]. If the reviewer has some suggestions on what algorithm/family of algorithms within TSCL would better suited for this comparison, we would gladly consider it.

[1] Soviany, Petru, et al. “Curriculum Learning: A Survey.” International Journal of Computer Vision, vol. 130, no. 6, June 2022, pp. 1526–65, https://doi.org/10.1007/s11263-022-01611-x.

[2] Wang, Xin, et al. “A Survey on Curriculum Learning.” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 9, Sept. 2022, pp. 4555–76, https://doi.org/10.1109/TPAMI.2021.3069908.
 
[3] Graves, Alex, et al. “Automated Curriculum Learning for Neural Networks.” Proceedings of the 34th International Conference on Machine Learning, edited by Doina Precup and Yee Whye Teh, vol. 70, PMLR, 2017, pp. 1311–20, https://proceedings.mlr.press/v70/graves17a.html.

[4] Matiisen, Tambet, et al. “Teacher-Student Curriculum Learning.” IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 9, Sept. 2020, pp. 3732–40, https://doi.org/10.1109/TNNLS.2019.2934906.

[5] Colas, Cédric, et al. “Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration.” arXiv [cs.AI], 21 Feb. 2020, http://arxiv.org/abs/2002.09253. arXiv.

[6] Portelas, Rémy, et al. “Teacher Algorithms for Curriculum Learning of Deep RL in Continuously Parameterized Environments.” arXiv [cs.LG], 16 Oct. 2019, http://arxiv.org/abs/1910.07224. arXiv.

[7] Racaniere, Sebastien, et al. “Automated Curricula through Setter-Solver Interactions.” arXiv [cs.LG], 27 Sept. 2019, http://arxiv.org/abs/1909.12892. arXiv.

[8] Turchetta, Matteo, et al. “Safe Reinforcement Learning via Curriculum Induction.” Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 12151–62