Paper: EduGym: An Environment Suite for Reinforcement Learning Education (ICLR-2024)

Reviewer comment (Reproducibility):
The authors do not provide information about whether open-sourcing their codes.

Author response:
To answer your questions: 

* Yes we definitely open-source the code, and invite everyone to contribute. The repository is already on Github, although we kept it private for now because of the double-blind review.

Reviewer comment (Experiments):
Do the authors plan to add more examples on more RL methods like target network, double Q networks, etc?

Author response:
To answer your questions: 

* We will add novel methods in the future, and have been discussing an “EduGym 2”. Topics to include could be multi-agent, multi-objective, goal-conditioned, etc., as we discuss in the last paragraph of the Discussion. Your suggestions are also very valid!

Reviewer comment (Novelty):
There is no key contribution to RL algorithm and research. This paper introduces an educational library for RL learners. Its key contribution is limited to RL teaching, which is not the key scope of ICLR conference.

Author response:
You indicate our work is outside the scope of ICRL, which we disagree with. In recent years, many papers have appeared at main machine learning conferences that present either an environment suite, benchmark and/or educational effort. Examples include [1,2,3,4,5]. Our work actually falls in the same category. We agree these papers don’t present a novel algorithm, but we disagree they don’t contribute to the research community (they usually get cited a lot). The research community also needs educational efforts for new students entering the field, as does it need benchmarks and small test environments that isolate a particular challenge (which did not exist yet as a collection). Since this type of work has appeared at ICLR, ICML and NeurIPS over recent years [1,2,3,4,5], we do not think it is fair to judge our paper as outside the scope of ICLR.

Reviewer comment (Novelty):
There are many excellent libraries for RL researchers and learners, such as OpenAI’s Spinning Up library and the code of Sutton and Barto’s RL book. Spinning UP provides simple and highly-consistent code for (Deep) RL researchers and practitioners. The code of Sutton and Barto’s RL book also provides example for students to understand concepts, such as the Markov chain example in Q-learning and Double Q-learning. Environments in EduGym are deliberately designed but are not conceptually novel in RL research. EduGym does not provide any guidance for RL practitioners.

Author response:
You also ask about the difference with OpenAI SpinningUp and the code of Sutton and Barto. Spinning Up is very different: it is educational (and very useful), but it implements complex specific deep RL algorithms, such as PPO and DDPG. This is very useful for more advanced students, but it does not isolate individual conceptual aspects of RL, such as partial observability, on/off-policy, sparse exploration, etc. We write about this in the Related Work section, in the paragraph called “Algorithmic frameworks”. 

The code of Sutton and Bart comes much closer, and the CliffWalking example to illustrate the difference between on/off-policy was a major inspiration. However, there are two main differences: 1) Sutton and Barto mostly focus on credit assignment, and do not include specific conceptual examples for most of the challenges we did include (state dimensionality, partial observability, language, continuous actions) etc., and 2) most importantly, the code of Sutton and Barto does not come with interactive notebooks, which prevents the mixing of text and code, and makes it way less accessible to students.

Reviewer comment (Reproducibility):
It lacks best practices on how to use deep neural networks in RL.

Author response:
Finally, you aks about best practices on deep learning. We do include a notebook about deep learning, which is the notebook on state dimensionality. Also, deep learning appears in several other notebooks, such as partial observability and continuous state/actions. However, we do indeed not discuss all practical aspects of tuning deep learning, but this is also clearly beyond the scope of our work. We intend to teach reinforcement learning fundamentals: exploration, credit assignment, partial observability, etc. Our main focus lies on small conceptual examples, while for complex deep RL many codebases actually do exist (such as Spinning Up).

Reviewer comment (Writing):
The lack of introduction to basic RL algorithms like value iteration or REINFORCE [1].

Author response:
As a weakness you mention the lack of value iteration and policy gradients. They are actually included: value iteration appears in the notebook on model-based RL, while the policy gradient appears in the notebook on continuous state/actions.

Reviewer comment (Novelty):
Lack of innovation.

Author response:
You finally mention that the innovation is limited, which we disagree with. It’s true that EduGym does not contribute a novel algorithm, but we believe the RL community as a whole has immensely profited in recent years from environment suites, benchmarks and educational efforts. These papers often get quite some traction (citation-wise), which indicates they have their merit for the community. We think EduGym also falls in this category of papers, and although there is no novel algorithm, we do not think this implies the innovation is limited.

Reviewer comment (Theory):
It would be beneficial to include information about the pedagogical principles or educational theories that guided the development of EduGym. How was the content structured to support effective learning, and were any instructional design models considered?

Author response:
You first of all ask about the learning theory below EduGym. This is mostly “Learning by Doing”, about which there is much literature [1,2,3]. The main idea is that we need to connect theory (equations, text) to practical interactive examples (code, notebooks) in a single document. In addition, we attempt to break the problem down in underlying conceptual topics. We have not included this in the paper since it seemed too far off for a machine learning conference, but of course see your point as well, and could include it.

Reviewer comment (Reproducibility):
What is the plan for maintaining and updating EduGym to keep it relevant and aligned with the latest developments in RL research and education?

Author response:
You also ask whether EduGym will be 1) extended and 2) kept up-to-date. Regarding the former, we definitely will extend Edugym, as we indicate in the last paragraph of the Discussion, where we for example indicate we think of a future EduGym extension with multi-agent, multi-objective, goal-conditioned RL, etc. Regarding the latter, we will also maintain EduGym, but additions will probably mostly focus on novel notebooks. The reason is that EduGym teaches the RL basics, such as the distinction between on- and off-policy updates. These conceptual topics are so fundamental to RL, that their basic explanation will probably stay quite stable. Of course, with novel research completely novel topics may develop, which we will then try to incorporate into EduGym.

Reviewer comment (Novelty):
The main contribution of this paper seems to be the creation of a new tool, software package, environment, and notebooks for learning RL. I am not sure how such contribution suits the themes at ICLR. While such new learning materials are welcomed, it lacks scientific rigor I am expecting from an ICLR contribution.

Author response:
You indicate our paper has an educational rather than technical contribution, which does not suit ICLR. We believe this criticism is not fair, when compared to the RL papers that have been published at major ML conferences in recent years. Our work contains an environment suite, of which several have appeared in big ML conferences in recent years [2,3], and can also be leveraged as a benchmark suite, which has also previously appeared at ICLR [1]. The same applies to structured codebases at the intersection of research and education [4]. All these papers got cited quite a lot, and have their impact on the community by making novel research feasible. Our additional educational focus should only add to this, since entering the research field can be challenging. In short, we believe our paper has merit to the research community, even though it does not present a new algorithm.

Reviewer comment (Evaluation):
Although the evaluation results show that EduGym improves learning outcomes, this is measured by self-reported survey. There is no other evaluative methods to assess learning, nor any kind of comparison (or A/B testing) to existing tools to benchmark the effectiveness of EduGym, without which it is difficult to understand the educational utility of the proposed environment.

Author response:
Your second critique is that our work is not properly empirically validated. Of course, empirical validation is important, but we actually did include this evaluation. We asked a range of bachelor, master and PhD students with experience in RL whether they valued EduGym, which most of them strongly agreed with. These students already know a bit of RL, and can therefore judge whether our resources help them compared to what they used so far. Note that nearly all other teaching resources (books, public codebases) available are just published, and never evaluated with A/B testing. We therefore believe this criticism is also not completely fair, since we 1) included a useful practical evaluation and 2) already go beyond the practical evaluation of other educational resources.

Reviewer comment (Novelty):
This paper, in my opinion, is a good educational contribution rather than a technical contribution, which seems to me an unsuitable contribution for ICLR.

Author response:
You indicate our paper has an educational rather than technical contribution, which does not suit ICLR. We believe this criticism is not fair, when compared to the RL papers that have been published at major ML conferences in recent years. Our work contains an environment suite, of which several have appeared in big ML conferences in recent years [2,3], and can also be leveraged as a benchmark suite, which has also previously appeared at ICLR [1]. The same applies to structured codebases at the intersection of research and education [4]. All these papers got cited quite a lot, and have their impact on the community by making novel research feasible. Our additional educational focus should only add to this, since entering the research field can be challenging. In short, we believe our paper has merit to the research community, even though it does not present a new algorithm.

Reviewer comment (Evaluation):
The paper lacks scientific rigor in terms of evaluating the utility of EduGym.

Author response:
Your second critique is that our work is not properly empirically validated. Of course, empirical validation is important, but we actually did include this evaluation. We asked a range of bachelor, master and PhD students with experience in RL whether they valued EduGym, which most of them strongly agreed with. These students already know a bit of RL, and can therefore judge whether our resources help them compared to what they used so far. Note that nearly all other teaching resources (books, public codebases) available are just published, and never evaluated with A/B testing. We therefore believe this criticism is also not completely fair, since we 1) included a useful practical evaluation and 2) already go beyond the practical evaluation of other educational resources.