  Training autonomous agents able to generalize to multiple tasks is a key
target of Deep Reinforcement Learning (DRL) research. In parallel to improving
DRL algorithms themselves, Automatic Curriculum Learning (ACL) study how
teacher algorithms can train DRL agents more efficiently by adapting task
selection to their evolving abilities. While multiple standard benchmarks exist
to compare DRL agents, there is currently no such thing for ACL algorithms.
Thus, comparing existing approaches is difficult, as too many experimental
parameters differ from paper to paper. In this work, we identify several key
challenges faced by ACL algorithms. Based on these, we present TeachMyAgent
(TA), a benchmark of current ACL algorithms leveraging procedural task
generation. It includes 1) challenge-specific unit-tests using variants of a
procedural Box2D bipedal walker environment, and 2) a new procedural Parkour
environment combining most ACL challenges, making it ideal for global
performance assessment. We then use TeachMyAgent to conduct a comparative study
of representative existing approaches, showcasing the competitiveness of some
ACL algorithms that do not use expert knowledge. We also show that the Parkour
environment remains an open problem. We open-source our environments, all
studied ACL algorithms (collected from open-source code or re-implemented), and
DRL students in a Python package available at
https://github.com/flowersteam/TeachMyAgent.
