In this report, we present a new reinforcement learning (RL) benchmark based on the Sonic the Hedgehog TM video game franchise. This benchmark is intended to measure the performance of transfer learning and few-shot learning algorithms in the RL domain. We also present and evaluate some baseline algorithms on the new benchmark. In the past few years, it has become clear that deep reinforcement learning can solve difficult, high-dimensional problems when given a good reward function and unlimited time to interact with the environment. However, while this kind of learning is a key aspect of intelligence, it is not the only one. Ideally, intelligent agents would also be able to generalize between tasks, using prior experience to pick up new skills more quickly. In this report, we introduce a new benchmark that we designed to make it easier for researchers to develop and test RL algorithms with this kind of capability. Most popular RL benchmarks such as the ALE [1] are not ideal for testing generalization between similar tasks. As a result, RL research tends to "train on the test set", boasting an algorithm's final performance on the same environment(s) it was trained on. For the field to advance towards algorithms with better generalization properties, we need RL benchmarks with proper splits between "train" and "test" environments, similar to supervised learning datasets. Our benchmark has such a split, making it ideal for measuring cross-task generalization. One interesting application of cross-task generalization is few-shot learning. Recently, supervised few-shot learning algorithms have improved by leaps and bounds [2]-[4]. This progress has hinged on the availability of good meta-learning datasets such as Omniglot [5] and . Thus, if we want better few-shot RL algorithms, it makes sense to construct a similar kind of dataset for RL. Our benchmark is designed to be a meta-learning dataset, consisting of many similar tasks sampled from a single task distribution. Thus, it is a suitable test bed for few-shot RL algorithms. Beyond few-shot learning, there are many other applications of cross-task generalization that require the right kind of benchmark. For example, you might want an RL algorithm to learn how to explore in new environments. Our benchmark poses a fairly challenging exploration problem, and the train/test split presents a unique opportunity to learn how to explore on some levels and transfer this ability to other levels. 1
The Sonic benchmark REF was designed to measure generalization in RL by separating levels of the Sonic the Hedgehog TM video game into training and test sets.
Gotta Learn Fast: A New Benchmark for Generalization in RL
ArXiv
ArXiv
Computer Science Mathematics