  Current deep reinforcement learning (RL) algorithms are still highly
task-specific and lack the ability to generalize to new environments. Lifelong
learning (LLL), however, aims at solving multiple tasks sequentially by
efficiently transferring and using knowledge between tasks. Despite a surge of
interest in lifelong RL in recent years, the lack of a realistic testbed makes
robust evaluation of LLL algorithms difficult. Multi-agent RL (MARL), on the
other hand, can be seen as a natural scenario for lifelong RL due to its
inherent non-stationarity, since the agents' policies change over time. In this
work, we introduce a multi-agent lifelong learning testbed that supports both
zero-shot and few-shot settings. Our setup is based on Hanabi -- a
partially-observable, fully cooperative multi-agent game that has been shown to
be challenging for zero-shot coordination. Its large strategy space makes it a
desirable environment for lifelong RL tasks. We evaluate several recent MARL
methods, and benchmark state-of-the-art LLL algorithms in limited memory and
computation regimes to shed light on their strengths and weaknesses. This
continual learning paradigm also provides us with a pragmatic way of going
beyond centralized training which is the most commonly used training protocol
in MARL. We empirically show that the agents trained in our setup are able to
coordinate well with unseen agents, without any additional assumptions made by
previous works.
