  Agents that interact with other agents often do not know a priori what the
other agents' strategies are, but have to maximise their own online return
while interacting with and learning about others. The optimal adaptive
behaviour under uncertainty over the other agents' strategies w.r.t. some prior
can in principle be computed using the Interactive Bayesian Reinforcement
Learning framework. Unfortunately, doing so is intractable in most settings,
and existing approximation methods are restricted to small tasks. To overcome
this, we propose to meta-learn approximate belief inference and Bayes-optimal
behaviour for a given prior. To model beliefs over other agents, we combine
sequential and hierarchical Variational Auto-Encoders, and meta-train this
inference model alongside the policy. We show empirically that our approach
outperforms existing methods that use a model-free approach, sample from the
approximate posterior, maintain memory-free models of others, or do not fully
utilise the known structure of the environment.
