  Although reinforcement learning has been successfully applied in many domains
in recent years, we still lack agents that can systematically generalize. While
relational inductive biases that fit a task can improve generalization of RL
agents, these biases are commonly hard-coded directly in the agent's neural
architecture. In this work, we show that we can incorporate relational
inductive biases, encoded in the form of relational graphs, into agents. Based
on this insight, we propose Grid-to-Graph (GTG), a mapping from grid structures
to relational graphs that carry useful spatial relational inductive biases when
processed through a Relational Graph Convolution Network (R-GCN). We show that,
with GTG, R-GCNs generalize better both in terms of in-distribution and
out-of-distribution compared to baselines based on Convolutional Neural
Networks and Neural Logic Machines on challenging procedurally generated
environments and MinAtar. Furthermore, we show that GTG produces agents that
can jointly reason over observations and environment dynamics encoded in
knowledge bases.
