  We present a self-improving, neural tree expansion method for multi-robot
online planning in non-cooperative environments, where each robot tries to
maximize its cumulative reward while interacting with other self-interested
robots. Our algorithm adapts the centralized, perfect information,
discrete-action space method from Alpha Zero to a decentralized, partial
information, continuous action space setting for multi-robot applications. Our
method has three interacting components: (i) a centralized, perfect-information
`expert' Monte Carlo Tree Search (MCTS) with large computation resources that
provides expert demonstrations, (ii) a decentralized, partial-information
`learner' MCTS with small computation resources that runs in real-time and
provides self-play examples, and (iii) policy & value neural networks that are
trained with the expert demonstrations and bias both the expert and the learner
tree growth. Our numerical experiments demonstrate neural expansion generates
compact search trees with better solution quality and 20 times less
computational expense compared to MCTS without neural expansion. The resulting
policies are dynamically sophisticated, demonstrate coordination between
robots, and play the Reach-Target-Avoid differential game significantly better
than the state-of-the-art control-theoretic baseline for multi-robot,
double-integrator systems. Our hardware experiments on an aerial swarm
demonstrate the computational advantage of neural tree expansion, enabling
online planning at 20Hz with effective policies in complex scenarios.
