  In this paper, we propose a neural motion planner (NMP) for learning to drive
autonomously in complex urban scenarios that include traffic-light handling,
yielding, and interactions with multiple road-users. Towards this goal, we
design a holistic model that takes as input raw LIDAR data and a HD map and
produces interpretable intermediate representations in the form of 3D
detections and their future trajectories, as well as a cost volume defining the
goodness of each position that the self-driving car can take within the
planning horizon. We then sample a set of diverse physically possible
trajectories and choose the one with the minimum learned cost. Importantly, our
cost volume is able to naturally capture multi-modality. We demonstrate the
effectiveness of our approach in real-world driving data captured in several
cities in North America. Our experiments show that the learned cost volume can
generate safer planning than all the baselines.
