  Humans have a remarkable ability to make decisions by accurately reasoning
about future events, including the future behaviors and states of mind of other
agents. Consider driving a car through a busy intersection: it is necessary to
reason about the physics of the vehicle, the intentions of other drivers, and
their beliefs about your own intentions. If you signal a turn, another driver
might yield to you, or if you enter the passing lane, another driver might
decelerate to give you room to merge in front. Competent drivers must plan how
they can safely react to a variety of potential future behaviors of other
agents before they make their next move. This requires contingency planning:
explicitly planning a set of conditional actions that depend on the stochastic
outcome of future events. In this work, we develop a general-purpose
contingency planner that is learned end-to-end using high-dimensional scene
observations and low-dimensional behavioral observations. We use a conditional
autoregressive flow model to create a compact contingency planning space, and
show how this model can tractably learn contingencies from behavioral
observations. We developed a closed-loop control benchmark of realistic
multi-agent scenarios in a driving simulator (CARLA), on which we compare our
method to various noncontingent methods that reason about multi-agent future
behavior, including several state-of-the-art deep learning-based planning
approaches. We illustrate that these noncontingent planning methods
fundamentally fail on this benchmark, and find that our deep contingency
planning method achieves significantly superior performance. Code to run our
benchmark and reproduce our results is available at
https://sites.google.com/view/contingency-planning
