  High capacity end-to-end approaches for human motion prediction have the
ability to represent subtle nuances in human behavior, but struggle with
robustness to out of distribution inputs and tail events. Planning-based
prediction, on the other hand, can reliably output decent-but-not-great
predictions: it is much more stable in the face of distribution shift, but it
has high inductive bias, missing important aspects that drive human decisions,
and ignoring cognitive biases that make human behavior suboptimal. In this
work, we analyze one family of approaches that strive to get the best of both
worlds: use the end-to-end predictor on common cases, but do not rely on it for
tail events / out-of-distribution inputs -- switch to the planning-based
predictor there. We contribute an analysis of different approaches for
detecting when to make this switch, using an autonomous driving domain. We find
that promising approaches based on ensembling or generative modeling of the
training distribution might not be reliable, but that there very simple methods
which can perform surprisingly well -- including training a classifier to pick
up on tell-tale issues in predicted trajectories.
