  Action prediction focuses on anticipating actions before they happen. Recent
works leverage probabilistic approaches to describe future uncertainties and
sample future actions. However, these methods cannot easily find all
alternative predictions, which are essential given the inherent
unpredictability of the future, and current evaluation protocols do not measure
a system's ability to find such alternatives. We re-examine action prediction
in terms of its ability to predict not only the top predictions, but also top
alternatives with the accuracy@k metric. In addition, we propose Choice F1: a
metric inspired by F1 score which evaluates a prediction system's ability to
find all plausible futures while keeping only the most probable ones. To
evaluate this problem, we present a novel method, Predicting the Likely
Alternative Next Best, or PLAN-B, for action prediction which automatically
finds the set of most likely alternative futures. PLAN-B consists of two novel
components: (i) a Choice Table which ensures that all possible futures are
found, and (ii) a "Collaborative" RNN system which combines both action
sequence and feature information. We demonstrate that our system outperforms
state-of-the-art results on benchmark datasets.
