  Action recognition is computationally expensive. In this paper, we address
the problem of frame selection to improve the accuracy of action recognition.
In particular, we show that selecting good frames helps in action recognition
performance even in the trimmed videos domain. Recent work has successfully
leveraged frame selection for long, untrimmed videos, where much of the content
is not relevant, and easy to discard. In this work, however, we focus on the
more standard short, trimmed action recognition problem. We argue that good
frame selection can not only reduce the computational cost of action
recognition but also increase the accuracy by getting rid of frames that are
hard to classify. In contrast to previous work, we propose a method that
instead of selecting frames by considering one at a time, considers them
jointly. This results in a more efficient selection, where good frames are more
effectively distributed over the video, like snapshots that tell a story. We
call the proposed frame selection SMART and we test it in combination with
different backbone architectures and on multiple benchmarks (Kinetics,
Something-something, UCF101). We show that the SMART frame selection
consistently improves the accuracy compared to other frame selection strategies
while reducing the computational cost by a factor of 4 to 10 times.
Additionally, we show that when the primary goal is recognition performance,
our selection strategy can improve over recent state-of-the-art models and
frame selection strategies on various benchmarks (UCF101, HMDB51, FCVID, and
ActivityNet).
