  Video action recognition is one of the representative tasks for video
understanding. Over the last decade, we have witnessed great advancements in
video action recognition thanks to the emergence of deep learning. But we also
encountered new challenges, including modeling long-range temporal information
in videos, high computation costs, and incomparable results due to datasets and
evaluation protocol variances. In this paper, we provide a comprehensive survey
of over 200 existing papers on deep learning for video action recognition. We
first introduce the 17 video action recognition datasets that influenced the
design of models. Then we present video action recognition models in
chronological order: starting with early attempts at adapting deep learning,
then to the two-stream networks, followed by the adoption of 3D convolutional
kernels, and finally to the recent compute-efficient models. In addition, we
benchmark popular methods on several representative datasets and release code
for reproducibility. In the end, we discuss open problems and shed light on
opportunities for video action recognition to facilitate new research ideas.
