  In this paper, we propose a conditional early exiting framework for efficient
video recognition. While existing works focus on selecting a subset of salient
frames to reduce the computation costs, we propose to use a simple sampling
strategy combined with conditional early exiting to enable efficient
recognition. Our model automatically learns to process fewer frames for simpler
videos and more frames for complex ones. To achieve this, we employ a cascade
of gating modules to automatically determine the earliest point in processing
where an inference is sufficiently reliable. We generate on-the-fly supervision
signals to the gates to provide a dynamic trade-off between accuracy and
computational cost. Our proposed model outperforms competing methods on three
large-scale video benchmarks. In particular, on ActivityNet1.3 and
mini-kinetics, we outperform the state-of-the-art efficient video recognition
methods with 1.3$\times$ and 2.1$\times$ less GFLOPs, respectively.
Additionally, our method sets a new state of the art for efficient video
understanding on the HVU benchmark.
