  Video action recognition (VAR) is a primary task of video understanding, and
untrimmed videos are more common in real-life scenes. Untrimmed videos have
redundant and diverse clips containing contextual information, so sampling
dense clips is essential. Recently, some works attempt to train a generic model
to select the N most representative clips. However, it is difficult to model
the complex relations from intra-class clips and inter-class videos within a
single model and fixed selected number, and the entanglement of multiple
relations is also hard to explain. Thus, instead of "only look once", we argue
"divide and conquer" strategy will be more suitable in untrimmed VAR. Inspired
by the speed reading mechanism, we propose a simple yet effective clip-level
solution based on skim-scan techniques. Specifically, the proposed Skim-Scan
framework first skims the entire video and drops those uninformative and
misleading clips. For the remaining clips, it scans clips with diverse features
gradually to drop redundant clips but cover essential content. The above
strategies can adaptively select the necessary clips according to the
difficulty of the different videos. To trade off the computational complexity
and performance, we observe the similar statistical expression between
lightweight and heavy networks, thus it supports us to explore the combination
of them. Comprehensive experiments are performed on ActivityNet and mini-FCVID
datasets, and results demonstrate that our solution surpasses the
state-of-the-art performance in terms of both accuracy and efficiency.
