  In the world of action recognition research, one primary focus has been on
how to construct and train networks to model the spatial-temporal volume of an
input video. These methods typically uniformly sample a segment of an input
clip (along the temporal dimension). However, not all parts of a video are
equally important to determine the action in the clip. In this work, we focus
instead on learning where to extract features, so as to focus on the most
informative parts of the video. We propose a method called the non-uniform
temporal aggregation (NUTA), which aggregates features only from informative
temporal segments. We also introduce a synchronization method that allows our
NUTA features to be temporally aligned with traditional uniformly sampled video
features, so that both local and clip-level features can be combined. Our model
has achieved state-of-the-art performance on four widely used large-scale
action-recognition datasets (Kinetics400, Kinetics700, Something-something V2
and Charades). In addition, we have created a visualization to illustrate how
the proposed NUTA method selects only the most relevant parts of a video clip.
