  For action recognition learning, 2D CNN-based methods are efficient but may
yield redundant features due to applying the same 2D convolution kernel to each
frame. Recent efforts attempt to capture motion information by establishing
inter-frame connections while still suffering the limited temporal receptive
field or high latency. Moreover, the feature enhancement is often only
performed by channel or space dimension in action recognition. To address these
issues, we first devise a Channel-wise Motion Enhancement (CME) module to
adaptively emphasize the channels related to dynamic information with a
channel-wise gate vector. The channel gates generated by CME incorporate the
information from all the other frames in the video. We further propose a
Spatial-wise Motion Enhancement (SME) module to focus on the regions with the
critical target in motion, according to the point-to-point similarity between
adjacent feature maps. The intuition is that the change of background is
typically slower than the motion area. Both CME and SME have clear physical
meaning in capturing action clues. By integrating the two modules into the
off-the-shelf 2D network, we finally obtain a Comprehensive Motion
Representation (CMR) learning method for action recognition, which achieves
competitive performance on Something-Something V1 & V2 and Kinetics-400. On the
temporal reasoning datasets Something-Something V1 and V2, our method
outperforms the current state-of-the-art by 2.3% and 1.9% when using 16 frames
as input, respectively.
