  A key challenge in self-supervised video representation learning is how to
effectively capture motion information besides context bias. While most
existing works implicitly achieve this with video-specific pretext tasks (e.g.,
predicting clip orders, time arrows, and paces), we develop a method that
explicitly decouples motion supervision from context bias through a carefully
designed pretext task. Specifically, we take the keyframes and motion vectors
in compressed videos (e.g., in H.264 format) as the supervision sources for
context and motion, respectively, which can be efficiently extracted at over
500 fps on the CPU. Then we design two pretext tasks that are jointly
optimized: a context matching task where a pairwise contrastive loss is cast
between video clip and keyframe features; and a motion prediction task where
clip features, passed through an encoder-decoder network, are used to estimate
motion features in a near future. These two tasks use a shared video backbone
and separate MLP heads. Experiments show that our approach improves the quality
of the learned video representation over previous works, where we obtain
absolute gains of 16.0% and 11.1% in video retrieval recall on UCF101 and
HMDB51, respectively. Moreover, we find the motion prediction to be a strong
regularization for video networks, where using it as an auxiliary task improves
the accuracy of action recognition with a margin of 7.4%~13.8%.
