To alleviate the expensive cost of data collection and annotation, many self-supervised learning methods were proposed to learn image representations without humanlabeled annotations. However, self-supervised learning for video representations is not yet well-addressed. In this paper, we propose a novel 3DConvNet-based fully selfsupervised framework to learn spatiotemporal video features without using any human-labeled annotations. First, a set of pre-designed geometric transformations (e.g. rotating 0 • , 90 • , 180 • , and 270 • ) are applied to each video. Then a pretext task can be defined as "recognizing the predesigned geometric transformations." Therefore, the spatiotemporal video features can be learned in the process of accomplishing this pretext task without using humanlabeled annotations. The learned spatiotemporal video representations can further be employed as pre-trained features for different video-related applications. The proposed geometric transformations (e.g. rotations) are proved to be effective to learn representative spatiotemporal features in our 3DConvNet-based fully self-supervised framework. With the pre-trained spatiotemporal features from two large video datasets, the performance of action recognition is significantly boosted up by 20.4% on UCF101 dataset and 16.7% on HMDB51 dataset respectively compared to that from the model trained from scratch. Furthermore, our framework outperforms the state-of-the-arts of fully self-supervised methods on both UCF101 and HMDB51 datasets and achieves 62.9% and 33.7% accuracy respectively.
Recently, REF propose a self-supervised learning model 3DRotNet for 3D sequences as input adopting a rotation prediction task to learn the rich spatiotemporal features.
Self-supervised Spatiotemporal Feature Learning by Video Geometric Transformations.
ArXiv
ArXiv
Computer Science