Season the steak with salt and pepper. Carefully place the steak to the pan. Flip the steak to the other side. Now let it rest and enjoy the delicious steak. Figure 1 : VideoBERT text-to-video generation and future forecasting. (Above) Given some recipe text divided into sentences, y = y 1:T , we generate a sequence of video tokens x = x 1:T by computing x * t = arg max k p(x t = k|y) using VideoBERT. (Below) Given a video token, we show the top three future tokens forecasted by VideoBERT at different time scales. In this case, VideoBERT predicts that a bowl of flour and cocoa powder may be baked in an oven, and may become a brownie or cupcake. We visualize video tokens using the images from the training set closest to centroids in feature space. Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube. Whereas most existing approaches learn low-level representations, we propose a joint visual-linguistic model to learn high-level features without any explicit supervision. In particular, inspired by its recent success in language modeling, we build upon the BERT model to learn bidirectional joint distributions over sequences of visual and linguistic tokens, derived from vector quantization of video data and off-the-shelf speech recognition outputs, respectively. We use VideoBERT in numerous tasks, including action classification and video captioning. We show that it can be applied directly to openvocabulary classification, and confirm that large amounts of training data and cross-modal information are critical to performance. Furthermore, we outperform the state-of-theart on video captioning, and quantitative results verify that the model learns high-level semantic features.
For example, VideoBERT REF applied BERT to learn a bidrectional joint distribution over quantized video frame features and linguistic tokens from video-text pairs.
VideoBERT: A Joint Model for Video and Language Representation Learning
2019 IEEE/CVF International Conference on Computer Vision (ICCV)
2019 IEEE/CVF International Conference on Computer Vision (ICCV)
Computer Science