Semi-Supervised Learning for Video Captioning (2020)

Abstract: Deep neural networks have made great success on video captioning in supervised learning setting. However, annotating videos with descriptions is very expensive and time-consuming. If the video captioning algorithm can benefit from a large number of unlabeled videos, the cost of annotation can be reduced. In the proposed study, we make the first attempt to train the video captioning model on labeled data and unlabeled data jointly, in a semi-supervised learning manner. For labeled data, we train them with the traditional cross-entropy loss. For unlabeled data, we leverage a self-critical policy gradient method with the difference between the scores obtained by Monte-Carlo sampling and greedy decoding as the reward function, while the scores are the negative K-L divergence between output distributions of original video data and augmented video data. The final loss is the weighted sum of losses obtained by labeled data and unlabeled data. Experiments conducted on VATEX, MSR-VTT and MSVD dataset demonstrate that the introduction of unlabeled data can improve the performance of the video captioning model. The proposed semi-supervised learning algorithm also outperforms several state-of-the-art semi-supervised learning approaches.

## introduction video captioning refers to the task that generating a description of a given video automatically and it combines computer vision and natural language processing (nlp) in a unified framework. it can be widely used in video retrieval, video recommendation, disabled supporting and scene understanding @xcite , @xcite . with the rapid development of deep learning, deep neural networks have dominated the video captioning task. venugopalan et al. @xcite extend encoder-decoder framework to video captioning which employs a cnn as the encoder and an rnn as the decoder and the following video captioning algorithms almost use this architecture. although recent video captioning algorithms have made great success, they are heavily dependent on supervised training data consisting of videocaption pairs. it is expensive to take long hours of laboring to collect such labeled data, thus there is a strong interest to develop the algorithm which does not need a lot of annotated examples. some studies embed the visual feature and text imformation into a mutual space and design unsupervised learning algorithm to reduce the requirement for annotated data @xcite , @xcite , @xcite , @xcite . however, the performances of such algorithms are poor because they do not use pairs of labeled examples at all. semisupervised learning, leveraging a small number of labeled examples and a large number of unlabeled examples at the same time, provides another solution to solve the problem of strong dependency on labeled examples. chen et al. @xcite proposed a semi-supervised learning (ssl) image captioning strategy which using unsupervised out-of-domain textual data to boost the captioning performance. @xcite proposed another semi-supervised image captioning algorithm which jointly using the labeled and unlabeled data and assigning pseudo-labels to unlabeled data via generative adversarial networks to learn the joint distribution of image and text. recently, some semi-supervised learning works which use the consistency of the output probability distribution between original data and augmented data have achieved excellent performances with the help of some latest data augmentation methods on several classification problems of computer vision and nlp @xcite , @xcite . although these data augmentation based methods have great potential, it is still challenging when applied to video captioning task, because the input figure 1: the architecture of the video captioning model, which includes an eco encoder and a gru decoder (or a transformer decoder). and output complexity of video captioning is much higher than that of image or text classification. in this paper, we design our algorithm based on output consistency of data augmentation, and apply a self-critical training strategy @xcite to exploit large amounts of unsupervised video data without requiring the corresponding caption annotations. first, we get a pseudo-label by monte-carlo sampling. then a reward score is obtained, while the score is the k-l divergence between the output distribution of augmented examples and real examples. based on the observation of figure 3 , k-l divergence is positive related with the quality of the sentence, thus k-l divergence can be used as the reward score in policy gradient. we use greedy decoding to get another pseudo-label and get another reward score as the baseline. then we use the difference between these two reward scores as the final reward. finally, we combine the reward with the log probability to get the policy gradient loss. in the meantime, we compute the cross-entropy loss for labeled data and train the model with these two losses jointly. in summary, the main contributions of the proposed algorithm are three-fold: ## conclusion in this paper, we make the first attempt to train the video captioning model in a semi-supervised learning manner. we train labeled data with the traditional cross-entropy loss. for unlabeled data, we leverage a self-critical policy gradient method to train the data. the reward function is the difference between the scores obtained by monte-carlo sampling and greedy decoding and the scores are the k-l divergences between output distribution of original video data and augmented video data. the final loss is the weighted sum of two losses mentioned above. experiments conducted on vatex, msr-vtt and msvd dataset demonstrate that the introduction of unlabeled data can improve the performance of the video captioning model significantly. the proposed method is robust for different tasks (english captioning task and chinese captioning task), different datasets (vatex, msr-vtt and msvd) and different models (gru and transformer). the proposed semi-supervised learning algorithm also outperforms several state-of-the-art semi-supervised learning approaches.