Cold Start Problem For Automated Live Video Comments (2021)

Abstract: Live video comments, or ”danmu”, are an emerging feature on Asian online video platforms. Danmu are time-synchronous comments that are overlaid on a video playback. These comments uniquely enrich the experience and engagement of their users. These comments have become a determining factor in the popularity of the videos. Similar to the ”cold start problem” in recommender systems, a video will only start to attract attention when sufficient danmu comments have been posted on it. We study this video cold start problem and examine how new comments can be generated automatically on less-commented videos. We propose to predict the danmu comments by exploiting a multi-modal combination of the video visual content, subtitles, audio signals, and any surrounding comments (when they exist). Our method fuses these multi-modalities in a transformer network which is then trained for different comment density scenarios. We evaluate our proposed system through both a retrieval based evaluation method, as well as human judgement. Results show that our proposed system improves significantly over state-of-the-art methods.

## introduction live video comments, or "danmu", is an emerging feature of video sharing platforms such as bilibili and nicovideo, which has been adopted by hundreds of millions of users in asia. danmu comments are a time-synchronous commentary subtitle system that displays user comments as streams of moving subtitles overlaid on the video playback screen (see fig. 1 ). danmu comments have become a key feature of these video platforms. so much so, that videos with many danmu comments stand a higher chance of being recommended or searched, and naturally attract more viewers. this new form of media consumption comes with a vast amount of annotated video data and figure 1 : a video frame from bilibili.com with damnu comments overlaid. the lower part of the image shows danmu comment distribution over the video. the subtitle says: "could you publish some danmu?" and the viewers are responding with a damnu burst. in this paper we propose to solve this "video cold start problem" by a method that can generate danmu comments on videos which have zero, few, or many comments. we propose a multi-density cold video transformer (mcvt) that can leverage multi-modal signals including surrounding comments, video frames, but also subtitles and audio signals in an end-to-end neural network (see section 4). the key idea is then to approach the task globally and train the network for different comment density scenarios (see section 5). to achieve this, we collect the publishing timestamps of comments from the video platform and look at the sequence of the comment publishing times (see section 3). this allows us to consider different snapshots of a video's commenting lifetime (ie. when the video was freshly uploaded with no comments, then when it had a few comments, and later with many comments). this information has not been exploited in existing work described in the literature, but we show that it can be used effectively in training of danmu generation. we evaluate our system in section 6 through both a retrieval based evaluation method and human judgement. results show that our system is able to produce comments that are close to the quality of human comments. the key contributions of this paper are as follow: to make our work fully reproducible, both the source codes and the dataset used have been made public available. 1 ## related work in this section we introduce existing work on automated danmu generation, detection of video highlights based both on manually contributed danmu and atomated analysis of video content, and automated creation of descriptive captions for videos. ## network architecture our proposed model, presented in fig 4 , applies standard transformer modules with an encoderdecoder architecture. during the encoding stage, visual, audio and text features are first encoded respectively, then three transformer modules are used to fuse the information for the three modalities recursively. in the decoder, the target comment is decoded through a transformer layer with multiple multi-head attention modules that attend to three encoded multi-modal representations respectively. ## experiments in this section we report results for our investigation of comment generation. we use the livebot model @xcite as a baseline. specifically, we use the code from @xcite , trained on our full dataset with only video frames and surrounding comments as input. the models proposed in @xcite are very recent and their code is not publicly available yet, so we do not consider these as one of our baseline methods. other older neural architectures such as lstm are also not included in this study since it is well established that transformers are the method of choice for modelling multi-modal signals. ## conclusions and further development in this paper we investigate the cold video start problem in automated danmu comment generation. we propose a multi-modal fusion network which includes processing of video frames, already published comments, and also audio and caption text. we train it for different comment density scenarios and perform extensive experiments on an expanded danmu video dataset. results demonstrate the advantage of our method over the state-of-the-art in solving the cold video start problem.