Grafting Pre-trained Models for Multimodal Headline Generation (2022)

Abstract: Multimodal headline utilizes both video frames and transcripts to generate the natural language title of the videos. Due to a lack of large-scale, manually annotated data, the task of annotating grounded headlines for video is labor intensive and impractical. Previous researches on pre-trained language models and video-language models have achieved significant progress in related downstream tasks. However, none of them can be directly applied to multimodal headline architecture where we need both multimodal encoder and sentence decoder. A major challenge in simply gluing language model and video-language model is the modality balance, which is aimed at combining visual-language complementary abilities. In this paper, we propose a novel approach to graft the video encoder from the pre-trained video-language model on the generative pre-trained language model. We also present a consensus fusion mechanism for the integration of different components, via inter/intra modality relation. Empirically, experiments show that the grafted model achieves strong results on a brand-new dataset collected from real-world applications.

## introduction in the age of information explosion, generating headlines of videos has been steadily gaining prominence on the short video platform. as the headlines can summarize the videos for people quickly acquiring their essential information. good headlines are also beneficial for various scenarios, such as video retrieval, recommendation, and understanding @xcite . specially, video headline generation can be regarded as a textual generation task with multimodal inputs @xcite , which is called multimodal generation as shown in figure 1 . given a video with related transcript, algorithm aims to generate a short, concise and readable textual attraction title. however, to build an effective model, collecting large-scale training data is the main challenge. especially, in the multimodal headline generation task, the form of triplet data (video, source transcript, target summary) further increases the difficulty of collecting data, which limits the application of video headline generation. to alleviate the issue in multimodal headline generation, the natural idea is to leverage pre-trained model (ptm). with large-scale corpus, such as gpt @xcite , bart @xcite , palm @xcite , etc., have shown great ability to generate readable and informative text. since the multimodal headline generation combine both video and textual information, we propose that the model can be grafted by ptms of language generation and video-text matching. the former provides the ability of headline generation, and the latter bridges the semantic gap between the multiple modalities @xcite . in addition, these two tasks have no concern of scarce data. for language generation, the existing pre-training model can be directly adopted. for video-text matching, the model can be trained with sufficient data from the internet without manual annotations. by this means, multimodal headline generation model can be constructed by fine-tuning the grafted model with limited data collection. in order to take advantage of the existing ptms and improve reusability, we propose a grafting mechanism for obtaining the multimodal summarization pre-training model (grammo). first, language generation and video-text matching tasks are introduced to pre-train the encoders and decoder, respectively. then we construct a unified architecture with a video encoder, a text encoder and a text decoder which grafted from different ptms to initialize the multimodal headline generation model. in addition, a joint-modality layer acted as a modalitybalance gate is designed to fuse the video and text features. unlike previous works which focus on retaining modality-shared feature @xcite , this layer uses a two-way attention strategy to capture the commonality and specialty of the modalities. in detail, the video modality can highlight the most relevant and important text tokens by video-text cross attention. the resulted feature is called video-enhanced text feature, which reflect the commonality. on the other hand, since video-enhanced text feature neglects the video specialty which is less related with text modality, we recombine the video embeddings according to video-text attention and exploit the video-specific feature for complementing the fusion representation. furthermore, dynamic frame sampling (dfs) and masked word prediction are designed in the encoder parts to reinforce the multimodal representation. we summarize the main contributions as follows: ## conclusion in this paper, we propose grammo, grafting a pretrained sequence-to-sequence language model and a video-language understanding model for multimodal headline generation. by fine-tuning the representation components (video-encoder&textencoder) and generation component (text-decoder) of the model, we alleviate the problem of lacking large-scale dataset in multimodal headline generation. to capture the commonality and specialty of the video and text features, we propose an extra fusion layer to balance modalities and maximally maintain the original architectures. with this approach, we can fully take advantage of the pre-trained models, including well-trained capacity for multimodal understanding and generation. experiments results show that our method can significantly improve the performance and outperform similar works. furthermore, the proposed method has also been applied effectively and efficiently in our online system. we will release the wb-news dataset, grammo code, and the grafted models.