Assist Non-native Viewers: Multimodal Cross-Lingual Summarization for How2 Videos (2022)

Abstract: Multimodal summarization for videos aims to generate summaries from multi-source information (videos, audio transcripts), which has achieved promising progress. However, existing works are restricted to monolingual video scenarios, ignoring the demands of non-native video viewers to understand the cross-language videos in practical applications. It stimulates us to propose a new task, named Multimodal Cross-Lingual Summarization for videos (MCLS), which aims to generate cross-lingual summaries from multimodal inputs of videos. First, to make it applicable to MCLS scenarios, we conduct a Video-guided Dual Fusion network (VDF) that integrates multimodal and cross-lingual information via diverse fusion strategies at both encoder and decoder. Moreover, to alleviate the problem of high annotation costs and limited resources in MCLS, we propose a triple-stage training framework to assist MCLS by transferring the knowledge from monolingual multimodal summarization data, which includes: 1) multimodal summarization on sufficient prevalent language videos with a VDF model; 2) knowledge distillation (KD) guided adjustment on bilingual transcripts; 3) multimodal summarization for cross-lingual videos with a KD induced VDF model. Experiment results on the reorganized How2 dataset show that the VDF model alone outperforms previous methods for multimodal summarization, and the performance further improves by a large margin via the proposed triple-stage training framework.

## introduction multimodal summarization (ms) for videos aims at integrating multimodal information such as videos and audio transcriptions to generate text summaries. with the rapid growth of videos on the internet, this task has attracted much interest from the communities and has shown its potential in recent years, figure 1 : an example of the mcls task, which seeks to generate a target language (e.g., english) summary based on the video and its transcript in the source language (e.g., portuguese). to assist non-native viewers, we propose a new task: multimodal cross-lingual summarization for videos (mcls). as illustrated in the example from fig. 1 , mcls seeks to generate summaries in the target language to reflect the salient video contents based on the videos and their audio transcriptions in the source language. there are two challenging issues for mcls: (1) messy multimodal and cross-lingual information: compared with recent summarization methods in multimodal @xcite @xcite or textual cross-lingual @xcite ) scenario, it is more challenging to effectively integrate multimodal and different language information simultaneously, and then generate summaries. (2) insufficient multimodal cross-lingual summarization data: due to the relatively small number of bilingualist, building a high-quality multimodal cross-lingual summarization dataset is costly. the experiment results of directly training on such limited multimodal data are typically under satisfactory. there are several textual low-resource cross-lingual summarization researches that focus on leveraging external toolkits @xcite or pre-trained language models (plms) @xcite . however, since mcls is a newly proposed task, there are no available tools or plms that can be directly adopted. in this paper, to overcome the first issue in mcls, we propose the video-guided dual fusion network (vdf) by designing dual diverse fusion strategies at both encoder and decoder structures to integrate multimodal and cross-lingual information. furthermore, to solve the second one, we propose a triple-stage training framework, which leverages the knowledge of the multimodal summarization model trained with prevalent mono language to assist the low-resource mcls generation. in the first stage, a vdf model is trained on sufficient multimodal summarization data in the target language. in the second stage, encoder-and vocab-level knowledge distillation methods are proposed to adjust the output features between the target language encoder (vocab) in vdf model and a new source language encoder (vocab). in the third stage, the target encoder (vocab) is replaced with the knowledge distillation induced source encoder (vocab), composing a new vdf model. the new vdf is fine-tuned on the limited mcls data and serves as the final model to generate the target language summaries. to simulate the mcls, we reorganize the how2 dataset @xcite , a large-scale multimodal understanding dataset of open-domain videos, for two scenarios according to the crosslingual video-summary data volume: 1) general mcls: contains portuguese videos with english summaries; 2) ms-augmented mcls: on the basis of 1), it is supplemented with sufficient english video-summary data and a small amount of bilingual english-portuguese transcripts. experiment results show that the vdf model alone outperforms state-of-the-art methods in the general mcls situation, and the performance further improves by a large margin via the proposed triple-stage training framework. what's more, with only 3k samples, our triple-stage training method outperforms strong baselines in general mcls, which is trained with more than 10k samples. the contribution of this work could be summarized as follows: ## related work to allow non-native speakers holistically understand the videos in other languages, we propose the multimodal cross-lingual summarization for videos (mcls) task. as mcls is a newly proposed task, there is no existing work. the most relevant research area includes: multimodal summarization (ms) @xcite @xcite @xcite and textual cross-lingual summarization @xcite @xcite . between the two, ms seeks to compresses multimedia documents. @xcite first released the how2 dataset for multimodal abstractive summarization for opendomain videos, which provides multisource information, including videos, audios, text transcriptions and human-generated summaries. with the rise of sequence-to-sequence learning @xcite @xcite , khullar and arora (2020) utilized hierarchical attention to fuse text, audio and video modalities. @xcite proposed a multistage forget gate to resist the flow of multimodal noise. meanwhile, textual cross-lingual summarization is the task of generating a target-language summary from the given documents in the source language, and some textual cross-lingual methods are reported to perform well @xcite . however, existing summarization approaches could not consider multilingualism and multimodality simultaneously. moreover, multimodal cross-lingual summarization task is likely to meet the data-insufficient problem. despite some textual cross-lingual summarization approaches utilising external toolkits @xcite or pre-trained language models (plms) @xcite to overcome such a lowresource dilemma, they can not be directly applied to the newly proposed mcls task. in this work, we propose the vdf model and the triple-stage training framework to address the above problems. in this work, we propose a new task: multimodal cross-lingual summarization for videos (mcls), which assists non-native viewers with native language summaries generated from non-native videos. concretely, we propose a video-guided dual fusion network (vdf) to integrate multimodal and cross-language information for summary generation. in addition, we introduce a triple-stage training framework to enhance models in mcls with monolingual multimodal summarization data. experiment results illustrate the effectiveness of the vdf model and the triple-stage training framework. specially, in the low-resource mcls scenario, the proposed methods achieve comparable or better performance than those baseline models trained with the full amount of data.