D ^2 TV : Dual Knowledge Distillation and Target-oriented Vision Modeling for Many-to-Many Multimodal Summarization (2023)

Abstract: Many-to-many multimodal summarization (M ^3 S) task aims to generate summaries in any language with document inputs in any language and the corresponding image sequence, which essentially comprises of multimodal monolingual summarization (MMS) and multimodal cross-lingual summarization (MXLS) tasks. Although much work has been devoted to either MMS or MXLS, little research pays attention to the M ^3 S task. Besides, existing studies mainly focus on 1) utilizing MMS to enhance MXLS via knowledge distillation without considering the performance of MMS or 2) improving MMS models by filtering summary-unrelated visual features with implicit learning or explicitly complex training objectives. In this paper, we first introduce a general and practical task, i.e. , M ^3 S. Further, we propose a dual knowledge distillation and target-oriented vision modeling framework for the M ^3 S task. Specifically, the dual knowledge distillation method guarantees that the knowledge of MMS and MXLS can be transferred to each other and thus mutually prompt both of them. To offer target-oriented visual features, a simple yet effective target-oriented contrastive objective is designed and responsible for discarding needless visual information. Extensive experiments on the many-to-many setting show the effectiveness of the proposed approach. Additionally, we contribute a many-to-many multimodal summarization (lmttM ^3 Sum) dataset with 44 languages to facilitate future research.

## introduction given a document input in the source language (e.g., english) and its corresponding image sequence, multimodal monolingual summarization (mms) aims to generate a summary in the same figure 1 : an example of our m 3 sum dataset. inputs: an article and corresponding image sequence; output: summaries in different languages. mms: the summary in the same language as the input article; mxls: the summary in a different language from the input article. the m 3 s setting covers both mms and mxls. language (i.e., english) while the goal of multimodal cross-lingual summarization (mxls) is to produce a summary in a different language (e.g., chinese). with the rapid increase of multimedia data, the mms @xcite @xcite @xcite @xcite @xcite @xcite and mxls @xcite tasks have attracted much attention in the research community because both tasks can help users quickly master the core idea from the cumbersome multimodal data. essentially, the many-tomany multimodal summarization (m 3 s) consists of mms and mxls tasks, which generate summaries in any language given the multimodal inputs (in any language), as fig. 1 shows. intuitively, the many-to-many setup should be more general and practical for its application in the multilingual and multimodal world @xcite . in the literature, although plenty of studies have been carried out on mms or mxls, there is only one study that involves both of them, i.e., @xcite devise a triple-stage training framework and distill the knowledge from mms to enhance mxls while ignoring the performance of mms. despite their effectiveness on mxls, to our knowledge, little research attention has been paid to simultaneously supporting both mms and mxls tasks and prompting both of them. besides, the visual features generally include noise which is summary-unrelated. thus the remaining work mainly focuses on improving mms models by filtering these noises with (a) implicit learning or (b) complex training objectives. for (a), researchers design various fusion methods to effectively model the interactions between textual articles and visual features @xcite @xcite . for (b), to explicitly filter needless visual information, @xcite present two well-designed auxiliary tasks, i.e., vision to summary and masked image modeling. albeit effective, implicit learning via the mms objective may limit the potential of visual features, and explicit training objectives are complex and time-consuming to be trained and applied in the real world. to address these issues, in this paper, we first introduce a more general task, i.e., m 3 s, which supports both mms and mxls tasks. further, we propose a dual knowledge distillation and target-oriented vision enhanced framework, named d foot_0 tv, for the new task. specifically, the dual knowledge distillation approach ensures that the knowledge from mms can be transferred to mxls and vice versa, and thus mutually improve both tasks. furthermore, to discard the summaryunrelated visual information, a target-oriented contrastive objective is devised to directly optimize the visual features. in this way, the model is enhanced to explicitly exploit the summary-oriented visual features, thereby yielding more accurate summaries. to validate the d 2 tv framework, we provide a many-to-many multimodal summarization (m 3 sum) benchmark dataset by reorganizing the cross-lingual summarization dataset @xcite and mm-sum dataset @xcite . the m 3 sum covers 44 languages and thus involves 44*44 language directions. to efficiently evaluate our approach, we randomly select 4 languages (i.e., english, indonesian, russian, and urdu 2 ), which consist of 4*4 language directions. we implement our approach grounding on two generative pre-trained language models, i.e., mt5 @xcite and mbart-50 @xcite . extensive experiments on both backbones show that our model significantly outperforms related methods in terms of rouge @xcite and bertscore @xcite ) scores, demonstrating its effectiveness. the human evaluation further suggests the superiority of our approach. in summary, our main contributions are: ## d 2 tv training framework based on the mms model described in § 2.2, we firstly introduce the proposed dual knowledge distillation (dkd) method in § 3.1, which improves both mms and mxls tasks. further, we present a simple yet effective target-oriented contrastive objective to filter needless visual information in § 3.2. finally, we describe the training and inference in § 3.3. ## conclusion in this paper, we first introduce a more general task, i.e., m 3 s, which can support both mms and mxls tasks. further, we propose a dual knowledge distillation and target-oriented vision (d 2 tv) enhanced framework for the new task. extensive experiments demonstrate that our model significantly outperforms related baselines in terms of rouge, bertscore scores, and human evaluation. furthermore, we contribute a many-to-many multimodal summarization (m 3 sum) dataset to the research community.