SCCS : Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment (2023)

Abstract: Multimedia summarization with multimodal output (MSMO) is a recently explored application in language grounding. It plays an essential role in real-world applications, i.e., automatically generating cover images and titles for news articles or providing introductions to online videos. However, existing methods extract features from the whole video and article and use fusion methods to select the representative one, thus usually ignoring the critical structure and varying semantics with video/document. In this work, we propose a Semantics-Consistent Cross-domain Summarization (SCCS) model based on optimal transport alignment with visual and textual segmentation. Our method first decomposes both videos and articles into segments in order to capture the structural semantics, and then follows a cross-domain alignment objective with optimal transport distance, which leverages multimodal interaction to match and select the visual and textual summary. We evaluated our method on three MSMO datasets, and achieved performance improvement by 8% & 6% of textual and 6.6% &5.7% of video summarization, respectively, which demonstrated the effectiveness of our method in producing high-quality multimodal summaries.

## introduction new multimedia content in the form of short videos and corresponding text articles has become a significant trend in influential digital media. this popular media type has been shown to be successful in drawing users' attention and delivering essential information in an efficient manner. multimedia summarization with multimodal output (msmo) has recently drawn increasing attention. different from traditional video or textual summarization @xcite , where the generated summary is either a keyframe or textual description, msmo aims at producing both visual and textual summaries simultaneously, making this figure 1 : we proposed a segment-level cross-domain alignment model to preserve the structural semantics consistency within two domains for msmo. we solve an optimal transport problem to optimize the crossdomain distance, which in turn finds the optimal match. task more complicated. previous works addressed the msmo task by processing the whole video and the whole article together which overlooked the structure and semantics of different domains @xcite @xcite @xcite @xcite . the video and article can be regarded as being composed of several topics related to the main idea, while each topic specifically corresponds to one sub-idea. thus, treating the whole video or article uniformly and learning a general representation ignores these structural semantics and easily leads to biased summarization. to address this problem, instead of learning averaged representations for the whole video & article, we focus on exploiting the original underlying structure. the comparison of our approach and previous works is illustrated in figure 1 . our model first decomposes the video & article into segments to discover the content structure, then explores the cross-domain semantics relationship at the segment level. we believe this is a promising approach to exploit the consistency lie in the structural semantics between different domains. previous models applied attention or fusion mechanisms to compute image-text relevance scores, finding the best match of the sentences/images within the whole document/video, regardless of the context, which used one domain as an anchor. however, an outstanding anchor has more weight in selecting the corresponding pair. to overcome this, we believe the semantics structure is a crucial characteristic that can not be ignored. based on this hypothesis, we propose semantics-consistent cross-domain summarization (sccs), which explores segment-level crossdomain representations through optimal transport (ot) based multimodal alignment to generate both visual and textual summaries. we decompose the video/document into segments based on its semantic structure, then generate sub-summaries of each segment as candidates. we select the final summary from these candidates instead of a global search, so all candidates are in a fair competition arena. our contributions can be summarized as follow: since msmo generates both visual & textual summaries, we believe the optimal summary comes from the video and text pair that are both 1) semantically consistent, and 2) best matched globally in a cross-domain fashion. in addition, our framework is more computationally efficient as it conducts cross-domain alignment at the segment level instead of inputting whole videos/articles. multimodal alignment aligning representations from different modalities is important in multimodal learning. exploring the explicit relationship across vision and language has drawn significant attention (wang et al., 2020a). xu et al. (2015); torabi et al. (2016); yu et al. (2017) adopted attention mechanisms, dong et al. (2021) composed pairwise joint representation, chen et al. (2020b); wray et al. (2019); zhang et al. (2018) learned fine-grained or hierarchical alignment, lee et al. (2018); wu et al. (2019) decomposed the inputs into sub-tokens, velickovic et al. (2018); yao et al. (2018) adopted graph attention for reasoning, and yang et al. (2021); gutmann and hyvärinen (2010); van den oord et al. (2018); radford et al. (2021) applied contrastive learning algorithms. multimodal summarization multimodal summarization explored multiple modalities, i.e., audio signals, video captions, transcripts, video titles, etc, for a summary generation. otani et al. (2016); yuan et al. (2019); wei et al. (2018); fu et al. (2020) learned the relevance or mapping in the latent space between different modalities. in addition to only generating visual summaries, li et al. (2017); atri et al. (2021); zhu et al. (2018) generated textual summaries by taking audio, transcripts, or documents as input along with videos or images, using seq2seq model @xcite or attention mechanism @xcite . recent trending on the msmo task has also drawn much attention @xcite fu et al., 2021, 2020; zhang et al., 2022). more related works are shown in appendix b. 3 methods sccs is a segment-level cross-domain semantics alignment model for the msmo task, where msmo aims at generating both visual and language summaries. we follow the problem setting in mingzhe et al. (2020), for a multimedia source with documents and videos, the document x @xmath0, x 2 , ..., x d } has d words, and the ground truth textual summary y @xmath1, y 2 , ..., y g } has g words. a corresponding video x v is associated with the document in pair, and there exists a ground truth cover picture y v that can represent the most important information to describe the video. our sccs model generates both textual summaries y ′ d and video keyframes y ′ v . sccs consists of five modules, as shown in figure 3 (a): video temporal segmentation (section 3.1), visual summarization (section 3.3), textual segmentation (section 3.2), textual summarization (section 3.4), and cross-domain alignment (section 3.5). each module will be introduced in the following subsections. ## conclusion in this work, we proposed sccs, a segment-level semantics-consistent cross-domain summarization model for the msmo task. our model decomposed the video & article into segments based on the content to preserve the structural semantics, and explored the cross-domain semantics relationship via optimal transport alignment at the segment level. the experimental results on three msmo datasets show that sccs outperforms previous summarization methods. we further provide interpretation by the ot coupling. our approach provides a new direction for the msmo task, which can be extended to many real-world applications.