Topic-aware Multimodal Summarization (2022)

Abstract: Multimodal Summarization (MS) has attracted research interest in the past few years due to the ease with which users perceive multimodal summaries. It is important for MS models to consider the topic a given target content belongs to. In the current paper, we propose a topic-aware MS system which performs two tasks simultaneously: differentiating the images into “on-topic” and “off-topic” categories and further utilizing the “on-topic” images to generate multimodal summaries. The hypothesis is that, the proposed topic similarity classifier will help in generating better multimodal summary by focusing on important components of images and text which are specific to a particular topic. To develop the topic similarity classifier, we have augmented the existing popular MS data set, MSMO, with similar “on-topic” and dissimilar “off-topic” images for each sample. Our experimental results establish that the focus on “on-topic” features helps in generating topic-aware multimodal summaries, which outperforms the state of the art approach by 1.7 % in ROUGE-L metric.

## introduction due to the continuous growth of multimedia content, users often look for ways to read and go through only the crucial information content, and to avoid redundancy as much as possible. to cater to this need for concise information availability, automatic summarization systems are the need of the hour. extensive research works have produced summaries of a single modality like text @xcite or video @xcite . however, researchers have also demonstrated that users are more satisfied with multimodal summaries than uni-modal summaries @xcite . thus, generating output summaries of different modalities like text and images makes sense. images play essential roles in help-ing users understand the text and make the summary more attractive, contextualized, and complete. topic information is crucial for correctly identifying pictures and text as a part of a multimodal summary. however, existing work @xcite in the field of multimodal summarization has not yet utilized the sample's topic information to improve the multimodal summary quality. (1) figure 1: in this example we select two images to be part of the pictorial summary. using topic information, i.e., "sport", the model can decide that the images (1) and (2) are highly related to "sport" topic as compared to others, and hence increase their probability of being used in pictorial part of the final summary. in this paper, we introduce topic-aware multimodal summarization (tms) where multimodal summaries consisting of texts and images are generated by also focusing on topic-centric information. incorporating topic information of the source content aids the summarization process because the generated multimodal summary also considers the key elements of that topic. for example the summary of an article in the "sport" topic should highlight how a player scored a goal in a football match as a part of the text summary, and the images of the player as a part of the image summary (as shown in fig. 1 ). in contrast, the multimodal summary of any article belonging to the "travel" topic should highlight details of the place mentioned, and the image summary should showcase the images of that place. thus different topics may require different kinds of focus. in our experiments we have investigated the following research objectives: i) the significance of the topic similarity classifier with respect to the combination of different modalities, ii) the impact of using similar "on-topic" image feature vectors instead of zero-padded vectors for samples having limited number of in-article images and iii) the comparison with respect to the existing state-ofthe-art technique. the key contributions of our work are as follows: the first task is our auxiliary task to extract more useful features from image and text modalities which in turn can help in generating better multimodal summaries. ## related works multimodal summarization has gained in popularity in the recent years due to the enhanced quality and user experience it is able to offer. @xcite provided an overview of the recent developments in the field of multimodal summarization. the summarization process can produce a single modality output @xcite @xcite or a multi-modal output @xcite @xcite . in our work we focus on the latter case, and we have considered the msmo model @xcite as the baseline which produces multi-modal output summary in the form of text and images. one of the recent works inspired by the msmo model is @xcite ; however, unlike our model, it does not focus on producing topic-aware summaries. zhu et al. (2020) also used an extended version of the msmo dataset for training its image selection module in a supervised fashion. in contrast, we have used an unsupervised approach similar to msmo for training our model's image selection module. training our model using the extended dataset used by @xcite might produce better results in the future. recently, transformer-based models like mtms @xcite and ctnr @xcite were developed based on the msmo dataset. however, these models either produce text-only summaries using multimodal input or use different input parameter sizes (max. encoder length, decoder length, max. number of images, etc.) for training the model. thus because of these factors, we have not considered these transformer-based models as a baseline for comparison. furthermore, mtms also uses 80% of the test data for fine-tuning its model with the image-saliency-based loss. our model does not require using any segment of the test data for training purposes. multi-task learning (mtl) involves sharing representations between related tasks which helps in achieving better performance in the target task. earlier mtl has been used for producing both textual @xcite and multimodal summaries @xcite . taking inspirations from these works we have used mtl for making our summarization model topic-aware. producing multimodal summaries, which relate well with the topic they belong to, helps users get a better understanding of the actual content. earlier, research has been done in developing systems that produce multimodal summaries related to a specific topics like sports @xcite , movies @xcite or e-commerce @xcite but those works have used datasets which are specific multimodal attention layer decoder multimodal summary c txt c mm c img β txt cov img textual attention layer topic similarity classifier visual attention layer similar or dissimilar (t) shared topic-aware encoder on-topic similar image or off-topic dissimilar image (s) (i) w 1 w 2 w 3 w 4 w m i 1 i 2 i n β img figure 2: proposed architecture. the label (t) is the text encoder's last time-step output, (i) is the projected in-article image feature vector and (s) is the corresponding similar/dissimilar image feature vector, to be passed as input into the topic similarity classifier. to a single particular topic. we are the first to introduce a model that not only can produce topic-aware multimodal summaries, but is also trained using a topic-generic dataset. ## qualitative analysis as shown in fig. 4 foot_3 , the textual summary produced by the baseline atg model could not cover the content related to sending the letter well. in contrast, our best-performing model tsc-msmo-simpad-tis captured the details of why the boy had sent the letter and gave insights into the match scores due to focusing on "sport"-topic-related features. our tsc-msmo-simpad-tis model selected the image of the letter (image no. (1) in fig. 4 ) as part of the pictorial summary. although the letter image was not in the human-annotated pictures list, its selection complements our textual summary well. the generated multimodal output indicates that the model maintains a balance between topic awareness and content relevance while producing the output. the balanced output may be because the classifier and the other summarizing components were given equal weights in the final loss function. the images chosen by the other tscvariants and the baseline atg were also not bad. in the given example, 4 images were chosen as part of the pictorial summary. a significant limitation of our work, is the highlighted extractive textual summaries (fig. 4 ) that resulted from using a pgn-based decoder. however, there are few extractive elements in the human written summaries, as seen from the blue highlighted text. thus, the model learns this extractive behavior from the training data itself. another limitation, as seen from the topic-wise results (sec. 5.2), is the dependence on data size for producing good quality output. the presence of low-sample-count sub-topics further adds to the problem of finding good "on-topic"-similar images for the classifier, thus leading to a deterioration in the quality of the summary produced by our model. ## conclusion and future study multimodal summaries help users absorb rich multimedia knowledge by generating brief and pertinent summaries. adding topic information helps our model learn the different representation styles of various topics resulting in better quality summaries. the improvement in rouge and ip scores in the overall test set and the topic-wise segments for all our experiments indicate that making the model learn topic-related information helps produce better quality multimodal summaries. furthermore, our experiments also established that using similar image features instead of the zero-padded vectors for samples having lesser in-article images does help in producing better summaries. in a future study, we can find similar images using temporal information already present in our augmented dataset. exploration of other techniques like comparing image-image, image-caption, or image-title embeddings for finding similar photos can also be done. a novel dataset can be created with lesser sub-topics and well-defined topics for studying our topic-based summarization technique.