The Current Landscape of Multimodal Summarization (2023)

Abstract: In recent years, the rise of multimedia content on the internet has inundated users with a vast and diverse array of information, including images, videos, and textual data. Handling this flood of multimedia data necessitates advanced techniques capable of distilling this wealth of information into concise, meaningful summaries. Multimodal summarization, which involves generating summaries from multiple modalities such as text, images, and videos, has become a pivotal area of research in natural language processing, computer vision, and multimedia analysis. This survey paper offers an overview of the state-of-the-art techniques, methodologies, and challenges in the domain of multimodal summarization. We highlight the interdisciplinary advancements made in this field specifically on the lines of two main frontiers:1) Multimodal Abstractive Summarization, and 2) Pre-training Language Models in Multimodal Summarization. By synthesizing insights from existing research, we aim to provide a holistic understanding of multimodal summarization techniques.

## introduction in today's information age, where vast amounts of data are generated and shared in diverse formats such as text, images, and videos, the ability to distill meaningful insights from this multimedia landscape is paramount. multimodal summarization, at the intersection of natural language processing, computer vision, and multimedia analysis, stands as a pivotal solution to this burgeoning challenge. unlike traditional text-only summarization methods, the essence of multimodal summarization lies in its capacity to synthesize information from multiple sources, creating condensed yet comprehensive summaries that capture the richness of textual, visual, and auditory data. this synthesis not only caters to the evolving needs of users seeking efficient content consumption but also finds applications in diverse domains such as multimedia retrieval, content recommendation, and intelligent data analysis. by delving deep into the methodologies, algorithms, and challenges faced in this interdisciplinary field, this survey aims to provide a panoramic view of the existing state-of-the-art techniques. we primarily concentrate our efforts on two research frontiers: the former is chosen as it lays a foundation for multimodal summarization research. the prominent datasets used for the same such as how2 @xcite are ideal (both from the length and quality point of view) for experimentation and can be used as benchmarks. a lot of research and experimentation models have thus been proposed on these datasets. the latter is a new paradigm that very recently started proliferating in multimodal summarization research. to take multimodal summarization to practical applications, however, the above is not enough. the pre-training and fine-tuning paradigm has been widely successful in nlp and computer vision tasks. it significantly reduces the training costs on downstream tasks and aids generalizability. hence the recent work has been shifted toward building a pre-trained language model specifically for multimodal understanding and summarization. however, certain challenges render the conventional pre-training approaches ineffective. we discuss the innovations and research efforts that have been put in to develop multimodal gplms(generatively pre-trained language models). based on these developments we also outline a general recipe and pattern in these works and provide a sample architecture/pipeline of these models. we also outline the challenges and future scope of research in this field. moreover, we provide details of other supplementary resources such as datasets and applications for multimodal summarization. in short, we provide an exhaustive survey of the past, the present, and the future of multimodal summarization research. the rest of the paper is structured as follows-comparative analysis of various available datasets (section 2), followed by advances in multimodal abstractive summarization are discussed in (section 3), discussion on pre-training in (section 4), results (sections 5) and challenges and future work(section 6). ## datasets datasets play a crucial role in the advancements of a particular field. as there are many different inputoutput modalities and the job is customizable, there isn't a single standard dataset that can be used as a baseline for evaluation across all techniques for the mms task. however, we've gathered details regarding the datasets utilized in earlier studies, also found in ## multimodal abstractive summarization(mas) the task of multimodal summarization was first formally defined in @xcite . the formal definition deduced from the paper is given by: "given a document containing a set of sentences, and a set of images, the output of automatic multimodal summarization is a pictorial summary, consisting of condensed textual information, and the subsequent set of most relevant images". to extend the pointer generator network @xcite , which was built for text modality, msmo @xcite introduced a similar pipeline for image modality. similarly, the attention mechanism was extended to multimodal attention by fusing both the text and image context vectors during the decode step. finally, the image output was given by an ancillary network that would rank all the images. however, this model was being trained only with text modality in mind, which led to the problem of modality bias, i.e. only the central modality gets trained properly, and the secondary modality's loss is suppressed in the total objective loss in this training mechanism. to alleviate this lack of multimodal reference, @xcite introduces a new multimodal objective function. this objective function consists of the conventional negative log-likelihood over the decoded text and an additional cross-entropy loss of the image selection. this image selection process is guided with the help of the image discriminator module, which gets trained along with the seq-toseq parameters, thereby mitigating the problem of modality bias. @xcite proposes a model with a hierarchical encoder-decoder model. the proposed model uses 3 multimodal attention mechanisms ( text-image caption, text-image and image-image caption) to compute the sentence context in the hierarchical decoder. the model uses rnn @xcite ) for text and image captions and cnn @xcite to obtain image vector representation, followed by bi-directional rnn to encode ordered image sets. the attention score of each image is computed with each sentence of the generated summary. based on the score, images are arranged in descending order and the top k is selected. previous abstractive approaches used only image-text as modalities until @xcite introduced the how2 dataset (details in dataset section). videos contain abundant data that is represented in chronological order, which is crucial for summarization. @xcite proposed baseline framework, in which the videos are encoded with the help of a pre-trained action recognition model: resnext-101 and text is encoded using bi-directional rnn. this is followed by a hierarchical attention mechanism to fuse the modalities and a standard rnn decoder step. resnext-101 3d convolutional neural network @xcite is trained to recognize 400 different human actions 6589 in the kinetics dataset. this model outputs 2048 dimensional features, ex- dataset input modalities output modalities data statistics domain (uzzaman et al., 2011) images, texts images, text wikipedia web crawling content. ## pre-training for multimodal summarization pre-training and fine-tuning paradigm is widely successful in a variety of natural language processing, and computer vision tasks. pre-training specific to text summarization is done by using a variation of masked language modelling, in which span corruption is used @xcite @xcite . this substantially improves the downstream summarization performance. similarly, research efforts have been put in to develop methodologies for building pre-trained transformer-style encoders that can be used for vision-language understanding tasks @xcite @xcite . for text-understanding capabilities, most works resorted to standard masked language modelling @xcite , and for vision understanding a similar idea was used, just at different levels of granularity. @xcite ) considered the pixel level granularity and employed mlm(masked language modelling), image text matching, and random pixel sampling as self-supervised objective functions. figure 2 : general architecture for pre-trained lm for multimodal summarization @xcite focused on object-level granularity, and introduced masked object detection as their objective function. furthermore, @xcite focused on improving vision representations on similar object detection pre-training lines. @xcite dealt with patch-level granularity. while these are language-vision understanding models, a similar plethora of research has been conducted on vision and language-guided text generation @xcite presented a model for both visual question answering and image captioning @xcite . similar to pre-trained encoder-decoder t5 @xcite , to unify all the tasks @xcite was introduced. although prior work has made much progress on vl pre-training, the problem of generating text given text and video input (e.g. the how2 dataset) is not well studied under the vl pre-training setting, except by @xcite , who proposed a dual-stream model for both vl classification and generation with video data. however, compared to gplms in nlp such as bart @xcite and t5 @xcite , their text generation ability is limited as the most prominent dataset how2 @xcite is also comparatively very small, and it contains only short videos and short 2-3 sentence long reference summaries and is thus prone to overfitting. therefore there are several challenges to replicating a similar success story in the multimodal scenario and any pre-training methodology that initializes a large model from scratch is bound to fail. the research community found an innovative workaround to this limitation. the research was then manoeuvred to find training objectives on already pre-trained large language models, to render them multimodal, i.e. finding objective functions for injecting visual information in the pre-trained models such as bart. notably, @xcite , discussed 2 injection strategies: cross-modal dotproduct attention and multi-head attention. to further improve the performance of the model, a forget gate (fg ) @xcite is used to filter out redundancy and noisy information in the visual features. the research provides a foundation by answering two main questions: how to apply injection and where to inject visual information. the most prominent work of pre-trained end-toend transformer architecture is seen in ((papalampidi and lapata, 2022) adapters for long video-totext summarization). firstly, to mitigate the issue of a small dataset they introduce summscreen, a dataset consisting of 4,575 episodes, their transcriptions, and corresponding reference summaries. as the video and text are very long in this dataset(fulllength episodal summary), they take inspiration from a long-document summary regime and explore two methods: making attention lightweightinspired by @xcite and content selection techniques. after selecting the utterances, the multimodal embedding vector is appended to the textual data. for @xmath0, u2, .. @xmath1, t1, t2, t3, . . . tk, m2, t1, t2, t3,.., mn, t1, t2, ..tn>, where mi is multimodal information vector, and ti is the textual input token. to process the multimodal information, they add hierarchical adapters to the encoder and the decoder model of bart, adding only a small fraction of new parameters. these adapters fuse the global-level utterance information with low-level textual information. they end up fine-tuning/training only 3.8% of the total model parameters. following the task of msmo( @xcite ) and @xcite , wherein a pictorial summary was to be generated with text output and relevant images, @xcite proposed a unified framework built upon bart @xcite that performs extractive, abstraction summary, as well as image ranking. firstly, they extend the bart encoder to accept image modality, the images are first divided into patches, flattened, and linearly projected. similarly, to factor in visual information, they added a visual cross-attention block after every self-attention block, that attended to the visual hidden states. figure 2 shows the general architecture of pre-trained lm for multimodal summarization. to overcome the same limitations of the how2 dataset, as mentioned above, @xcite developed a new dataset called aviate, which is a large-scale dataset for multimodal abstractive summarization with videos of diverse duration and developed from academic presentations. the authors of the paper use deepspeech @xcite for automatic speech recognition (asr) to extract transcripts of all the videos. as the dataset contains academic presentations, the useful data from the slides is extracted from google ocr vision api. initially, asr and ocr were both represented using pre-trained model bert @xcite , and then these representations jointly attended to reduce redundant words from both sources. these fused representations of asr-ocr and video and audio modalities are then fed to factorised multimodal transformer (fmt @xcite ) architecture for sequential multimodal learning. the proposed decoder-only network uses a pre-trained transformer @xcite as its basic module and generates summary-like language modelling. thus, this concept of transformer lm is extended in the multimodal environment using fmt. ## results most prominently the performance of the models is benchmarked on how2 @xcite , and the vmsmo @xcite dataset. however, some research papers employed their datasets such as @xcite , @xcite , etc. we report the widely used evaluation metrics for summarization tasks-rouge-1, rouge- @xcite attempted to visualize the impact of the added vision component. however, they only showcased the relationship between the input text and the images, in the encoder step, and not the relationship between the decoded text and the image coverage being considered. as the lm is largely still pre-trained on the text modality, the downstream tasks might face the problem of modality bias @xcite . the task of standard abstractive summarization 1 *-not provided in particular paper 2 *-not provided in particular paper saw exponential progress with the advent of pegasus @xcite , bart @xcite , z-code++ @xcite etc. the primary reason for this jump was that these approaches came up with specific self-supervised objective functions that were specifically helpful for downstream summarization tasks. for a pre-trained model to be successful it has to satisfy-1) the objective function will help downstream tasks, and 2) the dataset used for pre-training must be large, and generalized. it is hard for multimodal summarization to satisfy either of the criteria. the availability of such datasets is limited, moreover, the pre-training objective functions for multimodal understanding notoriously hinder the generation capabilities @xcite . most applications of multimodal summarization deal with long videos, and documents, long document summarization is a challenging field, as it is hard to capture long-range relationships, and is computationally expensive. recently, @xcite and @xcite have shown promising results that map from arbitrary length input to arbitrary length output in a transformer-style model, and fixed size parameters(however, they have not been used for summarization yet). research on such new frontiers could help solve the challenges of multimodal summarization. this survey paper presents a comprehensive exploration of the multifaceted landscape of multimodal summarization. we then offer a detailed overview of the datasets needed to complete the mms challenge. additionally, we provide a brief description of the several approaches taken to complete the mms assignment as well as the assessment metrics applied to determine the quality of the summaries generated. lastly, we also offer some potential avenues for future mms study. in essence, this survey serves as a compass guiding researchers, practitioners, and enthusiasts through the dynamic landscape of multimodal summarization.