Towards Unified Uni- and Multi-modal News Headline Generation (2024)

Abstract: Thanks to the recent progress in vision-language modeling and the evolving nature of news consumption, the tasks of automatic summarization and headline generation based on multimodal news articles have been gaining popularity. One of the limitations of the current approaches is caused by the commonly used sophisticated modular architectures built upon hierarchical cross-modal encoders and modality-specific decoders, which restrict the model’s applicability to specific data modalities – once trained on, e.g., text+video pairs there is no straightforward way to apply the model to text+image or text-only data. In this work, we propose a unified task formulation that utilizes a simple encoder-decoder model to generate headlines from uni- and multi-modal news articles. This model is trained jointly on data of several modalities and extends the textual decoder to handle the multimodal output.

## introduction the task of multimodal summarization was introduced as an extension of the traditional nlp task of text summarization. early works (e.g., @xcite @xcite explored to what extent enriching the textual document with additional context-specific information (e.g., visual clues from images attached to a product/service review or video clips attached to a cooking recipe) helps the automatic systems in refining the summary generation process. @xcite were the first to notice that the informativeness of a summary can be significantly improved by including the visual clues in the output, introducing the task of multimodal summarization with multimodal output (msmo). in their formulation, based on a textual document and a set of images, the model is tasked to generate the textual summary and pick a single image as the pictorial summary. @xcite introduced a variant of the task where the input is a pair of textual article and a short video. the following works (e.g., @xcite explored the challenging problem of multi-modal fusion and alignment by introducing auxiliary tasks during training and extending the model architecture with task-specific blocks. however, by doing so, the model is tailored to a specific data modality. in this work, we propose a novel msmo task formulation that supports the most common data modalities (text+@xmath0, text+@xmath1, @xmath2. we explore two approaches (section 3.2): i) extending a text-to-text baseline with visual features and ii) fine-tuning a multimodal foundation model. we show that the proposed unified formulation leads to results competitive with previously introduced task-specific solutions (section 4) while not being restricted to specific data modalities. ## unifying msmo previous works explored two variants of the msmo task: video-based and image-based. in the video-based one, the multimodal article is represented as a pair of a video clip and a textual document. the goal is to generate the textual summary and to choose a single frame that acts as a pictorial summary. in the image-based variant, the input is a set of images, i.e., there is no temporal dependency. the second difference comes from the ground truth image: in the image-based variant, we assume that the target is one of the input images. in the video-based one, there is no such assumption foot_0 -a similarity function is utilized to obtain the perframe labels for training using the most similar one as a positive target. our goal is to train a system capable of natively handling both msmo variants as well as the basic text-to-text problem (summarization or headline generation). we achieve that by transforming the visual inputs into a sequence of image features that are concatenated with the textual token embeddings. instead of using a dedicated module for image scoring, we realize the target image representations by appending an index token to the textual target -img_ind_1 indicates that the first image is the target, img_ind_2 that the second, etc. this formulation allows us to use the standard transformer architecture @xcite trained end-toend in a multi-task setting (see figure 1 ) -for the text-only input, we do not extend the textual embeddings and do not add the index token into the target sequence. ## related work historically, for both the video-based @xcite and the image-based @xcite msmo, the attention mechanism @xcite was used to condition the encoded text representation on the visual information, which in the next step was passed to the autoregressive text decoder. following works focused on improving the quality end efficiency of this process: @xcite and @xcite focused on the filtering mechanism that would allow the model to attend only to chosen relevant features avoiding potential noise. @xcite and @xcite worked on adapting strong pre-trained language models to the multimodal input. all of those works perturb the textual representation -the model is no longer capable of inference on text-only data. the reverse attention (@xmath3. using a learning signal from the pictorial target, the model was trained to produce image/frame-level scores. a step towards simplifying these modular approaches was recently made by @xcite , who generate pseudo-captions for input images and then pick the image with the highest similarity between the caption and the generated textual summary, and @xcite , who instead of using a textual decoder, predict sentence-level scores and extract top-k sentences as the textual summary. a one-for-all architectures unifying several vision-and-language tasks have also been explored in a wider context. @xcite introduce visual sentinel tokens corresponding to image regions, allowing them to realize visual grounding with a text-only decoder. the task-and modality-agnostic ofa framework @xcite unifies the multi-modal and text-only tasks with a sequence-to-sequence transformer. by design, it is however limited to tasks dealing with a single image, e.g., image captioning or visual question answering, not supporting inputs containing multiple images or videos. a recent line of research on multimodal llms @xcite @xcite transfers the knowledge from image-text models into video-text models. inspired by those works and the general-purpose multimodal foundation models (e.g., @xcite @xcite , we propose the unified formulation (section 2) -the multi-task training with a simplified encoder allows the model to natively handle both multi-modal and text-only input and the usage of index tokens that explicitly point to a particular input image allows us to drop the scoring module and train with a single text decoder. ## conclusions in this pilot study on multi-task multi-modal summarization, we propose a novel unified formulation for the msmo task. by training the textual decoder to generate index tokens, we make use of the training signal from the visual modality without a dedicated scoring module. our results indicate that multi-task training, which incorporates text-only data, is an alternative to text-only pretraining, which preserves the native capability to handle purely textual input. for the challenging task of video-based msmo, there is still some gap left when it comes to the visual output quality when compared to sophisticated task-specific architecture. based on our results, for this specific task, the visual generative approaches are still inferior to extractive ones.