Bridging Multimodal and Video Summarization: A Unified Survey (2025)

Abstract: Multimodal summarization (MMS) and video summarization (VS) have traditionally evolved in separate communities—natural language processing (NLP) and computer vision (CV), respectively. MMS focuses on generating textual summaries from inputs such as text, images, or audio, while VS emphasizes selecting key visual content. With the recent rise of vision-language models (VLMs), these once-disparate tasks are converging under a unified framework that integrates visual and linguistic understanding.In this survey, we provide a unified perspective that bridges MMS and VS. We formalize the task landscape, review key datasets and evaluation metrics, and categorize major modeling approaches into new taxonomy. In addition, we highlight core challenges and outline future directions toward building general-purpose multimodal summarization systems. By synthesizing insights from both NLP and CV communities, this survey aims to establish a coherent foundation for advancing this rapidly evolving field.

## introduction summarization is a core task in natural language processing (nlp), traditionally defined as the process of producing a concise and coherent version of a longer source while preserving its essential content @xcite . with the increasing ubiquity of multimodal content in the digital age, ranging from news articles with images to instructional videos with subtitles, the need to summarize information across multiple modalities has become both urgent and technically feasible. this has given rise to two closely related but historically distinct areas of research: multimodal summarization (mms) and video summarization (vs). multimodal summarization focuses on generating summaries from inputs that combine text, images, audio, and other non-linguistic modalities @xcite . it is often approached as a text generation task and studied within the nlp community @xcite . in contrast, video summarization, which is traditionally rooted in the computer vision (cv) domain, aims to produce a condensed version of a video by selecting keyframes or segments, typically without involving language generation @xcite . as a result, these two lines of work have evolved in parallel, with limited interaction between their research communities, benchmarks, and methodologies. however, the growing prevalence of multimodal digital content has catalyzed a convergence between mms and vs. a pivotal shift in mms was the introduction of multimodal summarization with multimodal output (msmo), a paradigm that generates summaries comprising both text and visuals for a richer digest of information @xcite . concurrently, video summarization has evolved beyond traditional extractive video-to-video (v2v) techniques @xcite . it now includes abstractive video-to-text (v2t) generation and hybrid videoto-text+video (v2vt) systems @xcite . this progression demonstrates a clear alignment with the multimodal output objectives pioneered by msmo, as both aim to integrate textual narratives with key visual highlights. recent breakthroughs in vision-language models (vlms) such as blip-2 @xcite , flamingo @xcite , and gpt-4v are now unifying these once-separate research threads. by capably processing and generating content across text, image, and video modalities, these models enable the creation of truly integrated summarization systems @xcite . this development mirrors the recent history of text-only summarization, where large language models (llms) catalyzed transformative progress @xcite . the success of llms provides a compelling blueprint for leveragtext image video input output neural fusion transformer llm/vllm paradigm supervised self-supervised zero/few-shot + fine-tuning figure 1: overview of multimodal/video summarization input/output modalities, and major paradigms shift. despite this convergence, the literature lacks a comprehensive and unified survey that brings together the full landscape of multimodal and video summarization. existing surveys typically focus either on multimodal summarization from a language perspective @xcite or on visual summarization from a frame-selection or temporal segmentation perspective @xcite . thus, there is a pressing need to synthesize these research directions under a holistic perspective in the llm era. this survey bridges this critical gap by providing a comprehensive and unified overview of mms and vs, focusing on their convergence. we synthesize research from both nlp and cv, examining common challenges, modeling paradigms, datasets, and evaluation metrics under a cohesive lens, as shown in figure 1 . specifically, this survey: 1) formalizes the unified task landscape and discusses pertinent evaluation metrics (section 2); 2) reviews key datasets and benchmarks from both traditions, grouping by input and output modalities (section 3); 3) proposes a new unified taxonomy to categorize existing modeling approaches for mms and video summarization, highlighting how recent pretrained vlms are reshaping the summarization landscape (section 4); 4) discusses open challenges and charts promising future directions for a truly unified multimodal summarization framework that can reason over and summarize complex, cross-modal content (section 5). by unifying these perspectives, we hope to support a broader understanding of multimodal summarization and foster collaboration between nlp and cv researchers working at the intersection of language, vision, and video understanding. ## datasets the advancement of multimodal summarization has been significantly propelled by the development of diverse and specialized datasets. these resources are crucial for training robust models, benchmarking research progress, and exploring the nuances of different summarization objectives. in this section, we survey key datasets and categorize them by modalities into four groups: (1) text + image to text datasets (ti2t), (2) video to video datasets (v2v), (3) video to text datasets (v2t), and (4) multimodal summarization with multimodal output (msmo) datasets, which involve richer modality combinations. ## methodology this section traces the evolution of multimodal and video summarization methods and models across three paradigms. we begin with (1) early neural architectures that addressed mms and vs with separate, task-specific models we then examine (2) transformer-based methods, whose attention mechanisms enabled deeper cross-modal reasoning and integration. finally, we explore (3) large language and vision-language models (llms and vlms), which unify summarization under a single, generative framework. our analysis focuses on key architectural shifts and the progressive integration of modalities that drove the convergence of these fields. ## conclusion this survey has bridged the traditionally distinct yet increasingly convergent fields of multimodal and video summarization, offering a unified analysis of their common tasks, datasets, evaluation metrics, modeling approaches, and future trajectories. we highlight the pivotal role of vision-language models in accelerating this convergence. continued research within this integrated framework is paramount for developing intelligent systems capable of holistically understanding and summarizing the rich tapestry of multimodal information.