Multimodal Fusion and Coherence Modeling for Video Topic Segmentation (2025)

Abstract: The video topic segmentation (VTS) task segments videos into intelligible, non-overlapping topics, facilitating efficient comprehension of video content and quick access to specific content. VTS is also critical to various downstream video understanding tasks. Traditional VTS methods using shallow features or unsupervised approaches struggle to accurately discern the nuances of topical transitions. Recently, supervised approaches have achieved superior performance on video action or scene segmentation over unsupervised approaches. In this work, we improve supervised VTS by thoroughly exploring **multimodal fusion** and **multimodal coherence modeling**. Specifically, (1) we enhance multimodal fusion by exploring different architectures using Cross-Attention and Mixture of Experts. (2) To generally strengthen multimodality alignment and fusion, we pre-train and fine-tune the model with multimodal contrastive learning. (3) We propose a new pre-training task tailored for the VTS task, and a novel fine-tuning task for enhancing multimodal coherence modeling for VTS. We evaluate our proposed approaches on educational videos, in the form of lectures, due to the vital role of topic segmentation of educational videos in boosting learning experiences. Additionally, to promote research in VTS, we introduce a large-scale Chinese lecture video dataset to augment the existing English lecture video datasets. Experiments on both English and Chinese lecture datasets demonstrate that our model achieves superior VTS performance compared to competitive unsupervised and supervised baselines.

## introduction the proliferation of digital video content over the last few decades has underscored the importance of efficient content navigation and comprehension. as the unstructured nature of videos poses significant challenges for users seeking to quickly grasp or reference specific topics, video topic segmentation (vts) has emerged as a vital tool in addressing these demands. by delineating videos into coherent non-overlapping topics, vts not only facilitates intuitive understanding of video content but also enables swiftly pinpointing and accessing topics of interest. this is particularly pertinent for the furtherance of various video understanding tasks, where vts serves as a foundational component. traditional vts approaches predominantly hinge on shallow features @xcite @xcite and unsupervised methods @xcite , due to scarcity of labeled data. these methods often fall short in capturing the semantic cues that signal topical shifts in video streams, hence suffer from limited precision. recent advancements in supervised learning paradigms have achieved notable performance improvements in multi-modal task @xcite @xcite and various video segmentation tasks, such as video action segmentation @xcite , scene segmentation @xcite , and topic segmentation @xcite @xcite , surpassing unsupervised methods. performance of supervised approaches can be further enhanced by pre-training on vast volumes of unlabeled data @xcite or initializing models from pre-trained models @xcite and then fine-tuning the model. hence, in this work, we focus on further improving supervised methods for vts. compared to text topic segmentation @xcite @xcite , videos contain rich and diverse multimodal contextual information. fully utilizing multimodal information, such as visual cues and textual data (e.g., screen text and subtitles), could facilitate more detailed content understanding and in turn more accurate semantic segmentation than relying on text only. our case studies in appendix h demonstrate the great challenges posed by vts, particularly to unsupervised approaches or supervised methods that rely solely on either visual or textual modality. the complexity inherent in video content-where multimodal signals must be effectively integrated-accentuates the difficulty. also, coherence is essential for understanding logical structures and semantics. enhancing coherence modeling has achieved significant improvements in long text topic segmentation @xcite . therefore, we improve supervised vts methods by thoroughly exploring multimodal fusion and multimodal coherence modeling. we enhance multimodal fusion from the perspectives of model architecture and pre-training and fine-tuning tasks. specifically, we compare various multimodal fusion architectures built upon cross-attention and mixture-of-experts (moe). we investigate the effect of multimodal contrastive learning for general pre-training and fine-tuning for strengthening crossmodal alignment. for enhancing multimodal coherence modeling, we propose a new pre-training task tailored for the vts task, and a novel fine-tuning task by elevating intra-topic multimodal feature similarities and inter-topic multimodal feature differences. the proposed approaches are extensively evaluated on educational videos, in the form of lectures, due to the pivotal contributions of topic segmentation of educational videos in bolstering the learning experiences. our contributions can be summarized as follows. ## related work text topic segmentation text topic segmentation aims to automatically partition text into topically consistent, non-overlapping segments @xcite . by automatically mining clues of topic shifts from large amounts of labeled data @xcite , contemporary supervised models @xcite @xcite demonstrate superior performance compared to unsupervised approaches @xcite . notably, supervised models that excel at modeling long sequences @xcite are capable of capturing longer contextual nuances and thereby achieve better topic segmentation performance, compared to models that model local sentence pairs or block pairs @xcite . in addition, recent works @xcite @xcite show that strengthening coherence modeling can improve text topic segmentation performance. inspired by these findings, in this work, we explore enhancing coherence modeling for video topic segmentation under the multimodal configurations. video topic segmentation for video topic segmentation, some approaches, such as bassl @xcite , explore visual-only information. however, many recent works have achieved enhanced semantic understanding of videos by leveraging multimodal data. ## methodology figure 1 depicts the overall architecture of our mmvts model. section 3.1 presents the problem definition of multimodal vts and the overall model architecture. we enhance multimodal fusion from the perspectives of model architecture, pre-training, and fine-tuning tasks. specifically, we compare different multimodal fusion architectures built upon merge-and cross-attention, and mixture-of-experts (section 3.1). we explore multimodal contrastive learning for cross-modality alignment and propose a new pre-training task tailored for vts (section 3.2). for fine-tuning, we also propose a novel task for multimodal coherence modeling (section 3.3). ## conclusion we propose a novel supervised vts model by thoroughly exploring multimodal fusion and coherence modeling. we also introduce a large-scale labeled chinese lecture dataset for vts. extensive experiments demonstrate the superiority of our model and effectiveness of critical algorithmic designs.