F ake SV - VLM : Taming VLM for Detecting Fake Short-Video News via Progressive Mixture-Of-Experts Adapter (2025)

Abstract: We present FakeSV-VLM in this paper, a new VLM-based framework for detecting fake news on short video platforms. Despite significant efforts to combat this issue due to the severe threat that fake news videos pose to public information security, existing methods still fall short in detection accuracy, often due to lack of knowledge to verify the news is real or not. However, large Vision Language Models (VLMs) have absorbed extensive real-world knowledge from massive multimodal datasets. Motivated by this, we adapt advanced VLMs for fake news detection in short videos. Upon close examination of news samples, we observe that short video samples can be categorized into four distinct scenarios: both video and text are real (for real samples), or both are fake, or either the video or text is fake (for fake samples). Inspired by this insight, we design four experts tailored to handle each scenario and integrate them into VLM via Mixture of Experts. Specifically, we develop the Progressive MoE Adapter (PMOE) module where detection experts first provide an initial analysis, followed by attribution experts for a comprehensive diagnosis, leading to a robust decision. Additionally, we also note the fake news videos often show inconsistency between two modalities. Consequently, we further design the Alignment-driven Event Checking (ADEC) module, which perceives the fake news by capturing the inconsistency between different modalities. Extensive experiments on two benchmark datasets, FakeSV and FakeTT, verify the superiority of our model. It significantly outperforms current state-of-the-art models by +3.32% and +5.02%, establishing a new benchmark in the field.

## introduction with the rapid development of short video platforms, the spread of fake news videos on these platforms has posed significant risks to society, covering key areas such as politics @xcite , economics @xcite , and more. according to relevant statistics, over 500 hours of videos are uploaded to youtube every minute @xcite . relying entirely on manual verification of the authenticity of news videos would consume a huge amount of human resources. to address this challenge, some methods for detecting fake news videos have emerged in the early stages @xcite @xcite . however, with the continuous evolution of video editing and synthesis technologies, manipulating and fabricating news videos has become increasingly easier @xcite . moreover, the ongoing development of large language models further enhances the ability to fabricate fake news @xcite , making detection even more difficult and posing an unprecedented threat to public information security. to address this challenge, several methods have been proposed. fakingrecipe @xcite predicts video authenticity by fusing visual, textual, and audio features, while need @xcite leverages graph neural networks to analyze relationships among videos from the same event. although effective, these approaches often rely heavily on cross-modal fusion and are constrained by limited training data, lacking the ability to fully utilize real-world knowledge. some methods incorporate knowledge from vision language models (vlms)-for instance, ca-fvd @xcite detects modality alignment using vlms, and exmrd @xcite distills knowledge through designed reasoning chains-but they typically depend on vlms to pre-generate text, limiting scalability and practical deployment. to address the aforementioned issues, we propose directly using vlms to predict fake news videos. pre-trained vlms contain real-world knowledge, which can be fully utilized during the detection process to infer the authenticity of news videos. however, since vlms are not specifically designed for the task of predicting fake news videos, applying them directly for detection often does not yield ideal results. based on our observations, news videos can be classified into two broad categories: real and fake. at a more fine-grained level, fake news videos can be subdivided further into content forgery, description forgery, and full forgery of both. to enhance the capability of vlms in modeling hierarchical forgery patterns in fake news videos, we propose the progressive mixture-of-experts adapter (pmoe) module. specifically, a set of learnable artifact tokens is introduced to aggregate potential manipulation cues, which are concatenated with multimodal features and fed into the early layers of the llm. in the intermediate layers, the contextualized artifact tokens are extracted for two-stage reasoning: the detection moe estimates the overall authenticity of the video, and the attribution moe infers the specific manipulation type, such as visual tampering, textual misrepresentation, or cross-modal forgery. this enables fine-grained identification of diverse forgery strategies. given the frequent semantic inconsistencies between visual content and textual descriptions in fake news videos, we propose the alignmentdriven event checking (adec) module to enhance cross-modal event-level consistency verification. adec first extracts intermediate textual representations from the llm and applies average pooling to obtain a compact event-centric semantic embedding. this embedding is then compared with the visual representation using a contrastive learning objective. supervised by ground-truth labels, the model is encouraged to align cross-modal semantics and detect event-level discrepancies indicative of manipulation. based on the above design, we propose the fakesv-vlm framework to advance multimodal fake news videos detection. the main contributions of this work are summarized as follows: ## conclusion this paper presents fakesv-vlm, a novel framework for multimodal fake news videos detection that fully leverages the reasoning capabilities of vision language models (vlms) within an endto-end training paradigm. fakesv-vlm integrates vlms into the entire optimization process, enabling more effective modeling of semantic authenticity. to this end, we design the pmoe module, which captures hierarchical manipulation patterns through a two-stage expert mechanism, and the adec module, which facilitates cross-modal semantic alignment via contrastive learning. extensive experiments on real-world datasets demonstrate that fakesv-vlm consistently outperforms state-of-the-art baselines, showcasing its strong generalizability and effectiveness.