MVT amper B ench: Evaluating Robustness of Vision-Language Models (2025)

Abstract: Multimodal Large Language Models (MLLMs), are recent advancement of Vision-Language Models (VLMs) that have driven major advances in video understanding. However, their vulnerability to adversarial tampering and manipulations remains underexplored. To address this gap, we introduce MVTamperBench , a benchmark that systematically evaluates MLLM robustness against five prevalent tampering techniques: rotation, masking, substitution, repetition, and dropping; based on real-world visual tampering scenarios such as surveillance interference, social media content edits, and misinformation injection. MVTamperBench comprises ~3.4K original videos, expanded into over ~17K tampered clips covering 19 distinct video manipulation tasks. This benchmark challenges models to detect manipulations in spatial and temporal coherence. We evaluate 45 recent MLLMs from 15+ model families. We reveal substantial variability in resilience across tampering types and show that larger parameter counts do not necessarily guarantee robustness. MVTamperBench sets a new benchmark for developing tamper-resilient MLLM in safety-critical applications, including detecting clickbait, preventing harmful content distribution, and enforcing policies on media platforms. We release all code, data, and benchmark to foster open research in trustworthy video understanding.

## introduction multimodal large language models (mllms) have catalyzed significant progress in video understanding, enabling a wide array of applications across domains such as surveillance, healthcare, and autonomous systems. however, their growing integration into high-traffic platforms (e.g., instagram, facebook, tiktok) has exposed critical vulnerabilities. specifically, tampered videos are increasingly exploited to bypass platform policies, disseminate harmful content, and promote clickbait, posing serious challenges for content moderation and policy enforcement @xcite . these threats underscore the urgent need to improve the robustness of mllms against real-world manipulations. although video tampering can involve audio modifications, synthetic speech overlays, or deepfake generation, our benchmark focuses exclusively on visual-only manipulations. this choice is driven by the current limitations of existing models, most of which currently lack comprehensive audiovisual processing capabilities. unlike adversarial robustness in static images, an area that has been extensively studied, video tampering introduces unique challenges that arise from the interplay of spatial and temporal dynamics. common manipulation techniques such as frame dropping, masking, repetition, substitution, and rotation disrupt this coherence, frequently resulting in catastrophic model failures. these methods mirror real-world adversarial tactics: substitution injects objectionable material (e.g., nudity, violence) to circumvent detection; dropping eliminates key surveillance evidence; repetition loops footage to obscure illicit activity; masking occludes critical regions; and rotation induces spatial distortions commonly seen in edited or re-uploaded content. existing adversarial and traditional approaches, including black-box @xcite and crossmodal @xcite attacks, primarily address isolated scenarios rather than systematically evaluating diverse tampering types. furthermore, existing multimodal benchmarks @xcite -such as mmbench-video @xcite , blink @xcite , and video-mme @xcite -focus on multimodal comprehension and temporal reasoning but overlook adversarial robustness. for example, mmbench-video evaluates cross-modal alignment without adversarial testing; blink targets longform temporal reasoning without addressing tampering impacts; and video-mme, while effective for vision-language alignment, omits tamperingspecific tasks. this leaves a critical gap in systematically assessing how mllms withstand real-world manipulations. to bridge this gap, we introduce mvtam-perbench, a benchmark specifically designed to evaluate mllm robustness against five prevalent tampering techniques. built from 3.4k original videos-expanded into over 17k tampered clips spanning 19 video tasks-mvtamperbench challenges models to detect manipulations by disrupting temporal and spatial coherence in a video. by focusing on tampering resilience, we believe that mvtamperbench provides a critical tool for improving mllm robustness, and eventually, it enables the development of tamper-resilient models applicable to real-world challenges such as clickbait detection, content moderation, and policy enforcement, advancing adversarial robustness in video understanding for high-stakes domains. our main contributions are as follows: the development of benchmarks for mllms has significantly advanced the evaluation of image and video understanding tasks. they have covered spatial reasoning, temporal comprehension, object detection, common sense inference, and so on. however, the robustness of mllms to adversarial manipulations, particularly video tampering, remains underexplored. this section reviews existing benchmarks, which can be categorized into image-based and video-based evaluations, and analyzes their contributions and shared limitations. ## mvtamperbench we introduce mvtamperbench, a comprehensive benchmark designed to systematically evaluate the robustness of mllms against video tampering techniques. through our mvtamperbench which introduces diverse manipulations, we aim to broaden the evaluation landscape, enabling a deeper understanding of model strengths and vulnerabilities under adversarial scenarios. in the following subsections, we detail its construction, design choices, and key features. more details can be found in our code foot_0 , data foot_1 and benchmark foot_2 repositories. ## experiments this section outlines the experimental setup, results, and analysis of 45 models evaluated on our proposed benchmark. our results highlight their strengths, weaknesses, and actionable insights. we provide further overview of the evaluated mllms in appendix a.3. ## limitation mvtamperbench provides a robust framework for evaluating mllm resilience against video tampering, but there are limitations that present opportunities for future work. first, while the benchmark evaluates five tampering techniques, expanding to additional manipulations/modalities to better capture subtle, and emerging techniques. second,the dataset currently relies on limited datasets. expanding the benchmark to incorporate videos from diverse sources, such as user-generated content, surveillance footage, or policy enforcement, would broaden its applicability and relevance to task-specific and domainspecific challenges. third, while the current evaluation focuses on binary detection, future benchmarks could assess a model's ability to classify the specific tampering type, providing deeper insights into its robustness. finally, scalability to closed-source and extremely large-scale models (> 100b parameters) remains a challenge due to computational and cost constraints. addressing these limitations will enable mv-tamperbench to further advance tampering detection and resilience in a broader range of applications.