Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward (2025)

Abstract: Large Vision Language Models (LVLMs) have shown impressive performance on various vision-language tasks. However, while objects in natural scenes inevitably exhibit visual variations in position, scale, orientation, and context due to changes in viewpoint and environment, the robustness of LVLMs to these fundamental visual variations remains largely unexplored. To address this gap, we introduce V²R-Bench, a comprehensive benchmark framework for evaluating Visual Variation Robustness of LVLMs, which encompasses automated evaluation dataset generation and principled metrics for thorough robustness assessment. Through extensive evaluation of 13 LVLMs, we reveal a surprising vulnerability to visual variations, affecting even advanced models that excel at complex vision-language tasks yet significantly underperform on simple tasks like object recognition. Interestingly, these models exhibit a distinct visual position bias that contradicts theories of effective receptive fields and demonstrate a human-like visual acuity threshold. To identify the source of these vulnerabilities, we propose a systematic framework for component-level analysis, featuring a novel visualization approach for aligned visual features. Results show that these vulnerabilities stem from error accumulation in the pipeline architecture and inadequate multimodal alignment. Complementary experiments with synthetic data further demonstrate that these limitations are fundamentally architectural challenges, underscoring the need for architectural innovations in future LVLM designs.

## introduction the rapid development of large vision language models (lvlms) @xcite has been driven by two key factors: innovations in model architectures and the avail-ability of high-quality training data. these models have demonstrated impressive results in complex vision-language tasks, achieving human-level performance across various challenges @xcite . to systematically evaluate such capabilities, numerous multimodal benchmarks have been developed to assess models' fundamental knowledge @xcite , perceptual capability @xcite , cognitive understanding @xcite , and reasoning skills @xcite across various downstream applications. while current benchmarks extensively evaluate models using images collected in specific scenarios, they overlook a more fundamental generalization capability: the robustness of lvlms to visual variations. in input images, objects naturally exhibit diverse variations: spatial positions shift with changes in camera angles and viewpoints; object scales vary depending on viewing distances; orientations deviate from standard poses through rotations and inversions; and objects appear in a range of visual semantic contexts. these visual variations raise concerns about lvlms' robustness: whether models maintain consistent perception capabilities across all spatial positions in input images; what visual acuity threshold determines reliable performance; and how changes in orientation and visual context influence model behavior. despite the importance of such robustness, current research has primarily focused on model robustness to socio-cultural factors @xcite , adversarial prompt attacks @xcite , or corrupted image inputs @xcite , leaving the impact of natural visual variations largely unexplored. in this paper, we propose v 2 r-bench, a comprehensive benchmark framework to evaluate lvlm robustness against fundamental visual variations. our framework consists of an automated data generation pipeline, and tailored evaluation metrics, (2) synthetic data generation for the fundamental visual tasks proposed (e.g., object, direction, coordinate, path and ocr tasks). the subfigure on the right shows metrics for robustness testing, including performance consistency, semantic and token-level stability, and llm-as-a-judge. through extensive evaluation on 21 lvlms, we uncover surprising findings: despite their excellence in complex multimodal tasks, these models exhibit unexpected vulnerabilities to visual variations, leading to poor performance even in basic tasks such as object recognition. specifically, we observe: (1) a counter-intuitive position bias where models achieve higher accuracy at image edges rather than the center; (2) a human-like visual acuity threshold where model reliability steadily decreases with object size, reaching and maintaining minimum performance below a critical scale threshold; to identify the root cause of these vulnerabilities, we conduct a systematic analysis of lvlm components. a key innovation in our analysis is a novel visualization method that reconstructs language tokens from aligned visual features, offering insights into how models process and transmit visual information across modalities. our investigation reveals that inadequate multimodal alignment is the primary bottleneck, as models fail to maintain stable visual representations across variations and struggle to effectively align visual semantics with the language model. to disentangle whether these limitations stem from architectural constraints or data deficiency, we conduct complementary experiments with synthetic training data. the results reveal that these vulnerabilities are fundamentally rooted in architectural design. these findings underscore two critical directions for future lvlm de-velopment: stronger multimodal alignment mechanisms to maintain semantic consistency across variations, and unified architectural designs to mitigate error accumulation in current pipeline structures. our contributions are summarized as follows: • we identify and formulate a novel problem: the robustness of lvlms to visual variations, a fundamental yet overlooked capability essential for reliable vision-language reasoning. ## related work the rapid progress of lvlms is driven by continuous advances in model architectures and the exponential growth of training datasets. on the architectural side, improvements in foundation language models @xcite @xcite and vision models @xcite , combined with increasingly sophisticated multimodal alignment modules @xcite @xcite @xcite , have established the core capabilities of lvlms in crossmodal understanding and reasoning. building on these foundations, the introduction of high-quality datasets @xcite @xcite for different training stages (e.g., multimodal alignment, instruction tuning, preference alignment) has enabled lvlms to achieve strong performance across diverse real-world scenarios. despite effectively leveraging pretrained vision and language components, the pipeline architecture can accumulate errors across modules. no systematic investigation has yet been conducted to attribute model failures to specific components, limiting interpretability and understanding of modality alignment. ## v 2 r-bench evaluation framework in this section, we propose a novel framework for automatically automatically generating diverse visual variations across task settings (3.1), along with a detailed description of the construction and char-acteristics of our evaluation dataset (3.2) and our evaluation protocols (3.3), as illustrated in figure 1 . ## cross-modality diagnosis to understand the underlying mechanisms of lvlm vulnerabilities to visual variations, we propose a systematic analysis framework that examines contributions of each model components to robustness issues. central to this framework is a novel visualization technique that provides straightforward insights into how visual features extracted by vision encoder are processed through the multimodal alignment module and aligned with language embedding space. ## experiments detailed experiment settings are provided in appendix d. ## mitigating robustness issues to determine whether the robustness issues stem from architectural limitations or insufficient training data, we explore improvements through two complementary approaches: first, we conduct controlled experiments on a subset of our test data while maintaining a held-out test set, directly probing the architectural capacity for robust visual understanding. second, we utilize a general visual instruction tuning dataset injected with visual variations to analyze whether a more diverse training distribution can enhance model robustness. we find that a more diverse dataset offers limited improvement in model robustness, likely due to insufficient data volume (as exhaustively covering these variations requires many more variants). moreover, while directly training the model on spatial visual tasks improves its performance on position and direction tasks, it does not enhance robustness against position and orientation variations. as figure 6 suggests, after training, despite improvements in the coordinate task, the path tracing task remains underperforming, due to the lack of sustained visual attention. this reveals that the vulnerability is fundamentally rooted in architectural design choices rather than data limitations. the design of the vision encoder utilizes patchbased tokenization @xcite and positional embeddings, which may lead to information fragmentation due to arbitrary patch partitioning and position sensitivity induced by explicit positional encoding. furthermore, the cascading pipeline architecture amplifies vulnerabilities at each component, underscoring the need for a new architectural approach. although unified architectures currently underperform on visual understanding tasks due to training stability challenges @xcite , this direction remains promising. ## conclusion we present an evaluation framework, v 2 r-bench, designed to assess the robustness of lvlms against visual variations. our results show significant vulnerabilities in existing lvlms and identify their origins as twofold: insufficient multimodal alignment and error accumulation inherent in pipeline model architectures. while synthetic data augmentation @xcite seems as a mitigation strategy, fundamental advancement in the field requires a shift towards native multimodal architectures rather than the current approach of concatenating separate language and vision modalities. we aim to draw greater attention to the importance of lvlm visual robustness and inspire future research toward more robust architectural paradigms.