Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability (2025)

Abstract: Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in processing both visual and textual information. However, the critical challenge of alignment between visual and textual representations is not fully understood. This survey presents a comprehensive examination of alignment and misalignment in LVLMs through an explainability lens. We first examine the fundamentals of alignment, exploring its representational and behavioral aspects, training methodologies, and theoretical foundations. We then analyze misalignment phenomena across three semantic levels: object, attribute, and relational misalignment. Our investigation reveals that misalignment emerges from challenges at multiple levels: the data level, the model level, and the inference level. We provide a comprehensive review of existing mitigation strategies, categorizing them into parameter-frozen and parameter-tuning approaches. Finally, we outline promising future research directions, emphasizing the need for standardized evaluation protocols and in-depth explainability studies.

## introduction recent large vision-language models (lvlms) have achieved significant progress in multimodal understanding. models such as gpt-4v (openai), gemini @xcite , llava @xcite , claude-3.5-sonnet @xcite , qwen2-vl @xcite , and llama 3.2 @xcite demonstrate remarkable capabilities in tasks like image captioning and visual question answering, not only processing visual and textual information independently but also reasoning across these modalities. these advances are built upon two key foundations: large language models (llms) and vision encoders. llms such as gpt-3.5 (brown, 2020), llama (touvron * corresponding author. et al., 2023a), llama 2 @xcite , and qwen @xcite ) transformed natural language processing, while visual understanding models like vision transformer (vit) @xcite and vision-language models like clip @xcite have advanced the ability to process visual information and create aligned visual-textual representations respectively. the key challenge in developing effective lvlms lies in achieving proper alignment between visual and textual representations @xcite . the predominant approach involves using representation alignment techniques, where visual features from an image encoder and textual representations from an llm are mapped into a shared embedding space, typically matching the llm's embedding dimensions @xcite @xcite . once both modalities are mapped into this shared space, alignment can be achieved through various training objectives and architectural designs that encourage the model to understand and reason about cross-modal relationships. this method has gained popularity due to its straightforward approach and generalizability across different model architectures. however, the current understanding of alignment mechanisms remains limited. a critical challenge lies in misalignment phenomena, which manifest in various forms. for instance, when shown an image of a green apple, the model might fail to recognize the apple altogether (object misalignment), incorrectly describe it as red (attribute misalignment), or generate incorrect relationships like "the apple is floating in the air" when it's sitting on a table (relational misalignment). these misalignments lead to reliability issues @xcite @xcite , where models generate textual outputs that are inconsistent with the visual input. understanding and addressing these misalignment issues is crucial for developing more reliable and trustworthy lvlms, as they directly impact the models' ability to generate accurate and consistent multimodal outputs. ## alignment of lvlms in this section, we examine alignment in lvlms across four essential dimensions. first, we define the concept of alignment in lvlms. second, we detail the procedural stages through which alignment is achieved in practice. third, we explore the theoretical foundations that make alignment possible between visual and textual modalities. finally, we discuss methods for measuring and evaluating alignment in lvlms in appendix c. ## misalignment of lvlms after introducing the alignment of lvlms, we now examine a critical challenge facing these models: their tendency to generate outputs that diverge from the visual input. despite significant advances in alignment techniques, lvlms still frequently exhibit misalignment between their visual and textual inputs. in this section, we provide a comprehensive analysis of misalignment phenomena in lvlms, beginning with a definition and taxonomy of different types of misalignment (see figure 1 ), followed by an examination of their underlying causes. ## mitigation methods building upon our analysis of misalignment causes in lvlms, we now examine strategies for mitigating these challenges (see figure 2 ). these mitigation approaches can be categorized into two groups: parameter-tuning alignment methods and parameter-frozen alignment methods. parametertuning alignment involves modifying specific components within the lvlm architecture to reduce misalignment through targeted parameter updates. in contrast, parameter-frozen alignment methods address misalignment while maintaining the lvlm's original parameters unchanged, offering solutions that preserve the model's structure while improving its cross-modal alignment capabilities. ## conclusions in this paper, we systematically survey alignment and misalignment in lvlms through an explainability lens. we show that achieving proper alignment involves complex interactions between data quality, model architecture, and inference procedures. we categorize misalignment into object, attribute, and relational levels, providing a clear framework to understand these challenges and develop targeted solutions. our review of mitigation strategies highlights approaches ranging from computationally intensive parameter-tuning methods to practical parameter-frozen solutions, each with trade-offs in effectiveness and feasibility. lastly, we outline key future directions for building robust vision-language systems (see appendix h).