Why do LL a VA Vision-Language Models Reply to Images in E nglish? (2024)

Abstract: We uncover a surprising multilingual bias occurring in a popular class of multimodal vision-language models (VLMs). Including an image in the query to a LLaVA-style VLM significantly increases the likelihood of the model returning an English response, regardless of the language of the query. This paper investigates the causes of this loss with a two-pronged approach that combines extensive ablation of the design space with a mechanistic analysis of the models’ internal representations of image and text inputs. Both approaches indicate that the issue stems in the language modeling component of the LLaVA model. Statistically, we find that switching the language backbone for a bilingual language model has the strongest effect on reducing this error. Mechanistically, we provide compelling evidence that visual inputs are not mapped to a similar space as text ones, and that intervening on intermediary attention layers can reduce this bias. Our findings provide important insights to researchers and engineers seeking to understand the crossover between multimodal and multilingual spaces, and contribute to the goal of developing capable and inclusive VLMs for non-English contexts.

## introduction language fidelity in large language models (llms) refers to whether the model replies in the same language as it was queried in. while seemingly a simple task for humans, models with multilingual capabilities will often bias towards english replies, especially for queries in low-resource languages @xcite . in this work, we identify a surprising parallel pathology in llavastyle vlms @xcite : when prompted with a multimodal query that includes an image, the model is more likely to reply in an incorrect language with respect to the query language. we term this image-induced fidelity loss (ifl). in the first part of the paper, we define this phenomenon and demonstrate the extent of this problem in existing llava-style models. using a collection of 7740 evaluation tasks drawn from three vqa benchmarks and spanning fourteen languages, we show empirically that adding an image to the query to a llava model causes the probability of the response being in the correct language to decrease between 6% and 53%. we analyze the cause of ifl through two complementary approaches. first we conduct a macrolevel analysis by training numerous llava-style models by systematically ablating the design space. this allows us to statistically estimate the impact of each design choice on the model's propensity for generating linguistically misaligned responses. second, we provide a micro-level investigation of the model's intermediary representations. by studying the clustering patterns of vision tokens in relation to tokens from different languages, and directly intervening on hidden states within the language transformer layers, we gain insights into the internal dynamics that give rise to the observed phenomenon. the synthesis of these two levels of analysis -exploring the design space at a statistical level and probing the model's internal representations -provides a comprehensive characterization of the factors influencing cross-lingual response generation in multimodal models. we find strong evidence that ifl occurs due to the language modelling component of the llava model. our statistical analysis indicates that changing the language backbone from an english-main llm to a bilingual llm has the strongest effect on reducing ifl. in our mechanistic analysis we find that visual inputs are not mapped to a similar space as text inputs, and that intervening on intermediary attention layers can reduce ifl. in sum, our contributions are as follows: ## image-induced fidelity loss the phenomenon of interest in this study is the change in fidelity seen when adding an image to the input to a vlm. we refer to this as image-induced fidelity loss. we argue that this phenomenon is surprising, given that visual inputs should be languageagnostic (barring the cultural-linguistic context as-sociated with the image) and therefore orthogonal to the language of the response. ## effects of design choices the llava architecture combines a pretrained vision encoder and language model by using a small multi-layer perceptron (mlp) to project the penultimate hidden states of the vision encoder into the input embedding space of the language model @xcite . this architecture is then fine-tuned with two stages of training. in the first, the vision and language models are frozen and the projection mlp is trained on 558k image-caption pairs. in the second, the vision encoder is kept frozen and the projection mlp and language model are trained on 665k visual instruction-following and examples @xcite ). foot_2 ## implications 6.1 multilingual multimodal understanding our results provide strong evidence in favor of the hypothesis that llava models do not treat visual inputs orthogonally with respect to multilinguality, and this bias occurs in the language backbone of the vlm. in the first part of our analysis, we demonstrate the prevalence of ifl across different llava models and languages. the second part of our analysis provides a controlled comparison of the effects of changing the linguality of the language backbone, the pretraining objective of the vision encoder, and the language of the training data. we find that substituting the monolingual vicuna language backbone for a bilingual yi or leo backbone reduces ifl in chinese and german. in their exploration of the latent representations of multilingual inputs in llama-2 models, @xcite find that the abstract "concept space" of these llms lies closer to english than other languages. correspondingly, if we think of llava as instructing llms to map visual inputs to latent linguistic representations, an explanation for the positive effect of using a bilingual backbone may be that the higher proportion of the target language in the training data mix may reduce the extent to which visual concepts are mapped to spaces closer to english. this interpretation is supported by our investigations of the intermediate representations of visual and text inputs to our suite of ablated llava models. the umap and cka results show that visual inputs are located in a separate space to textual inputs, and that this is remarkably consistent across training configurations. this indicates that llava "instructs" the language backbone to interpret visual inputs, instead of learning to map visual inputs to a linguistic semantic space. further evidence of the multimodal fusion occurring at an intermediary layer of the language model is shown in our mechanistic intervention analysis, where we show that the bias can be partially corrected through intervention on these layers. surprisingly, machine translating the training data language and changing the vision encoder to a language-agnostic one do not mitigate ifl. these null results should not necessarily be interpreted as negative ones; alternative explanations include limitations in performance stemming from the quality of the machine translated data, or there being a specific incompatibility between dinov2 and the language backbones used. on accuracy, we likewise find null results for a systematic effect of our ablations on accuracy. this is less surprising, given that our ablations were designed to affect ifl and not accuracy. additional analyses on accuracy provide further evidence for the lack of a trade-off between accuracy and ifl for the models, languages and benchmarks explored. foot_3 ## future work as the adoption of generative models proliferates, multilingual multimodal models that can service multiple regions will become an attractive solution. however, further efforts can be made to create more effective multilingual vlms. one possible approach is the consideration of language fidelity (image-induced or not) during the training process through a mix of multilingual data and/or language-controlling instructions for in-context learning. other in-depth approaches may include the design of better model architectures that integrate multitmodal representations at a suitable semantic level to prevent language bias. the phenomenon of ifl may also serve as a motivation to better understand the semantic role of visual representations within vlms and how they are interpreted by the base language model. we are particularly interested in understanding and improving the mechanistic intervention. we think this is a promising area of research, and that our simple approach could be improved either by automatic attribute selection conditioned on internal representation or better construction of the attribute. we also would like to study the broad class of fidelity loss in single modality models and to what extent mechanistic interventions are useful in those scenarios. ## conclusion in this work, we systematically examined the phenomenon of image-induced fidelity loss in multimodal language models. our analyses reveal that the addition of visual content to queries can unexpectedly bias the language output of these models, often resulting in responses in an incorrect language. through a combination of empirical evaluations and introspective techniques, we quantified the extent of ifl across various vlm architectures and explored how different design choices impact language fidelity. our findings are further substantiated by micro analyses within the models, illustrating the intricate interactions between image and text embeddings. our introduction of a targeted mechanistic intervention demonstrates a potential method to mitigate ifl, indicating ifl is localized within the llm itself, and suggesting a pathway for future enhancements to vlms. our work contributes to the broader understanding of multimodal systems and offers actionable insights for developing more robust and linguistically accurate ai technologies.