Language-Specific Layer Matters: Efficient Multilingual Enhancement for Large Vision-Language Models (2025)

Abstract: Large vision-language models (LVLMs) have demonstrated exceptional capabilities in understanding visual information with human languages but also exhibit an imbalance in multilingual capabilities. In this work, we delve into the multilingual working pattern of LVLMs and identify a salient correlation between the multilingual understanding ability of LVLMs and language-specific neuron activations in shallow layers. Building on this insight, we introduce PLAST, a training recipe that achieves efficient multilingual enhancement for LVLMs by Precise LAnguage Specific layers fine-Tuning. PLAST first identifies layers involved in multilingual understanding by monitoring language-specific neuron activations. These layers are then precisely fine-tuned with question-translation pairs to achieve multilingual alignment. Our empirical results on MMBench and MMMB demonstrate that PLAST effectively improves the multilingual capabilities of LVLMs and achieves significant efficiency with only 14% of the parameters tuned. Further analysis reveals that PLAST facilitates the language-specific visual information engagement in shallow layers.

## introduction large vision-language models (lvlms) have made remarkable progress in understanding visual information with human languages, achieving impressive performance in multimodal tasks like visual question answering (vqa) @xcite . however, they still struggle in multilingual scenarios because the training corpora are predominantly english-centric @xcite . this imbalance poses significant challenges for their real-world applications across the globe. figure 1 : the logit lens @xcite applies the language modeling head (llava-1.5-7b in our case) to intermediate layer embeddings, producing one nexttoken distribution per position (x-axis) and per layer (y-axis). the blue-to-red gradient indicates entropy levels, ranging from low to high. full visualizations for all languages are shown in figure 9 -13. recent efforts to enhance the multilingual abilities of lvlms primarily focus on two aspects. one line of research @xcite @xcite involves incorporating translation, either implicitly or explicitly, into the design of prompts, encouraging the model to first comprehend questions in english and then solve them step by step. however, the inadequate multilingual understanding capability of lvlms often triggers cascading errors, leading to inferior performance. another approach expands this idea by adopting a translate-then-train strategy @xcite , where english training data are first translated into multiple languages via machine translation, followed by fine-tuning for multilingual instruction alignment. despite effectiveness, these approaches rely on accurately translating large-scale complex multimodal data and full-parameter fine-tuning, which limits their applicability in data-deficiency and resource-constrained scenarios. when considering multilingual models, one might think of capturing language-specific representation in certain parts of these models. inspired by recent studies @xcite @xcite on the multilingual mechanisms in large language models (llms), which have verified that the multilingual working pattern of llms can be divided into three stages: language understanding, task-solving, and language converting. moreover, during this process, the activation of language-specific neurons @xcite can serve as a crucial indicator for distinguishing each working stage. we further investigate whether this pattern persists in lvlms ( §2). our pilot study reveals that lvlms also follow this three-stage process for handling multilingualism, with lower-level layers more engaged in learning language-specific representations. as shown in figure 1 , the chinese query undergoes an explicit process of language understanding (@xmath0. this motivates us to unlock the multilingual capabilities more efficiently by precisely enhancing language-specific representations. to this end, we propose plast, a training recipe designed to achieve efficient multilingual enhancement for lvlms, as outlined in figure 2 . specifically, starting with a visual instruction-tuned model, we first identify decoder layers mostly involved in language understanding by monitoring the activation of language-specific neurons using multilingual image-text pairs ( §3.1). more concretely, we calculate the mean squared deviation (msd) of the numbers of neurons activated by different languages and select the layers with higher msd scores. subsequently, plast precisely fine-tunes only these identified layers using a small amount of image-question translation pairs ( §3.2). this improves its multilingual understanding ability by precisely enhancing the language-specific representation while not affecting task-solving abilities at higher layers. to evaluate the effectiveness of plast, we conduct extensive experiments on two multilingual vqa benchmarks, mmbench @xcite and mmmb @xcite across three lvlms. experimental results demonstrate the effectiveness of plast in improving multilingual performance, with average gains of 8.0% and 4.0% on mm-bench and mmmb. furthermore, compared to fullparameter fine-tuning approaches, plast achieves superior efficiency with only 15.6% and 12.5% of ... projection vision encoder word embedding ar tr ru pt zh translation instructions english response ... ... ... the parameters within 7b and 13b models tuned. further analysis reveals that plast can be generalized to low-resource and complex visual reasoning tasks, facilitating the language-specific visual information engagement in shallow layers. ## preliminaries recent studies @xcite have underscored the pivotal role of languagespecific neurons within llms for multilingual processing. drawing from these findings, this section starts with a preliminary analysis aimed at exploring whether neurons within lvlms exhibit language-specific behavior and understanding their significance for multilingual visual understanding. ## methodology building upon our findings, we propose a training method designed to achieve efficient multilingual enhancement of lvlms, which includes two processes: (1) identifying layers responsible for multi-lingual understanding; (2) precisely fine-tuning the selected layers to enhance multilingual capabilities. ## experimental results we present the main results of llava-1.5-7b/13b and qwen-vl-chat on mmbench and mmmb benchmarks in lvlms exhibit significant performance imbalances in multilingual scenarios. as shown in plast enhances multilingual performance and shows robust cross-model generalization. it can be observed that plast consistently outperforms all baselines on both mmbench and mmmb, achieving state-of-the-art performance. specifically, it achieves an average improvement of 8.0% and 4.0% on mmbench and mmmb respectively across the three evaluation models. these results highlight the effectiveness of plast in enhancing the multilingual capabilities of lvlms. remarkably, plast leads to substantial gains for moderately low-resource languages, with improvements of 12.0% (41.7 → 46.7) in arabic and 16.2% (43.1 → 50.1) in turkish on the mmmb for the llava-1.5-7b model. this suggests that plast, by leveraging only a modest amount of question-translation data, effectively bridges language gaps to some extent. in addition, among all evaluated models of varying architecture and scales, plast consistently delivers improvements, highlighting its strong generalizability across different models. plast demonstrates superior efficiency. unlike training-based baselines that require extensive multilingual image-text data for full-parameter finetuning, plast achieves remarkable efficiency by utilizing less than half of the training data (see ## ablation study and further analysis we conduct extensive ablation studies and analysis to verify the effectiveness of plast. for more analysis, please refer to appendix e. ## related work with the acceleration of globalization, multilingual lvlms @xcite have gained great attention for their ability to handle multiple languages comprehensively. however, due to the training corpora being mainly englishcentric, these models perform significantly better in english than in other languages, leading to an imbalanced performance in multilingual scenarios. numerous approaches have been proposed to enhance the multilingual abilities of lvlms, primarily categorized into prompting-based and trainingbased methods. prompting-based methods lever-age models' inherent understanding capabilities to translate non-english questions into english before generating final responses. for instance, @xcite ; @xcite employ either implicit or explicit prompting to guide the model to solve tasks step-by-step in english. conversely, trainingbased methods focus on synthesizing multilingual training data via machine translation for multilingual visual instruction tuning @xcite @xcite . for example, @xcite constructs multilingual image-text pairs to train additional mixture-of-experts modules to convert english-biased features to languagespecific features for multilingual alignment. unlike these approaches, our work focuses on achieving efficient multilingual capability enhancement. rather than relying on extensive translated multilingual image-text pairs and full-parameter fine-tuning, plast precisely identifies the specific layers responsible for multilingual understanding, enabling more efficient multilingual alignment while maintaining superior performance. ## conclusion this work proposes plast, a novel training recipe for efficient multilingual enhancement of lvlms. plast first identifies layers predominantly engaged in multilingual understanding by monitoring language-specific neuron activations. these critical layers are then precisely fine-tuned with translation pairs to achieve multilingual alignment. extensive evaluations show that plast significantly enhances lvlm multilingual performance. moreover, compared to full-parameter tuning methods, plast achieves superior performance and efficiency with only 14% of parameters tuned. further analysis confirms its effectiveness across both low-resource languages and more complex visual reasoning tasks, demonstrating its broad applicability in diverse multilingual scenarios.