MVL - SIB : A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching (2025)

Abstract: Existing multilingual vision-language (VL) benchmarks often only cover a handful of languages. Consequently, evaluations of large vision-language models (LVLMs) predominantly target high-resource languages, underscoring the need for evaluation data for low-resource languages. To address this limitation, we introduce MVL-SIB, a massively multilingual vision-language benchmark that evaluates both cross-modal and text-only topical matching across 205 languages – over 100 more than the most multilingual existing VL benchmarks encompass. We then benchmark a range of of open-weight LVLMs together with GPT-4o(-mini) on MVL-SIB. Our results reveal that LVLMs struggle in cross-modal topic matching in lower-resource languages, performing no better than chance on languages like N’Koo. Our analysis further reveals that VL support in LVLMs declines disproportionately relative to textual support for lower-resource languages, as evidenced by comparison of cross-modal and text-only topical matching performance. We further observe that open-weight LVLMs do not benefit from representing a topic with more than one image, suggesting that these models are not yet fully effective at handling multi-image tasks. By correlating performance on MVL-SIB with other multilingual VL benchmarks, we highlight that MVL-SIB serves as a comprehensive probe of multilingual VL understanding in LVLMs.

## introduction large vision-language models (lvlms) extend large language models (llms) to take images as inputs, leveraging their advanced language capabilities for vision-language (vl) tasks like image captioning and visual question answering (vqa). however, lvlms are typically trained mainly on ## experimental setup models. we test state-of-the-art lvlms qwen2-vl (wang et al., 2024), internvl 2.5 @xcite , centurio-qwen @xcite , and gpt-4o(-mini) across available sizes. 4 smaller lvlms are evaluated on all languages, while larger ones (26b+) are tested on subsets (cf. §5.3). for cross-modal topic matching, we also evaluate on msiglip-base @xcite . trained explicitly for semantic similarity on multilingual imagecaption pairs, the vit represents a strong baseline. 5 its prediction denotes the choice that has the highest average cosine similarity to the k references. image preprocessing. we downsample the images to 640×480 pixels, as the tasks rely on higher-level visual cues for topically associating images and texts rather than finer image details. this can significantly reduce the number of visual tokens input to lvlms, enabling more efficient inference. ## results and discussion we categorize the languages in mvl-sib based on their 'resourceness'. to do so, we reorganize the language groups from joshi et al. (2020) into four tiers. we first rank the tiers w.r.t. wikipedia size and then merge (i) the two highest-resource tiers and (ii) the two lowest-resource tiers. 6 this both ## conclusion we present the massively multilingual visionlanguage benchmark mvl-sib for cross-modal (and text-only topic matching) in 205 languages that offers key advantages over prior multilingual vl benchmarks. notably, it covers over 100 additional languages without relying on machine translation. mvl-sib allows for a clear separation between textual language support and visionlanguage support in lvlms by comparing performance on mirrored cross-modal and text-only tasks. moreover, it allows us to study how lvlms handle single-image versus multi-image formulations of cross-modal topic matching by varying the number of images provided. in our comparative evaluation of state-of-the-art lvlms on mvl-sib, we find that model performance is strongly correlated with both model size and the volume of available pre-training data for each language. however, all lvlms experience a dramatic performance drop on the lowest-resource languages. our analysis further reveals that vision-language support deteriorates disproportionately relative to language support, highlighting the need to incorporate lowresource languages into vl training. moreover, providing multiple images does not benefit openweight lvlms in cross-modal topic matching, suggesting that lvlms are not yet fully effective in multi-image tasks. lastly, we validate that mvl-sib correlates well with existing multilingual vl benchmarks, underscoring its reliability as a source of evaluation data for 205 languages.