Babel- I mage N et: Massively Multilingual Evaluation of Vision-and-Language Representations (2024)

Abstract: Vision-and-language (VL) models with separate encoders for each modality (e.g., CLIP) have become the go-to models for zero-shot image classification and image-text retrieval. They are, however, mostly evaluated in English as multilingual benchmarks are limited in availability. We introduce Babel-ImageNet, a massively multilingual benchmark that offers (partial) translations of ImageNet labels to 100 languages, built without machine translation or manual annotation. We instead automatically obtain reliable translations by linking them – via shared WordNet synsets – to BabelNet, a massively multilingual lexico-semantic network. We evaluate 11 public multilingual CLIP models on zero-shot image classification (ZS-IC) on our benchmark, demonstrating a significant gap between English ImageNet performance and that of high-resource languages (e.g., German or Chinese), and an even bigger gap for low-resource languages (e.g., Sinhala or Lao). Crucially, we show that the models’ ZS-IC performance highly correlates with their performance in image-text retrieval, validating the use of Babel-imageNet to evaluate multilingual models for the vast majority of languages without gold image-text data. Finally, we show that the performance of multilingual CLIP can be drastically improved for low-resource languages with parameter-efficient language-specific training. We make our code and data publicly available: https://github.com/gregor-ge/Babel-ImageNet

## introduction clip models @xcite @xcite have become widely used visionand-language (vl) models, owing popularity to efficient inference based on separate yet semantically aligned encoders for the two modalities. their bi-encoder architecture makes them ideal for efficient image-text retrieval @xcite and zero-shot image classifica-tion @xcite , or to produce downstream features for supervised tasks such as image generation @xcite or cross-modal reasoning @xcite . motivated by the observation that performance on imagenet classification translates well to performance in many other image tasks @xcite , clip models are typically evaluated on zero-shot image classification (zs-ic), i.e., by comparing the representation of an image with text representations of class labels, whereby imagenet @xcite is the most prominent benchmark. with imagenet class labels available only in english, this supports only evaluation of monolingual english models (i.e., models trained with english captions only). although most clip models are trained on english-only imagecaption data, some effort has been put into creating multilingual and monolingual non-english models by (1) training them from scratch @xcite @xcite @xcite or (2) distilling them from english models @xcite @xcite , typically using parallel data as supervision. despite attempts to translate imagenet labels to other languages @xcite , the language coverage remains very limited. because of this, multilingual clip models have mainly been benchmarked on image-text retrieval datasets (aggarwal and kale, 2020; @xcite , which cover only limited sets of mid-to-high resource languages. creating massively multilingual datasets for vl tasks (e.g., image-text retrieval) is prohibitively expensive. existing efforts @xcite @xcite either hire native speakers to write image captions in target languages or resort to machine translation (mt) of english data, followed by manual post-editing by native speakers. the mt approach (the cheaper of the two), is, we argue, still too expensive for low-resource languages because mt models are less accurate when translating to those languages, which implies a bigger post-editing effort for bilingual annotators, native in the lowresource language and fluent in english; in addition, such annotators are more difficult to find for lowresource than for high-resource target languages. in this work, we thus seek to create a robust massively multilingual benchmark for evaluating the quality of representation spaces of multilingual vl models, without resorting to mt or requiring any manual annotation effort. to be useful, such a benchmark needs to satisfy a crucial requirement: models' performance across languages must be indicative of their performance for the same languages in tasks such as image-text retrieval, for which creating massively multilingual (gold-standard) evaluation datasets is too expensive. contributions. with this in mind, we create babel-imagenet, a massively multilingual dataset for zero-shot image classification that offers (partial) translations of the 1000 imagenet classes to 100 languages. to obtain robust translations of ima-genet labels in other languages, we leverage the connection between imagenet classes, which are derived from wordnet @xcite synsets, and babelnet @xcite , a massively multilingual lexico-semantic network, also (in part) derived from wordnet. relying on the multilingual babelnet synsets (and wordnet synset identifiers of imagenet classes) to pivot between languages, we avoid problems known to occur with machine translation of short phrases without context, e.g., due to polysemy foot_0 . exploiting babelnet allows us to automatically obtain labels for imagenet concepts in many languages, removing the need for mt and manual annotation. we evaluate 11 different multilingual clip models on babel-imagenet, observing that all of them exhibit poor performance for low-resource languages. crucially, we validate that babel-imagenet is a meaningful benchmark for measuring the quality of multilingual vl representations by comparing models' performance on babel-imagenet against their performance on established multilingual image-text retrieval datasets. babel-imagenet thus allows us to evaluate models in languages not covered by those datasets and it additionally expands the retrieval-focused evaluation with the zs-ic task in languages included in the established datasets. finally, we propose a computationally efficient approach for improving multilingual clip models for low-resource languages. this modular language specialization approach yields large performance gains (>20% for some of the low-resource languages). ## benchmarking clip models models. we present the clip variants we benchmark on babel-imagenet (overview in all use vision transformers (vit) @xcite albeit of different sizes and with differently sized input patches (e.g., b-32 = base transformer with 32×32-pixel patches). openai: the original clip @xcite , trained fully from scratch on 400m english image-caption pairs. openclip: openclip @xcite ) aims to replicate the openai models using the public laion datasets @xcite . two multilingual models have been trained: the b-32 model with the text encoder initialized with the weights of xlm-r-base and the h-14 model initialized with xlm-r-large. the b-32 variant is trained with the original contrastive clip objective, whereas the h-14 model was trained via locked image tuning (lit, @xcite ) in which the pretrained image encoder of the english h-14 openclip model is frozen and only the parameters of the text encoder are updated. sentencetransformer (st): one of the first distillation-based multilingual clip-like models using the approach of reimers and gurevych (2020), 6 with over 50m en-x parallel sentences (x being one of 49 other languages) as supervision. they distill a (distilled) mbert student @xcite from the english openai b-32 teacher. m-clip: multilingual-clip (m-clip) (carlsson et al., 2022) is a model distilled using mbert and openai b-32 with translations of 3m english image captions to 69 languages as parallel supervision. post-publication, they released a set of models 7 with xlm-r-large as the student and ope-nai b-32, l-14, and openclip b-16+ as teachers. these models were trained on 7m captions by @xcite , translated to 48 languages. ## clip-vit-b-32-multilingual-v1 7 github.com/freddefrallan/multilingual-clip altclip: this model by @xcite distills an xlm-r-large student with openai l-14 as teacher, targeting 9 languages and using as training data a mix of machine-translated captions, multilingual captions sampled from laion5b, and aligned english-x sentence pairs. after distillation, the authors additionally fine-tune the model via lit using selected image-text pairs from laion5b in the 9 target languages. siglip: the multilingual msiglip @xcite is a b-16-size clip model trained from scratch on google's multilingual webli dataset @xcite . they replace the softmax function used in the original clip with a sigmoid for more compute-efficient training. ## validating babel-imagenet we perform two additional analyses that establish the validity of babel-imagenet as a benchmark: ## improving multilingual clip for low-resource languages figure 3: r@1 text-to-image retrieval results on three datasets plotted against babel-imagenet performance (each dot denotes the performance of one model for one language) together with a linear regression estimate (95% ci). setup. we train language-specific adapters on top of (a) openclip b-32 model (trained from scratch) and (b) m-clip xlmr-l b-32 (obtained via distillation). we experiment with three training objectives: english-target language distillation with (i) mse and (ii) contrastive loss, and (iii) lit on image-caption pairs. the former two require parallel data, whereas the latter requires images paired with target-language captions. for comparability between languages, we sample 100k captions (with corresponding images) from the synthetic dataset provided by @xcite and translate them automatically to all target languages with nllb. we perform adapter-based specialization for 16 languages. one run (i.e., one modellanguage-objective combination) takes 3h on a sin-gle nvidia rtx 3090 card (see §b.3 for details). results. ## conclusion we introduced babel-imagenet, the first massively multilingual translation of the imagenet classes to 100 languages. we leverage the wordnet synsets as the link between imagenet and babelnet to obtain high-quality translations without relying on mt or human annotators. using babel-imagenet, we carried out the most comprehensively multilingual comparative evaluation of 11 public clip models on zero-shot image classification, demonstrating that all models fail for low(er)-resource languages. crucially, we validate our benchmark by showing that models' text-to-image retrieval performance (on three datasets) strongly correlates with their zs-ic performance on babel-imagenet for the corresponding languages. finally, we proposed a parameter-efficient fine-tuning procedure that drastically improves the performance of multilingual clip models for low-resource languages. the wide range of languages encompassed by our benchmark reveals that the theoretical "multilinguality" of clip models is practically very limited and points to the need for methods that derive robust vl encoders with much stronger performance especially for low-resource languages: e.g., better distillation procedures that retain more of the impressive performance of english clip.