m BLIP : Efficient Bootstrapping of Multilingual Vision- LLM s (2024)

Abstract: Modular vision-language models (Vision-LLMs) align pretrained image encoders with (frozen) large language models (LLMs) and post-hoc condition LLMs to ‘understand’ the image input. With the abundance of readily available high-quality English image-text data as well as strong monolingual English LLMs, the research focus has been on English-only Vision-LLMs. Multilingual vision-language models are still predominantly obtained via expensive end-to-end pretraining, resulting in comparatively smaller models, trained on limited multilingual image data supplemented with text-only multilingual corpora. We present mBLIP, the first Vision-LLM leveraging multilingual LLMs, which we obtain in a computationally efficient manner on consumer-level hardware. To this end, we re-align an image encoder previously tuned to an English LLM to a new, multilingual LLM using only a few million multilingual training examples derived from a mix of vision-and-language tasks, which we obtain by machine-translating high-quality English data to 95 languages. On the IGLUE benchmark and XM3600, mBLIP yields results competitive with state-of-the-art models and it greatly outperforms strong English-only Vision-LLMs like Llava 1.5. We release our model, code, and train data at https://github.com/gregor-ge/mBLIP .

## introduction the success of model and data scaling in nlp from bert @xcite to more recent large language models (llms) @xcite @xcite has prompted similar endeavors in visionlanguage pretraining from 'small' bert-size models @xcite @xcite trained on a few million image-text pairs to billionparameter models trained with billions of examples * work done during an internship at wünlp @xcite @xcite @xcite . the prohibitive cost of such end-to-end (pre)training, however, has resulted in increased interest in efficient modular methods that leverage existing large language models (llms). these align the output of a pretrained image encoder to the llm's input representation space @xcite @xcite , resulting in a vision-llm. pretraining vision-language models from scratch requires a massive amount of high-quality imagetext data, which is only available in english. because of this, multilingual pretraining of visionlanguage models @xcite @xcite @xcite commonly supplements limited-size multilingual image-text data with multilingual text-only data (the amount of which often surpasses that of imagetext data) to achieve strong results, despite initialization with weights of multilingual text encoders such as xlm-r @xcite . in this work, we recognize modular vision-llm methods as a potential solution to this problem, observing that: (1) once an image encoder is aligned to one llm, it requires significantly less data to re-align it to another llm @xcite and (2) since image encoding is, in principle, language-agnostic, it may be possible to successfully re-align the image encoder to a strong multilingual llm, even if it was initially aligned only with english image-text data. based on these observations, we present mblip, the first massively multilingual modular vision-llm, which we obtain by (re-)aligning an image encoder to a multilingual llm. putting together a range of recent advances in multimodal representation learning, we efficiently bootstrap a massively multilingual vision-llm using only ∼2.5 million images (and without any additional multilingual text-only data), training only 124 million parameters on consumergrade hardware. we achieve this efficiency by: 1) bootstrapping our model from a) an "english" image encoder @xcite , previously aligned to a monolingual english llm and b) a strong instruction-tuned multilingual llm @xcite @xcite ; 2) leveraging recent advances in massively multilingual machine translation @xcite , which we use to translate high-quality english data-both classic captions as well as task instructions @xcite -to 95 languages; and finally 3) coupling parameter-efficient training methods @xcite together with quantization @xcite to enable training on consumer-grade hardware. we extensively evaluate mblip on different multilingual vision-language tasks to confirm the efficacy of our approach: for multilingual image captioning, mblip (with mt0-xl) surpasses (zero-shot) pali-x (a model with 55b parameters, trained with billions of examples) @xcite on the xm3600 @xcite . on the visual reasoning and qa tasks of the iglue benchmark @xcite , mblip matches or surpasses the performance of state-of-the-art models, despite training far fewer parameters on far less pretraining data. we consistently outperform state-of-the-art english vision-llms outside of english, highlighting the multilingual prowess of our model. ## mblip we first briefly describe the modular blip-2 architecture @xcite which we adopt in this work, followed by the description of training tasks and data, which we translate to 95 languages. ## ablation we ablate the various components and design decisions for mblip, namely: 1) using our instruction mix compared to the 'classic' setting used for blip-2 with only image-caption data (using the 2m web capfilt examples as training data) and compared to the instruction mix translated following the mt5 language distribution, 2) using lora on (all) llm matrices to better align the llm to the visual input, and 3) using the warm-start where the projection between q-former and llm is trained briefly in a preliminary stage before the full re-alignment training. we use the zero-shot results on xgqa, xvnli, and xm3600 for evaluation. results are shown in p%/,3 p7 p%/,3 %/220= 8& &&/00 (uqlh 8ql; /l prgho dffxudf\ odqjxdjh eq gh lg nr sw ux ]k (a) xgqa p%/,3 p7 p%/,3 %/220= 8& &&/00 /l prgho dffxudf\ odqjxdjh lg vz wd wu ]k (b) marvl p%/,3 p7 p%/,3 %/220= 8& &&/00 (uqlh 8ql; /l prgho dffxudf\ odqjxdjh du hv iu ux (c) xvnli figure 2: cross-lingual transfer of models fine-tuned on english. the smaller gap of mblip mt0 between highand low-resource languages suggests better transfer capabilities. (cclm 4m from (zeng et al., 2022b) v1 on arxiv.) design & training: for zero-shot xgqa and xvnli, our complete mblip configuration yields the best performance. not using lora (i.e., preventing any updates to the llm) as well as training only on image captioning (compared to the full instruction task mix) both lead to substantially worse performance. moreover, training (with lora) only for image captioning results in a model that does not follow instructions but merely generates captions, making it (zero-shot) useless for other tasks, barring task-specific fine-tuning. for image captioning, both the warm-start and lora fine-tuning boost the performance. unsurprisingly, the realignment on captioning alone yields similar or slightly better captioning performance compared to re-alignment based on the full task mix (i.e., other tasks in the mix do not contribute to captioning ability of mblip). while the task mix brings additional quality captions from mscoco and llava (in addition to the web capfilt examples), the model also has to learn the other tasks; importantly, the ablation shows that including other tasks to re-alignment training does not harm the captioning abilities of the model. language distribution: our translation, proportional to the mc4 language distribution, results in 44% examples in english and, e.g., only 0.003% lao examples. to test how the language distribution affects performance, we adopt another distribution: that of the mt5's pretraining corpus (reduces english to 8% and pushes lao to 0.3%). as expected, this reduces the performance for higherresource languages, and improves it for low(er)- ## conclusion in this work, we presented mblip, the first modular and massively multilingual vision-language model based on multilingual llms. using a small task mix from quality english datasets, made massively multilingual by means of mt, we re-align an english blip-2 model to an instruction-tuned multilingual llm. our approach is highly efficient in compute and data requirements and -using recent engineering advances such as 8-bit quantizationcan be trained in a few days on consumer-grade hardware (e.g., nvidia rtx 3090 cards). we extensively evaluate mblip on multilingual visionlanguage tasks covering image captioning, visual qa, and image understanding to confirm the effi-cacy of our approach. results render mblip comparable or better than state-of-the-art multilingual vision-language models and strong english vision-llms, despite the fact that we train only a fraction of their number of parameters and on far less data. on computer vision and pattern recognition, cvpr 2021, virtual, june 19-25, 2021, pages 4155-4165. computer vision foundation / ieee. deyao zhu, jun chen, xiaoqian shen, xiang li, and mohamed elhoseiny. 2023. minigpt-4: enhancing vision-language understanding with advanced large language models. corr, abs/2304.10592. arxiv: 2304.10592.