Peacock: A Family of A rabic Multimodal Large Language Models and Benchmarks (2024)

Abstract: Multimodal large language models (MLLMs) have proven effective in a wide range of tasks that require complex reasoning and linguistic comprehension. However, due to a lack of high-quality multimodal resources in languages other than English, the success of MLLMs remains relatively limited to English-based settings. This poses significant challenges in developing comparable models for other languages, even those with large speaker populations, such as Arabic. To alleviate this challenge, we introduce a comprehensive family of Arabic MLLMs, dubbed *Peacock*, with strong vision and language capabilities. Through comprehensive qualitative and quantitative analysis, we demonstrate the solid performance of our models on various visual reasoning tasks and further show their emerging dialectal potential. Additionally, we introduce *Henna*, a new benchmark specifically designed for assessing MLLMs on aspects related to Arabic culture, setting the first stone for culturally-aware Arabic MLLMs. The GitHub repository for the *Peacock* project is available at [https://github.com/UBC-NLP/peacock](https://github.com/UBC-NLP/peacock).

## introduction empowered by progress in large language models (llms) and foundation models of other modalities, multimodal large language models (mllms) now have a remarkable understanding @xcite @xcite @xcite . for example, they can handle various complex reasoning tasks spanning from visual question answering to understanding sarcastic comics @xcite . these capabilities, however, are mostly seen in models serving the english language. this leaves behind the majority of the world's languages, furthering an already acute technological divide. we alleviate this challenge for arabic, a diverse collection of languages and dialects with a native population of more than 400 million speakers. more concretely, we draw inspiration from english counterparts @xcite to present a robust family of arabic mllms with powerful vision and language capabilities. our models adopt the approach of integrating an image encoder with an arabic text decoder. in our experimental setup, we explore two popular directions for aligning the vision and the language components: one involves employing a fully connected layer as a projection head on top of the vision encoder @xcite , while the other utilizes a q-former transformer @xcite . all models are trained in two stages, a pretraining stage and an instruction fine-tuning stage. for the first stage, we curate high-quality pretraining data from publicly available english datasets. we translate these datasets into arabic and apply a carefully designed pipeline to ensure the quality of our training data. similarly, we curate and translate an instruction fine-tuning dataset which is essential for achieving reasoning and conversational capabilities. we showcase the performance of our models across different tasks such as visual question answering (vqa) and visual reasoning. our models perform much better than a multilingual baseline mblip @xcite on different tasks and datasets, and we set the first comprehensive arabic vision-language benchmark to facilitate future work in this area. finally, we demonstrate the promising capabilities of our peacock models in interacting in dialectal arabic by conducting a case study on the egyptian dialect. when fine-tuned on a small set of egyptian dialect data, our models exhibit an interesting level of proficiency in their dialectal responses when prompted in the same dialect. we hope this acts as a spark for future works in dialectal arabic vision language models. to summarize, our contributions in this paper are as follows: (1) we introduce a suite of arabic mllms, dubbed peacock, capable of instruction following and visual reasoning, in addition to their intriguing dialectal affinity. for developing peacock, we use existing vision and language models. we also offer a new language model, arallama, based on llama2-7b @xcite . (2) we introduce a diverse collection of arabic translated datasets carefully curated for the training and evaluation of arabic mllms. (3) we adapt the popular llava @xcite benchmark and seed-bench @xcite for arabic mllms evaluation. (4) we present henna, a benchmark designed to measure model capabilities in interpreting images related to arabic culture. the rest of this paper is organized as follows: in section 2, we provide an overview of related work. section 3 introduces peacock, our family of mllms. in section 4, we describe our evaluation strategies and benchmarks. in section 5, we present our experiments, human evaluation, and a comprehensive analysis of our models. we conclude in section 6. ## conclusion in this work, we present the family of peacock models. peacock demonstrates significant advancements in arabic mllms, showcasing remarkable abilities in interpreting visual data in arabic language. these models bridge the gap in multimodal understanding capabilities for arabic and egyptian dialects by introducing a suite of models, with various reasoning skills, accompanied by a diverse collection of datasets and benchmarks carefully prepared. this includes our henna benchmark, designed to assess mllm tasks focused on the arabic culture. the development of peacock sets strong baselines and a new benchmark for future work in arabic mllms, highlighting the importance of high-quality data processing and the selection of language models for multimodal task performance.