X - LL a VA : Optimizing Bilingual Large Vision-Language Alignment (2024)

Abstract: The impressive development of large language models (LLMs) is expanding into the realm of large multimodal models (LMMs), which incorporate multiple types of data beyond text. However, the nature of multimodal models leads to significant expenses in the creation of training data. Furthermore, constructing multilingual data for LMMs presents its own set of challenges due to language diversity and complexity. Therefore, in this study, we propose two cost-effective methods to solve this problem: (1) vocabulary expansion and pretraining of multilingual LLM for specific languages, and (2) automatic and elaborate construction of multimodal datasets using GPT4-V. Based on these methods, we constructed a 91K English-Korean-Chinese multilingual, multimodal training dataset. Additionally, we developed a bilingual multimodal model that exhibits excellent performance in both Korean and English, surpassing existing approaches.

## introduction recently, large multimodal models (lmms) have evolved to respond in alignment with human intent through visual instruction-following (vif) @xcite @xcite @xcite . in llava1.0 @xcite , a method was proposed to automatically construct a vif dataset using gpt4, which demonstrated excellent performance in visual question answering (vqa). however, there are two main limitations to the data generated in llava1.0: first, it was constructed using a textonly version of gpt4, which does not accept images as input; and second, it targeted only english. subsequently, llava1.5 @xcite ) incorporated the multilingual instruction dataset sharegpt (sha), demonstrating its potential in multilingual processing. however, sharegpt uses an instruction following (if) @xcite dataset for llms, still suffers from a lack of vision information. to address this issue, sharegpt4v @xcite , a vif dataset created using gpt4-v, which accepts image information as input, was released. sharegpt4v is also limited because it consists only of english questionanswering, posing a constraint in aligning multiple languages to acquire multilingual information. in this context, we propose constructing a multilingual vif dataset based on object relational information and a multilingual lmm that efficiently utilizes this dataset. the proposed multilingual vif dataset was composed of 23,496 question-andanswer pairs centered around objects, locations, atmospheres, and conversations to ensure the diversity of expressions. the target languages were selected considering linguistic diversity by choosing english, chinese, and korean, which belong to different language families @xcite . we also propose the development of a multilingual lmm, x-llava, utilizing the proposed data. x-llava is a model that enhances llava1.5, by applying the following three enhancement methods: (1) vocabulary expansion for target language, (2) pretraining for connecting knowledge across multiple languages, and (3) multilingual vif. first, bilingual-based vocabulary expansion involves adding words to a pretrained language model to strengthen the relatively limited vocabulary of korean compared to english @xcite . second, additional pretraining was conducted to link the english and korean knowledge. third, we conducted multilingual training using the proposed vif dataset. experimental results showed that the x-llava model demonstrated an average improvement of approximately 5.2% in three korean quantitative evaluations compared to the previously proposed kollava model. in addition, it achieved the highest performance in two out of five english quantitative evaluations. in qualitative evaluations, preference assessments using gpt4-v demonstrated that our model generated responses in both english and korean that were 19-93% superior to existing models. through qualitative analysis, we highlighted that the proposed bilingual training enhanced specific language vocabulary, leading to better performance in writing evaluations. the contributions of this study can be summarized as follows: finally, we emphasize that the 91k datasets and models constructed in this study can be implemented with relatively small resources, costing approximately @xmath03,200.