COMMIT : Code-Mixing E nglish-Centric Large Language Model for Multilingual Instruction Tuning (2024)

Abstract: Recently, instruction-tuned large language models (LLMs) are showing prominent performance on various tasks, such as question answering. However, the majority of instruction-tuned LLMs are English-centric, which hinders their application to low-resource language QA. In this paper, we propose COde-Mixed Multilingual Instruction Tuning (COMMIT) to adapt English-centric LLM to low-resource language QA. We point out two main causes of English-centricness: imbalance of unlabeled data, and English-centric instruction tuning datasets. To deviate from English-centric instruction tuning, we propose to specialize code-mixing for instruction tuning, which blocks code-mixing in English templates, to leverage the potential of its superiority. To overcome data imbalance, we perform cross-lingual alignment. The majority of cross-lingual alignment works focused on making representations similar, which is not desirable to decoder-based LLMs, such as LLaMA. Therefore, we propose code-mixed continual causal language modeling to align the decoder. COMMIT improves the exact match score of low-resourced language QA by up to 32x. Code is publicly available.

## introduction recently, large language models (llms) have shown prominent performance on various natural language processing tasks @xcite @xcite , such as question answering (qa). moreover, instructiontuning @xcite @xcite further updates the llms to be more efficient. however, the majority of instruction-tuned llms are english-centric. the reasons are two-fold: both the pretraining corpora and the instruction-tuning datasets are english-centric therefore, the performance of qa with lowresourced languages is lacking. resolving two would boost performance, but it is not trivial. first, to alleviate the former problem, the imbalance in unlabeled data, a naïve approach would be pretraining the llm again with balanced data, which is tremendously costly @xcite . alternatively, cross-lingual alignment @xcite can be considered. these methods focus on making the representations of different languages similar, particularly on encoder-based architectures such as mbert @xcite or xlm-r @xcite . however, for decoder-based llms, such as llama @xcite , similar representation across languages may confuse what language should decoder generate, thus such an approach is undesirable. second, to deviate from english instruction tuning datasets, machine translation could be considered. however, assuming high-quality machine translation for low-resource languages can be impractical. moreover, it ignores cross-lingual transferability from high-resource languages. to overcome such shortcomings, in this paper, we propose code-mixed multilingual instruction tuning (commit). first, to efficiently utilize the english instruction tuning dataset, we codemix it using the provided lexicon. since a dictionary is much more available than machine translation @xcite , it is more practical to assume a dictionary. furthermore, code-mixing can leverage cross-lingual alignment @xcite . while promising, we notice more room for improvement than naïvely performing code-mixing to the all part of the data. thus, we specialize codemixing for instruction tuning. inspired by the fact that the english prompt is more effective even in multilingual llms @xcite , we keep the template in english to preserve its strength, without allowing code-mixing. second, to alleviate unlabeled data imbalance, we perform cross-lingual alignment beforehand. to align, we propose continual causal language modeling with code-mixed corpus, relying on the crosslingual alignment ability of the code-mixing @xcite . experiments on mlqa @xcite , and xquad @xcite show the effectiveness of commit-it increases the exact match up to 32x. our code is publicly available. foot_0 2 related works ## proposed method we assume that the given instruction tuning dataset is in english, and a dictionary is provided. this is a realistic scenario, considering the existing instruction tuning datasets @xcite , and the availability of a dictionary @xcite . we also assume that our englishcentric llm covers the majority of target language tokens, which is practical considering language contamination @xcite . ## conclusion we studied adapting english-centric llm to low-resource language qa. we proposed align+commit, aligning and then performing a specialized code-mixing method for instruction tuning. experiments show that each component contributes to improving the performance.