Multilingual Large Language Models Are Not (Yet) Code-Switchers (2023)

Abstract: Multilingual Large Language Models (LLMs) have recently shown great capabilities in a wide range of tasks, exhibiting state-of-the-art performance through zero-shot or few-shot prompting methods. While there have been extensive studies on their abilities in monolingual tasks, the investigation of their potential in the context of code-switching (CSW), the practice of alternating languages within an utterance, remains relatively uncharted. In this paper, we provide a comprehensive empirical analysis of various multilingual LLMs, benchmarking their performance across four tasks: sentiment analysis, machine translation, summarization and word-level language identification. Our results indicate that despite multilingual LLMs exhibiting promising outcomes in certain tasks using zero or few-shot prompting, they still underperform in comparison to fine-tuned models of much smaller scales. We argue that current “multilingualism’ in LLMs does not inherently imply proficiency with code-switching texts, calling for future research to bridge this discrepancy.

## introduction large language models (llms) have shown their potential in the context of zero-shot and few-shot prompting @xcite @xcite . the successes of these llms have also been effective in multilingual settings @xcite @xcite where models are specifically trained to learn individual languages, proven to be highly beneficial for monolingual tasks. however, in multilingual communities, people do not confine themselves to speaking only a single language; instead, they use two or more languages interchangeably during a conversation -a phenomenon known as code-switching (csw) @xcite . it allows individuals to communicate cultural-specific concepts more * equal contribution. effectively, signaling their group identity and reinforcing their social connection @xcite . yet, existing multilingual llms are not specifically trained with objectives for managing csw scenarios. hence, assessing the capabilities of the current multilingual llms in processing csw texts is essential to the development of multilingual language models that are fully compatible with code-switching. the main challenge of developing multilingual llms optimized for code-switching lies in data scarcity. given the highly colloquial characteristic of code-switching @xcite , existing resources dedicated to csw are rare and collection at large-scale requires considerable annotation efforts. to mitigate such deficiency, @xcite investigate the possibility of employing multilingual llms to generate high-quality synthetic csw texts. the study revealed that, hosted llms, such as instructgpt @xcite and chatgpt foot_0 outperform public models like bloomz @xcite and flan-t5-xxl @xcite in generating naturalsounding csw texts. however, the quality of the generated text by these hosted llms is mostly confined to singlish and significantly declines when prompted for other languages. despite the preliminary promising results, the generation of highquality csw texts still remains challenging. this observation motivates us to probe from a different perspective -can existing multilingual llms effectively understand csw? there have been previous studies on evaluating multilingual lms in csw scenarios @xcite , where codeswitched texts are simulated by replacing words from parallel corpora. @xcite also assesses models' effectiveness by experimenting with word embeddings constructed from different methods. these works are mainly built upon small sentiment analysis summarization positive qué mejor que pasar valentine 's thirdwheeleando , ricura , @xmath0: machine translation sentiment analysis acha tho is movie kis baare me hein? translate the text above from hinglish to english language identification for each token, identify its language (lang1: english, lang2: hindi, other) using [ word | tag ]. hello koi mere se frndship krlo mere se hor singel nai raha @xmath1: chloene bataya tum humare saath nahi aa rahe! carter: mera ek family reunion around that time...ivy: why? ... carter: plan toh yahi hai at least ivy: take care! multilingual llms alright that is fine. what is the movie? [hello|lang1][koi|lang2][mere| lang2][se|lang2][frndship|lang1] [krlo|lang2][mere|lang2][se|lang2] [hor|lang2][singel|lang1][nai| lang2][raha|lang2][jata|lang2] carter is not joining ivy and chloe due to a family reunion. carter's grandfather is very ill. summarization figure 1: illustration of tasks included in our benchmark study. bert-based models and are restricted by either the number of languages or tasks investigated. given the recent success of prompting methods on multilingual llms and the effects of scaling, this paper presents a more comprehensive empirical analysis of models' code-switching abilities, including a variety of languages, task types, model architectures, model sizes and prompting methods. our results suggest that the scaling law is applicable to multilingual llms across diverse csw tasks and model architectures. however, fine-tuned smaller-scale models substantially outperform the largest multilingual llm with prompting methods. in addition, while hosted llms achieve scores comparable to our fine-tuned models, such performance remains uninterpretable due to their closedness. we argue that existing multilingual llms exhibit limited proficiency in code-switching contexts, highlighting future research opportunities to transform them into true polyglots. ## results and discussion overall results figure 2 presents the results of various multilingual llms for the four csw tasks. 3 in general, we observe a scaling pattern when prompting multilingual llms across tasks. nevertheless, the performance of these models significantly falls short when compared to that of substantially smaller fine-tuned models. therefore, adopting a fine-tuned model is a more practical approach for dealing with csw tasks, especially in scenarios with constrained computational resources. for chatgpt, it demonstrates comparable performance to fine-tuned models across all tasks and datasets, except for the english to hinglish mt task. this exception may stem from the challenges in generating code-switched texts as outlined in previous research @xcite . for the remaining tasks, chatgpt notably outperforms publicly available multilingual llms. such discrepancy may be attributed to the rlhf objective in its pretraining process, although a comprehensive analysis is hindered by its proprietary nature. ## implications for future llms in this section, we walk through various implications of our work and provide recommendations for enabling better csw ability in llms. by highlighting this limitation, we compel researchers to consider csw as a core feature of many people's multilingual repertoire across the world. fairer data representation for code-switching our results in section 3 show that existing llms have similar scaling patterns between monolingual and csw. however, despite all the models under study having seen each of the languages during pretraining, there is still a performance gap between monolingual and csw. this suggests that the ability to code-switch is not acquired by llms after pretraining and/or instruction-tuning with multilingual data @xcite @xcite , indicating the need for adding better data representation for code-switching in the multilingual pretraining and/or instruction-tuning process. such an approach can be done through manual csw data collection and/or various data augmentation methods @xcite @xcite . aside from adding more csw data, one potential solution is to identify and include the code-switching language pairs into consideration of multilingual pretraining and/or instruction-tuning. this allows better resampling strategy @xcite @xcite @xcite for csw data during the multilingual pretraining and/or instruction-tuning. ## related work code-switching code-switching is a common practice observed in multilingual communities where people mix multiple languages within an utterance @xcite . while more than half of the world population speaks more than one language, the availability of resources and assessments for code-switching is much more limited compared to the extensive literature on monolingual cases. the key challenges of collecting high-quality codeswitching data lie in the colloquial nature of the practice and the language proficiency required for accurate annotation @xcite . the recent advances of "multilingual" large language models compel one to explore whether these models are proficient in code-switching contexts like a true polyglot. previous research @xcite has studied the code-switching capabilities of language models in ner and pos-tagging tasks, however, the work is limited to using only differ-ent word embeddings and encoder-only models. in this paper, we expand on previous works and provide a detailed analysis of more model variations, task objectives and downstream applications of diverse language pairs adopted from existing csw benchmarks like lince @xcite and gluecos @xcite . multilingual large language models models like mbert @xcite and xlm-r @xcite have become the goto multilingual options for supervised fine-tuning, given their impressive abilities and adaptability to many languages. with the success of largescale generative models, their capabilities have been enriched with multilingual objectives @xcite @xcite through pretraining on large multilingual corpora such roots @xcite , mc4 @xcite and xp3 @xcite . in addition to excelling in different monolingual and multilingual benchmarks via zero-shot prompting @xcite @xcite @xcite , research has shown that scaling up model sizes @xcite @xcite and incorporating in-context learning examples @xcite could help further boost their performance. yet, given the scarcity of csw evaluation resources, how these multilingual llms perform in code-switching scenarios still remains questionable. in this paper, we evaluate these models under various settings including fine-tuning, zero-shot prompting, and in-context learning, and provide recommendations for future improvements in code-switching proficiency. ## conclusion in this paper, we systematically study multilingual llms' capabilities in code-switching tasks along various dimensions, including but not limited to finetuning vs. prompting, task objectives, scaling laws and model architecture. we observe that, despite improvements with larger sizes, existing multilingual llms still yield inferior performance compared to fine-tuning smaller models. we argue that multilingual llms are not necessarily code-switching compatible. given that multilingual llms are not explicitly trained for codeswitching data, we recommend future development should incorporate a more comprehensive evaluation framework that encompasses code-switching texts. finally, our study is limited to models' performance in sentiment analysis, machine translation, summarization and language identification. we suggest that benchmarking across a broader set of tasks is required. however, the scarcity of high-quality open-source code-switching datasets and the challenges associated with their collection process imply future work should also include constructing code-switching data with more complexity, such as commonsense reasoning.