Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South E ast A sian Languages (2023)

Abstract: While code-mixing is a common linguistic practice in many parts of the world, collecting high-quality and low-cost code-mixed data remains a challenge for natural language processing (NLP) research. The recent proliferation of Large Language Models (LLMs) compels one to ask: how capable are these systems in generating code-mixed data? In this paper, we explore prompting multilingual LLMs in a zero-shot manner to generate code-mixed data for seven languages in South East Asia (SEA), namely Indonesian, Malay, Chinese, Tagalog, Vietnamese, Tamil, and Singlish. We find that publicly available multilingual instruction-tuned models such as BLOOMZ and Flan-T5-XXL are incapable of producing texts with phrases or clauses from different languages. ChatGPT exhibits inconsistent capabilities in generating code-mixed texts, wherein its per-formance varies depending on the prompt template and language pairing. For instance, ChatGPT generates fluent and natural Singlish texts (an English-based creole spoken in Singapore), but for English-Tamil language pair, the system mostly produces grammatically incorrect or semantically meaningless utterances. Furthermore, it may erroneously introduce languages not specified in the prompt. Based on our investigation, existing multilingual LLMs exhibit a wide range of proficiency in code-mixed data generation for SEA languages. As such, we advise against using LLMs in this context without extensive human checks.

## introduction code-mixing, also known as code-switching, is the linguistic practice of alternating between two or more languages in an utterance or conversation @xcite . it allows individuals to express culturally-specific ideas, connect with or differentiate from other interlocutors, and reify their identities @xcite ; grosjean, sea countries selected sea countries 1982; @xcite @xcite . despite its prevalence across many parts of the world, computational research into this area remains understudied @xcite @xcite @xcite . one longstanding challenge in this area involves acquiring high-quality and low-cost code-mixed data. for one, code-mixing is observed more frequently in colloquial settings and spoken communication, which makes procuring and curating extensive datasets logistically demanding and costly @xcite . moreover, despite code-mixing's prevalence across social media and digital messaging platforms, consolidating such data may be curtailed by legal guardrails and scalability issues. recognizing these challenges, we explore the feasibility of using generative large language models (llms) to ameliorate data scarcity in code-mixing research. as cm assuming that you are an english-malay bilingual speaker, how would you write a natural english-malay code-mixed sentence about traffic? (f) template: write a cm sentence (a) template: assume to be bilingual speaker (a) template: assume as bilingual speaker m: oh! it was gre w: i'm glad. didn the next time we m: yeah... all righ (flan-t5-xxl) m party? are they g you don't have to (b) tem cm imitate the speaking style of a person who can speak singlish in one sentence about family. (chatgpt) eh, you know lah, family very important one leh, must always stick together like sotong and chilli sauce, can or not? 3 (davinci-003) "my family lah, we so close, can go anywhere together one!" 3 (davinci-002) my family is the best! 0 (c) template: imitate speaking style code-mixing refers more languages in code-mixed english (chatgpt) sure !(zhá jī) fried c (davinci-0 (davinci tofu, tofu (bl ( (d) te cm imagine that you are a vietnamese speaker. how would you talk about weather using both english and vietnamese in a single sentence? (chatgpt) "ngày hôm nay thìi ti∏t khá n≠ng nóng, so i think i'll wear my shorts and t-shirt i chïi." 3 (davinci-003) i'm sure the weather today is beautiful, thìi ti∏t hôm nay có v¥ rßt µp. 0 (davinci-002) i like the weather here because it's not too hot and not too cold. tôi thích thìi ti∏t  ây vì nó không nóng quá và không l §nh quá. 0 (bloomz) hãy t˜ng t˜òng b §n là ng˜ìi nói ti∏ng viªt, và th˚dùng c£ hai th˘ti∏@xmath0: 0 (flan-t5-xxl) v khách hàng nh hàng tc áo c tn c i n v nhng lng bn s tm thy hàng n c áo c ó 0 (e) template: native speaker write an english and artificial intelligence (chatgpt) engli tificial intelligenc makakapagbigay translation: "artifi that can provide m (davinci-003) ai i at mabuhay ngay modernong solus ilang mga trabaho (davinci-002) ar should be thankfu (bloomz) an ex area of expertise who speaks both (flan-t5-xxl) an ng hariko ay naggangalambot, na papasagawa ng k (f) tem figure 17: prompt templates with different languages and topic fields and res code-mixed / non-code-mixed sentences. we also include a template where w note that the explanations are a part of chatgpt's original generation. 23 (b) template: imitate speaking style figure 2: example prompt templates with different languages and topic fields and responses from llms containing code-mixed / non-code-mixed sentences. note that the explanations are a part of chatgpt's original generation. "cm" indicates the level of code-mixing (section 2.2). see figure 15 in appendix for all prompt templates and responses from other llms such as bloomz and flan-t5-xxl. to this end, we hone in on languages in south east asia (sea). home to more than 680 million people and over 1200 languages, code-mixing is particularly prevalent in this region due to its countries' extended histories of language and cultural cross-fertilization and colonialism (figure 1 ) @xcite @xcite . marked by its distinctive multilingual and multiracial composition today, sea presents an opportunity to further research numerous marginalized languages and linguistic practices in nlp research 1 @xcite @xcite @xcite . nonetheless, publicly available code-mixed datasets relevant to sea communities remain limited @xcite . we prompt five multilingual llms, i.e., chat-gpt, instructgpt (davinci-002 and davinci-003) @xcite , bloomz @xcite , and flan-t5-xxl @xcite to generate code-mixed text that bilingually 1 major languages in sea countries belong to different language families such as indo-european, thai, austronesian, sino-tibetan, dravidian, and austro-asiatic. furthermore, there are at least thousands of major and minor sea languages. to the best of our knowledge, our work marks the first attempt at studying the generation of synthetic code-mixed data through prompting llms in a zero-shot fashion without any monolingual reference texts or explicit linguistic constraints @xcite @xcite . we find that publicly available multilingual language models such as bloomz and flan-t5-xxl are only capable of code-mixing with loanwords or topic-related nouns. most of the time, they fail to code-mix (despite being advertised as multilingual). while chatgpt stands out in its ability to generate codemixed texts, it is extremely sensitive to the prompt template and exhibits a considerable variance of success in generating natural-sounding code-mixed texts across different language pairs. additionally, it may erroneously introduce additional languages not specified in the prompt and wrongly explain the code-mixing of the text. our results lead us to conclude that code-mixing, at least as of today, is not considered an essential component of many multilingual llms. moreover, the opaque creation of models like chatgpt makes it difficult to ascertain the mechanisms that enable code-mixing. by highlighting the limited promises of llms in a specific form of low-resource data generation, we advise nlp researchers against using existing systems to produce synthetic codemixed data without extensive human evaluation. ## implications for llms and ways forward in this section, we walk through various implications of our work for understanding the deployment of llms for synthetic data generation in lowresource settings, multilinguality in llms, importance of code-mixing ability in generative llms, and research transparency. ## related work code-mixed data in sea unlike monolingual data, there is only a limited number of human-curated code-mixed datasets. this resource limitation is more severe in sea due to its marginalization in nlp research @xcite . popular current code-mixing evaluation benchmarks @xcite do not include sea languages, and ex-isting code-mixing studies in sea only cover a limited number of language pairs and creoles, e.g., english-tagalog @xcite , english-indonesian @xcite , javanese-indonesian @xcite , chinese-english @xcite @xcite and singlish @xcite foot_6 . the current corpus does not even scratch the surface of the sheer amount of code-mixedness in sea @xcite , where deployable data is practically non-existent. in this work, we try to close this gap by exploring the potential of generating synthetic code-mixed data for the sea region by prompting llms. ## conclusion to ameliorate the scarcity of code-mixed data for south east asian languages, we explore generating synthetic code-mixed data using state-of-the-art multilingual large language models (llms). on one hand, we find that publicly available llms such as bloomz and flan-t5-xxl have limited capability in generating syntactically diverse code-mixed data. on the other hand, closed-source models such as chatgpt and instructgpt are better at generating natural code-mixed text, but their performance varies substantially depending on the prompt template and language pairing. furthermore, many outputs suffer from syntactic, semantic, and reliability issues. therefore, we caution against using llm-generated synthetic code-mixed data without the involvement of native speakers for annotating and editing.