In-context Mixing ( ICM ): Code-mixed Prompts for Multilingual LLM s (2024)

Abstract: We introduce a simple and effective prompting technique called in-context mixing (ICM) for effective in-context learning (ICL) with multilingual large language models (MLLMs). With ICM, we modify the few-shot examples within ICL prompts to be intra-sententially code-mixed by randomly swapping content words in the target languages with their English translations. We observe that ICM prompts yield superior performance in NLP tasks such as disfluency correction, grammar error correction and text simplification that demand a close correspondence between the input and output sequences. Significant improvements are observed mainly for low-resource languages that are under-represented during the pretraining and finetuning of MLLMs. We present an extensive set of experiments to analyze when ICM is effective and what design choices contribute towards its effectiveness. ICM works consistently and significantly better than other prompting techniques across models of varying capacity such as mT0-XXL, BloomZ and GPT-4.

## introduction incontext learning (icl) has emerged as a widely accepted gradientfree training paradigm for large language models (llms) where a prompt at test time comprises instructions relevant to the target task, a few taskspecific labeled examples and the test instance for which we seek an output @xcite . the fewshot examples in the prompt help the trained model adapt to the task better and elicits more accurate predictions for the test in stances. creating prompts for icl with multilingual llms (mllms) is an emerging area of interest @xcite . when the test instances are in different target languages, the prompts are typically multilingual with the in structions appearing in english and fewshot exam ples appearing in the target languages. it is common to use english for instructions in the multilingual prompts as a bridge to the target language, since mllms are typically most proficient in english. in this work, we expand further on this intuition of using english as a bridge language. we introduce a simple and effective prompt ing technique for mllms that we call incontext mixing (icm). in icm, the fewshot examples in the target languages are altered to contain intra sentential codemixing, i.e., a fraction of content words in the target language sentences are randomly replaced by their english translations. we show that icm outperforms many other prompting techniques and also show it is comple mentary to techniques such as chainofthought prompting @xcite . we provide a very extensive analysis of icm including how perfor mance varies depending on the nature of mixing, the degree of mixing, the source of english trans lations for mixing (i.e., from a lexicon or parallel text), the choice of mixing language, etc. while it appears like one could easily construct alternate prompt modifications that include english and can rival icm (e.g., append english translations to the target sentences, append target language to english word mappings to the target sentences, etc.), we show that none of these variants provide consistent performance gains like icm does. in order to tease apart why icm works better than other prompting techniques, we probe the mllm representations for disfluency correction by training a simple mlp classifier that takes mllm encoder representations of the target sentences as input and labels each word as being disfluent or not. the probe trained on icm representations performs sig nificantly better on the disfluency classification task compared to all prompting techniques. while prompting has been extensively studied for english @xcite , prompting multilingual models to enable improved crosslingual transfer is relatively far less explored. thus, we think a simple prompting mechanism like icm that can consistently improve performance across llms and across languages that are underrepresented in the pretraining/instruction tuning corpora of the multilingual llm could be of broader interest. ## related work in the context of llms, @xcite intro duced the paradigm of incontext learning (icl) which means that the models first develop a range of skills at training time and utilize those skills at inference time to adapt to a target task. icl refers to combining a query example and an instruction together to form a prompt, which is fed into the model for predictive tasks @xcite . a salient feature of icl, particularly relevant to large scale models, is its ability to deliver predictions without changing any of the model parameters. in recent years, prior work in the area of multi lingual large language models (mllms) has ex plored various prompting techniques. translatetest is a strong baseline in the traditional pretrain/fine tune paradigm @xcite . recent studies show that it is also effec tive for prompting autoregressive language models (lin et al., @xcite , as these models exhibit differential performance depending on the input language @xcite . a newer vari ant of translatetest is called selftranslate @xcite that overcomes the need for an exter nal translation system. by leveraging the intrinsic fewshot translation capabilities inherent in multi lingual llms. however, these paradigms falter for samelanguage sequencetosequence tasks where the output must closely mirror the input. this is crucial for tasks requiring high inputoutput corre spondance, like grammar error correction (gec) and disfluency correction (dc). recent work has also delved into methodolo gies like intracrosslingual transfer (winata et al., @xcite , as well as the chainof thought (cot) style of crosslingual prompting @xcite . nonethe less, a recurring observation across these methods is their tendency to predominantly focus on few shot example composition within a single language. this approach, albeit comprehensive, often over looks the potential benefits in crosslingual transfer when languages are mixed. to boost crosslingual transferability in multilingual models, a fewshot prompting method was introduced where the source and target languages are mixed @xcite . while the latter explore the merits of language mix ing, our work is different in that we modify the fewshot examples that appear in the prompt with intrasentential codemixing. ## incontext mixing (icm) a prompt typically comprises an instruction for a given task, accompanied by fewshot examples relevant to that task, followed by a test instance. in "monolingual prompting" @xcite , the instruction is written in english, while the few shot examples, the test instance, and the output (when applicable) are all presented in the native language. a "crosslingual prompt" @xcite uses english for the instruction and english or a pivot language for the fewshot examples, with the test instance still being in the native language. in both of the above prompting techniques widely in use, note that the test instances always appear in the native language. incontext mixing (icm) injects code mixing within the fewshot examples. the examples are now mixed with english, i.e., we randomly switch words in the examples to english. to determine which words are eligible for switching, we restrict our choices to content words with one of the follow ing partofspeech (pos) tags: noun, verb, adjec tive, or adverb. whenever possible, we use bilin gual lexicons sourced from @xcite to facilitate the english replacement, relying on the spacy tagger @xcite to predict pos tags for each word. for languages lacking these resources, we use nllb @xcite to translate each sentence word by word into en glish and borrow the english pos tags. we now introduce english words in the example by only considering content words and randomly switching a predetermined fraction of them. we introduce three different degrees of switch ing in the examples. we name them "icm30", "icm50", and "icm80". as the name suggests, "icmx" means that x% of the content words are randomly switched in each fewshot example. fig ure 1 is an illustration of the switching process, and it is important to note that switching is done only in the fewshot examples, and the instruction and the test instance are kept intact in their respective languages. in the fewshot examples, switching is introduced both in the input and output of each ex ample. if a word is switched in the input, and that word is also present in the corresponding position in the output, the word in the output is likewise monolingual prompt icm50 prompt instruction: correct disfluencies... instruction: correct disfluencies... (in english) (in english) few shot examples: few shot examples: input: अ मु झे बताइए ये से वा नहीं है क्या? input: अ मु झे tell ये ser vice नहीं है क्या? output: मु झे बताइए ये से वा नहीं है क्या? output: मु झे tell ये ser vice नहीं है क्या? ... ... (in native language) (mixed with english) test instance: test instance: input: क्या हमें कल.. हमें कल चलना चा�हए। input: क्या हमें कल.. हमें कल चलना चा�हए। output: output: (in native language) (in native language) ## main results our baselines include monolingual, crosslingual, nativecot and englishcot prompts. if the steps or "chainofthought" is conveyed in the native lan guage of the input and output, it is "nativecot", and if expressed in english, it is "englishcot" @xcite . an example of englishcot prompt is in appendix a. cot is complementary to icm as a prompting technique. we merge the two to see if we get further gains; icm30cot, icm50cot, and icm80cot are the englishcot versions of the icm prompts. that is, the fewshot examples are mixed, and the reasoning appended is in en glish. an example of icm50cot prompt is in appendix a. for the tasks nli and sa, we also in clude translatetest as one of our baselines. that is, we use nllb to translate both the input sentences and the test instance to english. our main observations are summarized below: french and german are examples of high resource languages that do not benefit from icm in results in test whether our results scale to larger models, we also show dc experiments using chatgpt (gpt4) in ## conclusion incontext mixing (icm) involving intrasentential codemixing has been demonstrated to be an effec tive prompting technique to use with multilingual llms. it significantly improves the performance of nlp tasks that demand a close correspondence between the input and output sequences such as dis fluency correction, grammar error correction, and text simplification. this prompting technique par ticularly benefits lowresource languages such as telugu, korean, turkish, etc. we present extensive ablation experiments and a detailed probing analy sis to demonstrate the benefits of icm across many target languages and models of varying sizes.