Multilingual LLM s are Better Cross-lingual In-context Learners with Alignment (2023)

Abstract: In-context learning (ICL) unfolds as large language models become capable of inferring test labels conditioned on a few labeled samples without any gradient update. ICL-enabled large language models provide a promising step forward toward bypassing recurrent annotation costs in a low-resource setting. Yet, only a handful of past studies have explored ICL in a cross-lingual setting, in which the need for transferring label-knowledge from a high-resource language to a low-resource one is immensely crucial. To bridge the gap, we provide the first in-depth analysis of ICL for cross-lingual text classification. We find that the prevalent mode of selecting random input-label pairs to construct the prompt-context is severely limited in the case of cross-lingual ICL, primarily due to the lack of alignment in the input as well as the output spaces. To mitigate this, we propose a novel prompt construction strategy — Cross-lingual In-context Source Target Alignment (X-InSTA). With an injected coherence in the semantics of the input examples and a task-based alignment across the source and target languages, X-InSTA is able to outperform random prompt selection by a large margin across three different tasks using 44 different cross-lingual pairs.

## introduction the emergence of large-scale, pretrained, transformer-based language models (llms) has marked the commencement of an avant-garde era in nlp. departing from the traditional methods of neural language learning with temporally separated training-testing phases for downstream tasks, pretrained llms have shown the ability to infer labels from test inputs conditioned on the training data within a single pass. this is known as in-context learning -an llm is prompted et and sd contributed equally. et and sd designed the experiments. et and mb ran the experiments. sd and tc wrote the paper. tc mentored the project. with a few input-output pairs from the training data (commonly referred to as demonstrations) followed by the test input; for generative tasks (summarization, text-to-code, chain-of-thought reasoning, etc.) the llm is then required to produce an output; for classification tasks, the probabilities of the next tokens predicted by the llm are mapped to the label space. all of this is done without updating the parameters of the llm. in-context learning is particularly promising for two different aspects. firstly, it reduces the need for task-specific training data, and thus, the cost of human annotation. secondly, while the llm was trained in a compute-intensive environment, the removal of the need for task-specific gradientbased weight updates can significantly reduce the carbon footprint of automated nlp/nlu since the inference-time compute-necessity is orders of magnitude smaller than that of the training/finetuning phases. multiple recent advancements have been proposed to optimize the icl ability of the llms @xcite @xcite . challenges in cross-lingual icl: given that there is an order-of-magnitude discrepancy in the availability of annotated data in a high-resource language vs. a low-resource one, the ability to learn from the high-resource source context to solve tasks in low-resource targets sounds enticing. yet, the application of icl in a cross-lingual setting remains largely unexplored. previous attempts at multilingual icl @xcite use randomly selected input-label pairs to construct the prompt-context. this limits the ability of an llm to infer from the context. as xie et al. (2022) suggested, icl emerges as the ability to infer target labels from the pretraining distribution conditioned upon the context; each input-label pair in the prompt-context are, in turn, sampled from the prompt token distribution. theoretically, review: cannot operate this without using 2 hands. doesnt that defeat the point of using it in the .. ... review: they were nice but too big. rating: review: so unhappy with these! hold no charge and stopped working after a few,probably ... ... review: failed on first day of use. it worked fine for a while, maybe 30 minutes of intermittent us examen: bonjour, en fait, j'ai un probleme avec cette commande. certaines pieces manquent. je vous joins la photo du système après montage. les 2 pieces bleues en plastique a fixer aux extrémités des bras tournants n'ont pas été livrees dans le colis. pourriez vous me les faire parvenir. salutations. loic menez évaluation: review: piece of junk! it comes in a million little pieces with no instructions on how to put together. t... ... review: i don't know... i received had a cracked plastic piece review: cannot operate this without using 2 hands. doesnt that defeat the point of using it in the... ... review: they were nice but too big. rating:.. in french bad means mal and good means bien. examen: j ai commencé a écrire correctement puis au bout de 10 lignes l'encre commence a sortir difficilement je suis très déçu de la qualité de ces recharges je ne pense pas que ce soit des recharges mont banc malgré l'emballage.....je vais faire une réclamation évaluation: review: cannot operate this without using 2 hands. doesnt that defeat the point of using it in the... ... review: they were nice but too big. rating:.. in french bad means mal and good means bien. review: great pen set. i love the colors and the writing is very smooth. a few of the pens i received were broken upon... ... ..arrived cracked and broken. a very bad experience..... in french bad means mal and good means bien. input random: semantic aligned: task aligned: semantic aligned: input task aligned: input examen: après seulement une petite semaine d'utilisation ou une vingtaine d'heure, en plus utilisation peu intensive la carte a subitement décider de ne plus fonctionner. ayant retrouver plusieurs commentaires présentant le même problème le défaut doit donc être récurrent je déconseille vivement d'acheter cette carte sur amazon. évaluation: 1). 2). 3). task and semanticaligned: figure 1: working example of different icl prompts explored in this work. in example #1, randomly selecting the prompt examples fails as it prompts irrelevant contradictions, whereas semantic alignment succeeds as it makes the context with similar reviews. in example #2, semantic alignment fails; it extracts demonstrations about 'multiple pieces', but these are not helpful for the llm, whereas a simple task aligner works. in the last example, it is a combination of semantic and task alignments that works. the expected prediction error decreases as the number of examples in the prompt increases. however, such infinitely long prompts are practically infeasible to attain. xie et al. (2022) imposed that a distinguishability of the prompt-concept, shared across the prompt-examples, from all other possible concepts is essential for an optimal predictor. a random sampling of prompt examples is unlikely to construct a prompt with distinguishable concepts. furthermore, given (x i , y i ) and (x i+1 , y i+1 ) as two consecutive input-label pairs in the promptcontext, the transition probability from y i to x i+1 is a low-probability one under the pretraining distribution @xcite . the transition becomes even more improbable if we are to simply append a test example to the prompt-context of a different language. consider the following example of icl prompting for cross-lingual sentiment classification: the text segments are concatenated from left-toright and top-to-bottom; therefore, two english input-label pairs are followed by a spanish test input. there are irremovable, token-level lowprobability transitions from the labels to the next input sentences. on top of this, we have three completely unrelated sentences juxtaposed together with an abrupt change in language. intuitively, it is less likely for an llm to be able to map the third input to its correct label, positiva (positive in spanish) following the very much convoluted patterns presented in english. proposed approach: we seek to develop prompt-design strategies for icl in a cross-lingual setting that can overcome the foregoing challenges. a two-way alignment of the source and target examples is proposed. we start with injecting semantic coherence into the prompt-context by selecting similar examples; this aligns the labeled demonstrations as well as the test inputs to share a set of common concepts. next, we seek to enforce an alignment of task-level signals across languages. we introduce manually-designed task-specific mappings from the source language to the target language, thereby providing the llm with a 'natural' transition from the former to the latter. together, these two approaches constitute our proposed prompts-selection strategy, x-insta (crosslingual in-context source-target alignment, see figure 1 for working examples). x-insta shows a staggering 18% relative improvement over random prompt selection averaged across three different text classification tasks in multiple different languages with english being the source language. careful perturbations to these alignment methods disclose the importance of label space structure induced by llms for cross-lingual icl. our contributions are summarized below 1 : 1. we propose x-insta, a novel method of aligning prompt examples in a cross-lingual scenario. to the best of our knowledge, this is the first at-tempt to push prompt design techniques for icl in cross-lingual settings beyond the trivial strategy of random example selection. 2. we present the first, in-depth analysis of the role of semantic similarity between prompt examples for cross-lingual icl. 3. a novel concept of task-based prompt alignment is presented. we show its efficacy with 44 different source-target language pairs and empirically relate this to the underlying structures of multilingual representations of the llm. ## prompting techniques in this section, we lay out a step-by-step approach to aligning semantic coherence and taskbased signals across source-target examples for icl prompts. ## results and analysis we experiment on three datasets -multilingual amazon reviews corpus (marc) @xcite , cross-language sentiment classification (cls) @xcite , and hat-eval @xcite , spanning over twelve language-task pairs and totalling 44 cross-lingual setups (refer to appendix a for further description of the datasets). the results on marc, cls and hateval are shown in tables 1, 2, and 3, respectively. for our main experiments, we make use of xglm @xcite 7.5 billion variant. we experiment with various models with random prompting and select xglm 7.5b for its performance superiority on various tasks (refer to ## related works in-context learning (icl) @xcite introduced a new approach, called in-context fewshot learning using the gpt-3 model. subsequent efforts have been made to enhance the effectiveness of icl. @xcite evaluated the breadth and depth of model understanding to determine its weaknesses and strengths. techniques such as selecting semantically-similar examples, using differentiable soft prompts for backpropagation, and adjusting prompts to eliminate bias in four examples represent the major error characteristics (discussed in section 3.5). we omit most of the text in the test input of the 3rd example as it was too long. however, fine-tuning these large models for each task is infeasible due to computational limitations. while icl has been attempted for cross-lingual downstream tasks, these methods only involve random sampling of demonstrations for prompt construction @xcite . @xcite addressed the problem of crosslingual text-to-sql conversion using icl. however, their method relies on translating the input text in the source language to the target language before generating the corresponding sql code. @xcite demonstrated the effects of similar example selection in a few-shot machine translation setting which is much similar to our proposed semantic alignment. to the best of our knowledge, there is no study on optimizing prompts for crosslingual nlp tasks using icl. ## conclusion in this work, we described the first-ever attempt in the direction of cross-lingual prompt design for in-context learning. we found that a random selection of labeled training examples to construct the prompt-context limits the capability of a multilingual llm to infer target labels. instead, aligning the semantics as well as the task-specific textual signals across the source and the target language inputs in the prompt demonstrates superior performance in cross-lingual text classification. based on these findings, we introduced x-insta, a novel method of in-context prompt design for cross-lingual text classification. x-insta improves upon random prompt selection substantially across multiple different cross-lingual tasks. we found that the dynamicity of similarity-based example selection is able to guide the llm to learn better in-context predictors irrespective of the language pair under consideration. on the other hand, language pairs with proper alignment in the label space get more out of the task-based alignment. these findings may serve as paving stones toward better cross-lingual icl methods that incorporate an automated, dynamic transition from the source to target distributions.