Adaptive Cross-lingual Text Classification through In-Context One-Shot Demonstrations (2024)

Abstract: Zero-Shot Cross-lingual Transfer (ZS-XLT) utilizes a model trained in a source language to make predictions in another language, often with a performance loss. To alleviate this, additional improvements can be achieved through subsequent adaptation using examples in the target language. In this paper, we exploit In-Context Tuning (ICT) for One-Shot Cross-lingual transfer in the classification task by introducing In-Context Cross-lingual Transfer (IC-XLT). The novel concept involves training a model to learn from context examples and subsequently adapting it during inference to a target language by prepending a One-Shot context demonstration in that language. Our results show that IC-XLT successfully leverages target-language examples to improve the cross-lingual capabilities of the evaluated mT5 model, outperforming prompt-based models in the Zero and Few-shot scenarios adapted through fine-tuning. Moreover, we show that when source-language data is limited, the fine-tuning framework employed for IC-XLT performs comparably to prompt-based fine-tuning with significantly more training data in the source language.

## introduction the recent progress in the development of multilingual language models (lms) has allowed for effective cross-lingual transfer (xlt) with minimal need for architectural modifications @xcite . by simply training a multilingual model in a language with abundant resources its acquired knowledge can be extended to target languages, in either zero-shot or few-shot scenarios. cross-lingual transfer is a significant topic as it addresses the prevalent challenge of data scarcity in languages other than widely resourced ones, such as english @xcite . the ability to leverage the extensive linguistic resources available in high-resource languages for languages with limited training data enables the deployment of truly inclusive nlp systems. zero-shot cross-lingual transfer (zs-xlt) involves transferring a model trained in a source language to a target language without any demonstration of target-language examples @xcite . this approach is highly modular, as it requires no adaptations specific to the target language. on the other hand, few-shot cross-lingual transfer (fs-xlt) enhances targetlanguage accuracy by further fine-tuning a model using labeled target data @xcite @xcite . however, this method faces limitations when the available target-language data is limited, especially for higher-level tasks, leading to negligible enhancements or even detrimental effects on performance @xcite . furthermore, this approach incurs higher computational costs due to the fine-tuning step and diminishes the modularity that characterizes the zero-shot method. our perspective is that adapting to a target language should prioritize resource efficiency and modularity, where we can seamlessly deploy a single model trained in english (or another source language) across different languages without any finetuning. in this work, we aim to improve this aspect for text classification -a high-level task-by leveraging the language-specific abilities of a multilingual model by prepending a one-shot text-label targetlanguage demonstration to the input text to predict the correct label. specifically, we propose in-context cross-lingual transfer (ic-xlt), a simple yet effective method for one-shot cross-lingual transfer in text classification. this novel approach employs in-context tuning (ict) @xcite to train an encoderdecoder model in the source language tasking it to predict input texts with information derived from context demonstrations. ict is a meta-learning strategy that optimizes a model's ability to learn from in-context examples, originally designed for facilitating swift adaptation to new tasks by prepending target-task in-context demonstrations to the input during the adaptation process. to the best of our knowledge, this is the first study of ict application in the context of cross-lingual transfer. the proposed method is composed of a finetuning and an adaptation stage. first, we fine-tune on the source language through ict, where the model is trained for the classification task and also to learn from context demonstrations. then, we adapt to the target language at inference time by prepending a one-shot 1 demonstration in that language to the input. this method is modular and cost-effective at the adaptation stage as it does not require any gradient update. we evaluate ic-xlt on two multilingual text classification datasets, spanning 5 and 13 target languages, with english as the source language. we consider two distinct settings. first, we assume access to the entire source-language training dataset. for the second setting, we deliberately constrain the amount of source training data available. this limitation aims to gauge the robustness of the proposed approach in scenarios where the availability of source data is restricted. we hypothesize that leveraging context information may prove particularly beneficial in tasks where source data is limited. the contributions of this work are the following: ## our proposed approach: in-context cross-lingual transfer our method aims to simultaneously train a pretrained multilingual encoder-decoder model for (1) a downstream text classification task, and (2) learning from context demonstrations. then, we expect it to be able to generate predictions in a target language by including context demonstrations in this language. therefore, we leverage ict fine-tuning to transfer between languages at inference. as described above, our proposed procedure, called in-context cross-lingual transfer (ic-xlt), is comprised of two stages: in-context tuning during the meta-training stage, we fine-tune the base multilingual model for a specific task using data from the source language. let the set of pairs d @xmath0, y src 1 ), . . . , (x src |d| , y src |d| )} represent the source-language training dataset. the objective is to train the model to predict the label y src i for a given text x src i with the following input⇒output format: here, x @xmath1, y j 1 ), . . . , (x j m , y j m )) is a random sequence of m text-label pairs randomly sampled from d src without replacement, which excludes the pair (x src i , y src i ). in simpler terms, m is the number of source-language demonstrations prepended to each input during fine-tuning, not the number of k tgt -shot examples prepended during inference. in-context learning at inference, we adapt to a target language by prepending the samples from the target language training dataset d @xmath2, y tgt 1 ), . . . , ( x tgt n ktgt , y tgt n ktgt )} to each entry x tgt i of the test set to predict y tgt i . consequently, the input format mirrors the structure observed in the ict stage: where n is the number of classes and the sequence x tgt is a concatenation of d tgt entries comprising the k tgt -shot samples per class, prepended to each x tgt i entry at the inference stage. the intuitive idea for this approach is that, after the meta-training stage, we expect the model to understand both the classification task and the contextual relationships relevant to it. during the adaptation stage, the model leverages its multilingual pretraining to interpret context examples in the target language. note that the adaptation to the target language in this context is gradient free, as it occurs during inference, and thus no model weights are updated. ## experimental methodology in this section, we outline the methodology employed to evaluate the proposed approach. we as-sess ic-xlt effectiveness in adapting to a target language for the classification task and compare its performance in cross-lingual transfer under (1) full training data on the source language and (2) various source-language data budgets. we conduct these limited data experiments to assess how much ic-xlt improves over a traditional fine-tuning method by leveraging the one-shot demonstration. ## conclusion in this paper, we investigated the application of in-context tuning for one-shot cross-lingual transfer, introducing in-context cross-lingual transfer (ic-xlt). our evaluations conducted on a multilingual encoder-decoder model (mt5) demonstrate the efficacy of the proposed method in effectively adapting at inference time to target languages using only a one-shot demonstration in-context, all with-out incurring additional computational expenses (gradient free). furthermore, in comparison to zs-xlt and 1s-xlt ∇ , ic-xlt demonstrated superior performance and smaller transfer gap for the task of text classification, a high-level task where fs-xlt tends to underperform. in scenarios with limited source-language training data, we provide empirical evidence that ic-xlt learns better the source language at the metatraining stage and demonstrates a smaller transfer gap at the adaptation stage with the one-shot demonstration, compared to zs-xlt and 1s-xlt ∇ . this makes ic-xlt a valuable tool for cross-lingual transfer in resource-limited scenarios. our findings also show a significant correlation between the performance improvements in target languages and their token count in the mt5 pretraining corpus, indicating that languages with lesser representation tend to benefit more from target-language adaptation through ic-xlt. to our knowledge, this study represents the first exploration of in-context tuning for cross-lingual transfer. for future work, we aim to explore the potential and limitations of this approach by evaluating its applicability to other architectures, such as decoder-only or encoder-only models, and examining the impact of training with a greater number of examples in-context.