SSP : Self-Supervised Prompting for Cross-Lingual Transfer to Low-Resource Languages using Large Language Models (2024)

Abstract: Recently, very large language models (LLMs) have shown exceptional performance on several English NLP tasks with just in-context learning (ICL), but their utility in other languages is still underexplored. We investigate their effectiveness for NLP tasks in low-resource languages (LRLs), especially in the setting of zero-labelled cross-lingual transfer (0-CLT), where no labelled training data for the target language is available – however training data from one or more related medium-resource languages (MRLs) is utilized, alongside the available unlabeled test data for a target language. We introduce Self-Supervised Prompting (SSP), a novel ICL approach tailored for the 0-CLT setting. SSP is based on the key observation that LLMs output more accurate labels if in-context exemplars are from the target language (even if their labels are slightly noisy). To operationalize this, since target language training data is not available in 0-CLT, SSP operates in two stages. In Stage I, using source MRL training data, target language’s test data is noisily labeled. In Stage II, these noisy test data points are used as exemplars in ICL for further improved labelling. Additionally, our implementation of SSP uses a novel Integer Linear Programming (ILP)-based exemplar selection that balances similarity, prediction confidence (when available) and label coverage. Experiments on three tasks and eleven LRLs (from three regions) demonstrate that SSP strongly outperforms existing SOTA fine-tuned and prompting-based baselines in 0-CLT setup.

## introduction very large language models (llms) such as gpt-3.5-turbo & gpt-4 @xcite show remarkable performance on a variety of nlp and reasoning tasks via in-context learning (icl) @xcite . icl feeds a task-specific instruction along with a few exemplars, appended with the test input, to the llm. as llms can be highly sensitive to exemplars @xcite , efficient exemplar retrieval becomes essential for icl. while llms have shown excellent performance on english tasks, their effectiveness in other languages remains relatively underexplored. in this work, we study zero-labeled cross-lingual transfer (0-clt) to low-resource languages (lrls) -a setting where labeled task data from one or more related medium-resource languages (mrls) is available, but no labeled data exists for the target lrl. we additionally leverage the available test sentences (unlabeled) in the target language. the high cost of annotating the sentences in lrls for new tasks or domains highlights the relevance of the 0-clt setting. cross-lingual transfer has been addressed through standard fine-tuning @xcite , and language adapters @xcite @xcite , but there is limited work on cross-lingual icl. there are two exceptions @xcite , where icl is employed with exemplars from a source language, but they use uniformly random sampling for exemplar selection, resulting in performance inferior to cross-lingually fine-tuned models, such as mbert and xlm-r @xcite . in our preliminary experiments, we prompt the gpt-4 model with exemplars from source mrls, and compare its performance with the same llm prompted with exemplars from the target lrl. we vary the label noise on the target exemplars. unsurprisingly, llms show better performance with less label noise. more interestingly, we find that a reasonably-sized noise region exists (see figure 1 ), such that if the exemplar noise is within that range, then the overall performance is higher than prompting with accurate source language data. armed with this observation, we present self-supervised prompting (ssp) -a novel icl framework for 0-clt to lrls. since the target lrl training data is not available in 0-clt, ssp operates in two stages. in stage i, ssp labels all test instances of lrl using training data from mrl. this may be done by llm prompting (as in the experiment above), or using any other existing approaches for 0-clt, such as by fine-tuning or adapters. once (noisy) labels on target lrl are obtained, in stage ii, ssp uses icl using these noisy test data points (except itself) as exemplars for further performance improvement. additionally, to select the best exemplars, we develop a novel integer linear programming (ilp) based selection approach, which balances the various objectives of (1) similarity of exemplar with test sentence, (2) high confidence in label predictions, and (3) coverage of the various labels for better task understanding. figure 2 gives an overview of our proposed pipeline. we define 3 scenarios for our zero-labeled setup -(1) 0-clt: only the available test sentences of the target language are used, with no additional unlabeled data, (2) 0-clt-u: the full wikipedia data available for target language is utilized, and (3) 0-clt-t: a translation model supporting the target language is leveraged. the primary focus of this work is on 0-clt (setting 1). however, we also conduct stage 1 experiments for both 0-clt-u and 0-clt-t settings. this enables us to comprehensively assess ssp's effectiveness across varying degrees of noise in stage i labelings. we perform experiments on sequence labeling tasks (pos tagging and ner), and natural language inference (nli) -a text classification task. our datasets encompass eleven low-resource languages from typologically diverse language families and three regions: african, germanic and american. our experiments show consistent and substantial improvements over existing fine-tuning as well as simpler icl-based approaches. to encourage reproducibility, we make our code and prompts publicly available. 1 our contributions are summarized as follows: ## related work an icl prompt consists of (1) task description: to facilitate the understanding of task, (2) labeled input-output pairs: written sequentially in order of their relevance to input query, and (3) input itself. cross-lingual icl: in general, cross-lingual icl has not been systematically explored in literature. in existing works, prompting is primarily done in a high-resource language, typically english. this is called cross-lingual (cl) prompting. this differs from in-language (il) prompting, where examples are retrieved from the candidate pool of the target language itself. this assumes the availability of labeled data for target lrl, which is not true in our zero-labeled (0-clt) setting. in response, we develop novel techniques making use of both cl prompting and il prompting, while not utilizing the gold labels during il prompting stage. most existing cross-lingual icl methods use uniformly random input-output pairs for exemplar selection @xcite @xcite . recent approaches @xcite address this gap by utilizing semantic similarity for cross-lingual retrieval from a highresource language's labeled data, given the target lrl's instance as query. this is facilitated by embedding-based multilingual retrievers such as multilingual sentence-transformers @xcite . more recently, openai-based embeddings such as ada-002 2 have been used effectively for cross-lingual retrieval @xcite . we extend this line of work by also incorporating label confidence and label coverage in exemplar selection. self-adaptive prompting: wan et al. (2023) proposed universal self-adaptive (usp) framework, which has been explored for only monolingual (english) setting. usp uses an external unlabeled dataset of instances and labels them using llm in stage i. it then samples multiple chain-of-thought (cot) paths to estimate the logits using the same llm, and then utilizes the entropy of logits for exemplar selection for stage 2. our work has similarities to usp in that both methods are two-staged prompting approaches. usp is different from ssp in that the former is much more expensive, since it requires multiple llm calls to just estimate the 2 https://platform.openai.com/docs/guides/embeddings/ logits. usp also does not use any exemplars (and only uses task description) in stage 1, which are quite important for better performance. finally, usp has only been applied for english tasks, and has not been explored for cross-lingual tasks. fine-tuning approaches for cross-lingual transfer: most approaches rely on fine-tuning a pretrained lm (plm) such as bert or xlm-r on the source languages @xcite ; alabi et al. (2022)) and deploying on an unseen target language. recently, language-adapter-based approaches have been found more effective @xcite for cross-lingual transfer settings. for sequence labeling tasks (ner and pos tagging), zgul @xcite is a recent sota method that leverages ensembling language adapters from multiple mrls to label each word in a target language. we leverage this in our proposed ssp pipeline. cross-lingual label-projection techniques: recent methods @xcite @xcite utilize an off-the-shelf translation model @xcite for label-projection in 2 ways -(1) translate-train: translate from english to target language (x) to generate training data in x, or (2) translate-test: translate test data in x to english to perform labelprojection and obtain annotations in x. although our focus is 0-clt transfer, we also experiment with these translation models in stage i, to assess the robustness of ssp across multiple settings. ## self-supervised prompting we define the setting of zero-labeled cross-lingual transfer (0-clt) as follows. we are given source training data for a specific task: @xmath0, lg i , y i )}, where x i is the input text in language lg i , and the output is y i . we are additionally given a set of unlabeled test data points @xmath1. our goal is to train a model/create a protocol, using d, t and a large pre-trained llm, that outputs good predictions on t for the task, assuming that lg t is a low-resource language, due to which its training data is not available, and that languages lg i are related to lg t . our solution approach, self-supervised prompting (ssp), comprises two key stages as follows. in stage i, it proposes a noisy labeling for all data points in t using source data d. this may be done in different ways, as described next. in stage ii, it uses the llm and noisy labeling on t from stage i as exemplars to improve the labelings. furthermore, ssp uses a novel integer-linear programming based exemplar selection. we now describe each component of our system. ## experiments our main experiments assess ssp performance compared to existing state-of-the-art models for 0-clt. we also wish to compare various ssp variants, and estimate the value of the ilp-based exemplar selection. ## conclusions and future work we study the zero-labeled cross-lingual transfer (0-clt) setting for low-resource languages, when task-specific training data is available for related medium resource languages, along with unlabeled test data for target language. we present self-supervised prompting (ssp) -a novel two-stage figure 4: precision of selected exemplars for african ner and germanic pos figure 5: avg. f1 scores for african ner and germanic pos as a function of candidate pool size in ssp framework for the use of in-context learning over very large language models. at a high-level, ssp first noisily labels the target test set using source training data (either by training a model/adapter) or by in-context learning over an llm. ssp then uses these noisily labeled target data points as exemplars in in-context learning over the llm. a key technical contribution is the use of integer-linear program that balances exemplar similarity, labeling confidence and label coverage to select the exemplars for a given test point. thorough experiments on three nlp tasks, and eleven low-resource languages from three language groups show strongly improved performance over published baselines, obtaining a new state of the art in the setting. ablations show the value each ilp component in downstream performance. we release our code to enable further research in the community. in the future, we seek to extend our technique to more non-trivial applications such as open generation tasks @xcite . we also posit that smaller fine-tuned models, when calibrated properly, can result in more efficient selection of exemplars to an llm, as compared to poorly calibrated counterparts, in terms of ssp's downstream performance. we leave a careful and systematic investigation into this hypothesis for future work.