Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages (2023)

Abstract: Multilingual Pretrained Language Models (MPLMs) perform strongly in cross-lingual transfer. We propose Prompts Augmented by Retrieval Crosslingually (PARC) to improve zero-shot performance on low-resource languages (LRLs) by augmenting the context with prompts consisting of semantically similar sentences retrieved from a high-resource language (HRL). PARC improves zero-shot performance on three downstream tasks (sentiment classification, topic categorization, natural language inference) with multilingual parallel test sets across 10 LRLs covering 6 language families in unlabeled (+5.1%) and labeled settings (+16.3%). PARC also outperforms finetuning by 3.7%. We find a significant positive correlation between cross-lingual transfer performance on one side, and the similarity between high- and low-resource languages as well as the amount of low-resource pretraining data on the other side. A robustness analysis suggests that PARC has the potential to achieve even stronger performance with more powerful MPLMs.

## introduction multilingual pretrained language models (mplms) @xcite @xcite @xcite , pretrained on multilingual corpora @xmath0, exhibit strong multilinguality on downstream tasks @xcite . low-resource languages, for which little text data is available for pretraining monolingual pretrained language models (plms), benefit from mplms. however, the lack of lrl data leads to an imbalanced language distribution in the pretraining corpora of mplms @xcite . lrls are therefore under-represented in pretraining, resulting in bad performance. furthermore, the scarcity of domain-or task-specific annotated data of lrls makes it difficult to apply the pretraining-finetuning paradigm to lrls @xcite . given that the pretraining-finetuning paradigm always has a high demand for domainspecific labeled data, another line of researchprompt-based learning -emerges, focusing on exploiting large pretrained language models by reformulating the input. the prompt is designed to help plms "understand" the task better and "recall" what has been learned during the pretraining. in particular, @xcite propose a simple incontext learning approach without any finetuning, which adds training examples as additional context to test examples. instead of using random examples as context, kate @xcite and soup @xcite retrieve semantically similar examples as prompt for monolingual in-context learning. the above mentioned prompt-based learning techniques require no parameter updating, while there is also work employing sampled similar examples for prompt-based funetuning @xcite . unlike @xcite who created prompts with manually selected examples, these approaches augment the context by retrieving related information from external corpora, allowing the plms to capture more domain-or task-specific knowledge. the prompt-based method offers a new form of zero-shot or few-shot learning in multilingual nlp studies. it involves performing a specific task using prompts, without labeled data in the target language and has the potential of being an effective method for lrls lacking annotated data. our work improves the zero-shot transfer learning performance of lrls on three different classification tasks by taking advantage of cross-lingual information retrieval and the multilinguality of mplms. specifically, we retrieve semantically similar cross-lingual sentences as prompts and use the cross-lingual retrieval information to benefit the lrls from the multilinguality of mplms and achieve better performance in the zero-shot setting 1 . our main contributions are: (1) we propose prompts augmented by retrieval crosslingually (parc), a pipeline for integrating retrieved crosslingual information into prompt engineering for zero-shot learning (figure 1 ). (2) we conduct experiments on several multilingual tasks, showing that parc improves the zero-shot performance on lrls by retrieving examples from both labeled and unlabeled hrl corpora. (3) to find an optimal configuration of our parc pipeline, we conduct a comprehensive study on the variables that affect the zero-shot performance: the number of prompts, the choice of hrl, and the robustness w.r.t. other retrieval methods and mplms. 1 different from the zero-shot cross-lingual transfer learning where mplms are finetuned on hrls @xcite , our zero-shot setting does not involve finetuning. details in §6.4 retrieval methods external knowledge extracted by information retrieval is often leveraged to solve nlp tasks. two types of representations have been used for retrieval: (1) sparse bag-ofwords representations @xcite , and (2) dense representation learned by neural networks @xcite . dense representations come either from contextual token embeddings @xcite or from sentence encoders @xcite . @xcite propose sentence transformers to create semantically meaningful sentence embeddings by applying siamese and triplet network structures to transformer-based pretrained language models. by using knowledge distillation, sentence transformers can be expanded to support various languages as multilingual sentence transformers @xcite , allowing for cross-lingual retrieval. retrieval augmented prompt @xcite show that large-scale pretrained language models such as gpt-3 can learn to perform a task by putting examples of input-output pairs into the input as context. the in-context learning method simply concatenates the input with examples randomly extracted from the training set. recent studies @xcite augment the prompts for pre-trained models by sampling semantically similar examples. they apply the retrieval augmented method to discrete prompts, which are represented by tokens instead of vectors in a continuous space. they use them either for finetuning in few-shot settings or for zero-shot learning. chowdhury et al. (2022) use a similar knn-based retrieval method for tuning the soft prompts in a continuous space with a standard supervised training setup. previous work focused on monolingual retrieval-augmented prompts. our work applies cross-lingual retrieval to discrete prompts in a scenario without parameter updating. to the best of our knowledge, our work is the first to investigate prompt learning augmented by cross-lingual retrieval. multilingual prompt learning despite the success of prompting in english, prompting in multilingual tasks has not been extensively studied. @xcite show the multilingual skills of lms mainly trained on english data in prompt learning by giving them a few english examples as context but testing them on non-english data. some recent works investigate the prompt learning with multilingual plms @xcite . unlike our work, they focus on finetuning or prompt tuning requiring parameter updating. we apply our method to lrls in a zero-shot setting without adjusting the model parameters. ## methodology this work aims to improve the performance of mplms on lrls in the zero-shot setting by leveraging retrieved cross-lingual contents from hrls. for that, we design the parc pipeline that can be applied to labeled and unlabeled scenarios, i.e., the hrl information can be retrieved from either labeled or unlabeled corpora. as figure 1 shows, the parc pipeline consists of two steps: (a) cross-lingual retrieval from highresource language corpora, and (b) prediction with a retrieval-augmented prompt. figure 1 shows an example: a telugu input sentence from a sentiment classification task is firstly fed into the crosslingual retriever to fetch the semantically closest sample from the hrl corpus, i.e. english in this case. in the second step, the retrieved hrl sample together with its label and the lrl input sentence are transformed into a prompt. for prompt-based classification, we need (i) a pattern which converts the input sentence into a cloze-style question with a mask token, and (ii) a representative word (called verbalizer) for each possible class. converting the classification task into a cloze-style question aligns seamlessly with the framework of our proposed parc method, because it not only performs zeroshot learning well but, more significantly, facilitates better integration of the retrieved cross-lingual contexts. in our example, we use the pattern p (x) = x• "in summary, the product was @xcite ." to convert the retrieved english sentence into "wonderful! works as stated! in summary, the product was @xcite .", where • is the string concatenation operator. a verbalizer such as {@xmath1, @xmath2, which maps the original labels {pos, neg} onto words in the vocabulary, is then used to replace the [mask] token with the verbalized label word "great", standing for the correct label pos of this sentence. we call the resulting english sentence (in our example: "wonderful! works as stated! in summary, the product was great.") the "cross-lingual context". at last, we fill the same pattern with the input telugu sentence and append it to the cross-lingual context. we feed this cross-lingual retrieval augmented input to the mplm. the mplm returns for each of the verbalizers its probability of being the masked token. more formally, let x l @xmath3, (x h j , y j ) ∈ d h lb and x h @xmath4, respectively, where x j is the text sample and y j its class label from a label set y . as eq. ( 1 ) shows, the cross-lingual retriever clr takes the hrl corpora d h and a given lrl input sentence x l i . it returns an ordered list of hrl sentences d r i according to the semantic similarity. we then have (x the prompt pattern p (.) converts an hrl input sentence x r i k into a cloze-style form with a mask token. the verbalizer v(.) is a bijective mapping from the set of class labels y to a set of verbalized words v from the hrl vocabulary. we use the verbalized label word to fill in the mask token in the prompt pattern, and construct the cross-lingual context c i k for the input x l i with the k-th most similar hrl sample x r i k : the cross-lingual context c i k is then concatenated with the prompted lrl input as the input i to the mplm: the mplm m performs masked token prediction and returns the probabilities @xmath5. we predict the class ŷ whose verbalizer v(ŷ) received the highest probability from model m : in the labeled scenario, we obtain the correct label y r i k of the hrl sentence from d r i lb . in the unlabeled scenario, we predict the label using the same prompt-based classification method without cross-lingual context, similar to eq. ( 4 ). we call this the self-prediction method: in order to use more cross-lingual information, we retrieve the k most similar hrl samples. with each sample, we obtain verbalizer probabilities as described above and retrieve the class whose verbalizer has the largest sum of probabilities. we call this method the bag-of-retrieval (bor) strategy. we also tried concatenating the different crosslingual contexts (conc method), but the resulting performance has been worse (see amazon reviews dataset categorizes the shopping reviews into 5 star ratings from 1 to 5. in order to satisfy a binary classification setting, we select the reviews with rating 1 as negative (0) and 5 as positive (1) for our experiments. the following pattern p (x) and verbalizer v are defined for an input review text x: ag news is a collection of more than 1 million news articles for topic classification. the news topic categories contained in the dataset are world (0), sports (1) , business (2), and tech (3). the pattern and verbalizers are as follows: xnli is a multilingual version of the multinli dataset @xcite . we use a subset of the original xnli dataset in our experiment. the text in each data item consists of two parts. sentence a is the premise and sentence b is the hypothesis. the nli task is to predict the type of inference between the given premise and hypothesis among the three types: entailment (0), neutral (1) and contradiction (2). for a given sentence pair x 1 and x 2 , we design the pattern and verbalizer as: construction of multilingual parallel test sets parallel test datasets for evaluating cross-lingual transfer performance on lrls are rare. however, recent research conducted by hu et al. (2020); liu et al. (2022c) shows that automatically translated test sets are useful for measuring cross-lingual performance. hence, we adopt their methodology and construct datasets for different tasks by automatically translating english test sets to targeted lrls. we use the python api of the google translate system to implement the construction of multilingual parallel test sets in our experiment. we also validate the translation effectiveness and quality. the original xnli datasets include two low-resource languages that are used in our experiments (swahili and urdu), so we use them as the "gold" standard for our translation validation. we compare the cross-lingual transfer performance on translation test sets and original test sets of xnli. we also measure the translation quality by using the original sets as gold standard. through the validation conducted on these two languages within the xnli task, we infer that the translation method is effective and could be generalized to other languages and tasks. detailed results are shown in appendix §a. following wu and dredze (2020), we regard languages with a wikisize foot_0 of less than 7 as lrls. we construct a test set consisting of 10 lrls in 6 language families: indo-european (afrikaans -af, urdu -ur), austronesian (javanese -jv, tagalogta), altaic (mongolian -mn, uzbek -uz), dravidian (tamil -tl and telugu -te), sino-tibetan (burmese -my), and niger-congo (swahili -sw). hrl corpora to retrieve rich and diverse information, a large-scale general corpus or knowledge base in the different hrls would be the ideal sentence retrieval pool. in practice, however, a trade-off is necessary in order to save computational resources. following @xcite , we therefore use the task-specific labeled training set of english as the sentence pool in our experiments. the selection of the hrl will be discussed in §6.2. ## results although our proposed methods perform better than the baselines on all three tasks, the degree of improvement differs. a large improvement is found on ag news, the topic categorization task, while xnli only shows a slight improvement. an explanation for this could be that the natural language inference task is more difficult than topic categorization, especially in a zero-shot setup. also, prior work has shown that designing cloze-style patterns and searching the answer space for nli tasks @xcite is difficult. we also find that parc-labeled noticeably outperforms parc-unlabeled, indicating that the performance of self-prediction is limited by the capabilities of mbert. more powerful mplms and better pattern designs might further improve the performance. to analyze the performance for every language in detail, we present the complete experimental results for the topic categorization task on ag news in however, it is worth noting that the sensitivity to cross-lingual retrieval differs from language to language. for some lrls, e.g. urdu (ur) and uzbek (uz), parc's improvement from cross-lingual retrieval is smaller. others gain more, e.g. javanese (jv). retrieving more samples increases the performance up to @xmath6. we now turn to the following two questions: 1) how does k affect the performance on other tasks than topic categorization? 2) which lrls profit most from our parc method and which hrls are best suited to retrieve prompts? 6 analysis 6.1 effect of k we investigated how the performance changes as the number of retrieved hrl samples k increases. as shown in figure 2 , an abrupt accuracy increase can be seen in both labeled and unlabeled scenarios by concatenating the most similar cross-lingual sample. in labeled scenarios, the performance tends to increase up to @xmath7. this can be explained by the fact that later retrieved samples are less similar to the input sample, so their contribution as prompts decreases. in unlabeled scenarios, there is no clear improvement beyond @xmath8, where the accuracy increases monotonically except for @xmath9. the performance of xnli is less obviously influenced by the value of k than binary sentiment analysis and topic categorization. we assume that this could be attributed to the difficulty of the inference task. unlike the other two single sentence classification tasks, xnli identifies the relationship between a pair of sentences. transferring knowledge about sentence relationships is more complicated and requires more samples to learn, in contrast to the other two tasks where semantic information from similar cross-lingual sentences can be transferred directly. ## conclusion we propose parc, a pipeline that augments prompts for zero-shot learning on low resource languages by retrieving semantically similar crosslingual sentences from hrl corpora. we test parc on three classification tasks with parallel test sets across 10 lrls, and it performs better than the baselines in both unlabeled and labeled settings. increasing the number of retrieved prompts improves performance at first, but deteriorates it after a certain point. a robustness study shows that parc also performs well with other cross-lingual retrievers or mplms, suggesting potential applications of parc to a wider scope of scenarios.