How to Translate Your Samples and Choose Your Shots? Analyzing Translate-train & Few-shot Cross-lingual Transfer (2022)

Abstract: Translate-train or few-shot cross-lingual transfer can be used to improve the zero-shot performance of multilingual pretrained language models. Few-shot utilizes high-quality low-quantity samples (often manually translated from the English corpus ). Translate-train employs a machine translation of the English corpus, resulting in samples with lower quality that could be scaled to high quantity. Given the lower cost and higher availability of machine translation compared to manual professional translation, it is important to systematically compare few-shot and translate-train, understand when each has an advantage, and investigate how to choose the shots to translate in order to increase the few-shot gain. This work aims to fill this gap: we compare and quantify the performance gain of few-shot vs. translate-train using three different base models and a varying number of samples for three tasks/datasets (XNLI, PAWS-X, XQuAD) spanning 17 languages. We show that scaling up the training data using machine translation gives a larger gain compared to using the small-scale (higher-quality) few-shot data. When few-shot is beneficial, we show that there are random sets of samples that perform better across languages and that the performance on English and on the machine-translation of the samples can both be used to choose the shots to manually translate for an increased few-shot gain.

## introduction with the emergence of large-scale multilingual pretrained language models like mbert @xcite and xlm-r @xcite , a significant amount of research went into exploring the cross-lingual transfer capabilities of these models, allowing for an easier adaptation to a task in many various languages. this is achieved through a number of approaches. zero-shot cross-lingual transfer has become a research focus, e.g. xtreme / xtreme-r benchmark @xcite . in this approach, transfer to new languages is done by fine-tuning a multilingual plm on the task at issue, using only an english corpus (source language) and reporting the performance on multiple target languages. few-shot cross-lingual transfer was recently shown to give an advantage over zero-shot cross-lingual transfer @xcite . in this approach, it is shown that fine-tuning the model using a small amount of target-language task data (fewshot) improves the performance, especially for lowresource languages. translate-train is another common approach to improve the performance. here the full training dataset is machine translated to the target language and used for fine-tuning. there exists relatively good machine translation (mt) systems for the languages that are usually studied in the few-shot approach foot_1 that could be used in translate-train. in the following, we use few-shot to refer to fine-tuning using fewer samples of high-quality professional manual translation. translate-train is used to refer to fine-tuning using lower-quality machine translation that has the potential to be scaled to a larger number of samples. although some research has dealt with few-shot cross-lingual transfer and analyzing it @xcite , no systematic study was done to compare it to translate-train. given that both zeroshot and few-shot cross-lingual transfer assume the availability of a large-scale english corpus of the task for source training, we hypothesize that the translate-train approach might have an advantage over few-shot given the scale of data that would be available even if not at the best quality. on the other hand, when there is a need for fewshot cross-lingual transfer for some task and therefore a need for professional translation of some training samples, this entails significantly more effort and cost compared to mt. it is then important to find out which samples to manually translate given the high variance in performance depending on the choice of samples as shown in @xcite . we investigate both those research directions using 3 base models (mbert base , xlm-r base , xlm-r large ) on 3 high-level semantic tasks and datasets: xnli (natural language inference), x-paws (paraphrase detection) and xquad (question answering), spanning 17 diverse languages. we investigate the following research questions: q1. how does the performance of few-shot crosslingual transfer compare to that of translate-train? we show that there is a performance advantage for few-shot transfer over translate-train given the same number of samples, but that with the increase of samples used for translate-train, this gap shrinks, and using the full large-scale corpus in translatetrain results in a clear advantage over few-shot. we show that at a scale of 10x-100x of machinetranslation to manual-translation, quantity trumps quality and it is recommended in this case to use translate-train if mt is available for the language. few-shot transfer still has an advantage when less source data is available and it is therefore not possible to benefit from the scale gain of using mt. q2. are there sets of samples that have better few-shot performance if translated and how can those sets be identified? we show that when few-shot transfer is beneficial for the task, there are random sets of samples that perform better across most target languages and across different model initializations. we investigate using the performance on the english version of the samples and the machine-translated version to choose the best candidates to manually translate and use for few-shot transfer. we show that there is a correlation between the performance of the same set of shots across languages and that the few-shot samples that perform better on the source language, english, perform also better across languages. a similar observation is made also using mt of the samples. we further show empirically that choosing the sets of samples for few-shot transfer using those heuristics or a model, using such features of the samples, results in more bang for your shots. few-shot: there has been recently some focus on few-shot cross-lingual transfer and its analysis. @xcite shows the effectiveness of few-shot compared to zero-shot cross-lingual transfer especially in lower-resource and distant languages, where zero-shot is least effective and few-shot gives a large gain. @xcite analyzes few-shot cross-lingual transfer emphasizing that the choice of shots has a significant effect on the performance. the experiments are conducted at a small scale of around 10 samples. compared to this, we conduct larger-scale few-shot experiments with a size up to hundreds of samples and focus on choosing the best-performing samples. translate-train: is commonly used to boost the performance for a target language using a machine translation of the source corpus @xcite @xcite . @xcite shows that, similar to zero-shot, translate-train performance increases with the scale of the model. no systematic study tested the effect of the scale of the translated data in comparison with few-shot to understand the interplay of data quality vs. quantity in this context. choosing samples: two related areas are sample selection @xcite which is used for robust training on noisy data @xcite and active learning @xcite used to choose the best potential samples to annotate @xcite . both assume access to the actual sample input (with or without label). on the other hand, this work investigates choosing samples while only having access to the source-language sample input/output. ## datasets we focus on high-level tasks and conduct our experiments on 2 classification tasks and a question answering task ( the dataset consists of pairs of sentences and the task is to predict whether those two sentences are paraphrases of each other. xquad @xcite is a professional translation of the dev set from squad v1.1 @xcite into 10 other languages. the dataset consists of a paragraph and a set of questions. the task is to select the span of the paragraph that answers the question. 10 paragraphs from the multilingual split are reserved, similar to @xcite , as dev (total of 261 question/answer samples) and the rest as test set. ## experiments three main models are used: mbert (base), xlm-r b (base) and xlm-r l (large). we report results on xlm-r b if not specified otherwise, because it strikes a balance between good performance and efficient training. for each task, we fine-tune the model on the source language (english) corpus for 5 epochs with early stopping using the loss on the english dev set. we then continue fine-tuning the model on the target language either in a few-shot or translate-train setup as explained in the following sections. training details are in appendix a. ## conclusion and future work this work conducted a systematic comparison between translate-train and few-shot cross-lingual transfer. it quantified the performance gain for each and showed that starting from 1k samples, mt data could be used to improve over zero-shot performance, and that at 10k-100k, there is an advantage for translate-train over few-shot. for the tasks that benefit from few-shot, we show that there are random sets of samples that perform better across languages and that the english performance of the samples in those sets can help us identify them. the performance of the mt of the samples can also be used as another indicator. when not incurring gain, both help at least avoid the worst performing samples. further analysis in the future could help identify why some datasets do not benefit from few-shot transfer with certain models, and analysing the samples might lead to uncovering interesting properties in the best/worst performing sets of samples.