Improving Few-Shot Learning with Multilingual Transfer and M onte C arlo Training Set Selection (2023)

Abstract: In industry settings, machine learning is an attractive tool to automatize processes. Unfortunately, annotated and high-quality data is expensive to source. This problem is exacerbated in settings spanning multiple markets and languages. Thus, developing solutions for multilingual tasks with little available data is challenging. Few-shot learning is a compelling approach when building solutions in multilingual and low-resource settings, since the method not only requires just a few training examples to achieve high performance, but is also a technique agnostic to language. Even though the technique can be applied to multilingual settings, optimizing performance is an open question. In our work we show that leveraging higher-resource, task-specific language data can boost overall performance and we propose a method to select training examples per their average performance in a Monte Carlo simulation, resulting in a training set more conducive to learning. We demonstrate the effectiveness of our methods in fashion text reviews moderation, classifying reviews as related or unrelated to the given product. We show that our methodology boosts performance in multilingual (English, French, German) settings, increasing F1 score and significantly decreasing false positives.

## introduction in real-life settings, machine learning methods are being applied to automate and improve processes, from content moderation to search query filtering. large pretrained language models have the potential to bring further improvements, at the cost of data resources, with high quantities of labeled data required for effective training. collecting, cleaning, annotating and analyzing data is an expensive, time-consuming and challenging task. and zero-shot capabilities, able to solve tasks with little or no data @xcite . for example, with pet @xcite , models can be finetuned on a task using only a few training examples. annotating a small number of examples is desirable and applicable in academic as well as real-life settings. an added complication in many scenarios is the need to develop solutions for multilingual settings. annotating data, developing and evaluating models is increasingly more challenging when there are multiple languages to consider. this is exacerbated in settings where certain markets are larger and higher-resource than other markets. developing solutions that are scaleable to both high-and low-resource markets is challenging. we work in the domain of customer text reviews on a fashion platform. when a customer leaves a review on one of their purchased products, the review goes through moderation to verify whether it abides by the platform's code of conduct. for example, a review may be rejected because it contains offensive content or personal data. in our work, we are focusing on the task of identifying whether a review is related or unrelated to the product. when a submitted review is unrelated to the product (e.g., a review that only mentions delivery time), the review is rejected. further, on our platform, multiple languages are covered across markets, with the majority of content written in german. thus, in our work, german is the focus language, with expansion to the in-domain lower-resource english and french. in synopsis, in an industry setting dealing with multiple markets and languages (e.g., an online shopping platform), (i) annotating large quantities of data for all languages is expensive and, (ii) the language of the dominant market makes up most of the available data. in line with these two observations, our contributions are: we show that multilingual models finetuned on all languages perform better than their monolingual counterparts and that with our monte carlo selection method we can extract the training examples most conducive to learning to achieve improved performance, both in the monolingual and multilingual settings. ## related work it has been shown that large pretrained language models exhibit strong cross-lingual abilities, with cross-lingual transfer investigated extensively @xcite @xcite @xcite @xcite . in our work, we make use of cross-lingual transfer from higher-to lower-resource in-domain languages to improve performance. with the emergent abilities of large language models @xcite , large models are being applied to few-and zero-shot settings @xcite @xcite , showcased saliently in gpt-3, where prompting was shown to be effective across a range of tasks. to aid in few-shot learning, pattern-exploiting training (pet) was introduced in schick and schütze (2021a), allowing for training of large language models in few-shot settings via the use of prompts. it has been further shown that pet is competitive with models orders of magnitude larger @xcite . @xcite showed that prompting can be employed in multilingual settings, with the authors showing that multilingual and multitask settings can be modeled without the use of language or task specific modules or training, by using prompts to leverage transfer learning capabilities. in our work we make use of prompts in a similar fashion, employing prompting to improve language transfer. language prompt german [mask]: r r: die beurteilung ist [mask] r ist [mask] english [mask]: r r: the review is [mask] r is [mask] french [mask]: r r: l'avis est [mask] while training examples can have a large impact and add significant noise during training (due to the small size of the set), selecting examples most conducive to learning is still under-explored. ## conclusion in our work, we investigate how to improve discrimination between customer reviews related and unrelated to the product in low-resource settings for english, french and german. we show that via multilingual transfer learning we can improve performance of models in english and french, leveraging the in-domain higher-resource german data, while at the same time reducing the rate of false positives across all languages. selecting training examples most conducive to performance in few-shot learning is of paramount importance and still an open question. we propose a method to extract such examples through a monte carlo simulation, selecting training examples with the highest average performance across experiments. we show that with our method we can improve performance both in monolingual and multilingual settings, outperforming baselines with orders of magnitude more data as well as all models trained on randomly-sampled sets, consistently increasing f1 scores and decreasing false positives.