Boosting Zero-Shot Crosslingual Performance using LLM -Based Augmentations with Effective Data Selection (2024)

Abstract: Large language models (LLMs) are very proficient text generators. We leverage this capability of LLMs to generate task-specific data via zero-shot prompting and promote cross-lingual transfer for low-resource target languages. Given task-specific data in a source language and a teacher model trained on this data, we propose using this teacher to label LLM generations and employ a set of simple data selection strategies that use the teacher’s label probabilities. Our data selection strategies help us identify a representative subset of diverse generations that help boost zero-shot accuracies while being efficient, in comparison to using all the LLM generations (without any subset selection). We also highlight other important design choices that affect cross-lingual performance such as the use of translations of source data and what labels are best to use for the LLM generations. We observe significant performance gains across sentiment analysis and natural language inference tasks (of up to a maximum of 7.13 absolute points and 1.5 absolute points on average) across a number of target languages (Hindi, Marathi, Urdu, Swahili) and domains.

## introduction multilingual pretrained models are a mainstay in modern nlp. to create highly-performant taskspecific models across different languages, a commonly adopted paradigm is to finetune a multilingual pretrained model like xlm-r @xcite using task-specific labeled data. in the absence of labeled data for a target language, pretrained models finetuned on task-specific data in a source language (such as english) have been shown to facilitate zero-shot crosslingual transfer @xcite @xcite . given large language models (llms) and their superior generation capabilities, a natural question is whether they can be used to generate synthetic task-specific data in english. to create synthetic data in a (non-english) target language, these llm generations could be further translated into target language data using existing machine translation systems. in this work, we examine the following central question: how do we make best use of llm generations to improve zero-shot foot_0 crosslingual transfer to target languages without any labeled data? we stress here that we are interested in the realistic setting where task-specific data in the source language might vary in domain from the target language tasks; this setting is largely absent in zero-shot evaluations in prior work. our overall data augmentation pipeline is illustrated in figure 1 . we use an open-source llm such as llama-2 @xcite and prompt it to generate task-specific text in english. for all target languages, we assume access to task-specific data in english that may not be in the same domain as the target-language tasks. when domain information is available for the target tasks, we add this information in the prompt to generate text that appears to be more in-domain. pseudolabels for synthetic data. we examine two choices to generate pseudolabels for the llm generations: 1. via prompting the llm, to appear in the output as part of the generation. 2. via a teacher model trained on task-specific english data. we explore two ways in which the abovementioned pseudolabels can be used when training models with synthetic data. we train a single model on task-specific english data augmented with generations pseudolabeled via llm prompts. we also adopt teacher-student training where the teacher is trained on task-specific english data and the student model is trained on synthetic data with soft teacher labels (i.e. label distributions). given label noise, we find training a student model with soft teacher labels yields significantly more accurate models compared to training with hard teacher labels. ## related work our work is closely related to whitehouse et al. that studies generations from various open-source and commercial llms for cross-lingual performance over reasoning tasks. however, they rely on instances from the target sets as few-shots for generations and do regular finetuning over labels derived from the class-conditional prompts. furthermore, our work is grounded in ideas inspired by he et al., incorporating self-training on unlabeled synthetic text produced by language models. however, he et al. fine-tuned generators using target data, and their experiments were limited to glue tasks (in english). in contrast, our focus is on multilingual models, aiming for crosslingual transfer from source data in a high-resource language across arbitrary domains in different task languages. we achieve this by self-training on zeroshot generations from llms without utilizing any target data during generation or training. we draw inspiration from @xcite to design data selection techniques based on the principle of dataset cartography. @xcite use dataset cartography on a large nli dataset (mnli) to choose instances with complex reasoning patterns, and instructs gpt-3 to generate new examples with similar patterns. automatically generated examples undergo filtering, and ultimately, human crowdworkers review, revise, and label them. in a similar vein, khanuja et al. present language-agnostic methods to pick specific data points to be labeled from a large, unlabelled multilingual dataset. these points are chosen either by considering their distance from the target set, the uncertainty of model predictions over them, or finding a balance between minimizing distance and maximizing model uncertainty. while wanli @xcite only explores english generation/evaluation, both depend not only on human-in-the loop annotation of unlabeled text, but also depend on existing target data for generator finetuning @xcite or few-shot prompting @xcite . de raedt et al. propose in-place augmentation of data instances from high-resource languages for better out-of-distribution generalization by leveraging llms. however, these techniques are specifi-cally applicable to single-text classification tasks. for tasks like nli or qa, which involve multiple components in each instance, automatically making such edits or augmentations while preserving the intended relationships between components (and affecting labels) is not straightforward. in contrast to @xcite , and @xcite , we make use of an open-source llm to generate task and domain-specific synthetic data. we also explore the more realistic setting of having no access to source data that matches the target domain; this is not explored in the above two works. similar to our work, synthetic data generation has been explored in @xcite . however, both works do not use any form of data selection for the synthetically generated data. @xcite do not show evaluations on low-resource languages; their multilingual experiments are limited to high-resource languages such as english, french, spanish and german. @xcite show experiments in the few-shot setting, while we operate in the zero-shot setting. ## conclusion in this work, we focus on the broader problem of boosting zero-shot cross-lingual transfer using llm-based augmentations. we highlight the importance of using data selection strategies to select smaller subsets that result in more efficient training and improved performance on downstream target language tasks. we also compare and contrast the utility of pseudolabeling generations using labels from llm prompts versus using a teacher model to label the generations. one of the main takeaways is that llm generations, in conjunction with our data selection strategies, can help improve cross-lingual transfer regardless of whether task-specific source data matches the domain of the target tasks or not.