LLM -powered Data Augmentation for Enhanced Cross-lingual Performance (2023)

Abstract: This paper explores the potential of leveraging Large Language Models (LLMs) for data augmentation in multilingual commonsense reasoning datasets where the available training data is extremely limited. To achieve this, we utilise several LLMs, namely Dolly-v2, StableVicuna, ChatGPT, and GPT-4, to augment three datasets: XCOPA, XWinograd, and XStoryCloze. Subsequently, we evaluate the effectiveness of fine-tuning smaller multilingual models, mBERT and XLMR, using the synthesised data. We compare the performance of training with data generated in English and target languages, as well as translated English-generated data, revealing the overall advantages of incorporating data generated by LLMs, e.g. a notable 13.4 accuracy score improvement for the best case. Furthermore, we conduct a human evaluation by asking native speakers to assess the naturalness and logical coherence of the generated examples across different languages. The results of the evaluation indicate that LLMs such as ChatGPT and GPT-4 excel at producing natural and coherent text in most languages, however, they struggle to generate meaningful text in certain languages like Tamil. We also observe that ChatGPT falls short in generating plausible alternatives compared to the original dataset, whereas examples from GPT-4 exhibit competitive logical consistency.

## introduction the success of nlp models greatly depends on the availability and quality of training data. this poses a significant challenge for multilingual nlp, as data for languages other than english is typically limited @xcite @xcite . an approach to address the data scarcity challenge is through zero-shot crosslingual transfer or multitask training, in which a model is trained across data of diverse tasks and languages, exhibiting the capability to handle unseen tasks, particularly in larger models @xcite @xcite . however, when aiming for task-specific objectives, a smaller, fine-tuned model dedicated to that particular task often outperforms larger general-purpose, zero-shot models. in addition, a smaller task-specific model is more practical and cost-effective for training and deployment. nevertheless, developing a powerful task-specific model becomes challenging in the absence of training data @xcite . conversely, recent powerful large language models (llms) excel at handling general instructions and have shown promise in data generation tasks @xcite . in this work, we leverage llms to generate synthetic data for various multilingual commonsense reasoning tasks, xcopa @xcite , xwinograd @xcite , and xstorycloze @xcite , where the training data is limited even for english (see we develop task-specific models by fine-tuning multilingual pre-trained language models, namely mbert @xcite and xlm-r @xcite , using the generated data. we then compare their performance against models trained on a limited set of human-created data in the target language whenever available, and otherwise through zero-shot transfer learning from manually created english training data. our experiments demonstrate that training the models with relatively large synthetically generated datasets yields better performance than training with limited manuallycreated datasets. this finding empirically confirms the utility of synthetic data generated by llms for improving downstream task-specific models. we expand the multilingual data synthesis using chatgpt and gpt-4 on xcopa and find that generating multilingual datasets generally surpasses the effectiveness of the zero-shot cross-lingual transfer. we further assess the quality of the generated dataset in different languages by asking native speakers to evaluate the naturalness and logical soundness of the generated dataset compared to the human-written examples. the annotation results reveal that while chatgpt and gpt-4 successfully generate natural text in most languages, they struggle with generating understandable text in certain languages such as tamil. moreover, a noticeable gap is observed in terms of commonsense coherence when comparing chatgpt-generated data to human-constructed data. on the other hand, gpt-4 significantly narrows this difference. to summarise, our work has the following key contributions: ## dataset augmentation our experiments use xcopa, xwinograd, and xs-torycloze, which are selected due to (1) the limited availability of training data and (2) commonsense reasoning datasets present greater challenges for data synthesis. xcopa is a cross-lingual choice of plausible alternatives dataset that translates and re-annotates the validation and test sets of english (en) copa @xcite into 11 target languages (et: estonian, ht: haitian creole, id: indonesian, it: italian, qu: quechua, sw: swahili, ta: tamil, th: thai, tr: turkish, vi: vietnamese, and zh: chinese). 3 each instance consists of a premise, a question (cuase/result), and two alternatives. the task is to predict the more plausible alternative. xwinograd expands the original english winograd schema challenge (wsc) @xcite to five other languages (fr: french, ja: japanese, pt: portuguese, ru: russian, and zh), foot_3 which consists of pronoun resolution problems aim-ing to evaluate the commonsense reasoning ability of a machine. given a statement with two noun phrases and a pronoun, the challenge of wsc is to determine the referent of the pronoun, which can only be inferred from the context. xstorycloze is collected by @xcite , where the validation split of the original english storycloze dataset @xcite is translated into 10 other typologically diverse languages (ru, zh, es: spanish, ar: arabic, hi: hindi, id, te: telugu, sw, eu: basque, and my: burmese). each example consists of a foursentence commonsense story, a correct ending, as well as a wrong ending. ## experimental setups we first generate synthetic english examples for xcopa, xwinograd, and xstorycloze, with dolly-v2, stablevicuna, chatgpt, and gpt-4. the size of the final filtered synthesised data for the three datasets is 3.7k, 2k, and 1.7k, respectively. we then fine-tune mbert, xlmr-base, and xlmr-large with the synthesised data and compare the zero-shot cross-lingual transfer performance across different languages, where we use the original validation set in target languages. for xcopa, we additionally experiment with generating data points directly in non-english languages, by providing examples in the target language and specifying the language desired for the generated data (see and stablevicuna exhibit limitations in effectively generating multilingual text. the size of the multilingual synthesised data is ∼3.6k in each language. we fine-tune models on all datasets as multiplechoice tasks 8 by searching best learning rate from {5e -6 , 10e -6 }, and batch size from {8, 16, 32}. all the fine-tuning experiments are conducted on a single 40g a100. for generating data with dolly-v2 and stablevicuna, we use 2×40g a100. ## results and discussion this section presents the main results of fine-tuned models on the three datasets and compares performance with generated data in different llms, languages, and scales. ## human evaluation to better evaluate the quality of the generated datasets and compare them with the human-created data, we ask native speakers to annotate the multilingual data generated by chatgpt and gpt-4. for each dataset, we first select 50 generated examples in english, and then request two annotators to evaluate the examples in two categories: (1) text naturalness. the annotators are asked to choose one of the following options for each example: "the text sounds natural", "the text sounds awkward but understandable", or "the text is not understandable", and (2) logic soundness. this category focuses on the commonsense aspect of the examples. the annotators are required to select the most appropriate description from: "the correct option is (clearly) more plausible", "both options are equally plausible", "both options are implausible", or "the wrong option is actually more plausible". we only ask the annotators to evaluate the logic if the text is at least understandable. for xwinograd, we introduce an additional evaluation criterion. annotators are asked to determine whether the two noun phrases in the examples can be replaced by the same pronoun (refer to §3.2). for xcopa, we extend the annotations to non-english languages, where we choose the two languages that demonstrate the most notable improvement, namely zh and id, as well as the two languages that exhibit the least improvement or regres-sion in performance with chatgpt-generated data, namely ta and sw (see to ensure impartiality, all the examples are shuffled, and the annotators are not provided with information regarding the source of the examples (human-created, llm-generated, or translated). ## conclusions this paper explores the effectiveness of utilising llms for data augmentation in cross-lingual datasets with limited training data. we specifically focus on commonsense reasoning tasks that are challenging for data synthesis. our experiments including four llms for data generation on three datasets, showcase enhanced cross-lingual zeroshot transfer on smaller fine-tuned task-specific language models. however, the impact varies across different datasets and languages. notably, larger models such as xlmr-large, which have higher baselines, demonstrate more difficulty in achieving performance improvements with llm-generated data. among the four llms, gpt-4-generated data exhibits mostly consistent superior performance. expanding data generation directly in target languages also shows general improvements compared to cross-lingual zero-shot with the englishgenerated data. human evaluation of the synthesised multilingual dataset shows that the chatgpt and gpt-4 generated data demonstrate high naturalness in most languages, even surpassing the original data. however, in certain languages like ta, both models fail to generate natural text. additionally, when assessing the logical soundness of the dataset, examples synthesised by chatgpt reveal notable inconsistencies regarding more plausible options compared to the original human-created data. in contrast, gpt-4 exhibits a performance on par with human-written data. in conclusion, leveraging llms for data augmentation shows promise. however, the choice of llm used for data generation significantly influences the quality of the resulting data, as well as its applicability to the language under consideration. in circumstances where a more advanced model such as gpt-4 cannot be accessed, other models can be utilised, though this might result in performance difficulties in certain non-english languages -a challenge that also exists for gpt-4 -and concerns regarding logical coherence. a compelling direction for future research could involve exploring the efficacy of more recent instruction-tuned or aligned open-source llms, such as llama 2 @xcite or tülu @xcite , in enhancing data augmentation.