Data Augmentation with Adversarial Training for Cross-Lingual NLI (2021)

Abstract: Due to recent pretrained multilingual representation models, it has become feasible to exploit labeled data from one language to train a cross-lingual model that can then be applied to multiple new languages. In practice, however, we still face the problem of scarce labeled data, leading to subpar results. In this paper, we propose a novel data augmentation strategy for better cross-lingual natural language inference by enriching the data to reflect more diversity in a semantically faithful way. To this end, we propose two methods of training a generative model to induce synthesized examples, and then leverage the resulting data using an adversarial training regimen for more robustness. In a series of detailed experiments, we show that this fruitful combination leads to substantial gains in cross-lingual inference.

## introduction there is a growing need for nlp systems that support low-resource languages, for which taskspecific training data may be lacking, while domain-specific parallel corpora may be too scarce to train a reliable machine translation engine. to overcome this, zero-shot cross-lingual systems can be trained on a source language l s and subsequently also be applied to other languages l t despite a complete lack of labelled training data for those target languages. in the past, such systems typically drew on translation dictionaries, lexical knowledge graphs, or parallel corpora, to build a cross-lingual model that exploits simple connections between words and phrases across different languages @xcite . recently, pretrained language model architectures such as bert @xcite have been shown capable of learning joint multilingual representations with self-supervised objectives under a shared vocabulary, simply by combining the input from multiple languages @xcite @xcite . such representations greatly facilitate cross-lingual applications. still, the success of such cross-lingual transfer hinges on how close the involved languages are, with substantial drops observed for some more distant language pairs @xcite . for our study, we focus on natural language inference (nli), i.e., classifying whether a premise sentence entails, contradicts, or is neutral with regard to a hypothesis sentence @xcite . this is a useful building block for applications involving semantic understanding @xcite . however, the task is also very challenging, as it not only requires accounting for very subtle differences in meaning but also inferring presuppositions and implications that are not explicitly stated. due to these intricate subtleties, zero-shot cross-lingual models are often fairly brittle, while obtaining in-language training data is fairly costly. ## method our proposed method consists of two steps. the first involves inducing training examples with two data augmentation models. next, a task-specific classifier is trained on both the original and the newly generated training instances, with adversarial perturbation for improved robustness and generalization. ## conclusion while multilingual pretrained model have enabled better cross-lingual learning, we still often encounter data scarcity issues due to the high cost of collecting data, which weakens the generalization ability of the multilingual model. to address this, this paper proposes a novel data augmentation strategy with label rectification to build synthetic examples, outperforming even models trained with larger amounts of ground-truth data. we show that we can best learn from such noisy instances with adversarial training, which enables the classifier to transfer more information from the source language to other languages and to become more robust. remarkably, with this, our models trained without any target language training data at all are able to outperform models trained fully on in-language training data. moreover, the amount of augmented data from our seq2seq-based reorder augmentation used in training is much less than that required by the state-of-the-art eda method in order to achieve comparable performance. finally, in our series of follow-up experiments comparing different training regimens and variants, one notable finding is that our overall augmented approach can even outperform non-augmented supervision with twice as many ground truth labels. overall, this suggests our combination of data augmentation with adversarial training as a valuable way of learning substantially more accurate and more robust models without any target-language training data.