Vicinal Risk Minimization for Few-Shot Cross-lingual Transfer in Abusive Language Detection (2023)

Abstract: Cross-lingual transfer learning from high-resource to medium and low-resource languages has shown encouraging results. However, the scarcity of resources in target languages remains a challenge. In this work, we resort to data augmentation and continual pre-training for domain adaptation to improve cross-lingual abusive language detection. For data augmentation, we analyze two existing techniques based on vicinal risk minimization and propose MIXAG, a novel data augmentation method which interpolates pairs of instances based on the angle of their representations. Our experiments involve seven languages typologically distinct from English and three different domains. The results reveal that the data augmentation strategies can enhance few-shot cross-lingual abusive language detection. Specifically, we observe that consistently in all target languages, MIXAG improves significantly in multidomain and multilingual environments. Finally, we show through an error analysis how the domain adaptation can favour the class of abusive texts (reducing false negatives), but at the same time, declines the precision of the abusive language detection model.

## introduction few-shot learning (fsl) is a machine learning paradigm that allows models to generalize from a small set of examples @xcite . unlike traditional methods, fsl does not require training a model from scratch. instead, pre-trained models are extended with just a little information, which is useful when training examples are scarce or data annotation is expensive. ## dataset and experimental setup xhate-999 @xcite ) is an available dataset intended to explore several variants of abusive language detection. this dataset includes three different domains: fox news (gao), twitter/facebook (trac), and wikipedia (wul). in our work, we define all as the set of instances resulting from the union of all three domains. each domain comprises different amounts of annotated data (abusive/non-abusive) in english for training, validation, and testing (see appendix a). english test instances are translated into five target languages: albanian (sq), croatian (hr), german (de), russian (ru), and turkish (tr). we extended this dataset with texts in spanish. to generate the texts, we rely on machine translation and post-editing, following the monitored translation-based approach described in the dataset paper. thus, slight modifications were made in the spanish translation to reflect and maintain the level of abuse in the original english instances. ## few-shot cross-lingual transfer we first examine the ability of three vrm-based techniques in few-shot cross-lingual transfer learning for abusive language detection to address rq1. @xcite propose ssmba, a data augmentation method for generating synthetic examples with a pair of corruption and reconstruction functions to move randomly on a data manifold. in the corruption function, we use two strategies: 1) masking a word in each text in a random way (default) or 2) masking the salient abusive words in each text. to identify abusive words, we use hurtlex @xcite , a multilingual lexicon with harmful words. for texts that do not contain words in the lexicon, we follow strategy 1. in the reconstruction functions, we use mbert. ## unsupervised language adaptation in this section, we investigate the scenarios in which there is no information about the target language for the few-shot cross-lingual transfer. in § 4 we used a small amount of supervised data d train in the target language to fine-tune the pre-trained model. this allowed us to adapt the model to the abusive language of each particular language. in contrast, now we assume that the labels of d train are not available. this is a simulated experiment where we only have an unlabelled set of texts and the set d test in which we want to detect abusive language. previous works have examined this scenario by adjusting a model with unlabelled external data. in this work, we use only a few unlabelled instances from d train . basically, this strategy is a zero-shot crosslingual transfer learning in which the model is adapted to the abusive terms of the target language. as mbert is pre-trained on general-purpose and multilingual corpora, it is familiar with the target languages. however, it has not been adjusted to the particular case of abusive language. we follow then a two-step methodology: 1) continual pretaining for domain adaptation via masked language modeling (mlm) to make it familiar to the partic- ## conclusions and future work in this work, we studied three techniques to improve few-shot cross-lingual transfer learning in abusive language detection. these techniques are concentrated on data-level approach to deal with the problem of data scarcity that can lead to a high estimation error in few-shot learning. specifically, we focused on vicinal risk minimization techniques to increase the data in the vicinity of the in future work, we aim to further examine our proposed vrm-based technique for data augmentation. mixag uses as a parameter the angle between the new instance and one of the original instances being combined. in our experiments, we fixed the angle as half the angle between the original instances, but we consider that the flexibility of varying that parameter must be exploited.