DFKI SLT at G erm E val 2021: Multilingual Pre-training and Data Augmentation for the Classification of Toxicity in Social Media Comments (2021)

Abstract: We present our submission to the first subtask of GermEval 2021 (classification of German Facebook comments as toxic or not). Binary sequence classification is a standard NLP task with known state-of-the-art methods. Therefore, we focus on data preparation by using two different techniques: task-specific pre-training and data augmentation. First, we pre-train multilingual transformers (XLM-RoBERTa and MT5) on 12 hatespeech detection datasets in nine different languages. In terms of F1, we notice an improvement of 10% on average, using task-specific pre-training. Second, we perform data augmentation by labelling unlabelled comments, taken from Facebook, to increase the size of the training dataset by 79%. Models trained on the augmented training dataset obtain on average +0.0282 (+5%) F1 score compared to models trained on the original training dataset. Finally, the combination of the two techniques allows us to obtain an F1 score of 0.6899 with XLM- RoBERTa and 0.6859 with MT5. The code of the project is available at: https://github.com/airKlizz/germeval2021toxic .

## introduction toxicity classification, or, more generally, hatespeech detection, has become a highly important topic due to the explosion of social media use. the automation of this task is a challenge for the nlp field with an increasing amount of research on this subject @xcite @xcite . the germeval series has already looked into various aspects related to the detection of german language hatespeech with two shared tasks on offensive language identification @xcite . the first subtask of germeval 2021 follows in these footsteps with the classification of toxic comments. we want to take advantage of the proliferation of hatespeech datasets for various languages created in the last couple of years. additionally, in the meantime, a number of multilingual language models have been published @xcite @xcite with a high capacity for cross-lingual transfer. we use multilingual models and pre-train them on a multilingual dataset created out of 12 datasets for nine different languages on toxicity and hatespeech detection. we evaluate whether performing this type of pre-training on multilingual models can improve their performance. we assume that the cross-lingual transfer capacity of the multilingual models can be applied to task-specific pre-training and that this will improve final performance on the german-only dataset of the shared task. furthermore, we perform data augmentation by labelling unlabeled data, retrieved from facebook, using one of the multilingual models pre-trained and fine-tuned on the toxicity classification task. as the dataset of the shared task contains only 3244 examples, we hope that extending the number of training examples can improve the overall performance of the models. in summary, our main contributions are: the rest of this article is structured as follows. section 2 presents our methodology for taskspecific pre-training and data augmentation. section 3 introduces the task as well as the dataset and describes the models and training scenarios. sections 4 and 5 present and discuss the results obtained in these training scenarios. concluding remarks are provided in section 6. ## methodology pre-training is a technique that often enables state-of-the-art performance in many nlp tasks @xcite . task-specific pre-training has shown its efficiency to produce models that capture task-specific features and that, thus, exhibit better performance @xcite . we want to profit from the many existing hatespeech classification datasets by using these datasets to perform task-specific pre-training. we adapt task-specific pre-training to toxicity classification by taking 12 toxicity or hatespeech classification datasets and training language models on these datasets before fine-tuning them on the dataset of the shared task ( to take advantage of this task-specific multilingual pre-training, we work with multilingual models. indeed, these models have already demonstrated their ability to transfer what they have learned in one language into other languages @xcite . in this work, the models will be finetuned on the dataset of the shared task which is in german only, however, we assume that the multilingual models can benefit from the task-specific pre-training. ## results we evaluate the models on the test dataset provided by the organizers of the shared task after the training phase and the submissions (see first, adding task-specific pre-training and/or using data augmentation improves the results for both xlm-roberta and mt5. training with task-specific pre-training and data augmentation improves the f1 score by 0.0490 (+8%) for xlm-roberta and by 0.0836 (+14%) for mt5. gbert ## discussion in the two scenarios where only german data is used (fine-tuning only and with data augmentation), gbert performs better than xlm-roberta and mt5. this is easily explained by the fact that gbert was pre-trained only on german data, in contrast to mt5 and xlm-roberta. however, the small difference in f1 scores with the use of the augmented dataset (with data augmentation) implies that with more data, multilingual models can perform as well as monolingual models. additionally, we see that the task-specific pre-training of multilingual models on a multilingual dataset compensates for the poorer performance of mt5 and xlm-roberta when trained on a german only dataset compared to gbert. it is interesting to note that the task-specific pre-training of mt5 and xlm-roberta on a multilingual dataset allows them to perform better than gbert. the fact that multilingual models can benefit from hatespeech classification datasets in other languages allows them to perform better than the german-only model. it is also important to notice that xlm-roberta and mt5 use more recent architectures and/or pre-training methods than gbert. it may also partly explain that gbert's results are worse than those of xlm-roberta and mt5. moreover, as noted in section 4, xlm-roberta does not benefit from the task-specific pre-training as much as mt5. our hypothesis is that having less trainable parameters, xlm-roberta (270m parameters) does not have as much capacity as mt5 (580m parameters) to benefit from all the examples on which the models are pre-trained. the number of parameters of the models is an important aspect to take into consideration when doing pre-training in general, and we observe this again in our experiments with task-specific pre-training. ## conclusion we describe the methods used for our submissions to the germeval 2021 toxic comment classification task. specifically, we can benefit from hatespeech detection datasets in other languages to improve the performance of multilingual models through taskspecific pre-training. with this method, multilingual models (xlm-roberta and mt5) perform even better, +0.0576 (+10%) in average in terms of f1, than gbert, a german-specific language model. we show that by increasing the shared task dataset by automatically labeling additional comments from facebook, we are able to improve the results of the three models we evaluated (gbert, xlm-roberta, mt5) by 5% in average. we have shown that multilingual models can perform as well or even better than monolingual models by performing task-specific multilingual pre-training. this particularly applies to tasks for which many datasets are available in languages different from the dataset used for fine-tuning and where the fine-tuning dataset is relatively small (less than 10,000 samples) as is the case of the german toxic comment classification task. in addition, multilingual models have some other advantages. first, in a production setting, it might not be feasible to deploy multiple monolingual models due to resource constraints. replacing multiple monolingual models with a single multilingual model can be a solution. second, multilingual models, due to their cross-lingual transfer capacity, can be used in a language other than the language of the training dataset. this allows the creation of models for languages for which obtaining training data can be difficult.