KS @ LTH at S em E val-2020 Task 12: Fine-tuning Multi- and Monolingual Transformer Models for Offensive Language Detection (2020)

Abstract: This paper describes the KS@LTH system for SemEval-2020 Task 12 OffensEval2: Multilingual Offensive Language Identification in Social Media. We compare mono- and multilingual models based on fine-tuning pre-trained transformer models for offensive language identification in Arabic, Greek, English and Turkish. For Danish, we explore the possibility of fine-tuning a model pre-trained on a similar language, Swedish, and additionally also cross-lingual training together with English.

## introduction offensive language is a prevalent phenomenon in many online communities and social media platforms. due to the vast amount of content, it is often infeasible to manually moderate all user submitted content. computational methods for identifying this type of content is one possible way to help mitigate the problem. different aspects of the problem such as aggression @xcite , cyber bulling @xcite and hate speech @xcite have been studied in recent work. offenseval 2019 used a new three-level hierarchical annotation schema to capture multiple aspects of offensive language in one framework @xcite . while much of the previous work is focused on english, offensive language detection is a multilingual problem. apart from country specific communities, large social media platforms such as facebook and twitter have many users interacting in their native tongue. recently, offensive language detection addressed different languages such as german @xcite , arabic @xcite , italian @xcite , and spanish @xcite . in offenseval 2020, the first level task of offensive language detection has been expanded to cover five languages, arabic, danish, english, greek, and turkish. transfer learning is nothing new in nlp but over time, the pre-training has become more complex, incorporating more context. in recent years, language models based on the transformer architecture pre-trained on large amounts of unlabeled text and then fine-tuned on downstream tasks have been used to achieve state-of-the-art (sota) results on many natural language benchmarks @xcite @xcite . in offenseval 2019, seven of the top ten models used bert in some way @xcite . one of the advantages of transfer learning is that it can potentially reduce the amount of labeled data that is needed. the model can learn general features of language from a large unannotated corpus during pre-training. task specific features can then be learned from a smaller annotated corpus. on some datasets, using a pre-trained language model has shown to match the results of models trained from scratch on ten times more data. adding language model fine-tuning on unlabeled domain specific text can potentially reduce the need for labeled data even more @xcite . one obstacle to using large transformer models is that the pre-training step is expensive. the megatron-lm has 8.3 billion parameters and was trained over 9 days on 512 gpus @xcite . in comparison, the fine-tuning step is relatively inexpensive. this makes model sharing an important part of applying large transformer models to many tasks. the huggingface transformers library provides a platform for sharing models developed by researchers and the community, and a unified api for using them @xcite . one additional challenge with multilingual offensive language detection is low resource languages. such languages might lack both unlabeled data for pre-training and labeled data for fine-tuning. one possible solution in such cases is to use multilingual models. such models can achieve lower perplexity than monolingual models for language modeling of low resource languages @xcite . in some contexts, multilingual models can even outperform monolingual models on downstream tasks @xcite . in the case of lacking labeled data, they have also shown to perform well on zero-shot cross-lingual classification tasks. this type of transfer works best between typologically similar languages. however, transfer is possible to some extent even between languages with different scripts @xcite . this paper describes our system for offenseval 2020 @xcite . we participated in sub-task a: offensive language identification for all language tracks. based on the recent success of the transformer architecture, we compared monolingual bert models for arabic, english, greek, and turkish with the xlm-r multilingual model @xcite . we found that the monolingual models outperform the multilingual models for all languages on the development data. we used models available through the huggingface transformers library. since no monolingual models were available for danish, we initially compared a swedish bert model with multilingual xlm-r. we found that the swedish model worked reasonably well on the development data, while xlm-r only predicted the majority class for most runs. we hypothesized that this is due to the small and imbalanced danish dataset; similar high variance results have been seen for bert in @xcite and @xcite . to get around the problem of the small dataset, we tried cross-lingual training of danish and english using xlm-r which outperformed the swedish bert model. in section 2 we give a short description of the task and data used. section 3 presents our approach, describing data preprocessing, models and training approach. section 4 shows our results on the test data for offenseval 2020. ## task and data offenseval 2020 uses a multilingual dataset of posts from twitter, tweets, with annotations following the hierarchical annotation schema proposed by @xcite . only the first level of annotation is provided for all languages. this level discriminates between two kinds of tweets: the goal of the task is to distinguish between offensive and not offensive tweets. macro-averaged f1-score is used as evaluation metric. for english the labeled dataset is the same as for offenseval 2019. in addition to the manually annotated data, about nine million additional tweets labeled using unsupervised methods are provided for english @xcite . these tweets have a confidence score and a standard deviation. for english, we also use additional publicly available data from kaggle foot_0 and @xcite 2 . language off not total turkish (c ¸öltekin, 2020) 6131 (19%) 25625 (81%) 31756 english @xcite 4640 (33%) 9460 (67%) 14100 greek @xcite 2486 (28%) 6257 (72%) 8743 arabic @xcite 1589 (20%) 6411 (80%) 8000 danish @xcite 384 (13%) 2576 (87%) 2960 ## results ## conclusions in the context of offensive language detection for multiple languages, we found that fine-tuning transformer models works well. monolingual models outperform multilingual models for all languages studied. however, multilingual models can still be a viable alternative when no monolingual models are available. when the amount of labeled data is small, they can also be used for cross-lingual transfer. we showed the positive effect of cross lingual transfer when augmenting danish with english.