ANDES at S em E val-2020 Task 12: A Jointly-trained BERT Multilingual Model for Offensive Language Detection (2020)

Abstract: This paper describes our participation in SemEval-2020 Task 12: Multilingual Offensive Language Detection. We jointly-trained a single model by fine-tuning Multilingual BERT to tackle the task across all the proposed languages: English, Danish, Turkish, Greek and Arabic. Our single model had competitive results, with a performance close to top-performing systems in spite of sharing the same parameters across all languages. Zero-shot and few-shot experiments were also conducted to analyze the transference performance among these languages. We make our code public for further research

## introduction offensive language, hate speech, cyberbullying, and abusive language detection are topics that have summoned a lot of interest in the last years, particularly due to the necessity in social media to stop -or at least, to diminish-this omnipresent phenomenon. not only the social implications are behind this, but also practical implications for companies: recently, some large advertisers have removed their presence from social media as they consider that platforms full of "divisiveness and hate speech" do not give value to their companies @xcite . the differences between these categories are loosely defined. @xcite propose using two axes to understand the particularities of each discourse: whether the abusive language is directed to a specific individual or to a generalized group, and whether the offending discourse is explicit or implicit. even when online platforms prohibit behavior that crosses the line into abuse, 2 these rules are frequently violated. automated moderation algorithms are necessary to perform a faster and even better usergenerated-content moderation or to serve as a tool that helps human moderators to reduce the volume of offensive content still present in online platforms. to address this problem, offensive language detection is usually thought of as a binary classification problem in which the input is a post @xcite and the output is the classification of whether the post is offensive or not. several features of offensive language make it complex to detect. the offenders could intentionally hide offensive words by substituting letters with special characters or numbers. content with irony or sarcasm could be harmful even when all words are polite, and vice versa. the real intention behind speech is sometimes difficult to detect even for humans. according to @xcite , in the case of twitter, each tweet provides a fairly limited context. therefore, an offensive post may be identified as unoffensive if the context is not taken into account. despite several published proposals on detecting offensive language, hate speech, and other related concepts, there is no consolidated or effective strategy that could be applied to different languages and domains. building a quality labeled dataset is an expensive task in time and effort. most available datasets are in english, and often datasets are not published due to privacy concerns, making research on offensive language detection difficult for other languages. the generalization power of learning models would allow us to transfer knowledge to languages with poor resources. in this work, we describe our participation in semeval 2020 task 12: offenseval, an offensive language detection task in five different languages. we propose a single multilingual system based on multilingual bert @xcite to jointly address the offensive language detection. we trained our system using all the available training data and evaluated its performance for each of the languages instead of adjusting language-specific models. our single model performs fairly well, achieving performances comparable to the winning teams for each language. to further explore the multilingual dimension of bert, we analyze zero-shot and few-shot capabilities in a cross-lingual setting for the task in question. we make our code public for further research. ## background the detection of offensive language, cyberbullying and hate speech are tasks that are closely connected and often confused @xcite . several machine learning models addressing hate speech or offensive language detection have been proposed in the last years; in particular deep learning models @xcite @xcite @xcite @xcite have increased their popularity among researchers on this task. despite the growing interest in the area, the models are usually trained and evaluated inside very specific english datasets, and their generalizability to other contexts or languages is still a challenge. morever, building these datasets is difficult @xcite and achieved performances are often overestimated @xcite . there are only a few studies addressing multilingual detection of these subtypes of abusive language in the related literature. in these, authors proposed single systems that can be used to classify data in different languages. some of the common features used in this kind of models are multilingual word representations such as muse @xcite or multilingual bert @xcite . other authors combined word embeddings with tweet-level features @xcite or linguistic features @xcite . in most cases, the multilingual models are trained and tested independently for each language and do not combine different languages in a single evaluation. an exception is the approach proposed by @xcite , where the authors trained deep neural network architectures with a concatenation of english and spanish datasets to classify data in both languages. although multilingual bert models have been tested as end-to-end solutions for several tasks, they have not been widely explored for offensive language detection. @xcite tested the zero-shot capability of bert for transferring knowledge from one language to another in named entity recognition and part of speech tagging tasks obtaining high performing results. ## data the dataset used in this work is described thoroughly in @xcite . we decided to replace the distantly-supervised english training dataset @xcite presented in this task by the olid dataset of @xcite . this is because the focus of this work is the multilingual capabilities of bert, and the datasets of the languages added to this task (danish(sigurbergsson and derczynski, 2020), greek @xcite , arabic @xcite and turkish(c ¸öltekin, 2020)) resemble more the olid dataset in that they were manually annotated. ## system overview our model is a fine-tuned version of multilingual bert @xcite . this architecture (which has become the state of the art for most nlp tasks) consists of a stack of transformer blocks @xcite prediction. for the downstream task an output layer is added and the model is fine-tuned with very low learning rates. multilingual bert shares the same training as single-language bert but using a concatenated dataset of 104 languages, and has demonstrated to have surprising cross-lingual capabilities, even among languages that do not share scripts @xcite . the implementation used in this work is the pretrained multilingual bert-base model from the huggingface library @xcite . this model consists of 12 transformer blocks, 12 self-attention heads, and a hidden layer size of 768. on the top of it, we added a linear layer and applied a sigmoid function to the outputs. we trained the model for 10 epochs with a batch size of 32 using a dropout probability of 0.1, setting the initial learning rate at 5 * 10 -5 and binary cross entropy as the loss function. adam with linear warm-up of 10% of the steps was used to optimize the loss. ## experimental setup we tested several experimental configurations using the data described in for monolingual evaluation, we trained our model using each one of the training sets and the corresponding validation and testing sets. this experimental setup allows us not only to test the model in a specific language but also to obtain reference values to be compared to the ones obtained in the multilingual experimental setups. we refer to this setting as bert lang (bert greek, bert english, etc). as a second configuration, we opted for multilingual training. we trained our model with the concatenation of all the training sets and evaluated it over each test set. the purpose of this experiment is to find languages that contribute positively to the monolingual classification. we call this setting bert all. it might be argued that in monolingual settings it would have been a better option to simply use the bert version trained specifically for that language. however, we decided to use the multilingual version to have comparable results. to assess the multilingual potential of bert for this task, we also performed some zero-shot and fewshot experiments. zero-shot experiments consisted in training the model in one language and evaluating it in a different one. that is, training with language a and testing with language b. few shot experiments, on the other hand, trains the model in language a using also a little amount of instances from b, and tests their performance against b. this cross-lingual generalization is desirable to tackle the same problem in low-resourced languages. in section 5 we discuss the results in each case. the evaluation metric proposed for this task is macro f1. ## results in most cases, bert all showed results equal or slightly worse than the monolingual setting, telling us (at first sight) that adding other languages does not contribute to the overall performance. in the case the turkish language, however, there is a slight increase from 0.766 to 0.773 of f-score. more interesting is the case of the danish dataset, where the f-score increased from 0.74 to 0.77. to find out the impact of the other languages over the danish results, we conducted a data augmentation experiment. we augmented the training danish dataset with other languages data to classify the tweets in the danish dataset. regarding the zero-shot cross-lingual capability (the non-diagonal entries in ## conclusion in this work, we explored the capabilities of multilingual bert for offensive language detection in a multilingual scenario. we evaluated our model in different experimental setups: training and testing it individually for each language; training with one language data and testing in others (zero-shot mode); and training a single model on all languages. something to notice is that, in spite of the good performance of multilingual bert in zero-shot cross-lingual evaluation for other tasks, it did not work well for this one. the only exception is the english-danish cross-lingual evaluation, and it is mainly due to vocabulary intersection. further work is needed to analyze the reasons why these zero-shot experiments failed. nonetheless, the only experiment performed in few-shot mode showed slightly better results, letting a danish model with just a hundred of examples achieve a reasonable performance. on the other hand, training all the languages jointly resulted in a single model having a similar performance overall. this is interesting as having a single model instead of five has practical implications, in particular concerning the resources needed for big models such as bert. the relationship of offensive language among different languages is something to be studied in more depth. at first glance, our experiments showed no zero-shot transference; a few-shot experiment showed (in the case of two languages of similar typology) some transference. further study is needed to explain these results.