Team Oulu at S em E val-2020 Task 12: Multilingual Identification of Offensive Language, Type and Target of T witter Post Using Translated Datasets (2020)

Abstract: With the proliferation of social media platforms, anonymous discussions together with easy online access, reports on offensive content have caused serious concern to both authorities and research communities. Although there is extensive research in identifying textual offensive language from online content, the dynamic discourse of social media content, as well as the emergence of new forms of offensive language, especially in a multilingual setting, calls for future research in the issue. In this work, we tackled Task A, B, and C of Offensive Language Challenge at SemEval2020. We handled offensive language in five languages: English, Greek, Danish, Arabic, and Turkish. Specifically, we pre-processed all provided datasets and developed an appropriate strategy to handle Tasks (A, B, & C) for identifying the presence/absence, type and the target of offensive language in social media. For this purpose, we used OLID2019, OLID2020 datasets, and generated new datasets, which we made publicly available. We used the provided unsupervised machine learning implementation for automated annotated datasets and the online Google translation tools to create new datasets as well. We discussed the limitations and the success of our machine learning-based approach for all the five different languages. Our results for identifying offensive posts (Task A) yielded satisfactory accuracy of 0.92 for English, 0.81 for Danish, 0.84 for Turkish, 0.85 for Greek, and 0.89 for Arabic. For the type detection (Task B), the results are significantly higher (.87 accuracy) compared to target detection (Task C), which yields .81 accuracy. Moreover, after using automated Google translation, the overall efficiency improved by 2% for Greek, Turkish, and Danish.

## introduction the emergence of web 2.0 platform that enabled user-generated content and participatory culture has witnessed the proliferation of online hate speech at an unprecedented level, increasing the likelihood of a random people of any age group to be subject to online harassment and abuse through some internet forum, message board or social network platform. on the other hand, offensive language fosters discrimination against particular categories on the basis of gender, ethnicity, or race @xcite , which violates the universal declaration of human rights. this often leads to a profound negative impact on the society as a whole, especially for youth and vulnerable groups, as well as on the policy-makers and business entities. to overcome this issue, many internet companies put forward standards and generic guidelines that users must adhere to when publishing content on such platforms; at the same time, they employ manual annotators to identify offensive language and remove the post (s) accordingly. nevertheless, such manual effort is inevitably costly, non-scalable, and non-sustainable, which motivates the importance of automatic offensive language identification based approach. prior work has studied offensive language detection in twitter @xcite @xcite , wikipedia comments and facebook posts @xcite , and fromspring posts @xcite . several academic events, shared task competition organized in conjunction with high impact data mining, information retrieval and computational linguistics conferences (e.g., workshop series on abusive languages, automatic misogyny identification, authorship aggressiveness analysis, identification of offensive language at germeval, hate speech detection task at evalita, various related semeval tasks, etc.). some of these tasks involved more than one language. in this respect, one of the most commonly employed methodologies is to train systems that can automatically identify offensive content, which will then trigger action to remove such content without any human moderation. past research has also examined various characteristics of offensive language such as the cyber aggression @xcite , abusive language @xcite , hate speech (foong and oussalah, 2017; abderrouaf and oussalah, 2019), racism @xcite and offensive language @xcite . nevertheless, automatic identification of offensive language is still challenging, especially given the continuous evolution and variability of offensive language discourse and characteristics together with the inherent limitation of natural language processing based approach. besides, the use of multiple languages in a single post adds extra difficulty to such a task. in this context, @xcite used an ensemble classifier based approach to identify hate-speech against immigrants and women in spanish and english twitter datasets. @xcite as part of semeval2019 task, addressed the issue of categorizing offensive language in social media. as part of our contribution to the offensive language challenge at semeval2020, our work presents a three-level annotation schema for abusive language detection, categorization, and target detection. first, we created an offensive language identification dataset (olid) by using unsupervised learning and thresholding-like technique. all the five provided languages (english, turkish, greek, danish, and arabic) were employed for the offensive post detection task. moreover, we have used google online tools for creating automated translation of datasets for turkish, greek, danish, and arabic. we performed experiments using different machine learning models and features for each level of the annotation, which enabled us to compare the performance of the machine learning automated annotated datasets and google translated dataset. the outcome is then used to trigger subsequent data analysis tasks of multilingual detection. the paper is structured as follows. in section 2, our dataset annotation schema and description of datasets are detailed, section 3 details our methodology, including the machine learning model and the associated feature engineering. in section 4, the obtained results are detailed and discussed. finally, conclusive statements and potential future work are drawn in the conclusion section. ## conclusion and future work this paper summarizes our participation to semeval2020 in tasks a, b, and c. the methodology advocates a two-stage strategy with an initial preprocessing that involves both automated annotated using unsupervised learning and google translation dataset and machine learning-based classification using cnn and fasttext embedding. in addition, we have compared the results with the manually annotated datasets and artificially annotated datasets. our result shows that artificially annotated dataset using unsupervised learning yields 2-4% high performance, and in all cases, cnn and lstm outperform baseline classifiers. moreover, our work presented a sound comparison of the results across five distinct languages: english, greek, turkish, arabic, and danish. we observed that the developed approach could overcome the language barrier, maintaining high-performance levels across the five languages. the analysis has also shown that the performance can be slightly improved by tuning the preprocessing stage. moreover, we have observed that google translation of dataset yielded 2-3% higher performance for greek, turkish, and danish, except for the arabic language. this result promotes the viability of use of automated translated datasets in nlp projects. in addition, we found 'target' detection was hardest compared to 'offensive post' and 'type' detection. in future work, we would like to experiment with broader category detection and apply more specialized cnn classifiers that enable wordsense disambiguation.