C o L i at U d S at S em E val-2020 Task 12: Offensive Tweet Detection with Ensembling (2020)

Abstract: We present our submission and results for SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020) where we participated in offensive tweet classification tasks in English, Arabic, Greek, Turkish and Danish. Our approach included classical machine learning architectures such as support vector machines and logistic regression combined in an ensemble with a multilingual transformer-based model (XLM-R). The transformer model is trained on all languages combined in order to create a fully multilingual model which can leverage knowledge between languages. The machine learning model hyperparameters are fine-tuned and the statistically best performing ones included in the final ensemble.

## introduction in the past years, offensive language and other forms of maliciously intended online communication (trolling, cyber-bullying, fake news) have grown drastically on social media platforms such as facebook, twitter or youtube. it is in the interest of everyone to inhibit this form of communication as quickly as possible once it happens. a very efficient and effective way to achieve this is via machine learning and deep learning architectures that automatically identify a message, comment or tweet as maliciously intended. these architectures have been steadily improving at this task. the purpose of the competition in semeval-2020 task 12: multilingual offensive language identification in social media (offenseval 2020) @xcite is to encourage further progress in the above mentioned challenges. the organizers presented participants with tweets in english, arabic, greek, turkish and danish. for all languages, task a consisted of identifying whether a tweet was labeled as offensive ("off") or non-offensive ("not"). tasks b and c were only available for the english data. they consisted of identifying whether a tweet was targeted ("tin") or untargeted ("unt") for task b and in task c whether a targeted tweet was aimed at an individual ("ind"), group ("grp") or other ("oth"). we participated in task a for all languages and task b for english. data set sizes and label balance varied greatly between languages which provided additional challenges when deciding on how to approach the task. we chose to construct an ensemble of more traditional machine learning architectures (svm, logistic regression, etc.) together with a multilingual transformer-based model (xlm-r) @xcite with the thought that the strengths of the different models would help the ensemble generate the most well-rounded results. the ensemble generated promising results during experiments and was submitted for all languages in task a except for danish. for danish task a and task b (english), we submitted only the xlm-r model predictions. ## related work semeval 2019 task 6: identifying and categorizing offensive language in social media (offenseval 2019) @xcite ) saw many competitors make use of the recently released pre-trained transformer model, bert. @xcite , the top performing model in offenseval 2019, combined a bert model with several pre-processing steps, such as hashtag segmentation, converting emojis to textual descriptions, and lowercasing the text. compared to runner-up bert submissions for the 2019 subtask a such as the rank 2 submission by @xcite , the rank 3 team um@ling @xcite or the rank 7 submission "sentibert" by @xcite , this illustrates that proper pre-processing is often neglected over algorithmic advances. therefore, judging the impact of algorithmic advances proves difficult. @xcite compared the performance of a bert-large model to those of more traditional machine learning models, such as a logistic regression model and soft voting classifier (svc). while the bert-large model outperformed the other on the test data, the logistic regression model and the svc outperformed bert on the dev data. another approach from last year's competition, presented by @xcite , used a bi-directional recurrent architecture with gated recurrent units (gru) @xcite and rule-based sentence offensiveness calculations to categorize tweets. this worked especially well for subtask b, where the submission achieved the highest results. it was less successful for subtasks a and c, where the rule-based nature proved to be detrimental to the classifier's performance. ## conclusion our final ensemble performs above average across all languages and sub-tasks. the ensemble worked well in some cases (greek, english) and less so in other cases (turkish, arabic). this seems to not stem from the characteristics of the languages themselves but more so the differences in label balanced between the respective training and test sets and how we balanced precision and recall during training. however, error and performance analysis on gold labels showed that removing our custom thresholds for balanced precision and recall significantly boosted performance for the final ensemble on the competition test sets (see