Data Science Kitchen at G erm E val 2021: A Fine Selection of Hand-Picked Features, Delivered Fresh from the Oven (2021)

Abstract: This paper presents the contribution of the Data Science Kitchen at GermEval 2021 shared task on the identification of toxic, engaging, and fact-claiming comments. The task aims at extending the identification of offensive language, by including additional subtasks that identify comments which should be prioritized for fact-checking by moderators and community managers. Our contribution focuses on a feature-engineering approach with a conventional classification backend. We combine semantic and writing style embeddings derived from pre-trained deep neural networks with additional numerical features, specifically designed for this task. Ensembles of Logistic Regression classifiers and Support Vector Machines are used to derive predictions for each subtask via a majority voting scheme. Our best submission achieved macro-averaged F1-scores of 66.8%, 69.9% and 72.5% for the identification of toxic, engaging, and fact-claiming comments.

## introduction in the early years after establishing social media platforms, setting up online discussion forums and installing comment areas on newspapers' websites, a door into this new digital world has been opened, allowing people to interconnect all over the world. various communication platforms and social networks enabled new ways of sharing information with followers, exchanging opinions between politically interested people, and encouraging debates with the readers. unfortunately, recent trends revealed the ugly face and adverse effects of these platforms when an increasing number of users make improper, illegal, or abusive use of such digital services @xcite . nowadays, social media platforms are notorious for spreading toxic comments, in which the writers justify violence and discrimination against a person or groups of persons @xcite . additionally, a second steadily growing trend is producing and sharing fake news or misinformation, seeking to dominate current discussions, and frame public debates @xcite . both hate speech, fake news and their impact have become very prominent in recent years. however, the tremendous amount of shared and distributed toxic messages on social media platforms make it utterly infeasible to identify and tag or delete poisonous comments manually. the ger-meval 2021 shared task tries to encounter this negative trend and motivates participants to work on automated solutions towards safer and more reliable digital rooms of interaction @xcite . therefore, the organizers of the task increased the difficulty of the competition by expanding the focus not only on the identification of toxic messages in online discussions but also on distinguishing between engaging and fact-claiming comments. the first task is similar to the germeval tasks in 2018 @xcite and @xcite and deals with identifying toxic comments, including offensive, hateful and vulgar language or ruthless cynism. as novel subtasks, the participants are also invited to identify two additional categories of comments: the second category defines engaging comments, which are annotated as highly relevant contributions by the moderators. the third category concentrates on finding factclaiming comments that should be considered for a manual fact-check with a higher priority. ## system overview the general system architecture is shown in fig. 1 . as the number of samples in the training dataset provided for germeval 2021 is rather small, our proposed framework focuses on suitable feature engineering with a conventional classification backend. these features and further implementation details of our system are described in the following. ## evaluation our framework is trained using a specific crossvalidation and hyperparameter tuning scheme, which is described in the following. ## conclusions within this contribution to the shared task of the germeval 2021 we have developed a modular feature extraction scheme which incorporates semantic and writing style embeddings as well as task specific numerical features. less complex algorithms like logistic regression models and svms converge converge faster than complex models like deep neural networks and therefore need less training data. the combination with automated hyperparameter tuning and dimension reduction as well as the final agglomeration of multiple models in voting ensembles allow to achieve an macro-averaged f1-scores of 66.8%, 69.9% and 72.5% for the identification of toxic, engaging, and fact-claiming comments.