Precog- LTRC - IIITH at G erm E val 2021: Ensembling Pre-Trained Language Models with Feature Engineering (2021)

Abstract: We describe our participation in all the subtasks of the Germeval 2021 shared task on the identification of Toxic, Engaging, and Fact-Claiming Comments. Our system is an ensemble of state-of-the-art pre-trained models finetuned with carefully engineered features. We show that feature engineering and data augmentation can be helpful when the training data is sparse. We achieve an F1 score of 66.87, 68.93, and 73.91 in Toxic, Engaging, and Fact-Claiming comment identification subtasks.

## introduction facebook quickly rose in popularity around 2008, taking the world by storm single-handedly creating the initial social media buzz. its user base is steadily increasing ever since and has held its position as the most used platform ever since the early 2010s. 1 it has around 2.38 billion users, and the increase hasn't flattened yet. the initial purpose of such social media platforms was to establish a bridge for fruitful information exchange, which is currently inhibited by offensive language and misinformation spread. given the number of comments exchanged each day, it's impossible to manually classify and mitigate such behavior. germeval is a series of shared task evaluation campaigns that focus on natural language processing for the german language. germeval 2021 tasks are intended to classify comments on facebook into three categories of toxic, engaging, and fact-claiming comments. subtask a focuses on the identification of offensive language which could be used to ban/timeout these users. subtask b on fact-claiming can further be classified as misinformation, and subtask c on engaging comments 1 statistics https://bit.ly/3azdqtj promoting cleaner information exchange. the outline of this paper is as follows: we give a short overview of related work in section 2. we then describe the dataset provided in section 3 and the preprocessing techniques we use in section 4, explain the features we engineered in section 5, and the architecture of our solution in section 6. we then move onto the evaluation of our solution in sections 7-9 and conclude in section 10. ## dataset the dataset provided for the shared task @xcite is an annotated dataset of facebook user comments that four trained annotators have labeled. the dataset was collected from the facebook page of a political talk show of a german television broadcaster (information about which was not revealed to the participants), consisting of user discussions from february till july, 2019. the dataset provided is anonymized. links to users are replaced by @user, likewise links to the show replaced by @medium, and the links to the show's moderator replaced by @moderator. each comment of the dataset is annotated into three categories -toxic, fact-claiming, and engaging. the test set contains 944 comments extracted from different shows other than the one in the training data. this way, the participants were provided with a realistic use case and could possibly test a possible bias caused by topics of discussion. there is an imbalance in the distribution of classes in the given dataset. still, we let the models be biased with this class imbalance as we believe it provides our models a fair understanding of these distributions from the real world. ## data preprocessing the corpora is abundant in emojis. we transcribe all emojis into german text instead of removing them while cleaning the text as not to lose information present, such as emotions. for this, we use a transliteration mapping for emojis. 3 we use the googletrans library foot_3 to translate these to german. we remove hyperlinks, mentions, lower-case the text, remove punctuations except for apostrophes and periods, and perform unicode normalization. due to the limit on the number of tokens (512 tokens) for transformer-based models, we cut the text in the middle of the sentence i.e. we take the first 500 and last 12 tokens. ## feature engineering we look at the linguistic features of the text and explore their correlation with each categorical prediction. ## model architectures we formulate the tasks as a multi-label classification problem as we are trying to address all 3 tasks. to learn the correlation between these classes, all our models are three-headed which output probabilities for 3 classes corresponding to each subtask. @xcite achieve the best performance when they concatenate the last 4 hidden layers of the pre-trained network for sentence-level tasks, so in all our models, we use the same approach and concatenate the last 4 hidden layers. we experiment with the following models and techniques (after freezing the pre-trained weights): ## results experimenting with the models, we achieve the best performance with an ensemble of models mentioned in the architecture section, which is also our submission. the participants were provided with the gold labels for the test set to evaluate the models. in ## analysis we carry out an analysis of the test set gold labels to find where our models failed. we find that many misclassified comments were very long ones with more than 512 tokens truncated in the middle part. we truncated in the middle as most of the emotions must be concentrated at the two ends. a possible solution could be is to use hierarchical lstms with chunking of 512 token chunks of these texts and feeding them to the models or using longformer based models @xcite . we analyze some of the misclassified texts by our model below. they were translated by a native german, two nonnative speakers, and google translate. we all will have to pay more without any improvement. best practice. it is always easy to point finger at others." in comment 1, the gold label is toxic. without context, it could also be classified as non-toxic, since it is congratulatory in the first part. comments 2 and 3 were classified as toxic but are non-toxic. one could note that both are in a rude tone. this could be because of the context of the comment and to what it is referring to. for engaging comments, some misclassified comments in our analysis were both toxic and engaging, which is strange without context. for subtask 3, comment 4 was classified as fact-claiming by the model, but the comment seems to be claiming a practice. we find in our testing that hand-crafted features could be crucial in improving the performance of pre-trained finetuning for low-resource tasks. we also notice no discrepancy between precision and recall even though there was a class imbalance in the training set. hence, our hypothesis that the model benefits from learning the class distributions and their correlations in the real world is validated. ## conclusion participating in all three shared tasks, we submit predictions from a model ensemble. we perform feature engineering and dataset augmentation and show how this can help train neural networks in low-resource tasks. our model ensemble with hand-crafted features performs better than the baseline fine-tuned bert model. we also analyze the errors made by our model against the gold label to understand the flaws in the model. we have also made the source code public 19 for reference.