Felix&Julia at S em E val-2022 Task 4: Patronizing and Condescending Language Detection (2022)

Abstract: This paper describes the authors’ submission to the SemEval-2022 task 4: Patronizing and Condescending Language (PCL) Detection. The aim of the task is the detection and classification of PCL in an annotated dataset. Subtask 1 includes a binary classification task (PCL or not PCL). Subtask 2 is a multi label classification task where the system identifies different categories of PCL. The authors of this paper submitted two different models: one RoBERTa model and one DistilBERT model. Both systems performed better than the random and RoBERTA baseline given by the task organizers. The RoBERTA model finetuned by the authors performed better in both subtasks than the DistilBERT model.

## introduction with the rise of social media, online hate speech has skyrocketed and has posed a problem for social media platforms: how can the giant number of messages on social platforms be surveilled, so that hateful comments can be reported and deleted? the answer to this is hate speech detection, a discipline within natural language processing that has become increasingly popular and successful in recent years. however, apart from hate speech, there is also other harmful language that should be studied. this is what the research by perez-almendros, espinosa-anke and schokaert dives into. they collected an annotated dataset that focuses on a type of harmful language that is not so easily 1 the code can be found here: https://github.com/juliaecrevisse/semeval2022 detected: patronizing and condescending language (pcl). the researchers describe pcl as follows: "an entity engages in pcl when its language use shows a superior attitude towards others or depicts them in a compassionate way." @xcite one of the difficulties in detecting pcl as opposed to openly hateful speech, is that persons who use pcl often do not intend to do harm, but instead want to support the groups that they name. pcl is usually aimed at vulnerable communities which makes the detection of pcl even more important. the semeval 2022 task 4 competition is based on this research by perez-almendros et. al: the detection of pcl. in two subtasks, one a binary classification and one a multi-label classification, the participants of the semeval competition were tasked to submit up to two models. the authors of this paper participated in both subtasks and propose a finetuning based approach on a pre-trained roberta language model. two models were submitted to the task organisers, one roberta model and one distilbert model. the roberta model performed better in both subtasks. 1 in this paper, the authors will first describe the task set out by the semeval 2022 competition in sections 2. in section 3, the authors explain the relevance of related work to this topic. section 4 gives an overview over the bert model. in the subsequent section, the two pre-trained models are presented. in the section experimental setup, the two different finetuned models are described in detail. section 7 presents the results of the competition and section 8 the conclusion. ## task description this research paper contributes to task 4 of the semeval 2022 competition foot_0 : pcl detection. the starting point of the research task is the paper "don't patronize me! an annotated dataset with patronizing and condescending language towards vulnerable communities" @xcite . the researchers provide an annotated dataset with paragraphs taken out of news articles from english speaking countries foot_1 where vulnerable communities are mentioned. the following communities are included: disabled, homeless, hopeless, immigrant, in need, migrant, poor families, refugee, vulnerable and women. the dataset includes a total of 10,637 paragraphs extracted from the news on web corpus. 3,554 of the paragraphs had been labeled as pcl by the annotators. the aim of the semeval task is 1) to identify which paragraphs include pcl and 2) if the paragraph includes pcl which category or categories it belongs to. the two subtasks are described as follows: subtask 1: binary classification. given a paragraph, a system must predict whether or not it contains any form of pcl. subtask 2: multi-label classification. given a paragraph, a system must identify which pcl categories express the condescension. there are seven different categories: unbalanced power relations, shallow solution, presupposition, authority voice, metaphor, compassion and the poorer, the merrier. ## related work while other areas related to hate speech and hateful language have been a focus of nlp research in recent years, the research into pcl is still limited. however, there are a few researchers that have delved into this discipline from different angles. wang and potts (2019) discuss condescending language and its detection in their research and also point out that high quality data for this kind of language detection is still limited. they introduce the dataset "talkdown" which is a data set from the social media platform reddit. their approach concentrated on bert baseline models and the researchers concluded that condescending language is highly connected to context @xcite . taking a different angle, @xcite write about the social and power implications of language. they claim that by using language with, e.g. social bias, stereotypes and prejudices are reinforced. their collected data stems from different social media sites. they state in their results that previously successful models can categorize social bias relatively well, but they have difficulties classifying the social bias frames which were developed by the authors @xcite . ## system overview of pretrained models in the paper "don't patronize me!" which is the starting point of the semeval task, the researchers point out that an nlp model with bert achieves the best results. more specifically roberta performs slightly better than distilbert and bert-base (perez-almendros et. al., 2020). ## experimental setup fine-tuning was performed on google colab. the authors used the train and test set as it was provided by the task organizers. ## results an overview of the results can be found in ## conclusion the purpose of this work was to identify pcl in texts using nle. for this purpose, existing text classification models were used and adapted to the task by fine tuning. the goal was to achieve the highest possible f1 scores by the model, which is equivalent to the percentage detection of condescending terms. a complete detection of all these terms was not achieved, but in the binary classification with roberta a score above 0.52 could be obtained. the multi classification was the bigger challenge and could only finish with a score around 0.250. despite these results, a basis, if not an improvement, has been created for future work. the hyperparameters which were used can already be set to default in the next works at the beginning and thus increase the speed of the development process for researches in pcl detection by bert models. more research could also be done into the different categories of pcl. there were categories that consistently performed better than others (unbalanced power relations and shallow solution), while the systems had more difficulties with other categories (especially the poorer, the merrier). there are still a lot of directions that the research into pcl detection can continue. as stated above, pcl detection in nle is an emerging field that would benefit from further research. the task is more difficult than traditional hate speech detection as pcl is often more subtle. however, pcl can contribute to repeating and furthering stereotypes and discrimination, especially among vulnerable communities and therefore the research into pcl detection systems is a vital endeavor.