ULFRI at S em E val-2022 Task 4: Leveraging uncertainty and additional knowledge for patronizing and condescending language detection (2022)

Abstract: We describe the ULFRI system used in the Subtask 1 of SemEval-2022 Task 4 Patronizing and condescending language detection. Our models are based on the RoBERTa model, modified in two ways: (1) by injecting additional knowledge (coreferences, named entities, dependency relations, and sentiment) and (2) by leveraging the task uncertainty by using soft labels, Monte Carlo dropout, and threshold optimization. We find that the injection of additional knowledge is not helpful but the uncertainty management mechanisms lead to small but consistent improvements. Our final system based on these findings achieves F1 = 0.575 in the online evaluation, ranking 19th out of 78 systems.

## introduction despite invaluable contributions to the society, the internet can also serve as an infrastructure for a rapid spread of hurtful language, in part due to the anonymity it commonly provides @xcite . the spread of such language can have a serious impact on individuals, such as the increased development of mental health problems in children @xcite . to prevent this, the society has to establish moderation mechanisms. fully manual content moderation is infeasible both due to the large scale of the web as well as the possible negative psychological effects on human moderators @xcite . much attention has been devoted to the automatic detection of offensive language within the field of natural language processing @xcite . some examples include the detection of hate speech @xcite , toxic language @xcite , and cyberbullying @xcite , which use a relatively explicit form of hurtful language @xcite . in contrast, patronizing and condescending language (pcl) is more implicit in nature. pcl can roughly be described as an expression of a superior attitude towards others, possibly unconsciously. perez @xcite have shown that large language models are able to detect pcl to a various degree, but consistently better than random guessing or a machine learning approach using the bag-of-words representation. based on that, the authors propose semeval-2022 task 4 @xcite , which aims to encourage further research and improvements in the detection of pcl. we present our attempts at modeling pcl, based on the roberta model @xcite and following two main lines: ## task description given an updated version of the don't patronize me! dataset (perez almendros et al., 2020), the goal of semeval-2022 task 4 subtask 1 is the detection of patronizing and condescending language (pcl). the provided dataset consists of 10 469 paragraphs annotated by three annotators: two were tasked with annotating the examples as not containing pcl (0), containing pcl (2), or borderline (1). the third annotator resolved complete disagreements, i.e. examples annotated as {0, 2} by the two annotators. the annotations are aggregated into a five-point fine-grained class y f : we provide the distribution of these fine-grained class labels in figure 1 . for the task evaluation, the fine-grained five label class is binarized into a coarse-grained class y c , where y @xmath0, 3, 4}, and y @xmath1. although the final evaluation uses binary labels, the finegrained labels can provide additional information ## methods in this section, we describe our methodology. first, we describe roberta, which we use as the baseline model. then, we describe how additional knowledge is injected into the model in section 3.2. in section 3.3 we describe how we leverage the task uncertainty in our model. ## evaluation in this section, we evaluate our methodology and compare it to the baseline. we start by describing the experimental settings in section 4.1, and continue with the results in section 4.2. ## conclusion we have described our approaches for the detection of pcl as part of the semeval-2022 task 4. we attempted to inject knowledge into prediction models and leverage the uncertainty present in the task. the injection of additional knowledge did not increase the f 1 score significantly. leveraging the uncertainty in different ways produced mixed effects. training the models on soft instead of hard labels consistently increased the f 1 score, while using mcd on its own was not beneficial. however, using mcd in combination with soft labels and threshold optimization brought consistent improvements in the f 1 score and produced our best score. both directions of our research have potential for further work. in our knowledge injection experiments, we have only experimented with a single type of additional knowledge at a time. to inject multiple types simultaneously, we would need to create special tokens for each combination, which could lead to overfitting due to relatively small and imbalanced data. therefore, in further work we will try a different method for knowledge injection considering multiple types of additional knowledge simultaneously. in leveraging uncertainty, we have constructed the soft binary labels from the two annotations per example and aggregated the annotations by weighing them equally. a possible further work would experiment with different weighting schemes.