S em E val-2022 Task 4: Patronizing and Condescending Language Detection (2022)

Abstract: This paper presents an overview of Task 4 at SemEval-2022, which was focused on detecting Patronizing and Condescending Language (PCL) towards vulnerable communities. Two sub-tasks were considered: a binary classification task, where participants needed to classify a given paragraph as containing PCL or not, and a multi-label classification task, where participants needed to identify which types of PCL are present (if any). The task attracted more than 300 participants, 77 teams and 229 valid submissions. We provide an overview of how the task was organized, discuss the techniques that were employed by the different participants, and summarize the main resulting insights about PCL detection and categorization.

## introduction the study of unfair, misleading or offensive language has attracted the interest of many scholars from the nlp research community. most relevant tasks in this context focus on explicit, aggressive and flagrant phenomena, such as fake news detection or fact-checking @xcite @xcite ; detecting propaganda techniques @xcite ; modeling offensive language @xcite identifying hate speech @xcite ; and rumour propagation @xcite . however, there also exist subtler but equally harmful types of language, which have received less attention by the nlp community, and which, due to their subtle nature, we can expect to be more difficult to detect. this is the case, among others, for patronizing and condescending language (pcl), which was the focus of task 4 at semeval-2022. an entity engages in patronizing or condescending communication when its use of language reveals a superior attitude towards others. these attitudes, when normalized, routinize discrimination and make it less visible @xcite . furthermore, the use of pcl is often unconscious and well-intended, especially when referring to vulnerable communities @xcite . this good will can make pcl especially harmful, as the audience receives this discriminatory language with low defense and is often unaware of its effects. research in sociolinguistics presents pcl as a subtle, often unconscious but harmful and discriminative kind of language @xcite . it creates and feeds stereotypes @xcite , which result in greater exclusion, rumour spreading and misinformation @xcite . pcl also tends to strengthen power-knowledge relationships @xcite , calling for charitable action instead of cooperation and presenting those who can help as saviours of those in a less privileged position @xcite . furthermore, pcl tends to conceal who is responsible for very deep-rooted societal problems, sometimes by implicitly or explicitly blaming the underprivileged communities or individuals for their situation, and often involves ephemeral and simple solutions. @xcite . the use of pcl by privileged communities has also been related to the so-called pornography of poverty @xcite , a communication style that depicts vulnerable situations with a pity discourse to move a target audience to charitable action and/or compassionate attitudes. while the negative impact of pcl, both in social interactions and in corporate and political discourse, has been extensively studied in the social sciences, it still remains an under-explored phenomenon in nlp. nonetheless, we believe that pcl detection offers a number of important challenges for nlp research, which warrant more work in this area, especially given the societal benefits that would result. for instance, given its subtle and subjective nature, we can expect pcl detection to be harder than tasks that are focused on more flagrant phenomena. moreover, pcl detection often involves the need for an implied understanding of human values and ethics, which requires a form of commonsense reasoning that nlp models are likely to struggle with. in this context, we have organized semeval 2022 task 4: patronizing and condescending language (pcl) detection. this task has attracted more than 300 participants, organized in 77 teams, during the official competition. the competition remains open on codalab to encourage further research on this topic foot_0 . ## related work as already mentioned in the introduction, pcl has been extensively studied within the context of sociolinguistics @xcite @xcite . within nlp, however, modelling of patronizing discourse has only received limited attention. as a notable exception, @xcite compiled a corpus of reddit comments, which were annotated as using a condescending tone or not. note that in contrast to our semeval task, their work did not specifically focus on vulnerable communities. in our previous work @xcite , we introduced don't patronize me!, which is, to the best of our knowledge, the first annotated corpus of pcl towards vulnerable communities. this corpus was used as the training data for the semeval task. some other works have studied types of discourse that are highly related to condescension, including @xcite , who studied how certain uses of language indicate power relations, @xcite , who discussed the dehumanization of minorities through language and zhou and jurgens (2020), who investigated how some expressions of condolences and empathy interplay with authoritative voices in online communities. ## dataset the seed material for this task is don't patronize me! (dpm), an annotated dataset with patronizing and condescending language towards vulnerable communities, which was introduced in our previous work @xcite . this dataset contains 10,469 paragraphs, which were used as the training set for the semeval task. to create the test set for this task, we annotated 3,898 additional paragraphs, following the same process. all paragraphs were extracted from news stories from media in 20 english speaking countries, origi-nally provided by the news on web (now) corpus foot_1 @xcite . we used a keyword-based strategy to collect paragraphs, focusing on texts in which vulnerable communities are mentioned (e.g., refugees or homeless). the data was annotated by three annotators, with backgrounds in communication, media and data science. for the main dataset, two annotators annotated the instances with the following labels: 0 (not pcl), 1 (borderline), and 2 (pcl), achieving an inter-annotator agreement (iaa) of 41% for the raw annotations and 61% when removing borderline cases. for all the total disagreements (paragraphs labeled 0 by one annotator and 2 by the other), the third annotator acted as a referee, providing a final label. the final dataset uses a scale from 0 to 4, indicating the level of agreement between the annotators. labels 0 and 4 correspond to clearly not condescending and clearly condescending (i.e. both annotators assigned 0 or both assigned 2), label 2 means that both annotators marked that paragraph as a borderline case (1-1), and labels 1 and 3 correspond to cases where either one of the annotators assigned the borderline label (0-1 or 1-2), or there was a disagreement that was resolved by the third annotator. each positive example from the dataset is furthermore labelled with one or more pcl categories. we briefly recall the meaning of these categories. unbalanced power relations (unb): the author entitles themselves as being in a privileged situation, considering themselves as saviours of those in need @xcite . shallow solution (shal): a charitable, superficial and short-term action is presented as life changing. presupposition (pres): stereotypes and clichés are used to describe a community, relying on assumptions without having all the information. authority voice (auth): the author stands as spokesperson and defendant of the community or individual and/or allows themselves to give expert advice about how to overcome underprivileged situations. ## task description the aim of the proposed task is to identify the presence of pcl (subtask 1), and to identify the categories of pcl that are present in a given paragraph (subtask 2). training data the 10,469 annotated paragraphs from the dpm corpus were provided as training data. to frame subtask 1 as a binary classification problem, paragraphs with labels 0 and 1 were considered as negative examples, while paragraphs with labels 3 and 4 were considered as positive examples of pcl. the original labels on the scale from 0 to 4 were also made available. the 993 positive examples in the training data are labelled with the corresponding pcl categories. span annotations for these categories were also provided. test data a total of 3,898 paragraphs were released as test set, with the same format and metainformation as the training set, but without labels and span annotations. paragraphs initially labelled as 2 were excluded from the test data, as these correspond to borderline cases. ## results and discussion a total of 77 different teams participated in the evaluation phase of our task, with 145 valid submissions for task 1 and 84 for task 2. for the competition, we allowed a maximum of 2 submissions per team. a total of 42 out of 77 teams outperformed the baseline for subtask 1, while 37 out of 48 outperformed the baseline for subtask 2. tables 2 and 3 present the rankings for subtask 1 and 2, respectively, where we have only listed the best performing system for each team. for subtask 1, the best-performing systems used the following strategies: team pali-nlp used an ensemble of pre-trained roberta models @xcite . while training, they applied grouped layer-wise learning rate decay, a variant of llrd @xcite , based on the idea that different layers capture different types of information @xcite . by optimizing the learning rate in different layers, the model captures more diverse and fine-grained linguistic features of pcl. to tackle the class imbalance in the dataset, they use weighted random samples @xcite to emphasize the positive instances. team stce created adversarial examples to train an ensemble model of roberta and de-berta @xcite . they also used weighted samples to address the class imbalance and explored different loss functions, establishing cross entropy and the contrastive loss algorithm nt-xent introduced by chen et al. (2020) as first and second loss function, respectively. for subtask 2, the best-performing systems used the following strategies: team beike nlp participated with a system based on prompt learning @xcite . they first reformulate pcl detection as a cloze prompt task and then fine-tune a pre-trained deberta model. team pingan omini-sinitic proposed an ensemble model which used prompt training and a label attention mechanism, by adding a new label-wise attention layer ( @xcite . their system over-samples the positive examples. they also use a form of transfer learning from subtask 1 to subtask 2, by pre-training on subtask 1 and using the resulting model as the starting point for training a model for subtask 2. for both sub-tasks, unsurprisingly, most systems rely on pre-trained language models, although a few teams have used cnn, lstm, svm or logistic regression based systems (xu, pc1, i2c, ryan wang, mcrock, amrita_cen, satlab and team lego, among others), or an ensemble of some of the above together with language models (utsa_nlp, taygete). although the use of language models usually outperformed other systems in this task, some lstm models, such as the one submitted by team xu, achieved competitive results. the ensembling of different models has also been a popular technique. other strategies that proved successful include adversarial training, data augmentation and multitask learning. in the following, we summarize how these techniques have been used by the different systems. ensemble learning ensembling different models has previously been found useful for text classification @xcite @xcite . accordingly, ensembling was one of the most common strategies for improving on baseline pcl detection methods. most of the teams combined different language models (e.g. pali-nlp, stce, pingan omini-sinitic,pai_team, lrl_nc, ssn_nlp_mlrg, asrtrans, amsqr, umass pcl). considering the choice of language models, the most successful systems either used roberta, deberta or an ensemble which included the former ones and other models. for instance, these models were used by the best performing teams for both subtasks, i.e. pali-nlp and stce for subtask 1 and beike nlp and pingan omini-sintic for subtask 2. to fine-tune the language models effectively, incorporating a contrastive loss function, in addition to the standard cross-entropy loss, has also proved useful. finally, it should be noted that the combination of language models with different types of neural networks (taygete, utsa_nlp) has also proven useful. ## conclusions patronizing and condescending language detection is a relatively new challenge for the nlp community. however, the high level of participation in this task has provided the community with valuable new insights about how to tackle this problem. a total of 42 out of 77 teams in subtask 1 and 37 out of 48 for subtask 2 outperformed the roberta baseline. the performance of the best-performing systems shows that a judicious usage of state-ofthe-art text classification techniques can bring significant benefits to pcl detection, especially when it comes to addressing the relative scarcity of the available training data and closely related external resources. however, there still remains considerable scope for further improvements. it is our expectation that further improvements may need to rely on techniques that are specifically targeted at pcl, e.g. by exploiting insights from linguistics about the linguistic features of pcl, or by building explicit models of stereotypes of vulnerable communities.