HITMI & T at S em E val-2022 Task 4: Investigating Task-Adaptive Pretraining And Attention Mechanism On PCL Detection (2022)

Abstract: This paper describes the system for the Semeval-2022 Task4 ”Patronizing and Condescending Language Detection”.An entity engages in Patronizing and Condescending Language(PCL) when its language use shows a superior attitude towards others or depicts them in a compassionate way. The task contains two parts. The first one is to identify whether the sentence is PCL, and the second one is to categorize PCL. Through experimental verification, the Roberta-based model will be used in our system. Respectively, for subtask 1, that is, to judge whether a sentence is PCL, the method of retraining the model with specific task data is adopted, and the method of splicing [CLS] and the keyword representation of the last three layers as the representation of the sentence; for subtask 2, that is, to judge the PCL type of the sentence, in addition to using the same method as task1, the method of selecting a special loss for Multi-label text classification is applied. We give a clear ablation experiment and give the effect of each method on the final result. Our project ranked 11th out of 79 teams participating in subtask 1 and 6th out of 49 teams participating in subtask 2.

## introduction the effect of patronizing and condescending language (pcl) towards vulnerable communities in the media is not always conscious and the intention of the author is often to help the person or group they refer to (e.g. by raising awareness or funds or moving the audience to action). however, these superior attitudes and discourse of pity can routinize discrimination and make it less visible. while there has been substantial work on modeling language that purposefully undermines others, the modeling of pcl is still an emergent area of study in nlp since pcl is the speaker's unconscious superior speaking attitude, the special word that causes pcl is subtle compared to the keywords in other natural language processing problems. the authors decided to evaluate the questions separately. in semeval-2022 task 4: patronizing and condescending language detection @xcite , the purpose of subtask 1 is to identify whether a sentence is pcl. in contrast, the goal of subtask 2 is to indicate the presence of pcl at the text span level, which detects the exact categories in the seven categories of pcl.in subtask 1, the method of using data set retraining to make the pre-trained language model learn the specific distribution of the data set, adding keywords to the input, and integrating five roberta-based models, subtask 2 is to select k from 7 for classification tasks, task-specific loss calculation methods are designed. these methods will be explained in detail in the following sections. ## background research on pcl has been in various fields such as language studies (margić, 2017), sociolinguistics @xcite , politics @xcite or medicine @xcite . in recent years, natural language processing systems for recognizing pcl languages have also begun to emerge, for example, @xcite introduced the task of modeling humility in direct communication from an nlp perspective, and developed a dataset of annotated social media messages. in the same year, @xcite discuss the social and power implications behind the use of certain languages, an important concept in the imbalanced power relations that often arise in condescending treatment. but there has not been a standard in terms of accuracy and definition of pcl. therefore, this article will first explain the definition of pcl and define some categories of the linguistic techniques used to express pcl. ## system description in subtask 1 and subtask 2, we ensemble several models to obtain the results, which are all in roberta-based architecture @xcite . roberta learns an inner representation of the english language that can be used to extract features useful for downstream tasks. in subtask 1, we pretrain the model on task-specific data. in subtask 2, we utilize multi-label categorical cross-entropy loss to improve performance. ## conclusion the paper describes our system at semeval-2022 task 4, which uses several different models based on roberta. we used a series of methods such as pre-training, constructing prefix templates, and model fusion to achieve relatively good results. as we can see, using roberta as the network backbone achieves better performance in this task. also post-training and using the hidden vectors of the last few layers of roberta can improve the effect. at the same time, we also tried fgm, focal loss and other methods, while none of them seemed to be beneficial for our task. still, the f1 value is less than satisfactory, so we can see that identifying and categorizing patronizing and condescending language are difficult challenges. in the future, we will consider using some external knowledge to help the judgment of the model.