PALI - NLP at S em E val-2022 Task 4: Discriminative Fine-tuning of Transformers for Patronizing and Condescending Language Detection (2022)

Abstract: Patronizing and condescending language (PCL) has a large harmful impact and is difficult to detect, both for human judges and existing NLP systems. At SemEval-2022 Task 4, we propose a novel Transformer-based model and its ensembles to accurately understand such language context for PCL detection. To facilitate comprehension of the subtle and subjective nature of PCL, two fine-tuning strategies are applied to capture discriminative features from diverse linguistic behaviour and categorical distribution. The system achieves remarkable results on the official ranking, including 1st in Subtask 1 and 5th in Subtask 2. Extensive experiments on the task demonstrate the effectiveness of our system and its strategies.

## introduction recently, some works @xcite on pcl are gradually emerging in nlp community. remarkably, @xcite have shown that general pre-trained language models @xcite can achieve nontrivial performance. however, the behaviour of pcl is usually more unconscious, subtler, and subjective than other harmful types of discourse that are widely studied, i.e., hate speech @xcite , offensive language @xcite , intended sarcasm @xcite , fake news @xcite and rumor @xcite . these characteristics make pcl detection a difficult challenge, both for human judges and existing nlp systems. to address this, we propose a novel transformerbased model bert-pcl (and its ensembles) with two discriminative fine-tuning strategies, to accurately understand such language context for pcl detection. the two strategies are grouped layerwise learning rate decay (grouped llrd) and weighted random sampler (wrs), and both are beneficial for task-adaptive fine-tuning based on language models. a brief description of these two strategies is as follows: 1) as different layers capture different types of information @xcite , grouped llrd, a variant of llrd @xcite , is applied to group hidden layers of the pre-trained transformer @xcite into different sets and apply different learning rates to each in a certain extent. and then, we can make full use of different layers to capture more diverse and fine-grained linguistic features, which can boost understanding of the subtle and subjective nature of pcl. 2) there is a quite common phenomenon that positive samples (patronizing or condescending) have a smaller number than the negative, which reflects usage rates of pcl in public forums @xcite . but the positive samples are more important when detecting pcl, due to the harmful impacts. to deal with imbalanced classes scenarios, we introduce wrs to place more emphasis on the minority classes. under this strategy, our classifier can capture discriminative features from the categorical distribution and detect whether it contains pcl in an unbiased manner. this challenge is divided into two subtasks, each corresponding to a subset of the don't patronize me! (dpm) dataset @xcite . the 10,469 annotated paragraphs (i.e., sentences in context) from the dpm corpus are used as training data, where each paragraph mentions one or several predefined vulnerable communities. these paragraphs are collected using a keyword-based strategy and cover english language news sources from 20 different countries. a short description of the two subtasks and training data is as follows: in addition, the test set for the evaluation phase contains around 4,000 manually annotated paragraphs with the pcl annotation scheme. more details about the task can be found on the competition page foot_0 . ## system overview in this section, we review our system adopted in semeval-2022 task 4, where we design a novel transformer-based model bert-pcl (and its ensembles) with two discriminative fine-tuning strategies for both subtasks of pcl detection. ## conclusion in this paper, we propose an advanced bert-like model and its ensembles to accurately understand and detect patronizing and condescending language. based on the pre-trained transformer, we apply two fine-tuning strategies to capture discriminative features from diverse linguistic behaviour and categorical distribution. at semeval-2022 task 4, our system achieves 1st in subtask 1 and 5th in subtask 2 on the official ranking. extensive experiments demonstrate the effectiveness and superiority of the proposed system and its strategies.