PINGAN Omini-Sinitic at S em E val-2022 Task 4: Multi-prompt Training for Patronizing and Condescending Language Detection (2022)

Abstract: This paper describes the second-placed system for subtask 2 and the ninth-placed system for subtask 1 in SemEval 2022 Task 4: Patronizing and Condescending Language Detection. We propose an ensemble of prompt training and label attention mechanism for multi-label classification tasks. Transfer learning is introduced to transfer the knowledge from binary classification to multi-label classification. The experimental results proved the effectiveness of our proposed method. The ablation study is also conducted to show the validity of each technique.

## introduction patronizing and condescending language (pcl) is proposed by @xcite , which builds a dataset for pcl detection. the patronizing and condescending language detection @xcite contains two text classification tasks. subtask 1 is a binary classification task, which requires a system to predict whether the paragraph contains any form of pcl. subtask 2 is a multi-label classification task, which must identify which pcl categories express the condescension. pcl detection is a sentiment analysis task and we treated subtask 2 as an aspect-based sentiment analysis (absa) @xcite task, which is also a multi-label classification to classify the sentence sentiment on different aspects. early works on absa focus on feature engineering @xcite and subsequent neural network-based methods @xcite . recently, pre-trained language models (lms) such as bert @xcite , roberta @xcite , albert @xcite have been proposed and brought significant improvement in various nlp applications. as there is not much improvement in absa task through direct implementation of those pre-trained lms, bert-pair @xcite has been proposed to help them adapt to absa effectively. furthermore, @xcite proposed an attention between sentence embeddings and label embeddings to focus on the crucial tokens which are related to the label. recently, capsnet-bert @xcite used the attention between the label and the input with capsule network to improve the performance. nowadays, the pattern-exploiting training (pet) @xcite and p-tuning @xcite has been proposed to utilize the pre-trained lms more effectively, which trains finetune model by the pre-training tasks such as mask language modeling (mlm) by adding a prompt sequence to the input. considering the similarity between pcl and absa, we applied pet and bert-pair in pcl detection through the following steps: firstly, we treat the subtask 1 as a prompt training task (as described in pet) by using a pcl description as the prompt. secondly, we transform the subtask 2 from a multi-label classification task to multiple binary classification tasks by using the label names as the prompts. thirdly, we conduct transfer learning from subtask 1 to subtask 2 to further improve the performance. last but not least, we also proposed a label attention mechanism based on the multi-prompt training for multi-label classification. our contribution can be summarized as follows: we apply prompt training on pcl detection subtask 1 and 2, and prove its effectiveness on both tasks; in subtask 2, we propose a label attention mechanism with multi-prompt training; we apply transfer learning from subtask 1 to subtask 2, where the transfer learning is proved to be effective in absa. ## conclusion in this paper, we propose a multi-prompt training with label attention mechanism to improve the performance of multi-label classification task. pretrained models have made great performance gain compared to traditional neural network models in many natural language tasks. the above experimental results may suggest that the current pre-trained model mechanism still has room for improvement in absa tasks.