Team LRL _ NC at S em E val-2022 Task 4: Binary and Multi-label Classification of PCL using Fine-tuned Transformer-based Models (2022)

Abstract: Patronizing and condescending language (PCL) can find its way into many mediums of public discourse. Presence of PCL in text can produce negative effects in the society. The challenge presented by the task emerges from the subtleties of PCL and various data dependent constraints. Hence, developing techniques to detect PCL in text, before it is propagated is vital. The aim of this paper is twofold, a) to present systems that can be used to classify a text as containing PCL or not, and b) to present systems that assign the different categories of PCL present in text. The proposed systems are primarily rooted in transformer-based pre-trained language models. Among the models submitted for Subtask 1, the best F1-Score of 0.5436 was achieved by a deep learning based ensemble model. This system secured the rank 29 in the official task ranking. For Subtask 2, the best macro-average F1-Score of 0.339 was achieved by an ensemble model combining transformer-based neural architecture with gradient boosting label-balanced classifiers. This system secured the rank 21 in the official task ranking. Among subsequently carried out experiments a variation in architecture of a system for Subtask 2 achieved a macro-average F1-Score of 0.3527.

## introduction the aim of the current task, viz. patronizing and condescending language detection @xcite , is to identify presence of condescending and patronizing tones in text, particularly by the media when referring to vulnerable communities. preponderance of pcl in news and different social media texts, often targeted towards marginalised and under represented communities is a major social concern these days. it can feed stereotypes, tilt the scales of superiority towards a particular community, and fuel discriminatory behaviour. thus the task of identifying pcl partic-ularly that directed to vulnerable communities is a crucial task. the task is based on the english language don't patronize me! dataset @xcite . it consists of two subtasks, binary classification (subtask 1) and multi-label classification (subtask 2). in subtask 1, given a paragraph the aim is to predict whether the paragraph consists of any form of patronizing and condescending language (pcl). in subtask 2, the aim is to assign each paragraph a subset of labels which express different categories of pcl. the major challenges offered in such tasks emerge from various linguistic aspects, such as use of cryptic sentences, sarcasms used, polysemous nature of the english words among others. for the present task, further challenge emerges from the imbalance of dataset as discussed in section 2. experiments were conducted with various systems and the best performing ones for both the subtasks are discussed in detail. the two systems that perform best among these, for both the subtasks are ensemble techniques, which employ two or more classifier models at different stages of the system. the paper is organized as follows. the task background is discussed in section 2. sections 3 and 4 discuss the details of the systems and the experimental setup, respectively. the results from the systems are given in section 5. the paper is concluded in section 6. the code for the proposed systems have been made available at https://github.com/kushagrit/ semeval2022-teamlrl_nc ## background as mentioned in section 1, this task is based on the don't patronize me! dataset. the paragraphs for this dataset have been extracted form the news on web (now) corpus ( https://www.englishcorpora.org/now/), and have been manually anno-1 2 3 4 5 6 7 neg/pos 14 52 46 45 52 21 261 the number of negative examples per positive example, for the binary classification task, in the train + dev dataset (henceforth referred to as training dataset), is approximately 10. for subtask 2, the number of negative examples per positive example (approximated to nearest integer) for each label in the training dataset is given in work has been carried out in the field of nlp to identify different types of linguistic variations or harmful languages from text such as, sarcasm detection @xcite , hate speech detection @xcite , @xcite ), and fake news detection @xcite , @xcite ). since the introduction of bert @xcite , transformer-based language models have been used for a variety of nlp tasks, such as the use of bert for a regression task of predicting eye-tracking features for a given word of the sentence @xcite , or use of bert for the task of document classification (?, @xcite ). all the methods proposed in this paper use further pre-trained transformer-based language models to extract document embeddings. the motivation comes from some recent work that has been carried out in detecting condescending language from text. @xcite introduce a new labeled dataset, namely talkdown, of condescending acts in context, and establish baselines for this dataset using bert. ## system overview experiments were conducted with two types of models for each of the two subtasks. each system uses transformer-based language models for generating text embeddings. these language models were further pre-trained on masked language modeling (mlm) task on the given data. the paragraphs from the training dataset were prepared for mlm with the model specific tokenizer by masking tokens in the input with probability 0.15, and using truncation and padding to maximum length of 256. the present experiments use further pre-trained roberta @xcite and xlnet @xcite language models foot_0 . these further pretrained roberta (roberta-base) and xlnet (xlnetbase-cased) models are henceforth denoted as my-roberta and myxlnet, respectively. this further pre-training is done using the given data on the existing pre-trained roberta and xlnet models in order to fine-tune the models for the two specific subtasks, so they become adapted to the given corpus irrespective of the task at hand, whether it is binary classification or multi-label classification. the systems discussed in sections 3.1 and 3.2, use a custom attention head architecture foot_1 . the input to this attention head is of dimension batch size × sequence length × embedding dimension, and the output is of dimension batch size × embedding dimension. the attention mechanism is used so that the system learns the emphasis of different tokens towards the classification. this layer is henceforth referred to as attn_head. embeddings are generated using these models to extract document representations, which will be used as features in each system. for training the proposed models, discussed in sections 3.1 and 3.2, and for the training of the myroberta and myxlnet models, smart batching is used, with a maximum token length limit of 256. in smart batching the dataset is sorted by length of the sequences before creating batches, and padding is done to each sequence in each batch to the length of the longest sequence in that batch. in the following subsections the model algorithms are discussed in detail. smart batching has been used to optimize the training speed for the transformer-based models, since now most of the resulting batches will have shorter fully connected layer input dimension: 2*768 / 768 output dimension: 512 fully connected layer input dimension: 512 output dimension: 1 tanh softmax (weights) weighted sum attention head fully connected layer input dimension: 2*768 / 768 output dimension: 2 classifier argmax prediction concatenate last hidden states xlnettokenizerfast further pertained mlm roberta model further pertained mlm xlnet model input documents binary.1 / binary.2 sequence lengths which would prevent memory overload. the following notations have been used in the further sections. the two subtasks of binary and multi-label classification have been referred to as, subtask 1 and subtask 2, respectively. the two systems in for each of these tasks have been denoted as binary.1, and binary.2 for subtask 1, and multi.1 and multi.2 for subtask 2. multi.1 and multi.2, each have two sub-models which are denoted as multi.1.a and multi.1.b for multi.1, and multi.2.a and multi.2.b for multi.2. ## experimental setup the experiments were carried on google colaboratory in python 3.7.12 with nvidia tesla p100 gpu. pytorch @xcite and huggingface transformers @xcite are the key frameworks used to carry out the experiments. the text from the training data was used to further pre-train roberta and xlnet models, without any preprocessing. the text was tokenized using the fast implementations of robertatokenizer and xlnettokenizer from the transformers library i.e., robertatokenizerfast and xlnettokenizer-fast, respectively. the input text was tokenized using xlnettok-enizerfast for binary.1 and using robertatok-enizerfast for binary.2. for multi.1 and multi.2, the text in the training data was preprocessed using the following preprocessing steps. for multi.1 and multi.2, the preprocessed text is tokenized using robertatokenizerfast from the transformers library. the xgboost classifier has been implemented using the xgboost foot_3 library and the minibatchkmeans has been implemented using scikit-learn foot_4 python library @xcite . fuzzy c-means is implemented using the 'fuzzy-c-means' @xcite python framework. for multi.2, the parameters are given in ## results several experiments were conducted using the systems described in section 3 and some variations of these systems. the systems were trained on the training dataset (train + dev) and the metrics obtained on test set are discussed in this section. ## conclusion in this paper, systems for the task of binary and multi-label classification for detection of pcl in text, have been proposed. for the binary classification task, the first proposed system uses an ensemble transformer-based language model architecture. the other system detailed in the present work, for binary classification is a fine-tuned transformerbased language model with some custom layers. the first system proposed for the task of multi-label classification combines outputs from a transformerbased model fine-tuned for multi-label classification, and uses the embeddings from this model as features for individual label-balanced xgboost models. another system for multi-label classification has also been discussed, which uses a transformer-based model fine-tuned for multi-label classification, and uses the outputs from this model as inputs to a fuzzy-membership based ensemble classifier.