P i C k L e at S em E val-2022 Task 4: Boosting Pre-trained Language Models with Task Specific Metadata and Cost Sensitive Learning (2022)

Abstract: This paper describes our system for Task 4 of SemEval 2022: Patronizing and Condescending Language Detection. Patronizing and Condescending Language (PCL) refers to language used with respect to vulnerable communities that portrays them in a pitiful way and is reflective of a sense of superiority. Task 4 involved binary classification (Subtask 1) and multi-label classification (Subtask 2) of Patronizing and Condescending Language (PCL). For our system, we experimented with fine-tuning different transformer-based pre-trained models including BERT, DistilBERT, RoBERTa and ALBERT. Further, we have used token separated metadata in order to improve our model by helping it contextualize different communities with respect to PCL. We faced the challenge of class imbalance, which we solved by experimenting with different class weighting schemes. Our models were effective in both subtasks, with the best performance coming out of models with Effective Number of Samples (ENS) class weighting and token separated metadata in both subtasks. For subtask 1 and subtask 2, our best models were finetuned BERT and RoBERTa models respectively.

## introduction patronizing and condescending language (pcl) in the context of vulnerable communities refers to language that portrays a sense of superiority or has a tendency to view communities with a pitiful and passionate lens. pcl works subtly and is intricately associated with the way that words and phrases are used. this makes it difficult to classify pcl as compared to overtly offensive language where the nature of words and phrases is clearly negative. since the harms associated with pcl are not always evident, it is often used inadvertently by actors trying to help these communities. recognising pcl is important because the expression of pcl leads to a paradigm where discrimination and injustices are routinised and made less visible @xcite . use of pcl also feeds into stereotypes @xcite , reinforces societal power dynamics and avoids deeprooted societal problems, providing surface-level solutions for the same @xcite . task 4 of semeval 2022 @xcite aims to identify pcl with subtask 1 working towards binary classification and subtask 2 working towards the multi-label classification of pcl. our strategy involves using state-of-the-art pre-trained language models (plms) and finetuning them for our specific task. we work with bert @xcite , roberta @xcite , distilbert @xcite , albert @xcite as our models. additionally, we design effective and simple approaches to optimize our models, by experimenting with different costsensitive class weighting methods and working with token separated metadata to enhance performance. with an increasing number of plms, each having millions of parameters and being computationally expensive to train, it is essential to make the right model choice. this paper provides a comprehensive analysis of the performance of different models. this can help determine baselines for similar text classification tasks. for model replicability, our code is available online. foot_0 2 background ## experimental setup to ensure comparability, all models are trained on the same train, dev and test split. further, the traindev splits are the same splits provided by the task organizers in the practice splits. the models were developed on keras foot_1 @xcite , and implemented using the transformers library by huggingface foot_2 @xcite . we experimented with learning rates of 1e-5,2e-5 and 5e-5 for all models, finding the best results at 2e-5. for all the models, we fixed the max length parameter at 256 tokens and the batch size parameter to 16. the finetuning for the models was performed on google colab gpu. we trained each model for 1-2 epochs and found the best results at 2 epochs. the value of β for ens class weighting was taken as 0.9997 for subtask 1 and 0.99 for subtask 2 based on experiments with different values. class weighting was implemented using the class_weight parameter during model fitting. we used the autotokenizer offered by huggingface's transformers library to tokenize our inputs. we implemented the token separated metadata by setting the add_special_tokens parameter of the tokenizer as true and using the text_pair parameter. we used the adam @xcite ) optimiser by keras. the loss function used is binary crossentropy and categorical cross-entropy for subtask 1 and subtask 2 respectively. ## results and analysis based on the performance of different plms with different configurations ( our results in the given subtasks on the test set are shown in while we recognise that our model has performed well for subtask 1, the model still lacks in terms of learning what exactly represents pcl. recognising pcl is an inherently difficult task because of the subtle nature of the language used and the lack of an exact benchmark of what constitutes pcl. with further tuning of parameters and attempts at improving paragraph representations, we may improve the performance of our existing model. for subtask 2, on analysis, we believe that a possible issue with the chosen model for submission is that its performance on the development set is heavily biased by the distribution of labels in the training and development set. this is evident by the fact that 'unb' which is in the highest proportion in subtask 2 has a development f1 score of 0.838 on the 'unb' class (which also raises the average f1), signifying that a large number of samples have been correctly classified as 'unb' in the development set. we believe that since this model has a high tendency to classify samples as 'unb', it gained a high f1 for 'unb' on the development set where this class was statistically well represented with 71% of the samples with pcl having the class 'unb' in the development set, however, the same distribution may not present in the evaluation set revealing a pitfall of our model. ## conclusion the task of predicting patronizing and condescending language (pcl) is relatively new in the field of natural language processing and comes with its challenges as discussed in this paper. we used a finetuning approach to build models to identify and classify pcl and explored the performance of various models as well as training variations and present them as a comparative in this paper. we explore techniques to deal with class imbalance, which is a rampant problem in real-world datasets by considering various class weighting techniques which work based on cost-sensitive learning. we also explore the idea of using metadata to optimize our model by adding a context that represents the target community or situation being referred to in a given paragraph. in the future, we would like to explore other options that utilise the power of task-specific metadata. we would also like to work with other transformer-based models such as t5 @xcite and electra @xcite . we would also like to work on improving the ability of our model to recognise the subtle use of language which is embodied by pcl.