Team S tanford ACML ab at S em E val 2022 Task 4: Textual Analysis of PCL Using Contextual Word Embeddings (2022)

Abstract: We propose the use of a contextual embedding based-neural model on strictly textual inputs to detect the presence of patronizing or condescending language (PCL). We finetuned a pre-trained BERT model to detect whether or not a paragraph contained PCL (Subtask 1), and furthermore finetuned another pre-trained BERT model to identify the linguistic techniques used to convey the PCL (Subtask 2). Results show that this approach is viable for binary classification of PCL, but breaks when attempting to identify the PCL techniques. Our system placed 32/79 for subtask 1, and 40/49 for subtask 2.

## introduction the goal of the task is to be able to identify whether or not a piece of text contains condescending or patronizing language, and if it contains patronizing language identity which linguistic techniques are used to express that sentiment @xcite . studies have shown that pcl fuels discriminatory behavior, creates and feeds stereotypes, subtly blames needy individuals for their problems, and oversimplifies problems. in general, pcl makes it harder for needy communities to get the help they need and reach total inclusivity @xcite . this is obviously negative, and being able to combat it with ai may help automate the process and get past inherent biases that humans identifying pcl may have. our system's goal is to use contextualized word embeddings to feed the model, and thus have the model analyze the semantics of the text. we specifically focused on a purely textual analysis and did not provide the model with any metadata to see if the model could learn the pcl patterns just from the text, because although real world usage would likely include those features, the ability to learn from the text itself would be useful and more universally applicable. in this task, we learnt that it is easier for a model to simply detect the presence of pcl than the techniques used in a piece of pcl. this is likely because of both unequal distributions of data and the fact that the context for the types of pcl likely look very similar, making the model default to a category of pcl with a higher frequency when not incredibly clear. we ranked 32/70 on subtask 1, which is detecting the presence of pcl, and ranked 40/49 on subtask 2, which involves identifying the pcl techniques used. in particular, the model struggled with the 'authority voice' and 'metaphor' categories. the dataset provided was the don't patronize me! dataset (p'erez-almendros et al., 2020), which is a collection of paragraphs mentioning vulnerable communities and published in media in 20 english speaking countries. each paragraph has the country code where the paragraph was published and the keyword that was used to search for it. for subtask 1, the paragraphs are manually annotated with a label from 0-4 on how much pcl it contains; these are converted to binary labels on whether or not a paragraph contains pcl, where 2-4 indicate positive examples and 0-1 indicate negative examples. for subtask 2, all paragraphs in the dataset contain pcl and are annotated with spans containing categories of linguistic techniques that are used to express the condescension. these categories are: ## experimental setup we used berttokenizerfast to tokenize the text and fine-tuned on the pre-trained bertforsequenceclassification model, both from the huggingface transformers library @xcite . we conducted hyperparameter optimization using the hyperopt package @xcite , using populationbased training @xcite . it automatically generated sets of hyperparameters for us, and then based on the results of training with those hyper parameters updated the future hyperparameter sets. our final model was trained using adam optimization with a learning rate of 2.31468e-05 (kingma and ba, 2014); we trained the model for 6 epochs with a batch size of 8. we used a train/test split of 70/30 and evaluated based on accuracy and f1. ## results our model had precision 0.4017, recall 0.7666, and f1 0.5271 on the evaluation data for subtask 1, and has an average f1 of 0.0963 for subtask 2. we placed 32/79 for subtask 1, and 40/49 for subtask 2. based on the results, the model had a hard time with the types of pcl that showed up less frequently in the data, and tended to perform best on the categories that were more frequent. in retrospect, using the same model architecture and setup on the two subtasks was not the optimal way to approach the task, despite some compelling reasons to approach it that way. we did not perform any quantitative analysis or ablations, but given the chance we would augment the less frequent pcl categories and see if that would fix the prediction issues for subtask 2, even if it wouldn't necessarily improve accuracy. ## conclusion we developed a system which attempted to first classify whether or not a piece of text contained patronizing or condescending language, and then identify the technique used to convey the pcl. in particular, we focused on examining whether or not pure textual analysis using contextual word embeddings alone would be enough to perform the aforementioned tasks. based on our results, this approach is only viable for the binary classification of whether or not a text contains pcl. in the future, we would explore creating an ensemble of models, only one of which uses textual analysis, and the rest would focus on things like meta data and word frequencies which do not rely on context. comparing the result of that ensemble to a traditional approach to this problem which uses many of those methods simultaneously would show whether or not there is a strong overlap between sources, locations, etc. and pcl, or whether only the text itself is the best indicator of a sign of pcl.