UTSA NLP at S em E val-2022 Task 4: An Exploration of Simple Ensembles of Transformers, Convolutional, and Recurrent Neural Networks (2022)

Abstract: The act of appearing kind or helpful via the use of but having a feeling of superiority condescending and patronizing language can have have serious mental health implications to those that experience it. Thus, detecting this condescending and patronizing language online can be useful for online moderation systems. Thus, in this manuscript, we describe the system developed by Team UTSA SemEval-2022 Task 4, Detecting Patronizing and Condescending Language. Our approach explores the use of several deep learning architectures including RoBERTa, convolutions neural networks, and Bidirectional Long Short-Term Memory Networks. Furthermore, we explore simple and effective methods to create ensembles of neural network models. Overall, we experimented with several ensemble models and found that the a simple combination of five RoBERTa models achieved an F-score of .6441 on the development dataset and .5745 on the final test dataset. Finally, we also performed a comprehensive error analysis to better understand the limitations of the model and provide ideas for further research.

## introduction patronizing and condescending language (pcl) generally appears as an act to hold a superior attitude, resulting in language that "talks down" to others. for instance, pcl may describe someone in a power position as the potential "savior" of a vulnerable community (e.g., "a donation of one dollar can save a life"), masquerading a sense of superiority as compassion. there has been recent research suggesting that pcl can have adverse effects on the mental health of individuals @xcite , particularly in the context of ageism. while there has been substantial research on pcl in various contexts @xcite @xcite , unfortunately, there have been few ef-forts to develop pcl detectors in the field of natural language processing (nlp). hence, this paper describes our team's (utsa nlp) contributions on the semeval-2022 task 4 @xcite that introduced a new dataset for detecting pcl language. nlp has investigated a broad spectrum of problematic language usages, such as hate speech @xcite , sarcasm language @xcite , fake news @xcite , and the spread of rumors and disinformation. however, pcl has only recently been explored in the nlp community (?). to alleviate this issue, semeval-2022 task 4 expanded on the work by ?, releasing a large pcl dataset for two pcl subtasks. subtask 1 focuses on detecting the presence of pcl in news stories. pcl detection consists of a variety of sub-problems, for instance, identifying the exact pcl type expressed post (if any). there are multiple technical challenges for identifying pcl. for instance, accurate models must handle imbalanced data (most news stories do not contain pcl) and complex semantic understanding to relate shallow solutions for helping vulnerable populations. for instance, ? describe "shallow solutions" as a type of pcl, e.g., "raise money to combat homelessness by curling up in sleeping bags for one night". nevertheless, for a model to understand this is an example of pcl, it would need to understand that "curling up in sleeping bags for one night" is unlikely to help the general problem of homelessness. hence, we hypothesize that different models will learn to detect different types of pcl with varying accuracy; thus, combining multiple methods can result in better performance than a single method. overall, this paper describes our system for task 1. specifically, we evaluate multiple combined methods to handle pcl's complex nature better than a single method. hence, for our methodology, we trained a roberta model and two traditional deep learning models (a convolutional neural network and a long short-term memory network) for comparison. in addition, we experimented with different model hyperparameters, random seeds, thresholds, and pre-trained word embeddings using the performance on the validation set to assess model variants. finally, we evaluate multiple simple, yet effective, methods of combining the neural network models in an ensemble. ## background based on the work of ?, the semeval task 4 dataset contains 10,637 news stories about vulnerable people published in 20 english-speaking countries, with a novel pcl taxonomy consisting of three toplevel categories (the savior, the expert, and the poet) and seven low-level pcl categories describing the different types of condescension @xcite . thet task contains two subtasks: binary classification (subtask 1) and multilabel classification (subtask 2). the binary classification for subtask 1 annotated the data with one of two categories: pcl and not pcl. subtask 2, the multi-label classification task, categorizes the news stories into a subset of of seven different pcl categories: unbalanced power relations, authoritative voice, shallow solution, presumption, compassion, metaphor, and the pooer, the merrier. a complete description of each category can be found in ?. pcl has been studied in a wide array of contexts, from sociolinguistics to healthcare @xcite @xcite . however, much of the prior work has focused on interviews and general qualitative methods. thus, automated pcl detection models can provide social scientists with tools to understand the impact of pcl at scale. for instance, pcl models would allow linguistics to understand the implicit language actions related to condescension and aid social scientists in researching the link between condescension and other characteristics like gender or socioeconomic status because these superior attitudes and discourse of pity can routinize discrimination and make it less visible @xcite . however, much of the research on harmful language in nlp has concentrated on the explicit, offensive, and apparent phenomena like false news identification, trustworthiness prediction and fact-checking, modeling offensive language, both generic and community-specific @xcite @xcite ; or how rumors spread @xcite . re-cently, some work on condescending language has begun to surface. for example, based on the challenge that condescension is often undetectable from isolated discourse because it depends on discourse and social context, @xcite introduces the task of modeling the phenomenon of condescension in direct communication from an nlp perspective and developing a dataset with annotated social media messages. likewise, ? also trained various baseline models to examine how existing nlp approaches perform in this task. although they observe that recognizing pcl is achievable, it is still difficult. hence, the work by ? formed the basis of semeval task 4. ## system description overall, we developed an ensemble model strategy for the pcl challenge. specifically, we evaluated three individual methods: roberta, convolutional neural networks, and long short-term memory networks. furthermore, we experimented with various ensemble combinations. approaching the task we conduct multiple experiments with a variety neural network architectures using convolutional neural networks (cnn) @xcite , bi-directional long short term memory (bil-stm) @xcite , and the pre-trained transformer-based model, roberta @xcite . each model and ensemble method is described in the section below. ## dataset, experimental setup, and training details for subtask 1, we use the train dataset provided by the pcl organizers. we choose the best epoch and the best hyperparameters using performance assessed in terms of f1-score on this development set based on the random search @xcite . we saved the best epoch and best hyperparameters for each model variant. for evaluation, we use the provided test and validation datasets released by the organizers. we implemented our models on four gpus using pytorch @xcite to train binary classifiers for pcl. we use cross-entropy loss in all our experiments as the loss function. we ran the experiments on a server using a gpu cuda version: 11.4. we selected the epoch based on the f1 score on the development set to save the best version of each model. cnn lstm roberta ss 2 roberta ss 3 .30 .35 .40 .45 .50 .55 .60 .65 .70 f1 score =.3600 =.0283 =.3000 =.0082 =.5793 =.0116 =.5867 =.0197 to build a basis for comparison, all models were trained using the training data provided by the task organizers and evaluated against the provided validation dataset. the best-performing models were then submitted for evaluation against the test dataset during the task evaluation period. the training method was repeated three times for cnn and lstm and eleven times for roberta, each with a new random seed. this is because changing the random seed used in fine-tuning roberta models can provide significantly different outcomes, even if the models are similar in terms of hyper-parameters @xcite . the best-performing hyperparameters in each model were saved for the remainder of the ensemble study. we examined three random seeds (17, 42, and 99) for the best cnn and lstm models, with standard deviations .0283 and .0082, respectively. we also evaluated ten random seeds for the roberta model with step sizes of 2 and 3, yielding standard deviations for variance of .0116 and .0197, respectively. the distribution of each model's performance for different random seeds is shown in figure 1 . we attempted to build a robust ensemble classifier with softmax output aggregation. for the ensemble model, the default threshold for interpreting probabilities as class labels is 0.5, but due to the imbalanced classification problem, we adjust the optimal threshold range from 0.1 to 0.9 when converting probabilities to class labels. we found the optimal probability threshold of cnn, lstm, and roberta that resulted in the best f1 score on the validation dataset were .1, .35, and .35, respectively.an optimal threshold was also chosen for the ensemble model, which was found to be .35. avg p. avg r. avg f1 roberta @xmath0.5948 .5738 .5826 @xmath1.6006 .5916 .5952 cnn googlenews .2542 .7085 .3733 fasttext .2738 .5729 .3653 glove_word .2253 .4640 .3031 glove_twitter .2070 .6549 .3132 bilstm googlenews .3250 .4087 .3609 fasttext .3659 .4020 .3810 glove_word .3525 .4271 .3831 glove_twitter .3745 .3953 .3821 ## results in next, in figure 2 we report the results of averaging different number of models in our ensemble. more specifically, we evaluate averaging the best two models, best three models, and best n models, for n up to an ensemble of 30 different models. the model is chosen based on the top n performing models across all model types and random seeds. overall, we see that initially the result of an ensemble of size 1 (i.e., only using the best roberta model) has an f1 of around .63. however, that slowly increases beyond .64 at around the top five models. after that, the results slow decrease. overall, we find that while a few models with varying performance improves the results. the more inaccurate models slowly outweigh the best performing model, thus decreasing the overall results. however, we find that the results stabilize around 0.61. finally, in ## conclusion in this paper, we have presented our submission for the pcl detection system submitted to the semeval-2022 task 4. our team are focus on the subtask1 to identify whether the paragraphs contain the pcl or not. we proposed several ensemble models that leverages pre-trained word vectors and three different deep learning architectures. in future efforts, we plan to further improve our model by incorporating structured knowledge bases.