ML _ LTU at S em E val-2022 Task 4: T5 Towards Identifying Patronizing and Condescending Language (2022)

Abstract: This paper describes the system used by the Machine Learning Group of LTU in subtask 1 of the SemEval-2022 Task 4: Patronizing and Condescending Language (PCL) Detection. Our system consists of finetuning a pretrained text-to-text transfer transformer (T5) and innovatively reducing its out-of-class predictions. The main contributions of this paper are 1) the description of the implementation details of the T5 model we used, 2) analysis of the successes & struggles of the model in this task, and 3) ablation studies beyond the official submission to ascertain the relative importance of data split. Our model achieves an F1 score of 0.5452 on the official test set.

## background work on various sorts of harmful language in nlp has mostly concentrated on explicit aggressive and brazen phenomena @xcite . scholars are striving to distinguish between harmful and unhealthy language by identifying the fundamental characteristics of unhealthy language. @xcite proposed one of the most recent efforts in this regard. the research introduced a new dataset containing 44,000 comments with the unhealthy category sub-classified as either (1) hostile; (2) antagonistic, insulting, provocative or trolling; (3) dismissive; (4) condescending or patronizing; (5) sarcastic; and/or (6) an unfair generalisation. in their work, it is assumed that the language with a pcl tone will assume an attitude of superiority, implying that the other speak-ers/listeners are ignorant, naive, or unintelligent. in such scenarios, the language will usually imply that the other speaker should not be taken seriously. similarly, morris (2007) explains the high likelihood of using pcl language when there is discussion between two persons with different mental health conditions. he demonstrated that patronizing language is common when a discussion occurred between a cashier with no cognitive issue and a customer who suffers from cognitive disability. overall, pcl does not have an obvious negative or critical language and there is the challenge of limited, high-quality labelled data. there have been different efforts at automatically detecting pcl. @xcite showed that models with contextual representations are much better at identifying pcl and this bolstered the hypothesis that context is essential for pcl detection. they implemented the bert model, which deploys a transformer-based encoder architecture, on the talkdown corpus they introduced. both the base and large versions of the bert model are implemented and evaluated over the new proposed corpus for balanced and imbalanced data. @xcite added more context to their work by comparing the performance of bert to human performance in order to better understand the model's performance. in their experiments, they observed that the bert model detects pcl with a 78% accuracy, whereas the average over human annotators does so with a 72% accuracy. @xcite also finetuned a bert model to classify the unhealthy comments in norwegian data. this model was subjected to a variety of finetuning approaches to distinguish between condescending and non-condescending cases and in the binary classification subtask, the best accuracy was 0.862. ## system overview the t5 architecture @xcite ) is very similar to the originally proposed architecture of the transformer by @xcite . we use the pretrained base version of the model from the huggingface hub @xcite . input sequence of tokens are mapped to embeddings and then passed to the encoder, which has alternating set of multi-head attention and feed-forward layers. the attention mechanism @xcite replaces each element of a sequence by a weighted average of the remaining sequence @xcite . in addition to each self-attention layer of the decoder, there is the standard attention mechanism. as self-attention is order-independent, relative position embeddings are used in the architecture. the training method (for both pretraining and finetuning) uses maximum likelihood objective (i.e. teacher forcing) and a cross entropy loss @xcite . the model was pretrained on 34b tokens. adam optimizer is used for optmization during finetuning. the model has 12 layers each in the encoder and decoder blocks and a total of 220m parameters @xcite . when we refer to t5, we mean the base model, except where explicitly stated otherwise. the size of the model meant that a batch size of 64 or 32 required more memory than what is available on a single v100 gpu, so we lowered the batch size to 16. t5 takes a hyperparameter called a task prefix. we, hence, use 'classification: ' as the task prefix. we introduced a correction to the out-of-class prediction of the model, as shown in the flow chart in figure 1 . @xcite mentioned this issue as a possibility but they did not experience it. the issue appears to be because all the tasks the t5 model is trained on are framed as "text-to-text" before training. hence, sometimes, the model might predict tokens seen during training but that do not belong to the category of classes in a classification task. this behaviour seems more common in the initial epochs of training and may not even occur sometimes. we further observed that replacing target labels with numbers and explicitly typecasting them as string reduces this occurrence, as the model becomes more stable with predictions. we split 10% of the training set for validation (dev set) for both of our submissions to the competition. we explored different sizes, however, in further ablation studies, as explained in the next section. the 2 submissions of prediction files are based on 2 adaptive optimizers: adam and adamw @xcite . the predictions based on adam had the better f1 score. each experimental run was for 3 epochs and the model checkpoint with the lowest validation loss was saved and used to make prediction on the test set. the initial learning rate and scheduler for both submissions are 2e-4 and linear schedule with warmup, respectively. ## experimental setup all the experiments were conducted on a shared dgx-1 cluster of 8 × 32gb nvidia v100 gpus. the server runs on ubuntu 18 os and has 80 cpu cores. the experiments were conducted in a python (3.6.9) virtual environment with the pytorch framework (1.8.1+cu102). we use both the training & test data provided by @xcite . besides the 2 submissions of prediction files, we perform ablation studies over the training/dev set split ratio (95%/5%, 90%/10%, 85%/15%, and 80%/20%). the training set was shuffled before splitting each dev set. we evaluate all the models using macro f1 scores, precision (p) and recall (r). in the absence of the ground truth of the test set, we perform error analysis by constructing the confusion matrix on a split of the dev set (20%). further to that, in order to have a basis of comparison of the t5 model's strengths and struggles with the official roberta baseline, we removed the 10 examples provided in evaluation of the available data, by code, be-fore and after running the script provided by @xcite to categorize the labels into 0 (neg) and 1 (pos) (for subtask 1) reveals that there are a total of 10,469 samples. the script treated paragraphs with the original labels 0 and 1 as 0 (instances not containing pcl) and paragraphs with the original labels 2, 3 and 4 as 1 (instances containing pcl). after running the script, the following are obtained: 9,476 samples classified as 0 and 993 classified as 1 in the training set. the test set has 3,832 samples. before training, the following preprocessing steps were applied to all splits of the data: ## conclusion we describe the system involving the pretrained t5 model, which we use for our submission for the subtask 1 of the semeval-2022 task 4. we split 10% of the training set as dev set for hyperparameter evaluation in our official submission. typecasting integer values, which represent classes, as string before feeding the t5 model and adjusting for out-of-class predictions improved the stability of the model in making predictions. furthermore, in the post-competition phase, we performed ablation studies on the relative importance of dataset split by experimenting with different ratios of the training/dev set and showed what the model struggles with. our results show that the encoder-decoder t5 model is competitive in this binary task and can obtain better performance with more hyperparameter tuning. pred. paragraph gold pos from his personal story and real-life encounters with poor families , manpower correspondent toh yong chuan suggested shifting the focus from poor parents who repeatedly make bad decisions to their children ( """" lifting families out of poverty : focus on the children ; last thursday ) . pos pos he said their efforts should not stop only at creating many graduates but also extended to students from poor families so that they could break away from the cycle of poverty pos neg smyth told hkfp : """" the biggest challenge is the no work policy . i think that refugees who come here , or asylum seekers , they 're unable to work and they have kids here -their kids are stateless . that 's really the cause of a lot of stress in the community . pos neg the people of khyber pakhtunkhwa are resilient . i did not see hopelessness on any face , """" he said . pos pos teach kids to give back : when kang runs summer camps with kids , she includes """" contribution fridays """" -the kids work together as a team to make sandwiches for the homeless and dole out the food in shelters . pos pos these shocking failures will continue to happen unless the government tackles the heart of the problem -the chronic underfunding of social care which is piling excruciating pressure on the nhs , leaving vulnerable patients without a lifeline . neg neg lilly-hue : his ability to make sure our family is never in need -his sacrificial self .