Taygete at S em E val-2022 Task 4: R o BERT a based models for detecting Patronising and Condescending Language (2022)

Abstract: This work describes the development of different models to detect patronising and condescending language within extracts of news articles as part of the SemEval 2022 competition (Task-4). This work explores different models based on the pre-trained RoBERTa language model coupled with LSTM and CNN layers. The best models achieved 15 ^{th} rank with an F1-score of 0.5924 for subtask-A and 12 ^{th} in subtask-B with a macro-F1 score of 0.3763.

## introduction the use of patronising and condescending language (pcl) in text or speech can affect healthy communication channels adversely. the effect of pcl on the vulnerable sections of society have been widely studied. pcl acts as a catalyst for discriminatory behaviour @xcite against various vulnerable groups. it has been observed to promote exclusion and discrimination among communities and provide a conducive environment for rumour spreading and misinformation @xcite . these negative effects of pcl are unaffected by the intent of the writer/speaker who might have unknowingly used pcl. these reasons provide a strong argument for developing methods that can identify and prevent unwanted use of pcl in news articles, blogs, and other pieces of text. in subtask-a of the patronizing and condescending language detection task at @xcite , the goal is to develop a model which takes a sample text as an input and outputs a label indicating the presence or absence of pcl. in subtask-b, the model was required to identify the correct set of pcl categories. the model takes in a sample text as an input and return seven separate outputs each indicating the presence or absence of the pre-defined seven categories. the dataset for the task was shared by the task organisers in the english language. to tackle these tasks, roberta based models were developed. different variations of the models involved the use of feedforward layers, lstms, cnn and their combinations. for subtask-a roberta with lstm, cnn and feed-forward layers outperformed all the other variations with an f1 score of 0.5924. in subtask-b roberta with feed-forward layers got the best f1 score of 0.3763 as compared to the other variations. for subtask-a, this work achieved 15th rank in the leader board and 12th rank for subtask-b details of which are discussed in section 3.2. ## background identification and analysis of pcl in text is well explored in linguistics (aggarwal and zhai, 2012), politics @xcite , sociolinguistics @xcite and other fields. however, in nlp it is still heavily unexplored and starting to gain traction. in the past topics such as sentiment analysis @xcite , offensive speech identification @xcite and fake news identification @xcite have been significantly worked upon. one major roadblock in exploring pcl in the text is the lack of well structured and labelled dataset. recently, some new work has been developed to tackle this issue. wang et al. @xcite developed a model for identifying the condescending language in reddit threads and also developed an annotated dataset for the same. ## experimental setup to gauge the effect of data augmentation and loss weighting techniques on the performance of models for each subtask four experiments were carried out ( ## results for subtask-a rb-bls-cnn under wt experiment achieved the highest f1 score of 0.5924 with a precision of 0.5357 and recall of 0.6625 on the evaluation dataset. while on the validation dataset the same model received an f1 score of 0.6318 with a precision of 0.5685 and recall of 0.7109. for subtask-b rb-fnn performed best out of all the models under the aug experimental settings. the model achieved a macro f1 score of 0.4006 on the validation dataset and 0.3763 on the evaluation dataset. the minute difference in the f1 scores of the best models for the evaluation dataset and the validation dataset shows that the model did not overfit exp augment loss weighting basic no no aug yes no wt no yes aug+wt yes yes the effect on class re-weighting (refer 3.1.3) and data augmentation was also explored (refer similarly, the effect of data augmentation on model performance was also explored (refer to another interesting find in subtask-b was the significantly poor performance of the lstm and cnn based models as compared to the vanilla roberta model i.e. rb-fnn. this is not in line with the trend observed for the models in subtask-a ( ## conclusion this work explored the design and training of different roberta based models for pcl detection in text. the added benefits of using cnn and lstm layers along with roberta in boosting model performance was also shown. this work also explored the effects of using back translation as a data augmentation technique along with a class re-weighting technique to deal with low sample size and high class imbalance. finally, the challenges offered by the models under different problem statements were explored which gives a deeper insight into the impacts of different design methodologies. the best models achieved 15th and 12th rank for subtask-a and subtask-b respectively. future work can include expanding the dataset with more data as the current dataset includes 10469 samples. also, the original article can be provided against each sample which can be fed to the model as added context. this added context should significantly boost model performance.