MS @ IW at S em E val-2022 Task 4: Patronising and Condescending Language Detection with Synthetically Generated Data (2022)

Abstract: In this description paper we outline the system architecture submitted to Task 4, Subtask 1 at SemEval-2022. We leverage the generative power of state of the art generative pretrained transformer models to increase training set size and remedy class imbalance issues. Our best submitted system is trained on a synthetically enhanced dataset with 10.3 times as many positive samples as the original dataset and reaches an F1 score of 50.62%, which is 10 percentage points higher than our initial system trained on an undersampled version of the original dataset. We explore possible reasons for the comparably low score in the overall task ranking and report on experiments conducted during the post-evaluation phase.

## introduction task 4 of semeval-2022 focuses on the detection of patronising and condescending language (pcl) in news @xcite . pcl in popular media and news sources is detrimental to an emancipated and equal society, as it is usually targeted towards minorities and socially disadvantaged communities, often in an unsuccessful attempt to show solidarity @xcite . pcl has the potential to strengthen existing stereotypes by representing minorities either as passive entities to be pitied and supported, thus taking away their agency and focusing on their vulnerabilities or praising members of vulnerable groups for everyday achievements simply because of their background @xcite . in contrast to hate speech, pcl is usually subtle, well intentioned, and free of discriminatory phrases or racial slurs, which makes it an interesting natural language processing (nlp) problem. in other domains with more discriminatory classes such as hate speech detection, generative models have recently become increasingly popular and successful as a tool to increase classification performance @xcite . in our contribution to the shared task, we explored to what extent this approach is feasible for the presented use case, where classification of a text sample is less distinct and often relies on world knowledge @xcite . the dataset provided for the task was fairly small, with less than 10% of the data belonging to the positive class. we thus enhanced the original dataset in two ways for our system runs: we generally followed the approach used by @xcite and initially fine-tuned a bert classifier on the original dataset. we then finetuned gpt-3 @xcite and generated samples of pcl and npcl which were classified using our fine-tuned system. samples for which the bert classification did not correspond to the intended output were discarded. we then fine-tuned a new bert instance with the modified dataset pclenhanced including the synthetic data. although our system only ranked middle field in the competition, both classifiers trained on the modified datasets improve our initial classifier trained on the original dataset by multiple percentage points. we conclude that this approach does add value to classification, even in cases where the distinction between the positive and the negative class relies on subtleties. the code described in the following as well as the synthetic data used for the modification of the original dataset is available on github 1 . text class meanwhile "throughout this island, the high level of suicide is terrible and terrifying. "as christians" we can give hope, where a person feels only darkness and hopelessnes," he said ## background we participated in subtask 1 of the competition, which entailed the binary classification of news paragraphs as either patronizing or not patronizing. basis for the task was the don't patronize me! dataset (dpm) (perez almendros et al., 2020), which contains 10,469 paragraphs of annotated data from 20 english news sources. while all paragraphs include references to potentially vulnerable groups, only 993 are examples of patronising speech. the dataset included meta-information about the country each paragraph was published in, an article id, a keyword indicating which vulnerable group is addressed, and a label ranging from 1 to 4, where 0 and 1 are treated as non-patronizing and 2 to 4 as patronizing. the task organizers define pcl as often unconscious, subtle and subjective ways in which the speaker conveys a superiority "concealed behind a friendly or compassionate approach towards the situation of vulnerable communities" @xcite . they explicitly exclude hate speech and discriminatory speech from pcl, making it harder to be identified not only by nlp-systems, but also by humans. we include examples of both classes in transformer-based generative models such as gpt @xcite and its successors have become prevalent in various nlp tasks. for instance, @xcite explored the idea of synthetically constructing benchmark datasets to concur with existing benchmarks such as squad, while @xcite showed that a fine-tuned gpt model can accurately mimic the personal conversation style of an individual, leading to improvements in the persona-chat dataset. another increasingly popular use case is the generation of data on tasks with small labeled corpora to synthetically increase dataset size in order to train better performing classifiers. dekker and van der goot (2020) used synthetic data for lexical normalisation, while other researchers employed such data to train question answering models @xcite . even in maths, researchers have pro-posed ways of creating synthetic theorems @xcite . @xcite used gpt-2 @xcite for their approach to hate speech detection. their datasets were small to medium sized (6-53k labelled examples) and highly unbalanced, with as little as 1-6k hate speech samples per dataset. they created three mixed datasets containing 10k, 80k and 240k synthetic samples respectively, as well as 80% of the original datasets. the classification models trained on the largest created dataset outperformed those trained on the smaller datasets in most cases. @xcite generated data using gpt and improved sentence-level topic classification on three datasets, ranging from 4.2k to 17k entries. @xcite and anaby-tavor et al. (2020) fine-tuned the respective gpt models on relatively small datasets, and find statistically significant improvements on classifier performance through incorporating synthetic data in the datasets used for fine-tuning classifiers. while gpt and gpt-2 were trained on 117m and 1.5b parameters respectively, gpt-3 models were trained on up to 175b parameters @xcite @xcite . as it has been shown that an increase in model size systematically leads to improvements in text synthesis as well as common downstream tasks @xcite , gpt-3 is likely to produce higher quality and more natural sounding data than its predecessors. we thus expect gpt-3 generated data to have an even greater impact on performance in intricate language classification tasks such as pcl detection. we know of only few other research teams which used gpt-3 in their experiments, for instance to search for more suitable prompts for natural language understanding (nlu) tasks @xcite or using prompts for few-shot generation @xcite . both achieved strong results on classification benchmarks. while using foundation models for data generation has the potential to increase the power of language models and mitigate the data scarcity prob- ## system overview to generate the synthetic data, we used gpt-3's curie model. curie has about 13b unique parameters, while davinci has about 175b. although davinci performs significantly better on a number of nlp tasks than curie, we chose curie, as it is more financially viable than the larger model, while retaining a comparatively strong performance @xcite . for fine-tuning, we split the dataset into pcl and npcl data and modified it to meet the api's requirements. as the api requires a promptcompletion pairing, the prompt was set to be empty (") and the completion contained the data sample. afterwards, two gpt-3 curie instances were finetuned on the pcl and npcl data, respectively. we thus created two models, one to generate pcl and one for npcl phrases. following @xcite , we called the models with an empty (") prompt in the pipeline for synthetic data generation and the default parameters. we set max_tokens to the rounded mean length of the samples in the original dataset (60 for pcl and 54 for npcl). with each iteration, we generated the maximum number samples (128), resulting in a total of 24.321 synthetic phrases by the npcl model and 24.197 by the pcl model. like @xcite , we classified all synthetic samples after generating the data. we used an initial baseline classifier and discarded all samples where the intended and predicted class did not match. due to the high class imbalance of the original dataset, we randomly undersampled the negative class to the size of positive samples for training of the baseline classifier. we fine-tuned bert base -cased @xcite across three epochs using a learning rate of 1e-5 on the undersampled dataset. since the synthetically generated data consisted solely of text for each label, we did not use any of the meta-information or context provided in the dataset and fine-tuned solely on text and labels. in the future, it might be useful to take meta-information into account for text generation. 39% of the generated pcl samples were classified as such by the baseline classifier, whereas 85,5% of generated npcl samples were classified as npcl. we explain this with the much larger sample size of npcl in the dpm allowing the gpt-3 pipeline to generate better suited data. based on the predictions, we created two enhanced datasets: for dpm enhanced , we added a similar amount of synthetic pcl (9448) and npcl (9357) samples to the dpm. for dpm enhancedpos , we added 7086 pcl samples to balance the original dataset. for a comparison of sample sizes and share of pcl in the dpm and the different datasets used for fine-tuning see ## results and discussion the evaluation metric used for ranking in the task was f1 over the positive class. our baseline classifier reached an f1-score of 40.74% on the test set provided by the task organisers after the end of the competition's evaluation phase. although it had a high recall of over 80%, precision was very low, leading to a suboptimal f1-score. the classifier trained on dpm enhanced scored almost 10% higher than the initial classifier, but had neither the highest recall, nor the highest precision of the three classifiers trained before the post-evaluation phase. this was suprising, as we initially expected the classifier trained on dpm enhancedpos , which was the larger balanced dataset out of the three, to perform best. this leads to the assumption that with synthetic data, sheer amount might be more important than balancing out the dataset. although in the official task scoring, our system trained on dpm enhanced ranked in place 41 of 78 and surpassed the official baseline (fine-tuned roberta) by only about 1%, we note that using both synthetically enhanced datasets led to a boost in performance compared to our initial classifier. this might seem surprising, especially considering the low performance of the initial classifier used to filter the gpt-generated data. in the post-evaluation phase, we repeated the experiments from our two system runs without previous filtering of the gpt-output, to explore the role of the initial classifier in our system's performance. neither dpm enhancedunfiltered nor dpm enhancedposunfiltered led to better performance than dpm enhanced . thus, using a baseline classifier for filtering seems to be the most sensible option when working with synthetic data, regardless of its performance strength. we report on detailed classification results in we also looked at some of the synthetic data generated by gpt-3. both for pcl and for npcl, the generated samples were not always coherent on a semantic level and the occurrence of incoherent text appeared to be more common in the npcl condition. however, it seems like coherence did not impact classification, as in both cases incoherent synthetic samples could be found in the final dataset (see we also found a lot of text in languages other than english, possibly because of the small size of the dataset in comparison to the vast amount of training data used to create gpt-3. we expect that filtering out such samples would increase performance further. in addition, basic data-cleaning of the synthetic data before classification might be in order. both of this could potentially be achieved by only using data samples for which a confidence score above a certain percentage (i.e. 70%) is returned in classification. another approach might be using an unrelated dataset to filter out all synthetic data unrelated to the task at hand. in the context of pcl detection, this could help discard generated data that is not related to vulnerable groups. the approach of using an empty prompt (") while fine-tuning the models is debatable, because the prompt is such a powerful tool @xcite and should probably be utilized. a possible approach would be to train a single model on both pcl and npcl data, and put pcl/npcl information in each samples' prompt. the currently unused meta-information of the dataset could also be incorporated, possibly causing additional improvements in the quality of the generated data. ## conclusion we described our system submitted to task 4, subtask 1 of semeval-2022. although the system's performance did not score highly on the overall leaderboard, ranking 41st place, incorporating synthetic data in the original training set still boosted performance by up to 10% compared to our initial baseline system, which leads to the assumption that pairing this approach with more sophisticated classification systems has some potential to increase classification performance significantly. we derive some lessons learned from the presented experiments as follows: in the future, we plan to improve the baseline classifier and explore different data cleaning and filtering techniques, such as using confidence scores returned by the classifier for our data selection, using unrelated datasets to filter whether a data sample fits in the task-specific domain or making use of prompts during gpt-3 fine-tuning and data generation. exploring other augmentation strategies such as back-translation or synonym replacement of either the original data or the generated samples might further increase classification performance.