Noisy Self-Training with Data Augmentations for Offensive and Hate Speech Detection Tasks (2023)

Abstract: Online social media is rife with offensive and hateful comments, prompting the need for their automatic detection given the sheer amount of posts created every second. Creating high-quality human-labelled datasets for this task is difficult and costly, especially because non-offensive posts are significantly more frequent than offensive ones. However, unlabelled data is abundant, easier, and cheaper to obtain. In this scenario, self-training methods, using weakly-labelled examples to increase the amount of training data, can be employed. Recent “noisy” self-training approaches incorporate data augmentation techniques to ensure prediction consistency and increase robustness against noisy data and adversarial attacks. In this paper, we experiment with default and noisy self-training using three different textual data augmentation techniques across five different pre-trained BERT architectures varying in size. We evaluate our experiments on two offensive/hate-speech datasets and demonstrate that (i) self-training consistently improves performance regardless of model size, resulting in up to +1.5% F1-macro on both datasets, and (ii) noisy self-training with textual data augmentations, despite being successfully applied in similar settings, decreases performance on offensive and hate-speech domains when compared to the default method, even with state-of-the-art augmentations such as backtranslation.

## introduction online social media platforms are widely used by modern society for many productive purposes. however, they are also known for intensifying offensive and hateful comments, attributed in part to factors such as user anonymity @xcite . manual identification of hate speech is impractical at scale due to the massive number of posts generated every second and the potential harm to the mental health of moderators. therefore, there is a need for automatic approaches to detect offensive and hateful speech. in recent years, research on this topic has increased, resulting in new models and datasets published in various languages and sources @xcite . a common characteristic among available datasets is label skewness towards the negative class (non-offensive/hateful), which is usually more frequent than the positive class (offensive/hateful). apart from traditional ways of dealing with imbalanced classes (e.g. under or oversampling or applying class weighting), semisupervised techniques such as self-training can be used to extend the training set with unseen examples that introduce new learning signals without the costly burden of manual data labeling. self-training is a technique that involves iteratively training models using both labelled and unlabelled data. the process begins by training a model using human-labelled data only, which is then used to infer labels for a set of unlabelled data, creating a weakly-labelled dataset. the weakly-labelled dataset and the human-labelled dataset are then aggregated and used to retrain the model. this iterative process is repeated for a fixed number of steps or until no performance improvement is observed. self-training can be particularly useful when labelled data is scarce or expensive to obtain, and was successfully applied in a variety of domains such as computer vision @xcite , audio and speech processing @xcite , and natural language processing @xcite . several variants of self-training have been proposed over the years @xcite . one common approach is to use a teacher-student framework, in which the "student" model learns from the output generated by the "teacher" model @xcite @xcite . additionally, a confi-dence threshold filter may be applied to remove examples that are too ambiguous or non-informative. this process is summarised in figure 1 . recent research on self-training has reported further improvements in performance by introducing perturbations directly into the raw input or to its latent representation, improving generalisation and convergence @xcite @xcite @xcite . these perturbations are often introduced in the form of data augmentations, which are widely applied in computer vision tasks but are less commonly explored in natural language processing tasks, especially in the context of selftraining. these "noisy self-training" methods can be particularly useful in settings where the input data is noisy or subject to a high degree of variation, improving prediction consistency and adversarial robustness @xcite @xcite . @xcite argue that data augmentation depends on the underlying classification task, thus it cannot be effectively applied in all circumstances. previous work focusing solely on data augmentation methods, not coupled with self-training, has shown mixed results for the domain of offensive/hate speech classification (section 2.1). this indicates that there may not be a best method, while some may even negatively impact performance. an open question is whether noisy self-training with text data augmentations can contribute to text classification tasks using state-of-the-art transferlearning bert models that have been shown to be invariant to various data transformations @xcite . the task of offensive/abusive speech detection poses a difficult challenge for generating high-quality semantic invariant augmented examples, since it is a domain that is intrinsically associated with specific keywords that, if modified, can completely change the semantics of the text. in this paper, we innovate by providing an extensive experimentation setup using three different data augmentation techniques -backtranslation, random word swap, and random synonym substitution -in a self-training framework, with five different pretrained bert architectures varying in size, on two different datasets. we demonstrate that self-training, either with or without data noising, outperforms default finetuning regardless of model size, on both datasets. however, when comparing self-training without data noising vs 'noisy' self-training, we find that data augmentations decrease performance, despite the literature reporting the superiority of noisy selftraining in other domains. we further investigate how the augmentation methods fail to create labelinvariant examples for the offensive/hate speech domain. finally, we discuss future research ideas to address the limitations found in this work. in the domain of offensive/hate speech classification, @xcite experiment with three different text augmentation techniques to expand and balance their wikipedia dataset by augmenting negative (non-offensive) examples. from a binary view of the dataset, more than 85% of their examples are labelled as non-offensive, and from a multi-label view of the dataset, three of the six offensive classes are represented by less than 7% of the dataset. they report f1-score increases of +1.4% with unique words augmentation, +2.9% with unique words and random mask, and +3.6% with unique words, random mask, and synonym replacement. @xcite use a custom synonym replacement augmentation method to experiment with a 'toxic' dataset with 6 classes from a kaggle competition 1 . they experiment with character and word embeddings with a cnn architecture, and report a +3.7% and +5.1% roc-auc increase when applying their augmentation method with character embeddings on the public and private scores foot_1 , respectively. however, when coupled with word embeddings, they find that their augmentations result in a decrease of -0.09% and -0.21% roc-auc scores on the public and private scores, respectively. rizos et al. (2019) propose three text-based data augmentation techniques to address the class imbalance in datasets, and apply them on three english hate speech datasets named hon @xcite , rsn-1 @xcite and rsn-2 @xcite . their augmentation methods include (i) synonym replacement based on word embedding, (ii) warping of the token words along the padded sequence, and (iii) class-conditional rnn language generation. they compare the three methods on different architectures combining word embeddings, cnns, grus, and lstms, and they report an average across four different architecture configurations of -6.3% f1-macro using (i), +5% f1-macro using (ii), and -4% f1-macro using (iii). marivate and sefara (2020) experiment with four different data augmentation techniques: word-net synonym substitution, backtranslation between german and english, word embedding substitution according to cosine similarity, and mixup @xcite . authors experiment with three datasets from different domains: sentiment 140 @xcite , ag news @xcite and a hate speech dataset @xcite . they observe performance increases on both sentiment 140 and ag news across different augmentation methods, up to +0.4% and +0.5% accuracy score on ag news and sentiment 140, respectively. however, they report performance decreases with all methods on the hate speech dataset, with decreases of 0.0% with mixup, -0.3% with embedding similarity, -0.8% with synonym substitution, and -2.3% with backtranslation. ## materials and methods this section presents the description of the datasets, data augmentation methods and self-training architectures used throughout our experiments. our code is available at github foot_3 . ## experimental setup firstly, we experiment with each dataset to estimate the hyperparameters for the base models, which is the first teacher models in the self-training loop. we use a batch size of 128, maximum sequence length of 128, learning rate of 0.00001, 15% of the training set as warm-up batches, weight decay of 0.001 and 20 training epochs. we apply a dropout rate of 10% for both the attention and classification layers. the model with highest validation f1-macro score foot_5 obtained during training is loaded at the end of the last epoch. for the hyperparameters associated with the self-training method, we set the number of teacher-student iterations to 4 (including the first teacher model) and a confidence threshold filter of 80%, similarly to @xcite . also, we experiment with five different pre-trained bert models: distilbert, bert-base-cased, bert-large-cased, robertabase and roberta-large, aiming to investigate the impact of model size in performance gains associated with self-training. from the above-listed configurations, we designed two main classification scenarios. the first scenario accounts for a regular self-training loop without data noise injection through augmentations, while the second scenario uses the noisy self-training approach, introducing data noise with one of the three augmentation methods described in section 3.3. also, we highlight how self-training can make smaller models, which require fewer resources to maintain in practical applications, achieving the same performance as larger and more costly models that are trained with default fine-tuning. selftraining on a distilbert (66m parameters) outperforms a bert-large-cased (340m parameters) with default fine-tuning on both olid and convabuse. on olid, a roberta-base architecture (125m parameters) with self-training outperforms a roberta-large (354m parameters) architecture with default fine-tuning, although this does not hold true for convabuse. furthermore, we point out that olid and convabuse's data come from different sources, the first being twitter, and the second one representing conversations between humans and chatbots, thus their structure differs significantly. since our unlabelled dataset is composed of twitter data, it would be fair to assume that the benefits of self-training in our experiments would be more prominent for the olid dataset, but our results do not show this, since models trained with convabuse benefited from self-training with our twitter-originated unlabelled dataset just as much as models trained with olid. ## conclusion in this work, we analysed the impact of selftraining on offensive and hate speech classification tasks using five different pre-trained bert models text augmented text method i hate all of you all i hate of you ws maybe i dont respect all women maybe i respect dont women all ws bitches and sports females and sport bt wooooow what the fuck wooooow, what the hell? bt bitch you better be joking gripe you good be joking ss the nyt has been showing its whole ass [...] the nyt has follow showing its whole butt [...] ss finally, we investigated the three data augmentation methods and showed that the domain of offensive/hate speech classification is highly sensitive to semantic variances produced by them, and we discussed future research ideas to mitigate these problems.