Tsingriver at S em E val-2023 Task 10: Labeled Data Augmentation in Consistency Training (2023)

Abstract: Semi-supervised learning has promising performance in deep learning, one of the approaches is consistency training on a large amount of unlabeled data to constrain model predictions to be invariant to input noise. However, The degree of correlation between unlabeled data and task objective directly affects model prediction performance. This paper describes our system designed for SemEval-2023 Task 10: Explainable Detection of Online Sexism. We utilize a consistency training framework and data augmentation as the main strategy to train a model. The score obtained by our method is 0.8180 in subtask A, ranking 57 in all the teams.

## introduction sexism is an increasingly serious network problem. it may harm targeted women, make cyberspace inaccessible and unpopular, and perpetuate social asymmetry and injustice. explainable detection of online sexism task targets to predict whether a sentence is sexism @xcite . a small amount of labeled data limits the improvement of supervised learning model performance. in contrast, semi-supervised and self-supervised learning methods can learn from massive unlabeled data @xcite , even better than supervised learning. bidirectional encoder representation from transformers(bert) and robustly optimized bert approach(roberta) @xcite are classic models that are trained by unlabeled data in unsupervised learning tasks. in this work, we aim to apply a semi-supervised learning method to accomplish the task, using labeled data as the train data of supervised training and augmented labeled data as the train data of unsupervised training(consistency training), respectively. due to the difficulty of selecting domain-relevance unlabeled data in consistency learning and the fact that the labeled data is domain-relevance data that matches with task objective, we decide to augment labeled data instead of unlabeled data that mismatch the distributions of labeled data. improving model performance is as important as ensuring model stability @xcite , which is why we choose to use consistency learning. our method guarantees the same prediction result for the same sentence expressed with different words in a certain probability. in other words, it improves the model's performance in detecting synonymous sentences. to not lose bert's original performance as much as possible, we combine two strategies: consistency training and synonym replacement method based on tf-idf to fine-tune bert, instead of directly retraining bert @xcite . compared with the supervised learning method training our baseline model in this paper, our main training strategies have significantly improved model performance from 0.8004 to 0.8180. we have released the code at https://github.com/vincent-hyx/task_10 . ## background there are two main directions of research work, one is consistency training @xcite , and the other is unsupervised data augmentation(uda) @xcite . the consistency training method relies on unmarked data training to overcome the weakness of supervised learning that requires a large number of labeled data @xcite . it evaluates the consistency between corresponding sentences that contain a pair of sentences with similar semantics. data augmentation has achieved outstanding results @xcite . for instance, eda(easy data augmentation) @xcite applies to most nlp tasks limited by a small amount of labeled data. besides, to solve the problem of limited annotation data, computer vision tasks utilize auto-augmentation @xcite to generate additional data. it is worth noting that the proposed uda method provides a way to obtain high-quality, unlabeled and supplementary data for consistency learning. the general consistency training framework uses kl divergence as an evaluation of the consistency between the augmented data and the original data, and simultaneously combines it with a supervised learning approach. uda method using consistency training in the research of text classification proposes two data augmentation methods, back-translation and tf-idf word replacement @xcite , to generate the data required for consistency training. ## experimental setup in this work, our model accepts input data derived from given train data, while the model's output is the probability of binary classification(sexist or not sexist). the training data consists of 14,000 entries in english, of which 3,398 are sexist @xcite . in this task, we divide all tagged data into three parts, one accounting for 90% for training(train), one accounting for 5% for verification(dev), and one accounting for 5% for testing(test). in addition, the data used in the unsupervised learning setting is derived from the dataset for training augmented by the previously mentioned data augmentation approach, which has 14,000 examples. in the parameters setting, it is appropriate to set the learning rate at 1e-5, and we use a batch size of 20 for the supervised loss and use a batch size of 40 for the unsupervised loss. another superparameter about the temperature in softmax function is set to 0.8 for our experiment. if the focal loss is used as supervised loss, we recommend setting the parameter gamma at 2. because the original training data contains some unnecessary characters and emoji expressions, we performed a general data cleaning operation before using the data for training, and all the related code has been included in the project file uploaded to github. ## results in contrast to our method, we trained a baseline model which only utilizes the supervised learning method with cross entropy as a loss function, but it uses the same model structure as that used in our method.according to official requirements, we adopt macro f1(ave f1 called in figure 2 ) as evaluation metric to present the results. in this task, we compare the results with and without our method, and compare the effect of focal loss and cross entropy loss on our method. our method scored 0.8180 in task a, ranking 57 in all the teams. all results are shown in in addition to showing the macro f1 score obtained through our method, we also want to show the impact of different threshold selection on the model performance, and the degradation of performance caused by using unlabeled data from gab and reddit for the model trained by consistency training approach. 1 , and we put it here to show that the results of using unlabeled data from gab and reddit are not only inferior to those of using labeled data, but even worse than the baseline model. we tested some different thresholds, crucial for the data augmentation strategy, and found that 0.8 is the optimal threshold corresponding macro f1 score is 0.818. it is shown as follows: ## conclusion compared with the baseline model, we use the semi-supervised consistency training framework and the data-augmented method of synonym replacement based on tf-idf, which has significantly improved model performance from 0.8004 to 0.8180. in the future, we will explore different data augmentation strategies to provide more high-quality samples for consistency training, and research how to design better loss functions for consistency training and more training skills.