nice NLP at S em E val-2023 Task 10: Dual Model Alternate Pseudo-labeling Improves Your Predictions (2023)

Abstract: Sexism is a growing online problem. It harms women who are targeted and makes online spaces inaccessible and unwelcoming. In this paper, we present our approach for Task A of SemEval-2023 Task 10: Explainable Detection of Online Sexism (EDOS), which aims to perform binary sexism detection on textual content. To solve this task, we fine-tune the pre-trained model based on several popular natural language processing methods to improve the generalization ability in the face of different data. According to the experimental results, the effective combination of multiple methods enables our approach to achieve excellent performance gains.

## introduction gender discrimination is any mistreatment or negative feelings directed at women based on their gender, or based on their gender in combination with one or more other identity attributes (e.g., black women, muslim women, trans women). as the internet continues to grow, sexism has become a growing online problem. it can be harmful to women who are discriminated against while also posing a threat to the healthy functioning of cyberspace. large-scale pre-trained models have been widely used to automate the identification of discriminatory statements. however, these models are often capable of simple labeling and not interpretative descriptions. improving the interpretability of sexism detection can help to thereby enhance the understanding and behavioral decision-making ability of users and moderators of automated tools. to address this issue, semeval 2023 -task 10 supports the development of english-language models for sexism detection that are more accurate as well as explainable, with fine-grained classifications for sexist content from gab and reddit. this task proposes a new english sexism detection dataset * *corresponding author based on gab and reddit, which contains four finegrained and eleven more fine-grained category labels. in task a, each sentence was first asked to perform a dichotomous classification of the presence or absence of sexism. and tasks b and c require a finer-grained distinction between sentences with sex discrimination @xcite . in many past studies, language models called bert have been used extensively for several tasks @xcite . bert models can be pretrained with a self-supervised approach to generate word/tag or sentence representations that are rich in a priori knowledge @xcite . they can then be specifically fine-tuned for many downstream tasks, including text classification. moreover, various model optimization methods have been shown to be effective in enhancing the finetuning of pre-trained models. therefore, in this work, we employ the current state-of-the-art pretraining model deberta-v3 to exploit the prior knowledge in its pre-trained resources @xcite . and to adapt it to the linguistic context in the task, we pre-trained the model again using a label-free corpus provided by the organizer. in addition, we improve the model classification based on various model tuning methods including adversarial training @xcite and stochastic weight averaging (swa) @xcite . finally, based on the characteristics of the binary classification task, we constructed a dual model alternate pseudo-labeling (cascante-bonilla et al., 2021) approach using multi-sample dropout @xcite to complete the final classification. our approach finally achieved the eleventh place in the test phase leaderboard of task a. the experimental results demonstrate the effectiveness of the multiple methods we used and proposed. ## related work in recent years, there has been an increasing interest in sex discrimination detection tasks. such as by using glove embeddings @xcite and improved lstms to see sexism in the workplace, adding attention mechanisms to it @xcite . masked language modeling (mlm) is one of the pre-training methods used by bert. although the deberta model does not use this method, it can still improve the performance of the model from our experimental results. inspired by previous pre-training models such as bert, roberta @xcite and albert @xcite , deberta-v3 employs a series of innovative techniques to improve the performance and efficiency of the model. among the most notable features are: the use of a dynamic masking mechanism in the pre-training phase, which allows the model to better capture relationships in long texts; the use of a new cross-layer interaction mechanism that allows the integration of different semantic information at different levels; and the use of a global adaptive regularization method that reduces the overfitting problem. adversarial training is a training method that introduces noise and adds perturbations to the samples while trying not to change the distribution of the original samples, allowing the model to ignore such perturbations and thus improving the robustness of the model. the fast gradient method (fgm) that we use is mainly useful in nlp tasks @xcite . unlike the direct perturbation method in the image domain on the input samples, fgm perturbs on the word vectors because a sentence that changes a word (e.g., synonym substitution) may cause the meaning of the whole sentence to change, and the strength of the perturbation is not easy to control. swa is a technique for optimizing neural networks by using models with random weights during training to smooth out the loss curve of the model. this technique improves the generalization ability of the model, prevents overfitting, and generally leads to better test set performance. ## approach overview for task a, each sentence in the dataset was labeled as 0 if sex discrimination was present and 1 otherwise. we chose the large version of the deberta-v3 model as the backbone. the deberta-v3-large model comes with 24 layers and a hidden size of 1024. it has 304m backbone parameters with a vocabulary containing 128k tokens which introduce 131m parameters in the embedding layer. this model was trained using the 160gb data as de-berta v2. figure 1 shows the architecture of the whole approach. we first pre-trained the deberta-v3 model using the mlm method on one million gab texts and one million reddit texts provided by the organizer and then used fgm and swa to further improve the model's generalization and resistance to overfitting. in model 1, multi-sample dropout is used to improve the neural network and form an alternative dual-model structure with model 2. finally, we use the classification results of each model in the test set as pseudo-labels and add them to the training set of another model. several iterations of alternative training are performed until the classification effect no longer improves. ## experiments we conducted experiments on the training and test sets provided by the task organizer, where data from the development phase were also added to the training set. the training set contains 16,000 samples, of which 12,116 are "not sexist" samples and 3,884 are "sexist" samples. we divide 25% of them into validation sets and ensure that the sample proportion is constant. the test set contained 4,000 samples, including 3,030 "not sexist" samples and 970 "sexist" samples. ## conclusion this paper compares and summarizes the different methods used in solving task a. mlm, fgm, swa, and multi-sample dropout form the backbone of the whole approach. the pseudo-labeling technique based on dual-model interaction substantially improves the final performance of the model. the role played by multi-sample dropouts in this technique deserves further discussion.