HULAT at S em E val-2023 Task 10: Data Augmentation for Pre-trained Transformers Applied to the Detection of Sexism in Social Media (2023)

Abstract: This paper describes our participation in SemEval-2023 Task 10, whose goal is the detection of sexism in social media. We explore some of the most popular transformer models such as BERT, DistilBERT, RoBERTa, and XLNet. We also study different data augmentation techniques to increase the training dataset. During the development phase, our best results were obtained by using RoBERTa and data augmentation for tasks B and C. However, the use of synthetic data does not improve the results for task C. We participated in the three subtasks. Our approach still has much room for improvement, especially in the two fine-grained classifications. All our code is available in the repository https://github.com/isegura/hulat_edos .

## introduction sexism can be defined as behaviors or beliefs that support gender inequality, and result in discrimination, generally against women. contrary to what one might believe, sexism is still very present also in the most advanced and technologically advanced societies @xcite . proof of this is that many gender stereotypes are still present in our belief system today (for example, men should not wear dresses). unfortunately, social networks are used to spread hateful and sexist messages against women @xcite . during the last few years, various research efforts @xcite have been devoted to the development of automatic tools for the detection of sexist content. while these automated tools have addressed the classification of sexist content, this is a highlevel classification, without providing additional information that allows us to understand why the content is sexist. the goal of semeval-2023 task 10, explainable detection of online sexism (edos) @xcite , is to promote the development of fine-grained classification models for detecting sexism in posts written in english, which were collected from gab and reddit. the organizers of the task proposed three subtasks: a) binary sexism detection, b) category of sexism, a fourclass classification task, and c) fine-grained vector of sexism, an 11-class classification. a detailed description of these classifications can be found at @xcite . in our approach, we explored some of the most popular pre-trained transformer models such as bert @xcite , distilbert @xcite , roberta @xcite , and xlnet @xcite . moreover, we used different data augmentation techniques such as eda @xcite and nlpaug library to create synthetic data. 1 then, synthetic data and training data were used to fine-tune the models. based on our experiments during the development phase, we decided to use the roberta transformer model to estimate our predictions for the test dataset during the test phase. we participated in the three subtasks. in task a, our system obtained a macro f1-score of 0.8298, ranking 43th, with a total of 84 teams in the final ranking. the top system achieved a macro f1score of 0.8746, while the lowest macro f1-score was 0.5029. about half of the systems achieved a macro f1-score below 0.83. in task b, our system ranked in the 45th position out of the 69 participating systems. our macro f1-score was 0.5877, while the lowest and highest macro f1-scores were 0.229 and 0.7326, respectively. in task c, our team ranked in the 27th position out of the 63 participating systems. the lowest and highest macro f1-scores were 0.06 and 0.56, respectively. about half of the systems achieved a macro f1-score below 0.42, while our system had a macro f1-score of 0.44. our systems, which ranked roughly in the middle of the three rankings, show modest results on the three subtasks. our approach still has much room for improvement, especially in the two fine-grained classifications. the results showed that the use of synthetic data does not appear to provide a significant improvement in the performance of the transformers. all our code is available in the repository https://github.com/isegura/hulat_edos . ## dataset overview the goal of this task is to detect sexist content. the task is composed of three subtasks: a, b and c. task a is a binary classification task to distinguish between sexism and non-sexism texts. task b and c aim to make a more fine-grained classification with four and eleven classes, respectively. the full dataset consists of 20,000 posts written in english. half of the posts were taken from reddit and the other half from gab. gab is a social network known for its far-right users. the dataset was divided in three splits with a ratio of 70:10:20. that is, 14,000 posts were used for training, 2,000 for development, and 4,000 for the final evaluation. we have studied the class distribution in each task. in task a, a binary classification, the two classes are not balanced, where the not-sexist class is the majority class (see fig. 1a ). as expected, the same distribution was observed in the three splits (which have been provided by the organizers). we also plot the distribution of categories for task b (see fig. 1b . to obtain the distribution of these categories, we removed those records that were annotated as "not sexist". the majority category is "2. derogation". the second class with a larger number of instances is "3. animosity". the other two classes are the minority classes, "4. prejudiced discussions" and "1. threats, which have a similar number of instances. the same distribution is observed in the three dataset splits. regarding the distribution of the vectors in task c (see fig. 1c ), the vector subcategory "2.1 descriptive attacks" is the majority class, while "3.4 condescending explanations or unwelcome advice" is the minority class. the vectors follow a distribution similar to that of their corresponding categories. for example, the vectors with the largest number of instances are usually the vectors of the category "2. derogation", followed by the vectors corresponding to the category "3. animosity". we studied the length of the texts in tokens (see fig. 2 ) to set the max_length argument in the transformers. this argument controls the length of the padding and truncation. to calculate the number of tokens in a text, we split it by white space. the mean number of tokens is around 23.3 with a standard deviation of 11.7, and the maximum length is 58 tokens. we also want to know if there are differences in the length of the texts between the two main classes: sexist and non-sexist (fig. 3 ). the distribution of length for sexist texts is slightly more skewed towards longer texts than the non-sexist distribution. figure 4 shows the length distribution of the texts for each category in task b. we can see that the texts classified as "4. prejudiced discussions" appear to be longer than the other texts (mean number of tokens is around 27.8 with a standard deviation of 10.9). the category "1. threats, plans to harm and incitement" has the shortest texts, with an average length around 22.9 tokens and a standard deviation of 11.7. the other two categories, "2. . derogation" and "3. animosity", show very similar distribution with an average length of around 24 tokens for their texts. we also study the length distribution of texts for each vector. as there are eleven vectors, it is very difficult to compare their distributions. for this reason, we created a density graph for the vectors within each category (see appendix, fig. 5 ). all vectors have a very similar distribution of text length. texts classified as "4.1 supporting mistreatment of individual women" or "4.2 supporting systemic discrimination against women as a group" tend to have the largest average length between 27 and 30 tokens. the vector "2.1 descriptive attacks" has an average length of 26 tokens. the vector "1.2 incitement and encouragement of harm" has the smallest average length (around 22 tokens). the other vectors have an average length between 23 and 25 tokens. therefore, there do not seem to be significant differences between the length of the texts of each vector. bert @xcite is a popular transformer model due to its excellent results in many nlp tasks. bert is an encoder trained using two strategies: masked language modeling (mlm) and next sentence prediction (nsp). distilbert @xcite is a smaller version of bert, which can achieve similar results to bert but with less training time. roberta @xcite is based on bert. roberta was pre-trained using additional data. unlike bert, roberta does not use the next sentence prediction (nsp) strategy. regarding the mlm strategy, some tokens are dynamically masked during pre-training. another difference with bert is that roberta uses a byte-level bpe tokenizer, which has a larger vocabulary than bert (50k vs 30k). therefore, roberta has a larger vocabulary that can provide better results, but with an increase in complexity. xlnet @xcite ) is an autoregressive model. that is, it was pre-trained to predict the next token for a given input sequence of tokens. xlnet does not use any masked strategy. instead of this, it uses permutation language modelling that can capture context by training an autoregressive model on all possible permutations of words in a sentence. in this way, bidirectional contextualized word representations are obtained. like bert, this model was trained with wikipedia and bookscorpus, but also with giga5, clueweb 2012-b, and common crawl. ## results hulat participated in the three subtasks. below we present our results in each task. ## conclusion our team participated in the three tasks with an approach based on robert fine-tuned with training data and synthetic data created by data augmentation techniques. this approach shows very modest results on the three tasks (our systems approximately rank in the middle of the three rankings). we still have much room for improvement, especially in the two fine-grained classifications. while data augmentation does not achieve a significant improvement in task a, it obtains a positive effect on the results in task c. although the use of augmented data provided the best results for task b during the development phase, our final results on the final test dataset show the opposite. in future work, we plan to extend our research on data augmentation techniques to augment the training data. for example, we plan to use back translation @xcite . in addition, we will exploit other datasets for the detection of sexist content, such as the exist dataset @xcite or mami @xcite , to also approach the task from two different scenarios: multilingual and multimodal.