A lexa at S em E val-2023 Task 10: Ensemble Modeling of D e BERT a and BERT Variations for Identifying Sexist Text (2023)

Abstract: This study presents an ensemble approach for detecting sexist text in the context of the Semeval-2023 task 10. Our approach leverages 18 models, including DeBERTa-v3-base models with different input sequence lengths, a BERT-based model trained on identifying hate speech, and three more models pre-trained on the task’s unlabeled data with varying input lengths. The results of our framework on the development set show an f1-score of 84.92% and on the testing set 84.55%, effectively demonstrating the strength of the ensemble approach in getting accurate results.

## introduction sexist language has been a persistent issue in various forms of communication, including online communication. the use of sexist language perpetuates gender stereotypes and reinforces the marginalization of certain groups, particularly women. detecting and addressing instances of sexist language is crucial for promoting gender equality and reducing discrimination. in recent years, there has been a growing interest in developing methods for automatically detecting sexist language. the semeval-2023 task 10 provides @xcite a platform for researchers to explore and evaluate various approaches to identifying sexist text. this task involves identifying instances of sexist language in various forms of online communication, including social media posts, comments, and reviews. previous research has explored different methods for identifying sexist language, including rulebased approaches, machine-learning techniques, and deep-learning models. while these approaches have shown promising results, the challenge of detecting sexist language remains complex due to the subtleties and nuances of language use. our approach for identifying the sexist language in the semeval-2023 task 10 subtask a involves an ensemble of 18 different models, including deberta-v3-base and pre-trained bert-based models pre-trained on the task's unlabeled data. the model's predictions are combined to improve overall performance. our method achieved an f1score of 84.92% on the development dataset and 84.55% on the testing dataset, ranking 30th out of 89 participating teams. section 2 of this paper introduces an overview of the related work in the field of sexist text identification. section 3 gives an overview of the details of the dataset we used in this research. section 4 presents key characteristics of the ensemble approach and provides a detailed description of the models used. finally, in section 5, we conclude the paper and discuss future directions for research in this area. ## background sexism against women on social media has become a growing concern, leading researchers to develop automatic systems that detect sexist text. online competitions such as the semeval-2022 task 5 on "multimedia automatic misogyny identification" and "sexism identification in social networks (exist)" at iberlef 2021 @xcite have helped accelerate this research. the competition aimed to identify sexism in social media content using machine learning, with two sub-tasks: binary classification and fine-grained classification distinguishing between five different types of sexist content. the ait_fhstp team @xcite applies two multilingual transformer models, multilingual bert and xlm-r, using pre-training with additional data and supervised fine-tuning with augmented data. the best model in their work is the xlm-r with a macro f1-score of 77.52% for binary classification (task 1) ranking 13 and 55.89% for the multi-class classification (task 2) ranking seventh. @xcite presents a deep neural network approach to detect sexist text, including bert and distilbert. the best model for binary classification (task 1) uses bert and a multi-filter cnn model achieving an accuracy of 76.2%. the same model with data augmentation achieves the best performance for multiclass classification with an f1 score of 51.9% in earlier work, @xcite explore the use of knowledge graphs to improve the performance of sexist tweet classifiers. the authors propose using conceptnet and wikidata to improve sexist tweet classification by two methods: text augmentation and text generation. in the text generation approach, new tweets are generated by replacing words with data from conceptnet relations, increasing the size of the training set and preserving the label. in the text augmentation approach, words in the tweets are augmented with words from conceptnet relations and their descriptions from wikidata, increasing the length of each tweet without changing the number of tweets in each class. the authors find that their approach significantly improved sexist tweet classification across multiple machine learning models and claim it can be applied to other small dataset problems, such as hate speech or abusive language and text classification. ## dataset the subtask a dataset used in this research consists of 20,000 entries sampled from gab and reddit. the training dataset, which constitutes 70% of the overall dataset, contains 14,000 entries, out of which 3,398 are labeled as sexist. ## results the obtained results of the individual models and the ensemble approaches are presented in our results indicate that ensembling multiple models can be an effective approach for identifying sexist text. the combination of different models provides a more robust solution, leveraging the strengths of each individual model. it is possible that adding even more models to the ensemble would lead to further improvements in performance. additionally, we found that our approach had difficulty with detecting dehumanizing attacks and overt sexual objectification. these types of language often involved explicit references to the target's body and required a fine-grained analysis of the language use. ## conclusion in this research, we investigated the use of an ensemble approach for identifying sexist text. our approach combined multiple transformer models, including different variations of deberta-v3-base, and bert. we found that the ensemble approach outperformed the individual models, achieving 84.92% f1-score on the development set of the semeval task and 84.55% on the testing set. our results demonstrate the effectiveness of combining multiple models to identify sexist text. the combination of different models provides a more robust solution, leveraging the strengths of each individual model. our approach also showed that pre-training some of the models on the unlabeled data provided by the task can be an effective way of incorporating relevant information. additionally, we explored the impact of data augmentation on the results. our findings indicate that data augmentation techniques, such as using online data on the same topic, did not result in a significant improvement in performance. the improvement in results came from ensembling the models rather than data augmentation. our conclusion highlights the importance of ensembling multiple models for the task of identifying sexist text. further research is needed to determine the optimal number of models to include in an ensemble and to improve the performance of the ensemble approach. we discovered that our system struggles with identifying certain forms of subtle and implicit sexism, which is a common challenge in detecting sexist language. in future work, we plan to explore additional feature engineering techniques and alternative methods for model selection to further improve the performance of our ensemble system. overall, this research provides valuable insights into the use of an ensemble approach for identifying sexist text. the findings have important implications for future work in this area and demonstrate the potential for using ensembles to tackle complex nlp tasks.