PCJ at S em E val-2023 Task 10: A Ensemble Model Based on Pre-trained Model for Sexism Detection and Classification in E nglish (2023)

Abstract: This paper describes the system and the resulting model submitted by our team “PCJ” to the SemEval-2023 Task 10 sub-task A contest. In this task, we need to test the English text content in the posts to determine whether there is sexism, which involves emotional text classification. Our submission system utilizes methods based on RoBERTa, SimCSE-RoBERTa pre-training models, and model ensemble to classify and train on datasets provided by the organizers. In the final assessment, our submission achieved a macro average F1 score of 0.8449, ranking 28th out of 84 teams in Task A.

## introduction sexism is a very serious problem on the internet nowadays, where unsuspecting people attack women by making statements against them, exposing them to unfair treatment, and eventually causing harmful effects on society. therefore, tools that can automatically detect sexism are widely deployed @xcite , which is an important task in the field of nlp @xcite . sexism detection is a text classification problem. text classification is a classical problem in the field of natural language processing, and the main methods can be divided into two categories: machine learning-based methods and deep learning-based methods. for the machine learning-based methods @xcite , the main idea is artificial feature engineering plus shallow classification model @xcite . text feature engineering is divided into three parts: text preprocessing, feature extraction, and text representation, with the ultimate goal of converting text into a computer-understandable format and encapsulating enough information for classification, and then the classifier uses statistical classification methods, such as svm, plain bayesian classification algorithms. this approach overall relies heavily on discrete manual features. for deep learning-based text classification @xcite , the main idea is that the text to be classified is represented as a word vector, and then deep learning networks and other variants are used for automatic feature extraction, represented by models such as transformer @xcite . however, there are still some deficiencies in current sexism detection, which need to be improved, such as the abuse of language detection models, and the misestimation of sexism @xcite . task 10 of semeval-2023 is the explainable detection of online sexism @xcite , which aims to address the issue of automated detection of large-scale online sexism. by proposing a hierarchical taxonomy with three tasks for detecting sexist content, it ultimately aims to improve the accuracy and explainability of automated detection tools, and contribute to reducing the harmful effects of online sexism. in this paper, we describe the details of our research in subtask a and how it performed on the evaluation data. the specific task is described as performing sexism detection, so that the system must predict whether the post is sexist or not, making a binary classification. the paper is organized as follows. section 2 briefly describes the relevant background regarding text classification and text preprocessing. section 3 describes the system architecture used in the experiments to explain the model approach. in section 4, the experiments and results are presented, and section 5 presents a summary of the whole paper and areas for future improvements. ## related work the emergence of pre-trained models (ptms) has brought the field of natural language processing into a whole new era. pre-training has been an effective strategy for learning deep neural network parameters, and fine-tuning the downstream tasks based on pre-training models is beneficial for nlp tasks. the development of ptms goes through two stages from non-contextual embeddings to contextual embeddings @xcite . for non-contextual embeddings, the representative one is word2vec @xcite . this type of word embedding usually takes shallow networks for training, and when applied to downstream tasks, the rest of the whole model still needs to be learned from scratch. for contextual embedding, the main purpose is to solve the problem of multiple meanings of a word through a pre-trained encoder that can output contextually relevant word vectors, representative ones are openai gpt @xcite , bert @xcite .in pre-training models, roberta is an improved version of bert, which uses a larger training dataset, longer training time, and some other optimization strategies. roberta has achieved better results than bert in multiple natural language processing tasks @xcite , including text classification tasks. therefore, it is suitable to be used as the basic pre-training model for this task. in conclusion, in this competition, we adopt a deep learning-based approach for text classification, using the existing dataset to fine-tune the pretrained models roberta and simcse-roberta to obtain the embedding vectors, combining bil-stm @xcite and bigru @xcite neural network structures for task classification. finally, in order to improve the task classification, the model ensemble method is used to fuse the fine-tuned models together @xcite . ## system overview this section describes the models used. we use roberta and simcse-roberta pre-trained models as the coding layer for modification, and on top of roberta-large and sup-simcse-roberta-large base models, we then down-join bilstm and bi-gru structures to let the models capture more information in combination to contextual semantics. then, we construct 4 types of text classification models. finally, the trained models are fused and the final classification results are selected using the voting method. the system architecture used is described below. ## conclusion this paper describes our binary sexism detection system for subtask a of semeval-2023 task 10, including system design, implementation, and evaluation. the final experimental results demonstrate the effectiveness of our pre-trained language ensemble model for sexism detection and classification. due to the limited time of this event, there are still some improvement strategies that we have not tried. for future improvements, we can start from these aspects: (1) mitigating the impact of imbalanced sample distribution through sampling or loss function adjustments, such as using over-sampling, class weight adjustment; (2) using unlabeled data to perform semi-supervised training so that the system can learn more valid information; (3) using more effective model ensemble strategies, such as xgboost; (4) using multi-task deep pre-training @xcite .