JCT _ DM at S em E val-2023 Task 10: Detection of Online Sexism: from Classical Models to Transformers (2023)

Abstract: This paper presents the experimentation of systems for detecting online sexism relying on classical models, deep learning models, and transformer-based models. The systems aim to provide a comprehensive approach to handling the intricacies of online language, including slang and neologisms. The dataset consists of labeled and unlabeled data from Gab and Reddit, which allows for the development of unsupervised or semi-supervised models. The system utilizes TF-IDF with classical models, bidirectional models with embedding, and pre-trained transformer models. The paper discusses the experimental setup and results, demonstrating the effectiveness of the system in detecting online sexism.

## introduction sexism is a pervasive problem online, particularly for women who are disproportionately targeted. it can cause harm, make online spaces unwelcoming, and reinforce social inequalities. automated tools have been developed to identify sexist content at scale, but most current systems only provide generic, high-level classifications without further explanation. this lack of detail makes it difficult for users and moderators to understand and trust the decisions made by these systems. semeval 2023 -task 10 -explainable detection of online sexism (edos) @xcite aimed at releasing a dataset to assist in the development of english-language models for sexism detection that are more accurate and explainable. this paper handles the first subtask a in task 10. we experimented with systems relying on classical models, deep-learning models, and transformer-based models to tackle the first task of binary sexism detection on gab and reddit. the goal of our system is to provide a comprehensive approach that can handle the intricacies of online language, which often involves slang, colloquialisms, and neologisms. ## related work previous competition "sexism identification in social networks (exist)" @xcite was conducted during the iberlef 2021 international forum, which was exclusively focused on the field of natural language processing. the competition aimed to classify multilingual posts into five categories, including ideological and inequality, stereotyping and dominance, objectification, sexual violence, misogyny, and non-sexual violence. the semeval competition differs and involves the identification of 11 types of sexual posts, indicating a more complex and nuanced classification task. in addition, fortuna @xcite addresses the challenges that arise when applying conventional nlp methodologies to hate speech detection. the authors reflect on the inadequacies of frequently used conventions for text classification with respect to hate speech detection and discuss the future of current methodologies for this task. based on their analysis, they conclude that current models are incapable of detecting hate speech without causing harm to marginalized communities. therefore, they call for the scientific community to adapt nlp methodologies and center on the impact that used methodologies may have on marginalized communities. the authors believe that by critically reflecting on the potential impacts of methodologies for hate speech detection, the scientific community can identify methodologies that result in more just futures. ## dataset the dataset @xcite for the binary sexism detection task consists of three parts: a training set with 14,000 sentences, a development set with 2,000 sentences, and a test set with 2,000 sentences ( in addition to the labeled data, there are two unlabeled corpora, one from gab and one from reddit. these corpora are provided by task organizers as well. each contains 1,000,000 sentences. these corpora were included to provide additional unlabeled data for the development of unsupervised or semi-supervised models. the availability of this dataset provides a valuable resource for researchers and practitioners to develop more accurate and explainable models for detecting and combatting sexism online. train dev test 14000 2000 4000 ## system overview our system incorporates a variety of models and methodologies, including classical models such as random forest @xcite and logistic regression @xcite based on vectors generated by the tf-idf statistical measure. we also explored deep learning models with bidirectional layers and word embedding created by word2vec @xcite . additionally, we explored transformer-based models to further enhance our system's performance. ## experimental setup our implementation uses sklearn library @xcite for the classical models (tf-idf, logistic regression and random forest), gensim library @xcite for the embedding models, keras library @xcite for the bi-lstm model and the transformers li-brary by huggingface @xcite for the pre-trained transformer models and corresponding tokenizers. ## results the results of various models on the dev dataset, evaluated using the f1 metric score, are presented in among the classical models, the random forest model failed to learn the underlying patterns, resulting in a poor f1 score of 43%. in contrast, the logistic regression model achieved a better f1 score of 73.63% in the char approach and 73.65% in the words approach. using an unlabeled corpus for training gives a small improvement in the logistic regression model's performance, compared to using only the training corpus. in in general, the tf-idf model trained on the unlabeled corpus with a word-based approach produced the highest score. the bi-lstm models with glove-twitter-crossentropy and the word2vec model trained on the unlabelled corpus with crossentropy achieved a minor improvement, compared to their corresponding models with focal loss. in contrast, the pre-trained word2vec model achieved a minor improvement in focal loss compared to the crossentropy model. in contrast to the tf-idf, the leveraging of the unlabeled corpus did not improve performance. the results decreased by 0.636, and 0.429 compared to using the most effective bi-lstm models (glove-twitter embedding vectors with and without focal loss, respectively). however, the bi-lstm model's overall performance was lower than that of the logistic regression model. the transformer models, roberta and bertweet, delivered the best performance among all models. specifically, bertweet outperformed the roberta model, resulting in a higher performance improvement of 12.02% compared to the best tf-idf model and 16.09% compared to the bi-lstm model. hyper-tuning the bertweet model by limiting the pre-processing to lemmatization resulted in the best score on the dev test with 84.07%, which was submitted and ranked 40th with 83.62% performance on the test set. ## conclusion in this study, we proposed an approach for the explainable detection of online sexism (edos) task, which involved the development of various systems such as classical models, deep learning models, and transformer-based models to address the binary sexism detection challenge. through our experiments, we found that the use of similar corpora such as twitter led to improved performance across most models. the vast volume of data on twitter, coupled with its diverse range of topics, makes it an ideal source for training machine learning models. moving forward, potential avenues for future research include the development of the procedure of pre-training bert on gab and reddit as done on twitter in bertweet to improve performance. furthermore, an ensemble of models of all the models developed in this study could lead to even better performance. by leveraging the strengths of each individual model and mitigating their weaknesses, an ensemble of models has the potential to produce more reliable results. as such, we believe that our work contributes to the broader goal of addressing online sexism. it provides a foundation on future studies to build upon.