P anwar J ayant at S em E val-2023 Task 10: Exploring the Effectiveness of Conventional Machine Learning Techniques for Online Sexism Detection (2023)

Abstract: The rapid growth of online communication using social media platforms has led to an increase in the presence of hate speech, especially in terms of sexist language online. The proliferation of such hate speech has a significant impact on the mental health and well-being of the users and hence the need for automated systems to detect and filter such texts. In this study, we explore the effectiveness of conventional machine learning techniques for detecting sexist text. We explore five conventional classifiers, namely, Logistic Regression, Decision Tree, XGBoost, Support Vector Machines, and Random Forest. The results show that different classifiers perform differently on each task due to their different inherent architectures which may be suited to a certain problem more. These models are trained on the shared task dataset, which includes both sexist and non-sexist texts. All in all, this study explores the potential of conventional machine learning techniques in detecting online sexist content. The results of this study highlight the strengths and weaknesses of all classifiers with respect to all subtasks. The results of this study will be useful for researchers and practitioners interested in developing systems for detecting or filtering online hate speech.

## introduction the prevalence of sexist and offensive language online has severe consequences for the mental health and well-being of women. emotional distress caused by exposure to such language can lead to anger, sadness, and fear, which can negatively impact their overall well-being. given the severe harm caused by sexist language online, it is imperative that automated systems be developed to detect and filter such texts. the shared task (see @xcite for a detailed description) deals with the online detection of sexism in english and provides the right opportunity to test the performance of automated systems in doing so effectively. the task comprises three subtasks: in this study, we employ several different conventional machine classifiers like random forests, decision trees, etc. by conventional classifiers, we mean those that have been developed before the advent of more recent techniques like deep learning and transformer-based approaches. they are simpler in design, easier to interpret and understand, and require less computational resources compared to classifiers that rely on deep learning techniques. in spite of being "conventional", these classifiers are still widely used in many practical applications and have been effective in resolving many realworld problems. the aim of this study will be to judge the effectiveness of these classifiers in detecting sexist text and also the type of sexism (as seen in subtasks b and c). moreover, we deal with the highly imbalanced dataset by undersampling so that the classifiers do not become biased toward one class. from the results, it is pretty clear that random forest classifier is the best at detecting sexism in a binary setting. however, when we go a level or two deeper and need to identify the class of sexism as well, random forests are not such a good choice and lose out to support vector machines and decision trees. for task-a, our best classifier achieved the rank of 73rd among all participants, while for tasks-b and c, our best-performing classifiers were ranked 65th and 55th respectively, among all the participating teams. ## background as stated earlier, the task had three subtasks. only the training dataset provided by the task organizers was used in the study and no other dataset was utilized for training the conventional classifiers. the dataset consisted of 14,000 entries taken from posts/comments on two social networking platforms: gab and reddit. the entries also had their respective labels for each subtask. the task organizers had also provided a large amount of unlabelled data but it was not used in training the classifiers. the input for the system would be the text of the post/comment with everything preserved including the emojis, hyperlinks, and hashtags but not the usernames. the usernames are masked as [user] in the dataset for privacy reasons. each subtask had a different form of output. ## system overview as stated earlier, the focus of the study is to explore the effectiveness of conventional machine learning classifiers, and therefore, we consider the following classifiers (some are used only in one subtask while some are used for all three subtasks): the general outline of the model can be noticed in figure -1. the first stage is also one of the most important stages of the system, i.e., data preprocessing, where we make sure that the data being carried forward is fit for modelling our classifiers. then we come to the stage of feature engineering, which is mainly concerned with converting the text into vectors, which the classifier will be able to understand when modelling the data. once we have extracted the features, we split the vector dataset into two parts for training and testing. we discuss the implementation level details in the next section. from here on, the training data is used to train the classifier to understand the task and provide correct labels. finally, the trained classifier is fed with testing data in order to evaluate its performance and see how accurately and precisely it is able to predict the correct tag of sexism. ## experimental setup one of the most important things to note about this shared task is the dataset itself. the dataset is highly imbalanced (see figure -2 ). the number of non-sexist entries is almost 3 times the number of sexist entries. in order to deal with this imbalance, we tried different techniques like oversampling and undersampling. undersampling proved to be the best option as it was able to boost performance significantly which we will look at in the next section. undersampling is a technique used in machine learning to address the problem of class imbalance in a dataset. class imbalance occurs when one class (in our case, the sexist class) has significantly fewer examples than the other class (the non-sexist class). this can lead to biased models that perform poorly on the minority class. undersampling involves randomly removing examples from the majority class in order to balance the number of examples in each class. in our system, we structure the dataset such that the non-sexist class had only 1000 more instances than the sexist class. we dropped more than half of the examples in the original non-sexist dataset. this greatly improved our performance, as we will see in the later section. after undersampling, we do random splitting of the dataset to generate our train-test split. for the train-test split, we started with a 75-25% split but noticed a jump in performance in development set evaluation when the split was changed to 85-15%. the authors had provided both the development set and the test set. test set correct labels were not made accessible until the end of the task, so the system was developed only using the development set. next, we come on to the data preprocessing details. the steps in this stage are as follows: for the feature engineering stage, we make use of a simple tf-idf (term frequency-inverse document frequency) vectorizer. it converts our words or tokens into vectors/embeddings by quantifying the relevance of a term in a sentence or a corpus by weighing how often it appears in a sentence against how often it appears in the entire corpus. for implementing the conventional classifiers, we use the in-built classifiers of sklearn foot_0 library. for classifiers like support vector machine, we performed hyperparameter tuning to ensure that we get the best performance out of the model. we use a svm classifier with linear kernel, 1.0 as the penalty parameter (c), degree set to 3, and gamma set to auto. for evaluation, like the authors of the shared task, we stick to the macro average f-1 scores of the sklearn library. it is best to take the macro average instead of just the average because our dataset is highly imbalanced and the macro average technique treats each class of the data adequately. it does so by first calculating the f-1 score for each class separately and then taking the arithmetic mean of those scores. as a result of this, each class is given equal importance in the final score, regardless of its size or prevalence in the dataset. ## results regarding the performance of the system in the shared task, it stood at rank 73 for task-a, rank 65 for task-b, and rank 55 for task-c. the best f1 scores obtained for task-a were as expected, as these statistical classifiers do not employ any deep learning or language techniques like large pretrained language models or deep neural networks. still, as we see in ## conclusion in conclusion, through this study, we were able to explore the effectiveness of some of the conventional machine learning classifiers in the task of online sexism detection. although they performed commendably in the binary classification task, it is clear that they are not cut out for multi-class classification tasks. random forests seem to be the best choice for simple binary classification of sexist texts out of conventional machine learning classifiers. in terms of future work, deep learning neural networks and pre-trained language models can be used to improve upon the score achieved by our system. moreover, instead of just removing emojis and hashtags from our post data, we should try to inculcate them in the feature engineering stage and extract features from them. for example, @xcite showed how emojis can be converted to vector embeddings for deep neural networks to make use of.