NLP _ CHRISTINE at S em E val-2023 Task 10: Utilizing Transformer Contextual Representations and Ensemble Learning for Sexism Detection on Social Media Texts (2023)

Abstract: The paper describes the SemEval-2023 Task 10: “Explainable Detection of Online Sexism (EDOS)”, which investigates the detection of sexism on two social media sites, Gab and Reddit, by encouraging the development of machine learning models that perform binary and multi-class classification on English texts. The EDOS Task consisted of three hierarchical sub-tasks: binary sexism detection in sub-task A, category of sexism detection in sub-task B and fine-grained vector of sexism detection in sub-task C. My participation in EDOS comprised fine-tuning of different layer representations of Transformer-based pre-trained language models, namely BERT, AlBERT and RoBERTa, and ensemble learning via majority voting of the best performing models. Despite the low rank mainly due to a submission error, the system employed the largest version of the aforementioned Transformer models (BERT-Large, ALBERT-XXLarge-v1, ALBERT-XXLarge-v2, RoBERTa-Large), experimented with their multi-layer structure and aggregated their predictions so as to get the final result. My predictions on the test sets achieved 82.88%, 63.77% and 43.08% Macro-F1 score in sub-tasks A, B and C respectively.

## introduction the task and its importance sexism emerged as a term from the second-wave feminism of the 1960s through the 1980s to comprise discrimination, prejudice and stereotyping based on someone's sex, typically targeting women and girls @xcite . even though our way of life has improved since then, it appears that sexism persists. on top of that, considering the immense influence of social networks on our lives, sexism is expressed online through text, image, video and sound on a daily basis. it has become evident that the anonymity and invisibility of the online environment foster the online disinhibition effect, meaning the tendency to exhibit negative behavior online rather than in person without taking into account the consequences @xcite , @xcite . in recent years, the amount of sexist content in social media ranging from cases of discrimination, abuse, stalking, bullying, impersonation, defamation, verbal and sexual harassment, non-consensual pornography, misogyny to cases of hate speech mainly against female individuals or groups has been on the rise. this has serious repercussions for the the mental health, the social status and the online experience of the targets, not to mention the negative impact on the community. to illustrate, the majority of women, who have experienced or witnessed sexist behavior online, state that they are discouraged from pursuing their political careers and upholding their rights, while they are led to self-censor and limit their presence on social media @xcite . what is worse, acts of violence and abuse online can disrupt the targets' physical safety as well as their families' safety @xcite . online content moderators working in social media companies are responsible for mitigating such type of content according to each company's internal policies. nevertheless, manual moderation of social media content is a time-consuming and laborious process and may often depend on the moderator's judgement @xcite . online content moderators may also suffer from various mental health problems, for instance ptsd and depression, owing to their continuous exposure to offensive, violent and pornographic content @xcite . the challenges and severe ramifications of manual social media moderation along with the increase of social media content have revealed that the need to automatically detect and eliminate such content is imperative. task 10 of semeval-2023 competition named explainable detection of online sexism (edos) addressed this particular need by urging participants to develop machine learning models, which are able to detect sexist posts from two social networks, gab and reddit, by means of performing binary and finegrained multi-class classification on english texts @xcite . the edos shared task comprised three hierarchical sub-tasks: sub-task a was focused on identifying whether a post is sexist or not. sub-task b was devoted to recognizing four categories of sexism. sub-task c aimed at detecting sexism based on eleven fine-grained vectors of sexism. sub-task a contained both sexist and nonsexist texts, whereas the other two sub-tasks were exclusively focused on the sexist texts from the first sub-task. ## system overview transformer-based pre-trained language models such as bert @xcite and its variations, like albert @xcite and roberta, @xcite have attracted a lot of attention since 2018, as they have managed to achieve state-of-the-art results in various nlp tasks. they have been widely used for fine-tuning downstream tasks by simply adding an additional taskspecific output layer. during fine-tuning, the output of the transformer encoder's last layer, which is the contextualized representation of the input text, is passed to the task-specific layer. however, utilizing only the last layer's output may limit the power of pre-trained representation @xcite . considering the multi-layer and deep structure of the pre-trained transformer language models, different layers are able to capture various levels of contextualized representations (embeddings) of the input text. therefore, they encode very different kinds of linguistic information, for instance surface, syntactic and semantic features in the lower, middle and higher layers respectively @xcite . the authors of bert @xcite , followed a feature-based approach by extracting the contextual embeddings from different layers of bert-base and providing them as input to a bil-stm for a named entity recognition task. inspired by this feature-based approach, i experimented with four machine learning model architectures, which show how to utilize information from different pretrained transformer layers for sexism detection in social media. ## experimental setup environment setup the proposed system was implemented in python programming language and the code was written on a google colaboratory (colab) pro notebook. the experiments were conducted using the pytorch library and nvidia a100-sxm4-40gb gpu memory. data used although both labelled and auxiliary unlabelled data were provided for training, only the labelled data were utilized in each sub-task. due to the fact that the labelled data were already split by the task organizers into training, development and test sets, no further splitting was considered necessary. preprocessing steps a function including a number of regular expressions and other functions was developed to apply a series of preprocessing steps to the text of the training, development and test sets. the utf-8 byte order mark (bom), which identifies a file as being encoded in utf-8, was deleted and the data were encoded using the beautiful-soup library. 2 the url links and usernames, which were already normalized and set as the anonymous [url] and [user] by the task organizers, were lowercased. the emojis were converted to their textual representation @xcite . 3 the &amp; and & were replaced with and. the ascii encoding apostrophe was replaced with the utf-8 encoding apostrophe. the presence of certain punctuation marks (full stops, exclamation marks, question marks) was limited up to 3 consecutive characters. the consecutive non-ascii characters were replaced by whitespace and all extra whitespace was deleted. the ekphrasis library was utilized for hashtag segmentation, correction of spelling, elongated words, unpacking of contracted words as well as tokenization and lowercasing of all words @xcite . 4 the hashtags and uppercase words were annotated on both sides with the special @xmath0, </@xmath1. hyperparameter tuning firstly, the required input including the input ids and the attention mask in pytorch tensors was created for the pre-trained transformer models. the special tokens [sep] and [cls] were added and the sequences were padded according to the adjusted maximum sequence length of each training set. apart from the randomsampler which was used in the train dataloader of sub-tasks a and b, the imbalanced dataset sampler was utilized for the train dataloaders in sub-tasks b and c because it re-balances the class distribution when sampling from an imbalanced dataset, calculates the sampling weights automatically and avoids creating a new balanced dataset. 56 for all development sets, the sequen-tialsampler was utilized. for all classification sub-tasks, dropout and early stopping patience were used so as to prevent model from overfitting. the binary cross-entropy loss with logits (bcewith-logitsloss) was utilized for the binary classification task (sub-task a), while the focal loss with gamma 2.0 and weights was employed for the multiclass classification of sub-tasks b and c in order to deal with the issue of class imbalance @xcite . 78 the weights were calculated for each class so that the models could treat all classes equally during training. focal loss required the softmax activation function after the final classifier layer in every model architecture. the adamw was selected as the optimizer. an overview of all model hyperparameters is presented in ## results despite the experimentation to deal with class imbalance, the model fine-tuning and ensemble learning with a view to achieving high system performance, it became evident that the system did not perform as adequately as expected due to the low final rank. the models that achieved the highest macro-f1 and mcc scores in the development sets were involved in the ensemble and are demonstrated in ## limitations class imbalance, especially in sub-tasks b and c, was a major issue during participation in edos. the difficulty of the system to distinguish between categories of sexism and between vectors of sexism was apparent. this could be mainly due to the class imbalance or other features of the texts. it is also difficult for a system not only to to detect whether a text is sexist, but also explain why it is sexist based on a plain text without any additional information of the user and the target. ## conclusion the proposed system for semeval-2023 task 10 comprised fine-tuning of different contextual representations of transformer-based pre-trained language models and majority voting ensemble of the best performing models with a view to detect sexist texts from social media and identify the category and sub-category of sexism. unfortunately, due to a submission error, the macro-f1 score results from the test set submissions proved to be much lower than expected compared to the development