SINAI at S em E val-2023 Task 10: Leveraging Emotions, Sentiments, and Irony Knowledge for Explainable Detection of Online Sexism (2023)

Abstract: This paper describes the participation of SINAI research team in the Explainable Detection of Online Sexism (EDOS) Shared Task at SemEval 2023. Specifically, we participate in subtask A (binary sexism detection), subtask B (category of sexism), and subtask C (fine-grained vector of sexism). For the three subtasks, we propose a system that integrates information related to emotions, sentiments, and irony in order to check whether these features help detect sexism content. Our team ranked 46th in subtask A, 37th in subtask B, and 29th in subtask C, achieving 0.8245, 0.6043, and 0.4376 of macro f1-score, respectively, among the participants.

## introduction sexism is defined by the oxford dictionary as prejudice, stereotyping, or discrimination, typically against women, on the basis of sex 1 . sexism is in our daily life when people underestimate the opinions expressed by women, in conversations both oral or written that contains sayings and fixed expressions. nowadays, with social media platforms, sexist comments are very frequent and they are widespread quickly promoting more sexist reactions. furthermore, sexism can be expressed in different forms, making detecting sexist comments a difficult task. due to this reason, several shared tasks and many academic events are organized by the scientific community. for example, hateeval @xcite is based on hate speech detection against women and immigrants. other tasks such as evalita @xcite or the task on automatic misogyny identification (ami) @xcite are focused on misogyny detection. however, sexism is not only misogyny (violence against women) which detection consists of a binary classification task, but also contains different 1 https://www.oed.com/ types of sexism depending on how it is expressed, making the detection of sexism a multi-class classification task. for this reason, there are shared tasks such as exist @xcite that promote the development of tools to detect sexism and categorize it according to the facet of the women that are undermined or the edos shared task. this paper describes the participation of sinai in the explainable detection of online sexism (edos) shared task @xcite aimed at the identification of sexist language and its categories. the main purpose of this shared task is to promote the development of different english language models to detect sexism and the explainability of this phenomenon. for this purpose, the organizers proposed three different subtasks. subtask a: binary sexism detection is related to identifying if a comment is sexist or not. once a post has been classified as sexist, the subtask b: category of sexism consists of classifying the sexist posts into four different classes, and subtask c: fine-grained vector of sexism aims to predict one class between eleven different classes that are more specific than the subtask b. our team sinai has participated in the three subtasks. our proposal for addressing these subtasks is the integration of external knowledge into classifiers to more accurately predict the specific task. for example, in hate speech detection some previous works have incorporated knowledge from sentiment and emotion analysis which are related to the expression of offensiveness. for instance, @xcite @xcite proposed a novel approach that uses a multi-task learning paradigm to combine different phenomena that are inextricably related to the expression of offensive language such as sentiments, emotions, target, irony, sarcasm, and constructiveness, among others. pérez et al. (2022) evaluated the impact of incorporating contextual information in hate speech related to the news posted on so-cial media. in misogyny detection, @xcite introduce an approach based on aesthetic features captured by character n-grams, sentiment information, and a set of lexicons built by analyzing misogynistic tweets. the rest of the paper is structured as follows: in section 2 we describe the main strategy used to develop the system for the shared task. the data and the experimental methodology are described in section 3. the evaluation results from the development and test phases are shown in section 4. finally, we conclude with a discussion in section 5. ## system overview in this section, we describe the system we develop for the explainable detection of online sexism shared task at semeval 2023. we propose a system that incorporates some extra-linguistic information related to emotions, sentiments, irony, or a combination of these features, such as emotions + sentiments or emotions + sentiments + irony, of the posts in the dataset, provided by the organizers of the task. the reason for incorporating the information of polarity, emotion, and irony to detect sexism is that sexist comments are often emotional and express a negative polarity and emotion towards the recipient. in addition, such comments are usually expressed using literary figures such as irony or mockery, to mask the sexist content. the architecture of the system we develop can be seen in figure 1 . the system incorporate information associated with different concepts such as emotions, sentiments, and irony. these concepts are associated with distinct tasks to extract this information, such as emotion classification which recognizes the emotions expressed within a text, sentiment classification which analyzes the polarity of the text as positive or negative, and irony detection which is based on the identification of whether a text is ironic or not. in the system we propose, to extract this information from the posts in the dataset, we use models pre-trained on the specific task related to the information that we are interested in. these models give us the scores of the texts in the different categories in which the models classify. for example, the emotion classifier gives us the probability that the text is related to the emotion of anger, disgust, and joy, among others. then, the first step to developing our system is to obtain the different scores of the text in the different categories of emotions, polarity, and irony for each post in the dataset. secondly, we provide the posts to the tokenizer and the model. the model returns the [cls] token of the last hidden state and we concatenate this token with the emotion, polarity, and irony scores. the concatenation between the [cls] token and the scores are then passed to the head classifier (feed-forward network), which will give us a prediction about the classification of the data. ## results in this section, we present the results obtained by the systems developed as part of our participation in semeval 2023 task 10. to evaluate our systems, we use the official competition metrics given by the organizers. specifically, the macro f1-score for all of the subtasks. furthermore, we add two additional metrics, the macro precision, and the macro recall scores. the experiments are conducted in two phases, the model selection phase and the evaluation phase, which are explained in the following subsections. ## conclusion this paper presents the participation of sinai research group in the explainable detection of online sexism shared task at semeval 2023. in all of the subtasks, we explore how different finetuned strategies of the roberta model perform by adding extra-linguistic features (emotions, sentiment, irony). for the first subtask, we have seen that the incorporation of emotions and sentiments tends to help detect sexist content. in the second subtask, on the contrary, we observed that the baseline model roberta helps to detect the different categories of sexism, however, the other strategies have not outperformed this model, as we expected in the pre-evaluation phase. finally, in subtask c, we noticed that the incorporation of the different features (emotions, sentiments, irony) has contributed significantly to the detection of the different fine-grained categories of sexism. we conclude that, in general, the incorporation of extra-linguistic information helps the models to conduct the sexist tasks, especially when the number of classes increases, as we have observed in subtask c. in future work, we plan to explore other ways to add extra-linguistic information into the pre-trained language systems. in addition, we would like to analyze the impact of the model architecture while incorporating features.