Ometeotl@Multimodal Hate Speech Event Detection 2023: Hate Speech and Text-Image Correlation Detection in Real Life Memes Using Pre-Trained BERT Models over Text (2023)

Abstract: Hate speech detection during times of war has become crucial in recent years, as evident with the recent Russo-Ukrainian war. In this paper, we present our submissions for both subtasks from the Multimodal Hate Speech Event Detec- tion contest at CASE 2023, RANLP 2023. We used pre-trained BERT models in both submis- sion, achieving a F1 score of 0.809 in subtask A, and F1 score of 0.567 in subtask B. In the first subtask, our result was not far from the first place, which led us to realize the lower impact of images in real-life memes about feel- ings, when compared with the impact of text. However, we observed a higher importance of images when targeting hateful feelings towards a specific entity. The source code to reproduce our results can be found at the github repository https://github.com/JesusASmx/OmeteotlAtCASE2023

## introduction in recent decades, online platforms have gained increasing relevance in the worldwide sociopolitical scenario, to the extent that they have become significant representations of the so-called soft power @xcite . this growing importance has also led to an alarming spread of offensive, discriminatory, and harmful content, particularly during periods of significant political changes such as elections @xcite or wars @xcite . detecting hate speech, both in text and images, is crucial in order to mitigate its negative impact on digital platforms and safeguarding individuals from its harmful effects @xcite . as an example of this need, in 2022, social networks witnessed a surge in activity following the outbreak of the russo-ukrainian war; numerous content, full of hate speech from both sides, went viral, and the need for a specific focus to that particular conflict became evident. for this reason, the multimodal hate speech event detection contest was proposed during the case 2023 workshop @xcite to tackle this problem with a dataset of manually annotated text-image memes @xcite . this shared task was divided into two subtasks a and b. in subtask a, participants were required to determine whether a meme related to the russo-ukrainian war constituted hate speech or not. in subtask b, participants were tasked with identifying the target of a hate speech meme, classifying it as directed against an individual (such as volodymyr zelensky or vladimir putin), an organization (such as the ukrainian army), or a community (such as the russian speakers in the donbass region). in this paper, we present out participation in both subtasks, under the name of team ometeotl. our proposal consists on a fine-tunning of the pre-trained bert model @xcite , trained solely on the text extracted from the memes, without incorporating any image feature. surprisingly, those experiments outperformed models that considered image features, such as resnet152, and even multimodal ensemble learning approaches, such as resnet152+bert. these approaches achieved the sixth position in subtask a, with an f 1 score of 0.809, and the seventh position in subtask b, with an f 1 score of 0.567. the structure of the paper is as follows: in section 2, we describe the updated research on automatic hate speech detection. in section 3, we describe the database. in section 4, we detail the methodology used. in section 5, we show the results of our experiments. in section 5, we discuss the results. finally, in section 6 we present the conclusions. ## related work hate speech detection in social media are one of the most prominent classification tasks in recent years @xcite . one of the earliest known approaches is the general inquirer @xcite , an ibm system developed in 1961 that enabled content analysis for behavioral sciences. it focused on pattern detection in text to categorize words based on their semantics, particularly positive or negative sentiments. in 1997, a more targeted approach was proposed with the system smokey @xcite , designed to detect abusive messages. smokey utilized a rule-based approach to identify offensive language and contexts. from there, several new approaches were proposed to address the task and its variations. in @xcite , the authors proposed a lexicon-based approach for hate speech detection, starting from the hypothesis that the task can be related with word sense disambiguation. however, such approach was vulnerable in front of incomplete datasets, as they discovered when every method learnt jew as a inherent word for antisemitism speech. in order to deal to this sort of datasets, several methods and further methodologies has been developed: one of the most recent machine learning techniques who had brought promising results are the transformers @xcite , including bert models @xcite . in a nutshell, bert models are is a family of language models composed of transformer encoder layers. such architectures has been successfully used in transphobic-homophobic speech detection, as can be seen in the lt-edi-acl2022 homophobia/transphobia speech detection contest in english, tamil and tamil-english @xcite . team sammaan (upadhyay et al., 2022) employed ensemble transformers and obtained the second place in english; team nozza @xcite obtained the third position in english and used ensemble learning over fine-tunned models of bert @xcite , roberta @xcite , and hatebert @xcite . another hate speech detection contest in which transformers were used was in the iberlef2023 shared task of homo-mex: hate speech detection towards the mexican spanish-speaking lgbt+ @xcite . contrary to the previous contest, in which only the first places used transformers (the last place used tf-idf with traditional classifiers such as support vector machine @xcite ), here team lidoma, the last place of the competition, employed a bert model @xcite . however, in their paper, the authors explained how the lack of a preprocessing highly affected the efficiency of the attention mechanisms. to dive further, in this work we find a counterexample to their hypothesis in the shared task a, where preprocessing actually brought worse results. all the related works discussed so far have focused solely on text-based hate speech detection. this is because text has historically been the most prevalent format for hate speech across the internet, especially during the early days of the worldwide web. however, it is crucial to recognize that there exists a wealth of historical data on hate speech in images, such as visual propaganda @xcite , extensive datasets from the second world war @xcite and the cold war @xcite . nevertheless, it is worth noting that these works were handcrafted by artists, hence impossible to get mass-produced during the early stages of the worldwide web, unlike the solely text-based propaganda. this landscape has since changed with the advent of text-image memes, which are pre-designed images that can have accompanying text, making possible the massproduction of visual propaganda and hence attracting the attention of researchers all across the world. for instance, meta ai initiated the paid contest titled "hateful memes challenge and dataset for research on harmful multimodal contest" @xcite , in which they provided a dataset of memes and the task to detect hate speech on them. one of the most interesting aspects of the challenge was that the dataset considered the significant phenomena of text-image interaction through phrasesense disambiguation. for instance, a meme featuring the text i love the way you smell today could be classified as hate speech if accompanied by an image of a skunk (mephitidae), but as non-hate speech if accompanied by a picture of a rose. another relatable example of multimodal hate speech text-image detection is @xcite . in this work, the authors combined natural language processing techniques with computer vision models to analyze both text and images in greek social media. for text processing they fine-tuned a pre-trained bert model @xcite . for image processing, the authors finetuned a pre-trained @xcite in the imagenet dataset @xcite . their best result was an f1 score of 0.947. in @xcite , the authors proposed a multimodal hate speech detection approach that uses cross-domain knowledge transfer to improve hate speech detection accuracy. to address the semantic inconsistency between hate speech and sarcasm, the authors combined the contrastive attention mechanism with representational dissociation to design a semantic adaptive module. in addition, they applied curricular learning to accelerate the training process. experimental results showed that the proposed approach outperformed existing multimodal hate speech detection methods in terms of accuracy and f1-score on two public datasets: the facebook hateful memes dataset from the meta ai's contest, mentioned before, and the twitter sarcasm detection dataset @xcite . ## dataset the dataset for both subtasks consists in 6, 913 text-image memes concerning the ukraine-russia conflict. these samples were collected from social media platforms such as twitter, reddit and facebook with keywords for specialized searches. the labeling was done manually, and they used cohen's kappa statistical measure @xcite to assess the agreement between two or more annotators which ranges from -1 to 1, where the value 1 indicates perfect agreement, 0 indicates casual agreement and -1 total disagreement @xcite . ## methodology the first step was to encode each labeling into a numerical value. in the case of subtask a, 0 was used to represent no hate speech and 1 to indicate hate speech. in subtask b, we utilized 0 for hateful messages towards a community, 1 for individual, and 2 for organization. all these labelings were chosen following the indications of the organizers. the next step involved an optional preprocessing outside the bert processing of the text. it consisted of a function that removed special characters, converted to lowercase, and removed the stopwords using the spacy python library 2 . the main idea behind this function was to enhance the efficiency of the attention mechanisms as mentioned in @xcite . however, as anticipated in section 2, it only worked for subtask b. regarding the model specifications, we utilized the bertforsequenceclassification model with the bert-base-uncased architecture, which was pretrained on the english corpus. the employed parameters for the preparation of the data were: the input tokens and attention masks were concatenated into separate tensors using the torch.cat and torch.tensor libraries. the parameter for training the bert-baseuncased model were: the system infraestructure consisted in a cpu with a amd rysen 2 5600x processor with six kernels, along with 46gb of ram. with this system, the run for the subtask a spent around ten hours while the run for the subtask b spent around eight. ## results and discussion in subtask a, we did not utilized preprocessing and achieved an f1 score of 0.809. the first-place score was 0.856, which is only 0.047 points higher than ours. this difference is relatively low, especially when considering that we did not employ image features in our predictions. see in subtask b, we employed preprocessing and achieved an f1 score of 0.567. this time, the difference with the first-place score was more substantial (of 0.195 points), leading us to hypothesize that visual features may have a stronger correlation when determining the target of a hateful meme. the leaderboard of this subtask can be found in ## conclusions in this paper, we presented our approach to address both subtasks from the multimodal hate speech event detection at case 2023, which consists in a) detect hate speech in text-image memes spread during the russo-ukranian war, and b) given a hateful meme about that conflict, determine if the target is a community, an individual or an organization. we utilized text-based transformers, specifically fine-tunned pre-trained bert models, and achieved high results in subtask a using only text features. our methodology involved the numerical encoding of the labels, and a preprocessing step for subtask b consisting in lowercase conversion and the removal of stopwords and special characters. afterward, we conducted a four-epoch training of the fine-tunned pre-trained bert model bert-baseuncased. we discovered that visual features played a more significant role in determining the target of hate speech rather than determining whether the meme itself was hateful or not, at least in this particular database. as a result, further research and analysis are needed to explore this phenomenon comprehensively. exploring other datasets could provide valuable insights into the dynamics between visual features and hate speech, offering a more comprehensive understanding of the varying impact these elements have across different contexts and social settings. such investigations can shed light on the broader implications of visual cues and how they interact with textual content in influencing the perception and spread of hateful memes. tigación y posgrado of the instituto politécnico nacional, mexico. the authors thank the cona-cyt for the computing resources brought to them through the plataforma de aprendizaje profundo para tecnologías del lenguaje of the laboratorio de supercómputo of the inaoe, mexico and acknowledge the support of microsoft through the microsoft latin america phd award.