A Multi-Modal Dataset for Hate Speech Detection on Social Media: Case-study of Russia- U kraine Conflict (2022)

Abstract: This paper presents a new multi-modal dataset for identifying hateful content on social media, consisting of 5,680 text-image pairs collected from Twitter, labeled across two labels. Experimental analysis of the presented dataset has shown that understanding both modalities is essential for detecting these techniques. It is confirmed in our experiments with several state-of-the-art multi-modal models. In future work, we plan to extend the dataset in size. We further plan to develop new multi-modal models tailored explicitly to hate-speech detection, aiming for a deeper understanding of the text and image relation. It would also be interesting to perform experiments in a direction that explores what social entities the given hate speech tweet targets.

## introduction the internet has become an increasingly popular communication medium to express the views of people. people mostly express their opinions on various topics using social media, microblogging platforms, blogs, etc. with great internet penetration even in the rural parts of the world and ease of access to information in real-time, people mostly rely on social media platforms @xcite . at times of political events and tension in any region, the users of such platforms become more active than usual and post their thoughts and updates regarding the issues. during the expression of such opinions and ideas, there can be mixed emotions. some opinions lean towards supporting the people on the ground who are suffering in such political events whereas some opinions are about blaming each other, name-calling, exaggera-tion of information, etc @xcite . in political situations pertaining to invasion, the situation becomes even worse. social media sometimes get polarized into the ones supporting the invasion and the ones opposing the invasion. during such polarization, a lot of content can be found which uses extreme language, falsifies the information, and spreads hate. such content when directed towards certain people or groups of individuals (race, gender, nationality) with the intent to show anger and hate is called hate speech @xcite . while the legal definitions of hate speech vary from territory to territory, hate speech on the internet sphere is taken as hateful content on the internet that is directed toward certain individuals or groups of individuals. the cambridge dictionary defines hate speech as "public speech that expresses hate or encourages violence towards a person or group based on something such as race, religion, sex, or sexual orientation" @xcite . on february 24, 2022, russia started a full-scale invasion of ukraine by land, sea, and air @xcite . the world was again polarized into two, with one supporting the russian invasion and the other opposing it. many countries condemned the war, and sanctions were eventually imposed on russia. with the development of these events, social media started getting active. people started to express their opinions related to the humanitarian crisis and economic crisis that was caused due to the invasion. amid the healthy and respectful discourse and discussions, there was some hateful content targeted at various people (figure 1 ). hate speech can bring serious consequences to society. microblogging platforms and social media platforms put a lot of effort into managing the hateful content on their platforms. mostly, the platforms use human mediators for the mediation of posts related to hate speech. despite being an efficient method for regulating hate speech, it is not always possible for human mediators to flag 1 (a) tweet with no hate (b) tweet with hate speech figure 1 : examples of tweets with hate and no hate speech during russia-ukraine conflict the posts provided that the volume of the hateful content becomes extremely high in situations of political events like an invasion. thus, there has always been a need for an automated system to identify the contents related to hate speech. our contributions can be summarized as follows: ## related works despite hate speech detection being a hard task, much research is being done to address hate speech on the internet. with advancements in the field of deep learning, there is a multitude of problems that are being solved by deep learning @xcite . hate speech is one of the tasks that is being explored using deep learning techniques. most of the research on hateful content is focused on leveraging the information from the textual content. del vigna et al. (2017) curated a dataset of 17,567 comments from facebook posts and annotated for strong hate, weak hate, and no hate categories. the proposed long-short term memory (lstm) (hochreiter and schmidhuber, 1997) and svm models performed with an accuracy of 72.95% and 75.23% for hate and non-hate categories. similarly, the accuracy of 64.61% and 60.50% were reported across all three categories. similarly, gambäck and sikdar (2017) had proposed multiple cnn architectures in order to classify hate speech spanning across multiple classes, viz. racism, sexism, both (racism and sexism), and non-hate speech. the architecture with word2vec embeddings was able to achieve an f1-score of 0.7829. calderón et al. (2020) did a slightly different task of hate speech classification by curating a dataset (1977 tweets) of the hate speech directed towards the immigrants in spain and performing the task of topic modeling and meticulously studying linguistic cues of hate speech. apart from these, some research has been done on multi-modal hate speech detection. for instance, shang et al. ( 2021 ) proposed analogy-aware offensive meme detection (aomd) that was able to learn the implicit analogy from the multi-modal contents of the meme and detect the offensive analogy. the model that used resnet50 @xcite and glove-based lstm was able to achieve the accuracy of 69% and 72% for gab and reddit datasets. similarly, @xcite proposed a method that integrates the image captioning process into the memes detection process. the approach enhanced the cross-modality relationship and helped achieve auroc as high as 78.86. for their study, they used the famous dataset from hateful memes challenge @xcite . similarly, @xcite presented a method to identify propaganda techniques in memes by leveraging the multi-modal information and classifying them into 22 propaganda techniques. in recent days, the research relating to multimodal information has been growing @xcite . most microblogging sites allow users to post in various modalities like text, images, videos, etc. which add a dimension of research in addressing all the modalities. one modality often provides supplementary information to another modality which makes multimodal models more robust. ## conclusion and future work this paper presents a new multi-modal dataset for identifying hateful content on social media, consisting of 5,680 text-image pairs collected from twitter, labeled across two labels. experimental analysis of the presented dataset has shown that understanding both modalities is essential for detecting these techniques. it is confirmed in our experiments with several state-of-the-art multi-modal models. in future work, we plan to extend the dataset in size. we further plan to develop new multi-modal models tailored explicitly to hate-speech detection, aiming for a deeper understanding of the text and image relation. it would also be interesting to perform experiments in a direction that explores what social entities the given hate speech tweet targets. reproducibility: the dataset and resources for this work are available at our github repository foot_1 .