Lexical Squad@Multimodal Hate Speech Event Detection 2023: Multimodal Hate Speech Detection using Fused Ensemble Approach (2023)

Abstract: With a surge in the usage of social media postings to express opinions, emotions, and ideologies, there has been a significant shift towards the calibration of social media as a rapid medium of conveying viewpoints and outlooks over the globe. Concurrently, the emergence of a multitude of conflicts between two entities has given rise to a stream of social media content containing propaganda, hate speech, and inconsiderate views. Thus, the issue of monitoring social media postings is rising swiftly, attracting major attention from those willing to solve such problems. One such problem is Hate Speech detection. To mitigate this problem, we present our novel ensemble learning approach for detecting hate speech, by classifying text-embedded images into two labels, namely “Hate Speech” and “No Hate Speech” . We have incorporated state-of-art models including InceptionV3, BERT, and XLNet. Our proposed ensemble model yielded promising results with 75.21 and 74.96 as accuracy and F-1 score (respectively). We also present an empirical evaluation of the text-embedded images to elaborate on how well the model was able to predict and classify.

## introduction political events have been a perpetual part of governance to date and serve as a medium of expression for those involved directly or indirectly with the process. but at times, this medium of communication might turn out to be a source of unfortunate insensitive expressions, hate speeches, etc, through verbal forms, visual representations, and physically inconsiderate actions among others. in such cases, it becomes crucial to monitor political events and other potential contributors to the circulation of hate speech and insensitive content. according to legal publications, hate speech is defined as an expression that seeks to malign an individual for their immutable characteristics, such as their race, ethnicity, national origin, religion, gender, gender identity, sexual orientation, age or disability @xcite . hate speech detection is one of the most important aspects of event identification during political events like invasions @xcite . as is evident in today's scenario, the incorporation of multimodal data to meet incentives is highly prevalent and is a major concern for hate speech detection and analysis. in this paper, we elaborate on our submission for the shared task foot_0 , for multimodal hate speech detection through text-embedded images from the russia-ukraine war, which is a part of the bigger picture leading to a significantly demanding issue @xcite . multimodal content being advertised through physical spaces, social media, etc, is a mode of spreading hate speech and spiteful views being used extensively in the current scenario. a significant contributor to this phenomenon is the sharing of text-embedded images, representing the views of an individual or a group of individuals, either directly or indirectly. in accordance with this fact, we aim to categorically determine if a given text-embedded image conveys hate speech in any possible form or not. the rest of the paper is structured as follows: section 2 illustrates the existing work which has been carried out in this field of research; section 3 describes the dataset and task for our research study; section 4 elaborates our proposed model architecture including the individual blocks incorporated in the same; section 5 states the results obtained from this work along with its empirical analysis; section 6 provides a view of the future scope in this domain besides concluding the paper. ## literature review extensive work has been carried out to survey the extent of incorporating technology for hate speech detection. for instance, in @xcite , a survey has been carried out on the scope of hate speech detection using natural language processing. through this study, the features, terminologies, existing approaches, and techniques in this context have been highlighted. another similar research work @xcite , shows the comparison of the performance of three feature engineering techniques and eight machine learning algorithms on a publicly available dataset having three distinct classes. the results of this research work showed that the bigram features when used with the support vector machine algorithm best performed with 79% off overall accuracy. in another study @xcite , an experiment has been performed to emphasize the usage of deep learning for hate speech detection in tweets. a twitter dataset containing relevant tweets has been used to classify them as being racist, sexist, or neither. as per the results obtained in this study, the deep learning methods outperform stateof-the-art char/word n-gram methods by ∼18 f1 points. in recent times, multiple attempts have been made to deal with the concern of intelligently determining the spread of hate speech and related expressions through multimodal data. for instance, as a part of the research study @xcite , it was attempted to jointly analyze textual and visual information for hate speech detection, using a large-scale dataset from twitter, mmhs150k. the researchers associated with this study have compared the implementation of models working on multimodal data with those on unimodal data. another research work @xcite , featuring the detection of hate speech in multimodal memes, forms its basis for categorizing a meme as hateful or non-hateful. as a part of this, the visual modality using object detection and image captioning models to fetch the "actual caption" has been explored and combined with the multi-modal representation to perform binary classification. along with this, an effort has been made to enhance the predictions using sentiment analysis. another instance of research work @xcite , has been carried out on a dataset containing more than 10000 examples of multimodal content, wherein visualbert, which is meant to be the "bert of vision and language" was trained multimodally on images and captions and was augmented with ensemble learning. ## dataset and task as a part of the 6th workshop on challenges and applications of automated extraction of sociopolitical events from text (case @ ranlp 2023), the sub-task a for this research experimentation is to identify whether the given textembedded image contains hate speech or not @xcite . the dataset @xcite provided for this task consists of around 4700 text-embedded images, having annotations for the prevalence of hate speech. as a two-way classification task, the two classes in the given dataset correspond to "hate speech" and "no hate speech", with 2665 and 2058 samples corresponding to the respective classes. the training data consists of 1942 and 1658 samples against the "hate speech" and "no hate speech" labels (respectively). concurrently, the evaluation and testing data consists of 443 random samples each. all the images have a unique identifier called "index". in the training data, the classes are wellbalanced, implying the occurrence of 50 textembedded images each against the two labels, that is, "hate speech" as well as "no hate speech". ## model architecture this section describes the proposed model architecture for classifying text-embedded images as "hate speech" or "no hate speech". as depicted in figure 1 , we hereby propose an ensemble approach for this binary classification problem. due to the multimodal nature of the data, it is necessary to extract both visual and textual features from the provided content containing the same. to comprehend the context of a textembedded image, it is necessary to map the textual context to its visual context. so as to have both of these contexts, we propose an ensemble model that extracts both of these characteristics from an image. we have incorporated respective models based on convolutional neural networks (cnn), and pretrained transformer models, which provided good results on the given dataset. inceptionv3 optimizes the neural network for better adaptation as it has a 2 deeper network compared to its predecessors and uses auxiliary classifiers as regularizers. eventually, it serves as an enhanced option for visual comprehension required for this task. since transformer models are based on the mechanism of selfattention and differentially weigh the significance of each part of the input textual data, they serve as an ideal option for the textual comprehension of this task. our model is comprised of three backbones, one of which (inceptionv3) extracts visual information while the other two (bert and xlnet) extract textual information. ## results and discussion we have mentioned the results obtained on the validation data in the variation of accuracy level with respect to the number of epochs taken for model training has been depicted in figure 2 . along with this, the variation of the loss function with respect to the number of epochs taken for model training has been depicted in figure 3 . ## conclusion and future scope in this paper, we present our system paper submission for lexical squad@multimodal hate speech event detection 2023. we aim to classify textembedded images, indicating whether they contain hate speech or not. the proposed system is an ensemble learning model with fine-tuned in-ceptionv3, bert, and xlnet serving as the individual blocks of the proposed model. given textembedded images and their respective extracted text through the ocr model, the submitted model classifies each image instance into one of the two labels: "hate speech" and "no hate speech". the system performs quite well to accomplish the desired task with an accuracy of 75.21%. the proposed system can be incorporated for further applications including recommendation systems, personalized content viewing, etc. along with, it can find usage in further research studies centered on the overlooking field of interest. in the future, we intend to work on a multitask learning framework to handle social media postings related to other concerns pertaining to sentiment analysis, apart from hate speech detection. we also aim to develop models for multi-lingual postings featuring similar scenarios.