Attentive Fusion: A Transformer-based Approach to Multimodal Hate Speech Detection (2023)

Abstract: With the recent surge and exponential growth of social media usage, scrutinizing social media content for the presence of any hateful content is of utmost importance. Researchers have been diligently working since the past decade on distinguishing between content that promotes hatred and content that does not. Traditionally, the main focus has been on analyzing textual content. However, recent research attempts have also commenced into the identification of audio-based content. Nevertheless, studies have shown that relying solely on audio or text-based content may be ineffective, as recent upsurge indicates that individuals often employ sarcasm in their speech and writing. To overcome these challenges, we present an approach to identify whether a speech promotes hate or not utilizing both audio and textual representations. Our methodology is based on the Transformer framework that incorporates both audio and text sampling, accompanied by our very own layer called “Attentive Fusion”. The results of our study surpassed previous stateof-the-art techniques, achieving an impressive macro F1 score of 0.927 on the Test Set.

## introduction in recent years, the explosive growth of digital communication platforms has facilitated unprecedented levels of information exchange, enabling individuals from diverse backgrounds to interact and share ideas. however, this surge in online interactions has also led to the emergence of a concerning issue: the increase of hate speech @xcite . hate speech, characterized by offensive, discriminatory, or derogatory language targeting individuals or groups based on race, ethnicity, religion, gender, or sexual orientation, poses significant challenges to maintaining a safe and inclusive online environment @xcite . traditional methods of hate speech detection primarily focused on analyzing text-based content, leveraging natural language processing (nlp) techniques to identify offensive language patterns. while these approaches have yielded some success, they often struggle to capture the nuanced nature of speech, as the exact text might be interpreted differently when considering context, tone, and intent @xcite . to address these limitations, researchers are turning to a more holistic approach combining both text and speech modalities to enhance the accuracy and robustness of hate speech detection systems @xcite . this multidimensional approach referred to as multimodal hate speech detection, leverages not only the textual content of messages but also the acoustic cues and prosodic features present in speech. by simultaneously analyzing both text and speech-based characteristics, this approach aims to capture a more comprehensive representation of communication, considering not only the words used, but also the emotional nuances conveyed through speech intonation, pitch, and rhythm. figure 1 illustrates two examples each for "hate" and "not hate" using the multimodality. in the two cues shown (figure 1 ), represents the speech cue and represents the text cue. in this paper, we investigate multimodal hate speech detection exploring the synergies between text and speech for identifying hate speech instances. we examine the challenges posed by hate speech in the digital age, the limitations of traditional text-based detection methods, and the potential advantages of integrating speech data into the detection process. by leveraging insights from various disciplines such as nlp, audio signal processing, and machine learning, multimodal approaches hold promise in achieving higher detection accuracy and reducing false positives, ultimately fostering safer and more inclusive online environments. our methodology sets itself apart from other state-of-the-art (sota) methodologies in the subsequent manner: ## results in cases of unfrozen wav2vec-2.0. the differences are very nominal as wav2vec-2.0 provides an embedding knowledge of each token of the audio specimen. in contrast, we didn't use any embedding knowledge of speech tokens, which will be experimented with in our upcoming works. the researcher has shown two wav2vec-2.0 among which is one wav2vec-2.0 (9 layer). in this system, the researcher took the representation token from the 9 th layer. ## conclusion in this work, we proposed a framework that can classify whether a speech promotes hatred or not. for the speech feature extraction, we used a log mel spectrogram feature extraction technique. our framework consists of speech sampling and text sampling followed by two separate transformer frameworks that serve different efforts. each transformer framework is followed by an lstm layer, the output of which is fed to our proposed layer, and further sent to linear layer for classification. the whole framework was able to outperform the existing benchmark macro f1 score. the only limitation of our approach is it is limited to the english language. in future work, we would like to test its robustness for other languages.