Emojis Trash or Treasure: Utilizing Emoji to Aid Hate Speech Detection (2024)

Abstract: In this study, we delve into the fascinating realm of emojis and their impact on identifying hate speech in both Bengali and English languages. Through extensive exploration of various techniques, particularly the integration of Multilingual BERT (MBert) and Emoji2Vec embeddings, we strive to shed light on the immense potential of emojis in this detection process. By meticulously comparing these advanced models with conventional approaches, we uncover the intricate contextual cues that emojis bring to the table. Ultimately, our discoveries underscore the invaluable role of emojis in hate speech detection, thereby providing valuable insights for the creation of resilient and context-aware systems to combat online toxicity. Our findings showcase the potential of emojis as valuable assets rather than mere embellishments in the realm of hate speech detection. By leveraging the combined strength of MBert and Emoji2Vec, our models exhibit enhanced capabilities in deciphering the emotional subtleties often intertwined with hate speech expressions.

## introduction in the rapidly evolving landscape of online communication, social media platforms have emerged as crucial channels for individuals to share their ideas and perspectives. however, the unbridled nature of this digital freedom has occasionally given rise to a concerning issue: the rampant spread of hate speech. this perilous outcome has the potential to inflict profound harm and sow discord among communities. detecting and effectively addressing hate speech in such a diverse and globalized space has become an urgent digital-age imperative. this write-up sets out to present a holistic and innovative solution that harnesses the power of advanced natural language processing (nlp) techniques to counter the challenge of hate speech in the bengali language. by integrating the capabilities of multilingual bert and emoji2vec models, we hope to achieve a multi-dimensional approach that goes beyond superficial analysis. through an in-depth examination of the data ecosystem, we uncover the nuances that surround hate speech. the incorporation of emoji embeddings, a rich source of emotional context in digital conversations, is expected to provide valuable cues for detecting subtle instances of hate speech that might evade traditional approaches. central to our exploration is the multilingual bert model, a pioneering development in nlp. the model's architecture, rooted in pre-training and fine-tuning phases, equips it with the capacity to understand and contextualize the intricacies of language. we will dissect this architecture to reveal how it enables the model to discern the fine line between genuine expression and hate-driven content, especially when the medium is bengali. the culmination of this paper is the meticulous analysis of the results obtained through the proposed approaches. we will gauge the efficacy of our method in accurately identifying hate speech and dissect the cases in which the model excelled or faced challenges. ultimately, this comprehensive approach not only detects hate speech but also paves the way for more inclusive and respectful digital interactions, transforming social media platforms into safer and more conducive spaces for open dialogue and shared understanding. ## related work in recent years, hate speech detection has received significant attention, with the majority of work dedicated to monolingual hate speech detection. these skewed in attention to monolingual language has opened ample room to carry research forward further in other languages too. the task defined in @xcite presented a benchmark for chinese offensive language analysis named cold. they provided a dataset called coldataset and a system, namely coldetector adopting trans-former based bert architecture. they showed that the cold benchmark contributes to chinese offensive language detection which is challenging for existing resources. they then deploy the colde-tector and conduct extensive analysis on popular chinese pre-trained language models. they first assess the offensiveness of existing generative models and show that these models necessarily expose varied degrees of offensive concerns. furthermore, they analyzed the aspects that influence the offensive generations, and concluded that anti-bias contents and terms referring to certain groups or exhibiting unfavorable attitudes activate offensive outputs faster. the task of @xcite ) presented an automated system to detect offensive language and racism in french. this kind of task has been a pressing need as the surge in internet usage has led to a notable rise in toxic online content, posing a significant issue in recent years. the system tested on the novel french twitter racist (ftr) speech dataset, for racist speech detection. the experiments were carried out to achieve three primary objectives, viz. (1). the assessment and comparative analysis of diverse models and textual representations within the french language context; (2). the execution of cross-lingual experiments designed to ascertain the efficacy of transfer learning employing the proposed methodology, particularly in scenarios characterized by limited linguistic resources; and (3). the conduct of multilingual experiments investigating the potential enhancement of classification accuracy through the incorporation of an additional language (e.g., english) into the training dataset. the investigation and analysis were conducted as part of information 2022, with a specific focus on addressing the aforementioned research goals. they evaluated the dataset with the use of alternative text representations and supervised learning approaches for racist text detection in social media. they showed that extending the ftr dataset with extra french data containing hate speech is beneficial because it leads to better scores for practically all models, like random forest, logistic regression, extreme gradient boosting, bert transformer, and text representations. they also run cross-lingual and multilingual experiments for evaluating a theory about transfer learning. the study conducted by @xcite proposed various deep learningbased approaches (like lstm, cnn and bertbased models) on hindi and marathi datasets re-leased as a part of hasoc-2021 @xcite . they performed binary and multi-class classification tasks. for the binary classification task, they employed cnn and lstm incorporating random and fasttext embeddings. among these, the lstm + non-trainable fasttext configuration exhibited optimal performance for marathi, while for hindi, bilstm with non-trainable fast-text demonstrated superior efficacy. additionally, they explored transformer-based bert models such as indicbert, mbert, and robertabase for marathi, as well as roberta base and neural space bert for hindi. indicbert outperformed other models in the case of marathi, while roberta yielded the best results for hindi. the same roberta model was utilized in a hierarchical approach. the findings underscore the superior performance of transformer-based models in binary tasks, although even fundamental models demonstrated competitive performance. in the context of hindi task 2, the study establishes that the cnn + non-trainable fasttext model slightly outperforms the roberta hindi model. the task of @xcite presented a brief survey of hate speech detection in asian languages. the survey's primary goal is to promote the development of an automated hate speech detection system tailored specifically for malayalam. messages on social media that have negative social impacts, touching on subjects such as sex, caste, religion, politics, race, etc., are classified as hateful messages, posing a significant detection challenge. the study exclusively considers language-specific research on hate speech detection, analyzing the methodologies employed in each instance. three parameters are used to assess the overall landscape of this issue across asian languages. the study seeks to identify the optimal classification algorithm for this task and explore the relationship between classification approach, dataset type and size, and accuracy. the article demonstrated diverse language-specific papers to automate the detection of hate speech in asian languages. the researchers employed a spectrum of machine learning (ml) classification techniques and deep learning algorithms to address this issue. the findings indicate that the support vector machine (svm) emerged as a widely preferred algorithm for binary-level categorization in this context. however, the accuracy of each study was found to be contingent upon factors such as the nature of the dataset (balanced or imbalanced) and its size. the survey underscores a positive correlation between accuracy and dataset size, revealing that accuracy tends to increase with larger datasets. additionally, certain studies focusing on multi-label categorization demonstrated superior performance. the task defined in @xcite demonstrated a comprehensive exploration of different transformer-based models. additionally, a genetic algorithm-based technique for assembling diverse models has been introduced. the ensembled models, trained separately on each language, achieved the first, second, and third positions in tamil, kannada, and the malayalam sub-task, respectively. the research conducted involved the exploration of ensemble methodologies, recognizing the potential for improved predictive capabilities through the amalgamation of diverse models, as opposed to reliance on a singular classifier. acknowledging the limitations of conventional prediction averaging ensembles, especially when weaker models are present, the researchers adopted a strategy incorporating model weights based on individual performances. this approach aligns with established genetic algorithm (ga) techniques for determining optimal weights within the ensemble. it is observed that, among the individual transformer models, optimal performance is achieved by employing xlm-roberta-large (xlmr-large) for the tamil dataset and custom xlm-roberta-base (xlmr-c) for the kannada dataset. in the case of the malayalam dataset, both aforementioned models demonstrate comparable performance. the heightened effectiveness of xlm-roberta models can be attributed to their pretraining using a parallel corpus, which is consistent across different languages. subsequent pretraining with the specific dataset further contributes to performance improvement in the kannada dataset. the utilization of the larger xlm-r model was precluded due to limitations in gpu space. furthermore, the study addresses the performance of fusion models, emphasizing their nearly identical performance across various combinations. in the task of (khan et al., 2021) approximately 100,000 tweets originating from south asia, with potential hate-speech content, were collected through web scraping. subsequently, the tweets were manually parsed to identify those in roman urdu. an iterative approach was then employed to formulate guidelines for hate-speech detection, which were subsequently used to create a hatespeech corpus specifically for roman-urdu. the re-sulting corpus comprises over 5000 tweets, predominantly containing hate speech. the initial categorization involved classifying tweets into hostile and neutral, followed by a secondary categorization of hostile tweets into offensive and hate speech. the evaluation of this developed corpus was conducted using various supervised learning techniques. the research sought to illustrate the utility of the curated dataset for the hate-speech detection task. in pursuit of this objective, a series of experiments were conducted employing five distinct supervised ml techniques encompassing both classic and deep learning methodologies. this dichotomy between classic and deep learning approaches has been extensively explored in the existing literature, as evidenced by the works of various scholars. noteworthy observations in the literature, as exemplified in certain instances, suggest that classic techniques may outperform deep learning methodologies under specific conditions. for instance, when confronted with a relatively small training dataset, classic techniques have demonstrated superior performance. conversely, on larger datasets, deep learning methodologies have exhibited a propensity to surpass the efficacy of classical techniques. these nuanced findings underscore the contextual dependence and interplay between dataset size and the comparative performance of classic and deep learning approaches in hate-speech detection applications. various supervised learning techniques were employed to assess their efficacy in identifying objectionable language, discerning between basic and intricate sentences, and categorizing sentences as offensive or involving hate speech. additionally, rules were devised to distinguish between hate speech and offensive speech. the experimental results yielded the following insights: a). logistic regression emerged as the most effective technique for discerning between neutral and hostile sentences, with count vectors proving to be the most efficient features; b). logistic regression, when coupled with count vectors, was also identified as the most effective technique for differentiating between hateful and offensive sentences; c). both character n-grams and word n-grams exhibited suboptimal performance on the developed corpus, primarily attributed to issues related to spelling variations in roman urdu; d) the detection of sarcasm proved challenging, as there were instances where no explicitly hateful or offensive words were used in the comments, resulting in the classification of comment polarity as neutral. the shared task in association with the forum for information retrieval evaluation (fire) 2020 on detecting hate speech and offensive content in german has been described in @xcite . ## dataset we initially intended to utilize well-known benchmark hate speech datasets to conduct a comprehensive analysis and effectively compare our proposed methodologies with existing works. regrettably, the paucity of studies focusing on speech inclusive of emojis has constrained our options, leaving us with only two datasets that align with our requirements. for the english dataset, the emoji2vec dataset comprises a collection of tweets, meticulously stored in a pickle file and thoughtfully divided into both training and test sets. this dataset encompasses a total of 64,599 tweets, with 51,679 tweets allocated to the training set and 12,920 tweets assigned to the test set. each of these tweets has been diligently labeled as positive (1), neutral (0), or negative (-1), providing a comprehensive understanding of the sentiment. for the bengali, the dataset consists of a substantial collection of 30,000 comments, revealing a significant disparity as 10,000 comments are categorized as hate speech. it is particularly noteworthy that this bias is more prevalent in areas such as celebrity and politics, where instances of hate speech are relatively infrequent. however, accurately labeling hate speech poses challenges due to the intricate contextual nuances, often resulting in misclassification. analyzing the average length of the text reveals interesting variations, with meme comments being notably concise, while comments related to celebrities tend to be more extensive. comparing this dataset with state-of-the-art collections emphasizes the remarkable linguistic diversity present, underscoring the necessity of considering different categories for a comprehensive linguistic analysis. ## model the proposed model incorporates the sophisticated encoder-decoder architecture, a powerful framework that enhances the understanding and interpretation of sentences. by embedding words into a sequence of vectors, the encoder skillfully captures the essence of the sentence, enabling a comprehensive comprehension. the decoder subsequently utilizes this sequence of word embeddings to gen-erate accurate predictions, elevating the overall performance of our system. we chose the most popular transformer-based encoders and emoji2vec word embeddings to help us understand and analyze emojis. emoji2vec word embeddings were created by assigning numbers to emojis and their official descriptions. we also matched each emoji with a sequence of word embeddings from google word2vec that represents its official description. for each emoji i, we define a trainable vector of float numbers x i and a vector of float numbers v i that represents its official textual description. also, we match each emoji with its official description that is expressed as a sequence of vectorized word embeddings w 1 , ..., w n from google word2vec as follows: v @xmath0. to train the vector representation x i for emoji i, we generate emoji-description pairs that are either a true match (the emoji i matches the description j) or a false match (the emoji i does not match the description j, but is paired with a description of another emoji). then, we train the vectors xi's with log-likelihood loss: here, y ij represents the binary label indicating whether the pair is a true match (y @xmath1. the optimization process aims to minimize this loss function, contributing to the learning of accurate vector representations for the given emojis. in the realm of transformer encoders, a crucial component is the self-attention mechanism, which empowers the calculation of attention for each word embedding towards the context. this process unfolds through a relatively straightforward mathematical operation, as elucidated below. imagine you have a set of data represented as x, where each element is a word in a sequence. think of b as the number of sets, s as the length of each sequence, and d as the dimension of the initial word representations. now, there are three operations called query (q), key (k), and value (v) for the word sequences. these operations are calculated by multiplying the input data x with trainable weight matrices named w q , w k , and w v . in simpler terms: q (query) is obtained by multiplying x with w q . k (key) is obtained by multiplying x with w k . v (value) is obtained by multiplying x with w v . these operations form the basis of the self-attention mechanism, enabling each word's representation to adapt and consider its surrounding context. this adaptability is a key factor contributing to the transformer encoder's effective processing capabilities. attention-based word embeddings (x att ) can then be calculated by the scaled dot product attention formula: a formula needs to be introduced here. ## results and analysis: the model has been trained and evaluated using the datasets that are split into training, validation, and test sets. the training process involved optimizing model parameters to minimize the binary cross-entropy loss. after ten epochs of training each model and the two datasets, the results are shown in the following the average model stands out as the most effective approach for hate speech detection, demonstrating superior performance across both english and bengali datasets. this model's success lies in its distinctive strategy of averaging emoji2vec embeddings, creating a nuanced and contextually rich representation that significantly enhances the discrimination between hate speech and non-hate speech instances. averaging emoji2vec embeddings offers a multifaceted advantage. firstly, it captures the collective emotional essence conveyed by emojis in a sentence, resulting in a comprehensive and contextually rich vector representation. this approach is particularly effective in hate speech detection, where understanding the emotional context is pivotal. moreover, averaging helps mitigate potential noise introduced by individual emojis, allowing the model to focus on the overarching emotional context rather than relying heavily on specific emoji instances. one noteworthy aspect of the average model's success is its cross-linguistic applicability. the model demonstrates robust performance in both english and bengali datasets, showcasing its adaptability and effectiveness in diverse linguistic contexts. this indicates that the model's ability to leverage universal emotional signals from emojis contributes significantly to its effectiveness. the inclusion of emojis in the average model holds strategic significance. emojis, as universal symbols, contribute to a shared emotional language. the model's capability to integrate both linguistic and emotional features enables a more comprehensive understanding of textual content. emojis, in this context, serve as complementary signals to the textual information, enriching the model's understanding of emotional nuances. the average model's proficiency highlights the practical application of considering emojis as valuable contextual cues in nlp tasks. ## conclusion: the development of effective hate speech detection models is hindered by the challenge of obtaining a large and diverse dataset that accurately labels instances of hate speech. the limited availability of comprehensive datasets restricts the model's exposure to the various nuances of hate speech, which can lead to decreased effectiveness in detecting hate speech instances that deviate from the limited training data. hate speech is known for its nuanced expressions, often conveyed through the use of emojis to convey emotions and sentiments. however, the inclusion of emojis in the model introduces complexities, as the model must decipher the emotional subtleties conveyed by these visual symbols. this challenge is particularly demanding when hate speech instances heavily rely on emojis for expression, requiring a model that can adeptly capture these nuanced emotional cues. the constraint of limited computational resources further compounds the challenges. the depth and complexity of neural network architectures necessary for discerning subtle patterns in hate speech may be curtailed due to resource constraints. this limitation hinders the model's capacity to achieve a nuanced understanding of hate speech instances, especially those that require intricate pattern recognition. an additional challenge is the lack of annotations for emojis in existing datasets labeled for hate speech. this absence of annotated emojis in the training data impedes the model's ability to learn and generalize from hate speech instances where emojis play a pivotal role in conveying context and emotion. furthermore, relying on free gpu resources can add further constraints to model training. the availability and capacity of these resources may be limited, leading to slower model training times and potential restrictions on the complexity of the neural network. these limitations can impact the model's overall efficiency and its ability to achieve optimal performance in hate speech detection tasks. the proposed approach combines the strengths of multilingual bert and emoji2vec embeddings to create a robust model for detecting hate speech in the bengali language. by incorporating both textual and emoji-based features, the model gains a deeper understanding of context and sentiment. the results demonstrate promising accuracy in identifying hateful content, although further improvements can be made to address the false positives.