A Corpus of H indi- E nglish Code-Mixed Posts to Hate Speech Detection (2024)

Abstract: Social media content, such as blog posts, comments, and tweets, often contains offensive language, including racial hate speech, personal attacks, and sexual harassment. Detecting inappropriate language is crucial for user safety and prevention of hateful behavior and aggression. This study introduces the HECM (Hindi-English code-mixed tweets) to fill the gap in Hindi language resources. The corpus comprises approximately 9.4K tweets labeled as hateful and nonhateful. It includes detailed information on the data, such as the annotation schema, the label definitions, and an interannotator agreement score of 85%. The study evaluates the effectiveness of traditional machine learning, deep neural networks, and transformer encoder-based approaches. The results show a significant improvement in terms of macro-F1 and weighted F1 scores. Additionally, a lexicon containing 2000 lexicons tagged in 21 categories is created based on the multilingual HURTLEX lexicon. This lexicon is merged with the transformer encoder, resulting in a marginal improvement in macro-F1 and weighted-F1. The study also experiments with a Hindi-Devanagari dataset to assess the impact of the lexicon on performance metrics.

## introduction social networks have become an intrinsic part of the lives of many people. a phenomenon that accompanies social media platforms with serious impacts on society is the presence of socially unacceptable language. socially unacceptable language comprehends many different user-generated language phenomena, such * this work was conducted at iit patna. as toxic language, offensive language, abusive language, and hate speech, among others. an approach to filter offensive content is to use human experts (e.g., moderators) and manually review the posts or comments as soon as they get posted. there has been a growing interest in computational linguistics (cl) and natural language processing (nlp), as manually monitoring and flagging these phenomena is impossible. the automatic identification of abusive language phenomena has followed a common trend in nlp: feature-based linear classifiers ( @xcite , @xcite , @xcite , @xcite ) neural network architectures ( @xcite , @xcite , @xcite , @xcite ,(bashar and nayak, 2020),) (e.g., cnn or bi-lstm), and fine-tuning pre-trained language models, e.g., @xcite , @xcite , @xcite , @xcite ). this paper focuses on detecting offensive language in hindi and hindi-english code-mixed posts. although there are numerous studies on automatic detection of offensive content in resource-rich languages such as english ( @xcite , @xcite , @xcite , @xcite , @xcite ), there are limited data and work available for a resource-poor hindi language. the key contributions of this paper are listed below. additionally, a lexicon has been developed based on hurtlex, consisting of 2000 offensive words across 21 categories. the experiment also utilizes the existing dataset hhsd @xcite . the corpus for the hindi covers both singlelayer @xcite (kumar et al., 2018) (mathur et al., 2018) (jha et al., 2020) (bhardwaj et al., 2020) and multi-layer (mandl et al., 2019) (mandl et al., 2020) (mandl et al., 2021)(kapil et al., 2023) (bhattacharya et al., 2020) textual-tagged data. (kapil et al., 2023) crawled hindi data in devanagari script and tagged it hierarchically, covering multiple layers of hate. the experiment involved using m-bert and muril for multitask learning with related datasets from the english, hindi, and urdu domains, resulting in significant accuracy and f-score on the single-task learning framework. the multi-channel multichannel transfer learning-based model (mimct) described in @xcite uses several feature inputs in conjunction with transfer learning to identify offensive (hate speech or abusive) hinglish tweets from the proposed hinglish offensive tweet (hot) dataset. in order to address hate and offensive detection on the data by @xcite , @xcite investigated deep learning architectures such as cnn, lstm, and variants of bert like m-bert, indicbert, and monolingual roberta. a complementary approach to supervised learning towards the detection of abusive and offensive language is the use of language resources such as lexicons and dictionaries. @xcite describe the creation of hurtlex, a multilingual lexicon of hate words. @xcite proposes to utilize lexical features derived from a hate lexicon towards improving the performance of bert in such tasks. ## experiment setup the experiments were performed using a 5-fold cross-validation approach. the 4-fold training set is split into 15% validation and 85% training, while the last fold is treated as the test set to evaluate the model. all the deep learning models were implemented using keras @xcite with tensorflow @xcite as the backend. the number of filters used in cnn is 100, and the kernel width ranges from 1 to 4. for the bilstm, the number of hidden nodes is set to 100. categorical cross-entropy is used as a loss function, topics (caa), (nrc), (article 370), (ram mandir), (beef ban), (triple talaaq), (award wapsi), (demonetization), (gst), (liquor ban), (mannkibaat), (pulwama attack), (saheenbagh), (swachh bharat), (sabrimala mandir), (fatwa), (love jihad), (azadimarch) and adam (kingma and ba, 2015) optimizer is used for optimizing the network. we use a learning rate of 2e-5 for the transformer models. the batch size of 30 and an epoch of 2 are found to be optimal. the value of bias is randomly initialized to all zeros, the relu activation function is utilized at the intermediate layer, and softmax is utilized in the last dense layer. the evaluation criteria used are macro-f1 and weighted-f1. the nvidia gpu is used for the evaluation. ## results and analysis ## conclusion and future works this paper released approximately 9.4k posts tagged into hateful and non-hateful, named as hecm. extensive experiments are conducted on hecm and the existing dataset hhsd to train svm, cnn, m-bert, muril, and xlm-r. the experiment is also conducted over the merged data set (hecm + hhsd). the transformer encoder features are fused with lexicon features to obtain significant macro-f1 and weighted-f1. the future work intends to augment the dataset with additional boosted data. since a lot of tweets require contextual information, localized knowledge graphs can be created for this by collecting intra-user and inter-user tweets to obtain valuable features. the contextual knowledge can easily be verified against this knowledge base.