CUET _ B inary_ H ackers@ D ravidian L ang T ech EACL 2024: Hate and Offensive Language Detection in T elugu Code-Mixed Text Using Sentence Similarity BERT (2024)

Abstract: With the continuous evolution of technology and widespread internet access, various social media platforms have gained immense popularity, attracting a vast number of active users globally. However, this surge in online activity has also led to a concerning trend by driving many individuals to resort to posting hateful and offensive comments or posts, publicly targeting groups or individuals. In response to these challenges, we participated in this shared task. Our approach involved proposing a fine-tuning-based pre-trained transformer model to effectively discern whether a given text contains offensive content that propagates hatred. We conducted comprehensive experiments, exploring various machine learning (LR, SVM, and Ensemble), deep learning (CNN, BiLSTM, CNN+BiLSTM), and transformer-based models (Indic-SBERT, m- BERT, MuRIL, Distil-BERT, XLM-R), adhering to a meticulous fine-tuning methodology. Among the models evaluated, our fine-tuned L3Cube-Indic-Sentence-Similarity- BERT or Indic-SBERT model demonstrated superior performance, achieving a macro-average F1-score of 0.7013. This notable result positioned us at the 6th place in the task. The implementation details of the task will be found in the GitHub repository.

## introduction the contemporary digital landscape is heavily influenced by the pervasive role of social media in facilitating online communication. platforms such as youtube, instagram, facebook, and twitter have not only provided users with avenues for creating and sharing content but have also become arenas where individuals can freely express their views and thoughts at any given moment @xcite . the evolution of social media has brought forth a darker side, where individuals are defamed, targeted, and marginalized based on factors such as religion, physical appearance, or sexual orientation @xcite . given the impracticality of manually identifying offensive texts at scale, there arises a crucial need for an automated system capable of detecting hate speech. such a system can empower relevant authorities to take necessary actions against offensive content. natural language processing (nlp) emerges as a pivotal solution, offering various techniques to address these challenges effectively @xcite . while the problem of identifying offensive language has been tackled from multiple angles, including detecting cyberbullying, aggression, toxicity, and abusive language @xcite @xcite @xcite , there is a pressing need for more focused attention on hatespecific contexts in diverse languages. over the past few years, numerous studies have been conducted on detecting hate and offensive content in several high-resource languages such as english, spanish, arabic @xcite , and others that have ample linguistic resources, datasets, and related facilities. however, the challenge persists in addressing this issue efficiently for low-resource languages @xcite . in this particular task @xcite , the organizers presented a telugu code-mixed hate speech dataset @xcite , framing it as a binary classification problem. the objective is to discern whether a given text represents any hate and offensive speech or not. this task serves as a crucial step toward addressing the gap in efficient hate speech detection for low-resource languages like telugu, especially in the context of code-mixed text. as part of the participants in this task, the main contributions of our work are outlined below: we organized the rest of our presentation as follows: section 2 delves into related work, section 3 describes the task and dataset, section 4 outlines our methodology, section 5 describes the experimental setup, section 6 presents the results analysis, section 7 conducts an in-depth error analysis, and finally, section 8 concludes with insights and outlines directions for future work. ## related work in the evolving landscape of hate and offensive text detection, researchers have explored a spectrum of techniques, each contributing to the continuous refinement of models. an influential bengali abusive text detection endeavor was conducted (eshan and hasan, 2017) assessing the efficacy of support vector machine (svm), random forest (rf), and naive bayes (nb) classifiers. their framework, achieving an accuracy of approximately 95%, laid a foundation for subsequent investigations into more advanced methodologies. @xcite investigated offensive language identification in social media code-mixed tanglish (tamil+english) and manglish (malayalam+english) text, as well as malayalam script-mixed. the n-gram tf-idfbased mnb classifier achieved a weighted f1-score of 0.90 in tamil code-mixed text whereas lr led with 0.78 for malayalam code-mixed content. the vanilla neural network (vnn) outperformed in handling malayalam script-mixed text, achieving an impressive weighted f1-score of 0.95. as the field matured, a notable shift emerged from traditional machine learning to deep learning, exemplified by omar et al. (2020)'s work on the detection of arabic hate speech. using recurrent neural networks (rnn), they achieved an exceptional 98.7% accuracy which outperformed convolutional neural networks (cnn). another study @xcite employed a multi-label approach for hate speech detection on social media, utilizing pre-trained bert and ensemble learning architectures that include bilstm and bigru models. integrating recent word embedding techniques and dl models, the proposed approach achieved a remarkable roc-auc score of 98.63%. the exploration of transformer-based models added a layer of complexity to hate speech detection. a weighted ensemble technique @xcite , incorporating m-bert, distil-bert, and bangla-bert, demonstrated the adaptability of these models in handling diverse linguistic nuances, particularly in bengali aggressive text datasets. in dravidianlangtech2021 foot_1 , the author @xcite addressed the challenge of detecting offensive text in code-mixed social media data, employing effective transformer-based models like xlm-r, m-bert, and indic-bert for tamil, kannada, and malayalam languages. extending this exploration, another study @xcite within the same task also delved into a diverse set of transformer-based models, including muril, distil-bert, and others. in hasoc 2023 foot_2 , a study @xcite evaluated the efficacy of various sentence-bert models, including bengali-sbert, gujarati-sbert, assamese-bert, and l3cube indic-sbert, showcasing state-of-the-art results in detecting hate speech within indian linguistic contexts. ## task and dataset description in this shared task @xcite , a telugu codemixed dataset was introduced for the detection of hate and offensive language @xcite . the dataset, designed for binary classification, comprises diverse social media posts and comments containing both hate/offensive text and non-hate/non-offensive text. for participants, both the training and test datasets were provided, without any separate validation set. the training dataset consisted of 4,000 samples, comprising 2,061 non-hate-labeled and 1,939 hate-labeled samples, demonstrating a well-balanced distribution. some other useful insights are mentioned in in this section, we will delineate our methodology step by step. figure 1 shows a schematic diagram of the methodology. ## experimental setup the hyper-parameters used in this task were determined through an iterative process involving frequent trials. the choice of the parameters depicted in ## result analysis ## error analysis to comprehensively analyze the performance of l3cube indic-sbert, we provide a detailed error analysis in this section, utilizing a confusion matrix depicted in figure 2 . out of 250 samples, 179 hate speech and 172 non-hate speech samples were correctly classified. however, there were 71 misclassified hate speech samples and 78 misclassified non-hate speech samples. the misclassification rates for both hate and non-hate classes are 28.4% and 31.2% respectively. the minimal difference suggests a close misclassification rate between the two labels, potentially influenced by slight variations in the number of types of training samples. additionally, similar code-mixed words between the two classes may contribute to this issue when the model attempts to understand the text's meaning. to reduce misclassification, a detailed analysis of each word in misclassified samples using the named entity recognition (ner) method can be done to remove redundant codemixed words. ## conclusion and future work this paper delves into the exploration and evaluation of various ml, dl, and transformer-based approaches. our initial investigation involved tf-idf and embedding features (word2vec & fast-text), followed by systematic experiments with ml and dl methods. the results indicate that svm outperformed other ml and dl models with an f1-score of 0.65. however, incorporating the transformer model significantly enhanced overall performance. specifically, indic-sbert, stood out by achieving the highest f1-score of 0.70. future exploration can involve incorporating contextualized embeddings like gpt, elmo, and flair, or experimenting with ensembling transformers and fusion models tailored to hate speech contexts. besides, alternative embedding techniques such as glove and bert-based embeddings can be applied to enhance the performance of dl models.