Fida @ D ravidian L ang T ech 2024: A Novel Approach to Hate Speech Detection Using Distilbert-base-multilingual-cased (2024)

Abstract: In the contemporary digital landscape, social media has emerged as a prominent means of communication and information dissemination, offering a rapid outreach to a broad audience compared to traditional communication methods. Unfortunately, the escalating prevalence of abusive language and hate speech on these platforms has become a pressing issue. Detecting and addressing such content on the Internet has garnered considerable attention due to the significant impact it has on individuals. The advent of deep learning has facilitated the use of pre-trained deep neural network models for text classification tasks. While these models demonstrate high performance, some exhibit a substantial number of parameters. In the DravidianLangTech@EACL 2024 task, we opted for the Distilbert-base-multilingual-cased model, an enhancement of the BERT model that effectively reduces the number of parameters without compromising performance. This model was selected based on its exceptional results in the task. Our system achieved a commendable Macro F1 score of 0.6369%.

## introduction in recent times, the exponential growth of internet technology has led to a surge in the user base, accompanied by the emergence of various social platforms. these platforms provide a space for netizens to freely express their opinions, often leveraging anonymous features. consequently, this freedom has led to an increase in hate speech @xcite and offensive content on the internet. addressing this issue is crucial, as it not only causes distress but also poses severe risks, including mental health concerns and potential instances of suicide. given the enormous volume of comments generated daily on the internet, manual moderation is impractical. therefore, the integration of artificial intelligence methods becomes imperative. however, identifying hate speech and offensive content poses several challenges. firstly, social media posts encompass multiple languages and diverse writing styles. the presence of irregular writing and the evolution of new internet expressions further complicate the detection task. additionally, some comments may not overtly contain derogatory language but instead employ implicit or ironic attacks, adding another layer of complexity. furthermore, the absence of a clear standard for the definition of hate speech contributes to the intricacy of the task. model performance is highly contingent on the training dataset, influenced by the annotator's perspectives to a considerable extent. in response to these challenges, the nlp community has introduced various tasks focused on hate speech identification, including the dravidianlangtech-eacl2021 shared task @xcite @xcite . this particular task is dedicated to identifying hate speech and offensive content in both telugu and english languages.in the realm of nlp, numerous tasks such as identifying hope speech @xcite , sentiment analysis @xcite , and detecting hate speech utilize various models like deep learning @xcite , transformers @xcite , and traditional machine learning @xcite . in this paper, our focus is on hate speech using the pre-trained model distilbert-base-multilingualcased @xcite , which builds upon the bert model @xcite , by significantly reducing the number of parameters, consequently enhancing training speed. the structure of this article encompasses an exploration of related research on hate speech and offensive speech recognition (section 2), an explanation of the model used in the task (section 3), an elucidation of the exper-imental procedure (section 4), a presentation of the experimental results and their analysis (section 5), and a comprehensive summary of this work (section 6). ## related work numerous research endeavors have sought to identify and address abusive comments across various languages; however, there is a noticeable research gap in the domain of low-resource languages. @xcite and addressed this gap by conducting a shared task at acl 2022, focusing on detecting categories of abusive comments on social media. their study encompassed comments in tamil and a codemixed language featuring both tamil and english scripts @xcite contributed a comprehensive investigation into abusive language detection, specifically in urdu and roman urdu comments. employing a diverse set of machine learning and deep learning models, the author evaluated the performance of five ml models (naive bayes, support vector machine, instance-based learning, logistic regression, and jrip) and four dl models (cnn, lstm, blstm, and convolutional lstm) across two datasets-one comprising tens of thousands of roman urdu comments and another with over two thousand comments in urdu. the cnn exhibited notable superiority, achieving accuracy rates of 96.2% for urdu and 91.4% for roman urdu, establishing itself as the most adept model in identifying abusive language in these linguistic contexts. some researchers have explored multiple methods independently to determine the most effective model rajalakshmi2022dlrg employed three methodologies-machine learning, deep learning, and transformer-based modeling. for machine learning, eight algorithms were implemented, with random forest yielding the best results for the tamil-english dataset. in deep learning, bi-directional lstm outperformed other models, especially with pre-trained word embeddings. in transformer-based modeling, indicbert and mbert with fine-tuning were employed, with mbert delivering the most favorable results. (eshan and hasan, 2017) delved into machine learning algorithms for bengali abusive text detection, revealing that svm with a linear kernel performed optimally using trigram tfidfvectorizer features. @xcite proposed a binary classification model for hate speech detection, utilizing advanced deep learning techniques such as continuous bag-of-words (cbow) and para-graph2vec to represent text in a low-dimensional vector space, achieving an auc value of 0.80. (kedia and nandy, 2021) developed an offensive content classification model for dravidian code-mixed languages (tamil, malayalam, and kannada) using transformer-based models-bert and roberta. renjit and idicula (2020) employed word embedding for the manglish dataset, achieving a weighted f1-score of 0.53 and 0.48 using keras embedding and doc2vec approaches for sentence representation, respectively. @xcite focused on hate speech detection by extracting unito-five-gram features from a dataset of 450,000 tweets. using ensemble learning, logistic regression, and svm classification techniques, they predicted hate speech with an accuracy of 89% using the ensemble learning model. ## methodology the methodology employed in this study leverages the distilbert-base-multilingual-cased model @xcite , a pre-trained transformer-based language model, to address the research objectives. the model, developed by hugging face, has been fine-tuned for multilingual understanding and exhibits capabilities across various languages. to train and evaluate the distilbert-base-multilingualcased model a diverse and representative dataset encompassing multiple languages is gathered. this dataset spans domains relevant to the research focus. the collected data undergoes preprocessing to ensure uniformity and compatibility with the model's requirements. this step involves tokenization, stemming, and the removal of irrelevant information to enhance the model's efficiency. the distilbert-base-multilingual-cased model is configured with specific parameters to align with the research objectives. this includes setting appropriate learning rates, batch sizes, and training epochs for optimal performance. the pre-trained distilbertbase-multilingual-cased model is fine-tuned on the task-specific dataset as seen in figure 1 . this involves updating the model's weights based on the unique characteristics of the dataset and the objectives of the research. fine-tuning enhances the model's ability to grasp nuances within the target domain. to assess the generalizability of the model, a cross-validation strategy is employed. the dataset is partitioned into training, validation, and test sets, ensuring that the model is rigorously evaluated on diverse samples. this step aids in mitigating over-fitting and enhances the robustness of the model. the model's performance is evaluated using appropriate metrics tailored to the research task. common metrics such as precision, recall, and f1 score are computed to quantify the model's effectiveness in capturing patterns and making accurate predictions. the outcomes of the experiments are analyzed comprehensively to draw meaningful conclusions. this involves an in-depth examination of the model's predictions, identification of potential challenges, and exploration of areas for improvement. throughout the implementation of the distilbert-base-multilingual-cased model, ethical considerations are prioritized. data privacy and confidentiality are ensured, and the research adheres to established guidelines for responsible ai usage. ## result the performance parameter used to assess the detection model's overall efficacy is the macro average f1-score. it is computed by taking the average of all classes after determining the f1-score for each class separately. the macro average f1-score assigns equal weight to each class by computing the average after considering each class's performance individually, regardless of class size or imbalance. in telugu, we obtained a macro f1-score of 0.6369. securing the 18th position among the 27 participating teams. for a comprehensive summary of the outcomes attained by all competing teams, as outlined in ## error analysis the distilbert model excels in detecting hate speech, boasting high accuracy with a notable true positive count. however, challenges arise with false positives, even in balanced datasets. this necessitates careful analysis, urging strategic adjustments like fine-tuning parameters to enhance overall efficiency. ongoing evaluations on validation and test sets are vital for adapting the model and ensuring reliable performance. ## limitations the utilization of pre-trained transformers, specifically multilingual distilbert, presents a significant consideration in our endeavors to detect hate speech. while leveraging these pre-trained models can enhance our comprehension of textual data, their effectiveness may be limited by the specificity of the pre-training corpus. this potential limitation could result in a mismatch with the unique characteristics of hate speech, underscoring the necessity for meticulous fine-tuning to ensure optimal performance. moreover, the inherent linguistic complexities of telugu codemixed text may pose challenges impacting the model's ability to discern subtle patterns. consequently, further investigation and refinement are warranted to address these challenges and enhance the model's accuracy in fake news detection for telugu codemixed text. ## conclusion hate and offensive posts on social media can put the victim in hazardous circumstances and increase their risk of mental health issues like depression, insomnia, and in extreme cases, suicide. as a result, identifying such hateful and harmful social media information is crucial for jobs involving natural language processing. our work presents an improved distilbert-base-multilingual-cased model for identifying hateful and abusive tweets from telugu text. for telugu language tweets, the suggested fine-tuned distilbert-base-multilingualcased model achieved 0.6369% accuracy. the role of embedding with an improved bert model for higher classification performance may be investigated in subsequent work.