Athena@ D ravidian L ang T ech: Abusive Comment Detection in Code-Mixed Languages using Machine Learning Techniques (2023)

Abstract: The amount of digital material that is disseminated through various social media platforms has significantly increased in recent years. Online networks have gained popularity in recent years and have established themselves as goto resources for news, information, and entertainment. Nevertheless, despite the many advantages of using online networks, mounting evidence indicates that an increasing number of malicious actors are taking advantage of these networks to spread poison and hurt other people. This work aims to detect abusive content in youtube comments written in the languages like Tamil, Tamil-English (codemixed), Telugu-English (code-mixed). This work was undertaken as part of the “DravidianLangTech@ RANLP 2023” shared task. The Macro F1 values for the Tamil, Tamil-English, and Telugu-English datasets were 0.28, 0.37, and 0.6137 and secured 5th, 7th, 8th rank respectively.

## introduction the detection of abusive language has become a major area for research and development in computer linguistics and natural language processing (nlp). the requirement to recognise and filter out harmful or abusive content has multiplied enormously with the rise of internet platforms and online social media. detection of abusive or offensive content in english language has been done by many researchers in a wide manner @xcite @xcite . recent days, the interest goes on the low resource languages and native languages. in multilingual societies like india, where tamil and telugu are widely spoken, this is especially crucial. due to the complexity of their linguistic structures and the lack of sufficient labelled data for the training of strong models, the identification of abusive language presents particular difficulties in tamil and telugu. a wide range of harmful information is included in abusive language, such as hate speech, cyberbullying, vulgarity, and disparaging remarks. for the sake of upholding a secure and civilised online community, it is essential to accurately identify such offensive language in tamil and telugu. the identification of abusive language in english has advanced significantly in recent years, according to academics. however, adapting these methods to tamil and telugu necessitates taking into account these languages' unique traits, cultural quirks, and lack of labelled data. furthermore, due to variances in syntax, morphology, and the existence of dialectical changes, existing models might not generalise effectively to tamil and telugu especially the deep learning techniques as most of them are built for english language. this work aims to contribute to the development of efficient abusive language detection systems for tamil and telugu by tackling the difficulties unique to these languages, promoting a safer and more welcoming online environment for users who communicate in tamil and telugu. in this study, multiple machine learning models were tested in an effort to create an effective system for identifying hate speech and abusive language in tamil and telugu comments on youtube. the paper is organised as follows: earlier studies on the identification of abusive language in dravidian languages like tamil and telugu are discussed in section 2. the proposed system along with the architecture diagram and the system modules are explained in section 3. the datasets and methodologies used in the suggested system are presented in section 4. the results are discussed in section 5. the conclusion and future work are presented in section 6. charangan vasantharajan and uthayasanker thayasivam @xcite offered a novel and flexible way of selective translation and transliteration operations to improve the results of adjusting and assembling bert, distilbert, and xlm-roberta. the experiment's findings proved that ulmfit is the best model for the task. ulmfit and mbertbilstm beat other well-known transfer learning models like distilbert and xlm-roberta as well as hybrid deep learning models for this tamil code-mix dataset. in the work done by shantanu patankar et al. @xcite recurrent neural networks, ensemble models, and transformers were used to optimise the results. for the tamil data, muril and xlm-roberta were utilised. the macro-averaged f1 score generated by the models was 0.43. with a macro-averaged f1 score of 0.45, the top models muril and m-bert both generated excellent results for the code-mixed data. malliga subramanian et al. @xcite worked to identify the offensive utterances, models based on traditional machine learning techniques-such as bernoulli naive bayes, support vector machine, logistic regression, and knearest neighbor-were constructed. the multilingual transformer-based pre-trained models of natural language processing mbert, muril (base and large), and xlm-roberta (base and large) were also used in the experiments. pradeep kumar roy et al. @xcite investigated the use of different machine learning and deep learning approaches. combining the output of transformer and deep learning-based models, an ensemble model was proposed to detect hate speech and objectionable language on social networking sites. the experimental findings of the suggested weighted ensemble framework outperformed state-of-the-art models for the malayalam and tamil code-mixed datasets, achieving weighted f1-scores of 0.802 and 0.933, respectively. ## abusive content detection system we have used the deep learning transformer model (bert), machine learning models (logistic regression, support vector machines, decision trees, naive bayes) and ensemble model (random forest) for abusive content classification. the training dataset is used to build the model by learning the data, development dataset is used to evaluate and fine tune the trained model and test dataset is used for final prediction. the features are extracted from the input text and its matching label are used for learning and training. the texts in the dataset are vectorized using the term frequency -inverse document frequency (tf-idf) vectorization technique for feature extraction. the model that produces the greatest macro f1-score is selected as the final model to be used for detection after various machine learning models are tested. the model is trained using the training dataset. the performance of the trained model is assessed using the development dataset. by tweaking the parameters, the model is re-trained based on the performance. finally, predictions are made using the test data and the model. figure 1 ## implementation modules this section elaborates on the datatsets and the methodologies used. ## results several machine learning models are experimented using the given datasets. evaluation metrics like accuracy (acc), precision (prec), recall and f1score are used to evaluate the performance of the models. the model that gives the best accuracy and f1-score is chosen to run finally on the test data. the task on the tamil and the tamil-english dataset are multi-class classification tasks. svm has proved to work best on them due to the factors like effective separation of classes and resistance to overfitting. the task on the telugu-english dataset is a binary-class classification problem. lr has performed best on this dataset due to its simplicity, interpretability, efficiency with small dataset and robustness to outliers. ## performance analysis during testing, it was found that simpler machine learning models like svm and lr performed better than more complex transformer models like bert and mbert. this can be due to the dataset's quantity and quality. traditional ml models generally require less data than more complex models to perform successfully. ml models might do better than bert if the dataset isn't too big.bert excels at a number of nlp tasks, however improved training and classification require a large dataset with a wide variety of texts. the performance of the bert may suffer if the data are unstable or skewed. it has been identified that even though the accuracy for all the three datasets (multi class and binary classification) are more or less same, the f1-score of multi-class classification models are very less when compared to the binary classification models. this is due to the imbalance in tamil and tamil-english dataset. we belive that this can be rectified using data augmentation in future. we have also noticed that deep learning transformer model has not performed well as it could not learn the features effectively from the small dataset of low resource language. in order to improve the performance of the deep learning model we have planned to augment the dataset as a future work, so that it can solve the data imbalance as well as increase the number of samples in the dataset. for the test dataset of tamil, tamil-english, telugu-english languages we have achieved the f1-score of 0.28, 0.37, and 0.6137 respectively. ## conclusion and future work this task was taken as a part of "abusive comment detection in tamil and telugu at dravid-ianlangtech@ranlp 2023" shared task. for the tamil and tamil-english datasets (multi class classification) support vector machine showed the maximum performance and hence was used to run on the test data and the results were submitted. for the telugu-english dataset (binary classification) logistic regression showed the best performance hence that was used to run on the test data. our team "athena" was ranked 5 th , 7 th , 8 th for the tamil, tamil-english and telugu-english datasets respectively. in the future we would like to improve the results by using larger and more balanced datasets. we would also like to experiment on vectoriza-tion feature extraction techniques more suitable for code-mixed and non-english languages.