S et F it: A Robust Approach for Offensive Content Detection in T amil- E nglish Code-Mixed Conversations Using Sentence Transfer Fine-tuning (2024)

Abstract: Code-mixed languages are increasingly prevalent on social media and online platforms, presenting significant challenges in offensive content detection for natural language processing (NLP) systems. Our study explores how effectively the Sentence Transfer Fine-tuning (Set-Fit) method, combined with logistic regression, detects offensive content in a Tamil-English code-mixed dataset. We compare our model’s performance with five other NLP models: Multilingual BERT (mBERT), LSTM, BERT, IndicBERT, and Language-agnostic BERT Sentence Embeddings (LaBSE). Our model, SetFit, outperforms these models in accuracy, achieving an impressive 89.72%, significantly higher than other models. These results suggest the sentence transformer model’s substantial potential for detecting offensive content in codemixed languages. Our study provides valuable insights into the sentence transformer model’s ability to identify various types of offensive material in Tamil-English online conversations, paving the way for more advanced NLP systems tailored to code-mixed languages.

## introduction text classification is a part of information extraction (ie), an emerging area of research and application that explores how to discover knowledge (information) from a vast amount of text @xcite . so many data resources are available on the internet, like social media, e-commerce sites, blogs, news portals, personal websites, and others, where people share their thoughts and opinions in their native language @xcite . code-mixed language is a phenomenon in which speakers interchange two or more languages or language varieties within a single conversation or text. it is common in multilingual societies, particularly in urban areas where people with diverse linguistic backgrounds interact regularly @xcite . code-mixing is prevalent in many parts of the world, including india, africa, and the americas. fig. 1 is an example youtube comment with tamil-english code-mixed languages. in code-mixed language, speakers often borrow words or phrases from one language and incorporate them into another, resulting in a hybrid language combining multiple languages' grammatical structures and vocabulary @xcite . the prevalence of code-mixed language in social media and online platforms has posed significant challenges for nlp systems. traditional nlp models are designed to work with monolingual texts and struggle with the complexity of code-mixed language. consequently, there is an increasing demand for developing nlp systems that can effectively process code-mixed language, especially in identifying offensive language @xcite . detecting offensive content in code-mixed lan-guage is challenging due to the linguistic complexity of code-mixed language and the nuances of language use (chakravarthi, 2023), @xcite . offensive language in code-mixed language can involve gender-based language, stereotypes, derogatory terms, and hate speech @xcite . developing nlp systems that effectively detect offensive language in a code-mixed language is crucial for promoting safe and inclusive online spaces @xcite , @xcite . this study focuses on detecting offensives in a code-mixed tamil-english dataset using sentence transformer fine-tune (setfit). it is a substitute for few-shot text classification and involves fine-tuning a sentence transformer using task-specific data. this approach can be effortlessly implemented using the sentence-transformers library. the main objective of this research is to evaluate the performance of the setfit model and compare it with other nlp models in offensive content detection within code-mixed tamil-english language data on social media and online platforms. the study aims to provide insights into the effectiveness of setfit in identifying offensive content and its potential applicability in enhancing the robustness of nlp systems for code-mixed languages. the following research questions are conceived from the above-mentioned objective of this research work. rq1 how does the performance of setfit compare to lstm, bert, multilingual bert (mbert), indicbert, and labse in the task of offensive content detection within a code-mixed tamil-english dataset? rq2 to what extent does the performance of setfit vary when detecting different types of offensive content (e.g., offensive targetediinsult group, offensive untargeted, not offensive, not tamil, offensive targeted insult individual, offensive targeted insult others) within code-mixed tamil-english online discourses? rq3 how does the performance of the nlp models (setfit, mbert, lstm, bert, in-dicbert, labse) differ in terms of precision, recall, and f1-score when identifying offensive content in the code-mixed tamil-english dataset? the findings from our research can be beneficial for developing nlp models that are more effective in detecting various offensives in code-mixed languages, which can help mitigate the harmful effects of such language on online platforms. the findings can benefit further research and practitioners developing nlp applications for social media and online platforms. additionally, developing nlp models capable of detecting various offensives in code-mixed languages can help mitigate the harmful effects of such language in online platforms, promote safer and more inclusive online spaces, and contribute to building a more equitable and inclusive society. ## related works in this related work section, we review the existing literature on this topic, focusing on studies explicitly addressing the problem of various offensive language detection in code-mixed text. we also discuss the existing approaches to code-mixed text classification, including languagespecific and language-agnostic models, and highlight the strengths and weaknesses of each approach. offensive language identification is a task that has been receiving considerable attention in recent years. however, identifying offensive spans in long sentences has been a challenging problem, especially for context-dependent ones. to address this issue, a study @xcite evaluated several models, including bi-lstm crf, muril, and lime. the rationale extraction-based approach involving a combination of muril and lime performed significantly better than the other models. however, the bi-lstm crf model was sensitive toward shorter sentences and performed worse than the random baseline. extracting offensive spans for long sentences was also found to be complicated. in the future, the researchers plan to re-do the offensive span identification task, where participants will be required to identify offensive spans while simultaneously classifying different types of offensiveness. the researchers also plan to release this study's baseline models and datasets to encourage further research. @xcite ) propose a model called told (tamil offensive language detection) that detects offensive language in code-mixed social media comments written in tamil and english. the study uses multilingual bert (mbert) with feature-based selection to generate contextualized word embeddings and selects relevant features using the chi-square test. the proposed told model outperforms the accuracy, precision, recall, and f1 score of existing models. the study also analyzes the importance of various features and finds that character-based features and part-of-speech tags are highly effective in detecting offensive language. the study demonstrates the effectiveness of using mbert with feature-based selection in identifying and moderating offensive language in social media platforms, promoting a safer and more inclusive online environment. ravikiran and annamalai (2021) @xcite present a new dataset called dosa (dravidian code-mixed offensive span identification dataset) for identifying the offensive language in code-mixed social media comments in four dravidian languages: tamil, telugu, kannada, and malayalam. the authors collected and annotated the dataset with offensive spans, the smallest text segment containing an offensive word or phrase. the dataset contains 7,500 comments and 15,547 offensive spans, and multiple annotators did the annotations to ensure high inter-annotator agreement. the paper also provides baseline experiments on the dataset using various models and shows the effectiveness of the dataset in identifying offensive language in code-mixed comments in dravidian languages. chakravarthi et al. (2021) @xcite present a study on sentiment analysis of codemixed text in four dravidian languages: tamil, telugu, kannada, and malayalam. the authors collected a dataset of code-mixed social media comments and used various preprocessing techniques to prepare the data for sentiment analysis. they then used various machine and deep learning models to perform sentiment analysis and evaluated the performance using various metrics. the results showed that deep learning models such as lstm and gru outperformed traditional machine learning models in sentiment analysis of code-mixed text in dravidian languages. the study provides insights into the challenges and opportunities in sentiment analysis of code-mixed text in dravidian languages. it can help develop practical tools for sentiment analysis in these languages. @xcite present an approach for offensive language identification in code-mixed tamil using a transformer-based model. the authors describe their participation in the dlrg shared task at the dravidian language technologies workshop 2021, which aimed to develop models for identifying the offensive language in code-mixed social media comments. the authors used a transformer-based model called roberta and fine-tuned it on the provided dataset. they also used data augmentation techniques to improve the performance of the model. the results showed that their approach achieved high accuracy in identifying the offensive language in code-mixed tamil, outperforming the baseline models provided by the shared task. the study demonstrates the effectiveness of transformer-based models for offensive language identification in code-mixed tamil and provides insights into developing models for similar tasks in other dravidian languages. ## proposed methodology: in this section, we described the experimental setup for our experimental work on classifying different types of offensive content in code-mixed text data in tamil and english @xcite . figure 2 illustrates the workflow of the general architecture of the offensive language detection framework. ## results and discussion in this experimental work, we proposed sentence transfer fine-tuning (setfit) to detect offensive content in code-mixed languages. it achieved an impressive 89% accuracy on the test set, surpassing other models like lstm, bert, mbert, in-dicbert, and labse, which recorded accuracies of 76.3%, 78%, 86%, 80%, and 84.5% respectively. figure 5 and while a thorough analysis of setfit's varied performance in identifying different kinds of offensive content is pending, its overall efficiency solidifies its position as a promising approach for improving nlp systems in code-mixed language contexts, aligning with the objectives outlined in research question 2 (rq2). further analysis conducted to examine the impact of dataset size on model efficacy revealed that setfit maintained consistent performance when the dataset size was increased from 10k to 35k samples. this finding suggests that setfit is adept at handling larger datasets effectively. these results endorse setfit as a highly effective method for classifying various forms of offensive code-mixed text. the high accuracy achieved by setfit likely stems from its ability to capture the complex linguistic patterns within the dataset effectively. moreover, our findings stress the importance of employing large datasets for training models in this specialized area. ## conclusion in conclusion, our experimental work, which introduced the sentence transfer fine-tuning (setfit) approach, marks a significant advancement in natural language processing (nlp), particularly in code-mixed languages. the primary objective of our research was to enhance the detection of offensive content in such languages, a challenge that has grown increasingly relevant in today's digital communication landscape. additionally, our analysis of the impact of dataset size on model performance revealed a key strength of setfit: its ability to maintain stable performance with the increase in dataset size from 10k to 35k samples. this robustness in handling larger datasets is crucial, especially in code-mixed language processing, where data variability is high. setfit's high accuracy is attributed to its advanced capability to capture the complex linguistic patterns inherent in the dataset. this highlights the importance of using comprehensive and large datasets in training models for this domain. our research indicates that setfit is a highly effective and reliable method for classifying different types of offensive content in code-mixed text. its performance provides a new benchmark in the field and opens avenues for future research and development in creating more sophisticated and nuanced nlp systems for multilingual and culturally diverse digital communications.