Identifying Aggression and Offensive Language in Code-Mixed Tweets: A Multi-Task Transfer Learning Approach (2025)

Abstract: The widespread use of social media has contributed to the increase in hate speech and offensive language, impacting people of all ages. This issue is particularly difficult to address when the text is in a code-mixed language. Twitter is commonly used to express opinions in code-mixed language. In this paper, we introduce a novel Multi-Task Transfer Learning (MTTL) framework to detect aggression and offensive language. By focusing on the dual facets of cyberbullying, aggressiveness and offensiveness, our model leverages the MTTL approach to enhance the performance of the model on the aggression and offensive language detection. Results show that our Multi-Task Transfer Learning (MTTL) setup significantly enhances the performance of state-of-the-art pretrained language models, BERT, RoBERTa, and Hing-RoBERTa for Hindi-English code-mixed data from Twitter.

## introduction social media encompasses a variety of internetbased applications that enable people to connect globally and share user-generated content. platforms like twitter and facebook are among the most popular applications on the internet today. however, there has been a significant rise in bullying behavior on these platforms, including snide remarks, abusive language, personal attacks, and even threats of rape and violence, impacting children, individuals, and communities. this situation underscores the need for technological advancements to automatically detect offensive content and create safer environments. machine learning models, leveraging recent techniques in natural language processing, can be utilized to effectively identify such harmful behaviors. in countries where english is not the native language, such as india, most social media users communicate using at least two languages, predominantly english and hindi. these texts are classified as bilingual. in a bilingual context, an entire post may be written in the script of one language while incorporating words from both languages, a phenomenon known as code-mixed (or mixed-code) text. in this paper, we introduce a pioneering multi-task transfer learning (mttl) framework aimed at identifying aggression and offensive language in hindi-english code-mixed tweets. our method delves into the correlation between aggression and offensive language. as illustrated in figure 1 , it reveals that offensive language frequently accompanies expressions of aggression, suggesting an inherent connection between the two. we validate our mttl framework using the dataset provided for the seventh workshop on online abuse and harms (woah) @xcite . derived from twitter, this dataset classifies tweets based on two primary dimensions of cyberbullying: aggressiveness and offensiveness. each tweet is annotated with the following labels. the key contributions of this work are the following: the rest of the paper is organized as follows. section 2 presents the associated literature. section 4 describes the proposed mttl approach and associated loss function. section 3 describes the dataset. section 5 presents the experimental setup. section 6 elaborates the results and section 7 concludes the paper. ## related work previous research on aggression/hate speech detection has explored various approaches. these include a unified multi-modal deep learning architecture that integrates deep pyramid cnn, pooled bilstm, and disconnected rnn (khandelwal and kumar, 2020). additionally, studies have investigated the utilization of word-level semantic information and sub-word knowledge to counter character-level adversarial attacks @xcite . another approach involves a tabnet classifier-based model trained on features extracted by muril from transliterated code-mixed data, which has demonstrated efficacy even with devanagari text @xcite . moreover, techniques such as data balancing using generative pretrained transformer (gpt-2) have been explored due to its contextual understanding and capability for more realistic data generation @xcite . recent studies on offensive language detection have explored different machine learning algorithms and n-gram feature sets to identify offensiveness in social media messages @xcite . additionally, researchers have combined various multilingual transformer-based embedding models with machine learning classifiers to detect hate speech and offensive language in code-mixed text in dravidian languages @xcite . furthermore, leveraging lstm architecture, zypherand, openchat-3.5, along with prompt engineering and qlora, has shown promising potential in addressing the challenges of hate and offensive comment classification @xcite . research on multi-task learning and transfer learning has explored various methodologies. these include proposing an unsupervised multitask learning network that estimates bullying likelihood using a gaussian mixture model @xcite , utilizing cross-lingual contextual word embeddings and transfer learning for predictions in low-resource languages @xcite , enhancing arabert with multitask learning to effectively learn from limited arabic data @xcite , employing multinomial naive bayes for textual data and resnet50 for pictorial data, and integrating the results from both to identify misogynistic memes @xcite . additionally, combining adapterfusion with language adapters on a multilingual large language model (llm) has been explored for classifying code-mixed and code-switched social media text @xcite . moreover, a multi-task model based on the shared-private scheme has been proposed to capture both shared and task-specific features @xcite . in this paper, we also introduce a multi-task transfer learning approach, leveraging the intrinsic relationship between aggression and offensive language. ## dataset and preprocessing the dataset @xcite consists of 10000 tweet ids, each labeled with offensiveness labels (off or not) and aggressiveness labels (oag,cag,or nag) in addition with codemixed labels (codemixed or monolingual). we were able to retrieve text from 8281 tweets from the tweet ids provided in the dataset,the remaining tweets were most probably deleted. we partitioned this data randomly into an 80% training set, 10% validation set, and 10% evaluation set. ## proposed model we based our approach on the multi-task model based on the shared-private scheme that captures the shared-features and task-specific features @xcite ) and leverage the pretrained language models that have achieved a state-of-theart performance in multiple hindi-english nlp tasks. our best model is based on augmenting the pretrained language model with task-specific layers and sharing the knowledge between them through transfer learning to achieve multi-task learning. we chose this approach to explore the relationship between aggressiveness and offensiveness of the text, and the results are more impressive than the models that achieved state-of-the-art performance in detecting aggression and offensive language from the text. 4.1 multi-task transfer learning (mttl) multi-task learning (mtl) is an approach in machine learning where a model is trained simultaneously on multiple tasks. by sharing representations between related tasks, the model can often improve performance on individual tasks compared to training separate models for each task. the core idea is that learning to perform multiple related tasks can help a model generalize better because it captures commonalities and differences among the tasks. transfer learning (tl) is a technique where a model developed for a particular task is reused as the starting point for a model on a second task. it leverages the knowledge gained while solving one problem and applies it to a different but related problem. the key idea behind multi-task transfer learning (mttl) is to combine the ideas of multi-task learning and transfer learning. this approach transfers the knowledge learned from multiple source tasks to improve learning for one or more target tasks. the aim is to leverage the shared information between the tasks to enhance the learning efficiency and performance of the target tasks. in our case, we have two sub-tasks, aggressiveness and offensiveness of the text; we employ the mttl approach to augment the pretrained language model such that it can learn both tasks simultaneously, and we mainly focus on optimizing the performance of the model on both tasks by sharing the task-specific knowledge. our mttl model architecture consists of two components, as can be seen in figure 2 . 1. common layers: these layers include the pretrained language model, which is fine-tuned based on the combined weighted loss of both tasks to extract general features representing shared information between the tasks. from figure 1 , we can see that the number of aggression instances is almost the same as the combined task instances, while the number of offensive instances is nearly half of the combined task instances. this explains why adding task-specific hidden layers to the offensive task model helps capture task-specific features effectively, whereas adding such layers to the aggression task model leads to overfitting. ## experimental setup we fine-tune the two tasks using the following pretrained language models: bert @xcite , roberta @xcite which are trained on english data, xlm-roberta @xcite which is trained over multilingual data, hing-roberta (nayak and joshi, 2022) a multilingual language model specifically built for hindi-english code-mixed language as seen in the indian context. these are the state-of-the-art models chosen by the authors of the dataset to evaluate their dataset. we perform the experiments using the huggingface transformers library @xcite . we monitor the validation set's macro-f1 scores to find the best hyper-parameter values, using the following range of values for selecting the best hyperparameter: ## results the individual performance of these models on the two tasks, corresponding with codemixed (hindi+english), monolingual (only english), and combined data (codemixed+monolingual) as input is shown in ## conclusion cyberbullying on social media platforms is a significant issue affecting many individuals, with the diverse languages and dialects in india posing a substantial challenge for automated offensive language detection systems. in this paper, we propose a multi-task transfer learning (mttl) framework enhanced with pretrained language models like hing-roberta to efficiently learn multiple tasks and improve performance in detecting aggression and offensive language in hindi-english code-mixed text. we explored the use of individual weighted loss functions for training task-specific layers and a custom overall loss function for training common layers. our results demonstrate signif-