Findings of the First Shared Task on Offensive Span Identification from Code-Mixed K annada- E nglish Comments (2024)

Abstract: Effectively managing offensive content is crucial on social media platforms to encourage positive online interactions. However, addressing offensive contents in code-mixed Dravidian languages faces challenges, as current moderation methods focus on flagging entire comments rather than pinpointing specific offensive segments. This limitation stems from a lack of annotated data and accessible systems designed to identify offensive language sections. To address this, our shared task presents a dataset comprising Kannada-English code-mixed social comments, encompassing offensive comments. This paper outlines the dataset, the utilized algorithms, and the results obtained by systems participating in this shared task.

## introduction addressing offensive content holds immense importance for various parties engaged in content moderation, such as social media companies and individuals @xcite . typically, moderation methods involve either human moderators reviewing content to flag offensive material or the use of semi-automated and automated tools employing basic algorithms and predefined block lists @xcite . despite the appearance of content moderation as a straightforward decision between allowing or removing content, this process is complex @xcite . this * work done during graduate school complexity is amplified on social media platforms due to the overwhelming volume of content, making it challenging for human moderators @xcite . with the continuous rise in offensive social media content, particularly offensive comments and statements, there's a preference for semi-automated and fully automated content moderation approaches @xcite @xcite . kannada, an ancient dravidian language, holds a significant historical legacy @xcite . predominantly spoken in the indian state of karnataka, kannada serves as the official language in the state, carrying cultural significance that extends beyond regional boundaries @xcite . with the emergence of digital communication platforms, code-switching has also found its way into kannada discourse, especially in informal online exchanges. this blending of languages and linguistic variations within social media has led to the integration of code-switched content in discussions, including offensive content, impacting the nature of online conversations in kannada-speaking communities. despite recent advancements in natural language processing (nlp), addressing offensive code-mixed content in dravidian languages, including kannada, remains challenging due to limitations in available data and tools @xcite . however, there has been a noticeable surge in research focused on offensive code-mixed texts in dravidian languages @xcite , although few of these studies concentrate on pinpointing the specific segments within a comment that render it offensive @xcite . identifying these specific segments could significantly aid content moderators and semi-automated tools that prioritize the detection and categorization of offensive content. the existing body of research on identifying offensive spans primarily stems from the works of @xcite . post this there are multiple iterations of shared tasks focusing on offensive span identification in tamil ravikiran and annamalai (2021); lekshmiammal et al. (2022); rajalakshmi et al. (2022); ravikiran et al. (2023 ). however to date there are no works in code-mixed kannada language. to address this gap, we introduced the first phase of code-mixed social media text in kannada, encompassing offensive segments. we invited participants to develop and submit systems under two distinct settings for this collaborative task. our codalab website foot_0 will remain open to encourage further research in this domain. ## task description our task of offensive span identification required participants to identify offensive spans i.e, character offsets that were responsible for the offensive of the comments, when identifying such spans was possible. to this end, we created two subtasks each of which are as described. ## dataset for this shared task, we build on top of the dataset from earlier work of ravikiran and annamalai 4 competition phases ## system descriptions overall we received only a total of 14 submissions from 7 teams all these were only for subtask 1. no submissions were made for subtask 2. ## evaluation this section focuses on the evaluation framework of the task. first, the official measure that was used to evaluate the participating systems is described. then, we discuss benchmarking of overall results. finally we present remark on the approaches used and the analysis of the results from these submitted systems. in line with work of pavlopoulos et al. (2021) each system was evaluated f1 score computed on character offset. for each system, we computed the f1 score per comments, between the predicted and the ground truth character offsets. following this we calculated macro-average score over all the 444 test comments. if in case both ground truth and predicted character offsets were empty we assigned a f1 of 1 other wise 0 and vice versa. the overall results of benchmarked systems are as shown in throughout this shared task we can see the trend to shift more towards language specific pretrained language models. especially top three systems all employ language models. meanwhile explainable ai method finds its place inside the rank list with ensemble of simple classifiers. at the same team few teams employed significant preprocessing indeed leading to improvement in results. besides, we also see that bi-lstm methods are still there in the overall list. ## conclusion in this research, we initiated a first shared task focused on identifying offensive spans within codemixed kannada-english text. unlike our previous attempt, we worked with 2k+ social media comments that were annotated to pinpoint offensive sections. among 45 registered participants, 7 teams submitted their systems. we detailed their approaches in our study and discussed their respective outcomes. notably, a strategy that employed pretrained language models and explainable ai have shown the best results. conversely, the lstm model performed notably worse particularly dis-playing sensitivity to offensiveness. we've made the data and related information publicly accessible to support future investigations. looking ahead, our plan involves revisiting the identification of offensive spans within a multitask framework, encompassing various forms of offensiveness alongside the identification of offensive language spans for kannada.