Findings of the Second Shared Task on Offensive Span Identification from Code-Mixed T amil- E nglish Comments (2023)

Abstract: Maintaining effective control over offensive content is essential on social media platforms to foster constructive online discussions. Yet, when it comes to code-mixed Dravidian languages, the current prevalence of offensive content moderation is restricted to categorizing entire comments, failing to identify specific portions that contribute to the offensiveness. Such limitation is primarily due to the lack of annotated data and open source systems for offensive spans. To alleviate this issue, in this shared task, we offer a collection of Tamil-English code-mixed social comments that include offensive comments. This paper provides an overview of the released dataset, the algorithms employed, and the outcomes achieved by the systems submitted for this task.

## introduction combating offensive content is crucial for different entities involved in content moderation, which includes social media companies as well as individuals @xcite . to this end, moderation is often restrictive with either usage of human content moderators, who are expected to read through the content and flag the offensive mentions @xcite . alternatively, there are semi-automated and automated tools that employ trivial algorithms and block lists @xcite . though content moderation looks * corresponding author, work done during graduate school like a one-way street, where either it should be allowed or removed, such decision-making is fairly hard @xcite @xcite . this is more significant, especially on social media platforms, where the sheer volume of content is overwhelming for human moderators especially @xcite . with ever increasing offensive social media contents focusing on offensive comments and statements semi-automated and fully automated content moderation is favored @xcite @xcite . tamil is an classical ancient language (subalalitha, 2019a; @xcite @xcite with a history dating back to 580 bce @xcite . it is primarily spoken in tamil nadu, india, and also in sri lanka, malaysia, and singapore. tamil holds official language status in tamil nadu, sri lanka, singapore, and the union territory of puducherry @xcite @xcite . additionally, there are significant tamil-speaking communities in kerala, karnataka, andhra pradesh, telangana, and the andaman and nicobar islands. the tamil diaspora is spread across countries across the world and is recognized as a scheduled language in the indian constitution. it has a rich literary tradition dating back to the 6th century bce @xcite , with rock edicts and "hero stones" serving as some of the earliest known written records. de-spite its own script, with the advent of social media, code-switching has permeated into the tamil language across informal contexts like forums and messaging outlets @xcite . as a result, code-switched content is part and parcel of offensive conversations in social media. despite many recent nlp advancements, handling code-mixed offensive content is still a challenge in dravidian languages @xcite including tamil owing to limitations in data and tools. however, recently the research of offensive code-mixed texts in dravidian languages has seen traction @xcite @xcite ). yet, very few of these focus on identifying the spans that make a comment offensive @xcite . however, highlighting these specific spans segments can greatly assist content moderators and semi-automated tools that prioritize identifying and attributing offensive content. in line with this objective, we presented second iteration of code-mixed social media text in tamil, including offensive spans, and invited participants to develop and submit systems under two distinct settings for this shared task. our codalab website foot_0 will remain open to foster further research in this area. ## task description our task of offensive span identification required participants to identify offensive spans i.e, character offsets that were responsible for the offensive of the comments, when identifying such spans was possible. to this end, we created two subtasks each of which are as described. ## dataset for this shared task, we build upon dataset from earlier work of @xcite , which originally released 4786 code-mixed tamil-english comments with 6202 offensive spans. we released this dataset to the participants during training phase for model development. additionally, the test set from the same work with 1006 samples were released for development/validation purposes. meanwhile for testing we extended this dataset with new additional annotated comments. following previous research @xcite , we created span-level annotations for 361 newly selected test comments. we followed the same process and guidelines for annotation, anonymity maintenance etc. profanity in data was explained apriori with an option to withdraw from the annotation process if necessary. to ensure quality each annotation was verified by one or more annotation verifier, prior to merging and creating gold standard test set. the overall dataset statistics is given in the ## system descriptions overall we received only a total of 3 submissions from three teams out of 48 registered participants. all these were only for subtask 1. no submissions were made for subtask 2. each of their respective systems are as described. ## analysis, discussion and remarks in general, we were pleased to witness the level of engagement in this shared task, with numerous participants signing up, expressing interest in obtaining datasets, and seeking potential baseline codes for the project. although only three teams ultimately submitted their systems, the variety of approaches taken to tackle the problem is quite promising. nevertheless, we have included some of our observations below, which stem from our evaluation and the insights gained from the results. benchmark 1 achieves a considerably high score and, hence, is very highly ranked with character f1 of 37.24%. combination of bart with lime interpretability by model ajns is behind benchmark 1 by 9%, indicating the language models ability to not so effectively rationalize and identify the spans. meanwhile dlrg-r1 and dlrg-r2 has large gap compared to random baselines, indicating the proposed approaches by these teams are not suitable for practical use. to this end, these methods employ direct token labeling which is more surprising. ## conclusion in this work, we set up a second shared task that was centred on locating offensive language spans in code-mixed tamil-english text. compared to our earlier iteration, we had 6,153 social media com-ments that were tagged to identify abusive spans. only three teams submitted their systems out of 48 registered participants. we described their strategies in this study and talked about the results they got. it's interesting that a strategy for reason extraction that combines bart and lime was effective but was not able to beat random baseline. the lstm/gru model, on the other hand, performed noticeably worse than the random baseline and showed sensitivity to shorter sentences. we have made the baseline models and information available to the public in order to aid future research. moving forward, we intend to redo the offensive span identification task under multitask setup with identification of different types of offensiveness alongside the offensive spans.