Findings of the Shared Task on Offensive Span Identification from C ode-Mixed T amil- E nglish Comments (2022)

Abstract: Offensive content moderation is vital in social media platforms to support healthy online discussions. However, their prevalence in code-mixed Dravidian languages is limited to classifying whole comments without identifying part of it contributing to offensiveness. Such limitation is primarily due to the lack of annotated data for offensive spans. Accordingly, in this shared task, we provide Tamil-English code-mixed social comments with offensive spans. This paper outlines the dataset so released, methods, and results of the submitted systems.

## introduction combating offensive content is crucial for different entities involved in content moderation, which includes social media companies as well as individuals @xcite . to this end, moderation is often restrictive with either usage of human content moderators, who are expected to read through the content and flag the offensive mentions @xcite . alternatively, there are semi-automated and automated tools that employ trivial algorithms and block lists @xcite . though content moderation looks like a one-way street, where either it should be allowed or removed, such decision-making is fairly hard. this is more significant, especially on social media platforms, where the sheer volume of content is overwhelming for human moderators especially. with ever increasing offensive social media contents focusing "racism", "sexism", "hate speech", "aggressiveness" etc. semi-automated and fully automated content moderation is favored @xcite @xcite . however, most of the existing works @xcite @xcite are restricted to english only, with few of them permeating into research that focuses on a more granular understanding of offensiveness. tamil is a agglutinative language from the dravidian language family dating back to the 580 bce @xcite . it is widely spoken in the southern state of tamil nadu in india, sri lanka, malaysia, and singapore. tamil is an official language of tamil nadu, sri lanka, singapore, and the union territory of puducherry in india. significant minority speak tamil in the four other south indian states of kerala, karnataka, andhra pradesh, and telangana, as well as the union territory of the andaman and nicobar islands @xcite @xcite @xcite . it is also spoken by the tamil diaspora, which may be found in malaysia, myanmar, south africa, the united kingdom, the united states, canada, australia, and mauritius. tamil is also the native language of sri lankan moors. tamil, one of the 22 scheduled languages in the indian constitution, was the first to be designated as a classical language of india @xcite @xcite . tamil is one of the world's longest-surviving classical languages. the earliest epigraphic documents discovered on rock edicts and "hero stones" date from the 6th century bc. tamil has the oldest ancient non-sanskritic indian literature of any indian language @xcite . despite its own script, with the advent of social media, code-switching has permeated into the tamil language across informal contexts like forums and messaging outlets @xcite @xcite . as a result, codeswitched content is part and parcel of offensive conversations in social media. despite many recent nlp advancements, handling code-mixed offensive content is still a challenge in dravidian languages @xcite including tamil owing to limitations in data and tools. however, recently the research of offensive code-mixed texts in dravidian languages has seen traction @xcite @xcite ). yet, very few of these focus on identifying the spans that make a comment offensive @xcite . but accentuating such spans can help content moderators and semi-automated tools which prefer attribution instead of just a systemgenerated unexplained score per comment. accordingly, in this shared task, we provided code-mixed social media text for the tamil language with offensive spans inviting participants to develop and submit systems under two different settings. our codalab website foot_0 will remain open to foster further research in this area. ## task description our task of offensive span identification required participants to identify offensive spans i.e, character offsets that were responsible for the offensive of the comments, when identifying such spans was possible. to this end, we created two subtasks each of which are as described. example of offensive span is shown in figure 1 3.1 subtask 1: supervised offensive span identification given comments and annotated offensive spans for training, here the systems were asked to identify the offensive spans in each of the comments in test data. this task could be approached as supervised sequence labeling, training on the provided posts with gold offensive spans. it could also be treated as rationale extraction using classifiers trained on other datasets of posts manually annotated for offensiveness classification, without any span annotations. ## dataset for this shared task, we build upon dataset from earlier work of @xcite , which originally released 4786 code-mixed tamil-english comments with 6202 offensive spans. we released this dataset to the participants during training phase for model development. meanwhile for testing we extended this dataset with new additional annotated comments. in line with earlier works (ravikiran and annamalai, 2021) for the 3742 comments we create span level annotations where at least two annotators annotated every comment. additionally, we also employ similar guidelines for annotation, anonymity maintenance etc. besides, no annotator data was collected other than their educational background and their expertise in the tamil language. additionally, all the annotators were informed in prior about the inherent profanity of the content along with an option to withdraw from the annotation process if necessary. for annotation, we use doccano @xcite which was locally hosted by each annotator. within doccano, all the annotators were explicitly asked to create a single label called cause with label id of 1, thus maintaining consistency of annotation labels. (see figure 2 ). to ensure quality each annotation was verified by one or more annotation verifier, prior to merging and creating gold standard test set. the overall dataset statistics is given in the ## system descriptions overall we received only a total of 4 submissions (2 main + 2 additional) from two teams out of 30 registered participants. all these were only for subtask 1. no submissions were made for subtask 2. each of their respective systems are as described. ## evaluation this section focuses on the evaluation framework of the task. first, the official measure that was used to evaluate the participating systems is described. then, we discuss baseline models that were selected as benchmarks for comparison reasons. finally, the results are presented. ## analysis and discussion overall we were happy to see the degree of involvement in this shared task with multiple participants registering, requesting access to datasets and potential baseline codes for the shared task. though only two teams submitted the systems, the resulting diversity of approaches to this problem is fairly encouraging. however we include some of our observations below, from our evaluation and what we have learned from the results. ## conclusion overall this shared task on offensive span identification we introduced a new dataset for codemixed tamil-english language with total of 5652 social media comments annotated for offensive spans. the task though has large participants, eventually had only two teams that submitted their systems. in this paper we described their approaches and discussed their results. surprisingly rationale extraction based approach involving combination muril and lime performed significantly well. meanwhile bi-lstm crf model was found showing sensitivity towards shorter sentences, though it performed significantly worse than the random baseline. also extracting offensive spans for long sentences were found to be difficult especially as they are context dependent. to this end, we release the baseline models and datasets to foster further research. meanwhile in the future we plan to re-do the task of offensive span identification where we could require the participants to identify offensive spans and simultaneously classify different types of offensiveness.