On the Errors in Code-Mixed T amil- E nglish Offensive Span Identification (2023)

Abstract: In recent times, offensive span identification in code-mixed Tamil-English language has seen traction with the release of datasets, shared tasks, and the development of multiple methods. However, the details of various errors shown by these methods are currently unclear. This paper presents a detailed analysis of various errors in state-of-the-art Tamil-English offensive span identification methods. Our study reveals the strengths and weaknesses of the widely used sequence labeling and zero-shot models for offensive span identification. In the due process, we identify data-related errors, improve data annotation and release additional diagnostic data to evaluate models’ quality and stability. Disclaimer: This paper contains examples that may be considered profane, vulgar, or offensive. The examples do not represent the views of the authors or their employers/graduate schools towards any person(s), group(s), practice(s), or entity/entities. Instead, they emphasize the complexity of various errors and linguistic research challenges.

## introduction offensive span identification from code-mixed tamil-english social media comments @xcite focuses on extracting character offsets corresponding to tokens contributing to offensiveness. identifying such offensive spans is helpful in multiple facets ranging from assisting content moderators for quicker moderation to the development of semi-automated tools which can provide thorough attribution related to the intervened offensive content. recently there are numerous methods @xcite that are capable of identifying these offensive spans with accuracy as high as 60% on very hard-to-understand short sentences with limited contextual information. however many of these methods rely on large code-mixed datasets @xcite @xcite and pre-trained language models (ravikiran and annamalai, 2021). nevertheless, these methods are still far away from solving offensive span identification despite such large success. to advance further with this, we need to understand better the sources of errors in the offensive span identification. such an analysis will, in turn, help introduce inductive biases to extract the spans effectively. thus, we analyze errors on the tamil-english code-mixed offensive span identification dataset (dosa-v2) which consists of 4816 (train) and 876 (test) offensive comments obtained from youtube movie trailers with span annotations @xcite . specifically, this work focuses on models' prediction errors and data-related errors. for the former case, we comprehensively investigate the predictions of 8 different models that currently exist for offensive span identification. accordingly, we find that all the existing models suffer from issues ranging lack of identification of words or phrases that are commonly used to making mistakes due to context ambiguity. based on this, we create eight different error categories suitable to measure the quality of models' predictions. in the latter case, we find very few works to focus on error analysis of offensive span identification, with a predominant concentration on the english language @xcite . additionally, some works focus on error analysis of sequence labeling method @xcite @xcite , but not from the point of offensive spans. in this work, in line with @xcite we use human intervention for error analysis. more specifically, we create multiple error analysis teams consisting ## methods in this work, we start our analysis by reproducing selected models for the dosa-v2 dataset. following this, the models' errors and errors in the test dataset itself are analyzed multiple times across each sentence. after reviewing the various errors, we define different error categories that help identify and diagnose common and important errors (section 2.2). finally, we re-annotate the dataset based on identified dataset errors to find a few improvements in overall results (section 2.4). ## conclusion overall in this work, we studied errors in offensive span identification models. to this end, we considered both zero-shot and supervised sequence labeling approaches. we started with analyzing predictions of 8 different models and creating various error categories. based on the analysis, we re-annotated the dosa-v2 test set and rebenchmarked the results to find the re-annotation was fruitful in improving the outcomes of sentences with less than 30 characters simultaneously highlighted the failure of methods across large sentences. we additionally developed diagnostic datasets to assist in identifying critical errors. finally, we discussed some of our key findings, which could be adapted in future works, including developing metrics that effectively capture models' performance development of cross-domain data and knowledge sources for context understanding.