DOSA : D ravidian Code-Mixed Offensive Span Identification Dataset (2021)

Abstract: This paper presents the Dravidian Offensive Span Identification Dataset (DOSA) for under-resourced Tamil-English and Kannada-English code-mixed text. The dataset addresses the lack of code-mixed datasets with annotated offensive spans by extending annotations of existing code-mixed offensive language identification datasets. It provides span annotations for Tamil-English and Kannada-English code-mixed comments posted by users on YouTube social media. Overall the dataset consists of 4786 Tamil-English comments with 6202 annotated spans and 1097 Kannada-English comments with 1641 annotated spans, each annotated by two different annotators. We further present some of our baseline experimental results on the developed dataset, thereby eliciting research in under-resourced languages, leading to an essential step towards semi-automated content moderation in Dravidian languages. The dataset is available in https://github.com/teamdl-mlsg/DOSA

## introduction fighting offensive content is imperative for social media companies and other entities involved in content moderation. currently, much of moderation is relatively limited on most community platforms @xcite with most of them relying on detection of repeatedly used words 1 and block-lists @xcite . additionally, most social media companies employ human content moderators, who are frequently swamped by offensive mentions and their volume @xcite . on the other hand, precise moderation leads to content delay leading to user attrition. furthermore, smaller entities cannot utilize human moderators on a large scale due to their sheer cost. as a result, they shut down their comments sections entirely. although content moderation to some degree has utilized semi-automated approaches @xcite , most of them are not yet available for non-english languages and code-mixed texts. code-switching or code-mixing is a mixing of linguistic units from two or more languages in a single conversation or sometimes even a single utterance and is widely used across the world @xcite . in india, due to widely employed educational and cultural guidelines, english largely influences all the indian spoken languages, including dravidian languages like @xcite . however, with the advent of social media, code-switching has permeated to mediums with informal contexts like forums and messaging platforms. as a result, code-switching is part and parcel of offensive conversations in social media. despite many recent nlp advancements, handling code-mixed offensive content is still a challenge in dravidian languages @xcite . the primary reason is data scarcity, as it appears relatively less in standard textual resources and instead spread across the world wide web. however, recently the research of offensive code-mixed texts in dravidian languages has seen traction @xcite . however, these are restricted to the whole comment's classification for offensiveness and do not identify the spans that make a text offensive. but emphasizing such offensive spans can assist human moderators who often deal with lengthy comments and prefer attribution instead of just a systemgenerated unexplained score per post. accordingly, the contributions of this paper are as follows the rest of the paper is organized as follows. in section 2, we discuss literature on offensive language and span identification. following this in section 3, we present the dataset collection and annotation process. section 4 offers the experimental setting used for the baseline creation. in section 5, we discuss our results and errors so identified. finally, in section 6, we conclude with a summary and possible directions for future work. ## dataset collection and annotation in this work, we reuse tamilmixsentiment (chakravarthi et al., 2020), and kancmd (hande et al., 2020) datasets consisting of 15k and 7k youtube code-mixed comments respectively in tamil-english and kannada-english languages. reusing the existing dataset is beneficial. it encourages the development of multitask models with span identification as one of the tasks, analysis of model interpretability during offensive language identification, and developing a unified benchmark dataset for multiple nlp tasks in code-mixed dravidian languages. in this work, we considered only a subset of the comments that were already annotated as offensive foot_2 for our span annotation process. out of this subset, we rechecked and removed non-code mixed comments resulting in 9049 and 1311 comments in tamil-english and kannada-english, respectively. for the final annotation process, we considered all of the code-mixed kannada-english comments and 5000 tamil-english comments. ## experimental settings to establish a baseline performance, we applied multiple state-of-the-art multilingual language models to determine the span of offensive comments. in this section, we present various models, hyperparameters, and other experimental settings used as part of the baseline estimation. ## experiments, results, and discussion the experimental results for various state-of-theart multilingual language models are as shown in tables 4 5 6 7 8 9 . since the focus of these experiments is to just establish baselines and provide some starting pointers for further exploration, we restrict ourselves from in-depth error analysis and instead focus on unique errors which we came across during the experiments. to start with, we compute results for each of the fold where we identify span/entity level precision (p), recall (r), and f1-score (f1) inline with past works @xcite . computing entity level p, r, and f1 measures consider only those word sequences which precisely match the annotation, thus eliminating partially identified offensive spans. this measure is also in line with pavlopoulos et al. 2021. ## distilbert-multi-finetuned-for-xqua-on-tydiqa 9 xlm-roberta-base 10 xlm-roberta-large-xnli-anli to start with, for both tamil-english and kannada-english code-mixed text, all the models perform poorly with the best average f1 of 0.403 for bert-m2 and xbert-m2, respectively for kannada-english. meanwhile, for tamil-english comments, we found the maximum average f1 of 0.405 for dbert-m1. besides, across all the folds on each language model, the results are in similar ranges. such poor performance can be attributed to two reasons: the training process and the complexity of the code-mixed text. during our experiments, for kannada-english, we found most models tend to overfit, and in some cases, the model training saturated at a training loss of 0.1. for the former case, we employed precision control of the learning rate to control overfitting; however, the net effect was relatively limited due to the small dataset size. however, the latter case tends to be more challenging to handle even after significant learning rate changes. additionally, since the experiment was a baseline, we didn't perform any hyperparameter tuning for other parameters, leaving them for future work. furthermore, unlike kannada-english experiments, we found overfitting for tamil-english only in the case of xlm-roberta models. moreover, for both kannada-english and tamil-english, we found that comments with only one or more profane words causing offensiveness were correctly identified in their spans. however, for both tamil-english and kannada-english, we can see overall lower results for which one of the reasons we thought was the nature of code-mixed text themselves. to verify this, we cross-check the errors and found the following issues. complete sentence annotations -for cases where the complete comment or more than 70% of the characters are accounted in offensive span, we found the errors to be highest where one or more words are not tagged as offensive, leading to a drop in span level f1. example sentences with ground truth and predicted spans are as shown below. ground truth: sir intha cinema madida mele dodda mattadalli prachara madbekittu. nnodi prem avru dabba film madudru hype build-up create madodralli etthida kai. prediction: sir intha cinema madida mele dodda mattadalli prachara madbekittu. nnodi prem avru dabba film madudru hype build-up create madodralli etthida kai. translation: sir, after making this kind of movie, there was need of more publicity. please see mr. prem, even after making useless movie still he is king in creating hype. word pronunciation -another unique case of errors involves words that are the same except the texts are written differently. these are again not correctly identified as offensive. in the example below both devidya and thevidya translates to whore, which is often used as an abuse word. sarcastic sentences: sarcastic comments where the complete sentence is annotated for spans were also cases where the model fails to work. one such example is as shown with ground truth and prediction. ground truth: puratchi thalavi kamika sona tun tun aunty kamikerenga. prediction: puratchi thalavi kamika sona tun tun aunty kamikerenga translation: rather than showing purtachi thalavi (honorable nick name given to former tamil nadu cm), why are you showing tun tun aunty (a cartoon character in chota bheem). ## conclusion in this paper, we presented dosa, a dataset for offensive span identification for tamil-english and kannada-english code mixed texts. we also achieved an inter-annotator agreement of 0.6 for the annotations. we also created baselines and reported p, r, and f1 for span identification using the developed dataset and state-of-the-art multilingual language models. in the due process, we presented some of the challenges in the training language model-based baselines, which is a possible future work. moreover, in this work, we did not present results on simpler models like lstm-crf and its variants, which is also a possible exploration. most importantly, we could see cases where the complete comment was annotated either due to their sarcastic nature or because they had only offensive terms. in this regard, a possible question to explore includes evaluating the need for such annotations by considering larger datasets and improving models' performance under such conditions. additionally, we think this resource is useful for multitask learn-ing and interpretability for code-mixed offensive language models.