ROFF - A R omanian T witter Dataset for Offensive Language (2021)

Abstract: This paper describes the annotation process of an offensive language data set for Romanian on social media. To facilitate comparable multi-lingual research on offensive language, the annotation guidelines follow some of the recent annotation efforts for other languages. The final corpus contains 5000 micro-blogging posts annotated by a large number of volunteer annotators. The inter-annotator agreement and the initial automatic discrimination results we present are in line with earlier annotation efforts.

## introduction the use of words to hurt others is a curious aspect of natural languages. besides the scientific curiosity, however, certain forms of offensive language can be harmful to individuals, may have discriminatory, toxifying effects on the society, and can be against law. an annotated corpus of offensive language is, hence, a valuable resource for understanding, identifying and preventing offensive content, particularly in online communication. this paper describes such an annotated corpus of romanian on social media. the study of offensive content online goes back to the earlier days of the internet @xcite . however, identifying various forms of offensive or abusive language online, such as hate speech or verbal harassment, has attracted considerable recent interest. the increased interest is evidenced by a number of recent shared tasks @xcite @xcite to the extent that offenseval 2020 shared task in semeeval evaluation campaign @xcite received submissions from 145 teams (out of 528 registered teams). an important ingredient of these studies is an annotated corpus. recent annotations efforts with the aim of detecting offensive language typically focus on a particular form of offensive content, such as hate speech @xcite @xcite @xcite @xcite to name a few), or cyberbullying @xcite @xcite @xcite . however, it is often difficult to study a particular form of offensive language in isolation @xcite . fore example, a classifier trained on a data set that annotates only instances of hate speech may detect other forms of offensive language as hate speech. as a result, some of the recent studies annotate multiple forms of offensive language together @xcite @xcite . in particular, the annotation scheme for the olid data set of english tweets by @xcite has been used for annotating a variety of languages, including arabic @xcite , danish, (sigurbergsson and derczynski, 2020), greek, @xcite , and turkish (c ¸öltekin, 2020). although not identical, the labels used in germeval shared tasks @xcite are also similar to the label set used by these data sets. in this study we use a similar label set and follow similar guidelines for annotation of romanian tweets. following earlier annotation efforts for offensive language, we try to maximize the number of offensive tweets by obtaining part of the tweets to annotate using a list of offensive words, but also include a larger number of randomly selected tweets. the annotations were performed by volunteers. for the remainder of this paper, we first describe the data set and the annotation process, present evaluation of the data and initial experiments with identifying offensive language and types of it for the present data, and conclude with a brief discussion and outlook. ## dataset this dataset, to our knowledge, is the first of its kind for romanian. it includes tweets from a wide range of topics between the second and third week of march in 2020. since covid-19 has been one of the main topics of the year, the data in this dataset consists largely of opinions about the virus, but also includes political slur and day to day tweets. in the following subsections, we provide an overview of how the data was collected and the annotation guidelines and process. ## classification experiments in this section, we report initial classification results on the data set. since we use a 'closed experiment' setup without any external information or data augmentation, the results can be improved. however, we believe the classification scores presented here can serve as a strong baseline. following offenseval shared tasks @xcite , we present results of three separate tasks. the first task, task 1, is identifying whether a given post is offensive or not. in task 2, we aim to identify whether an offensive tweet is targeted or not. finally, task 3 is about classifying the target of a targeted offensive tweet. for all tasks, we use an lstm-based classifier. the model uses words (tokenized using regular expressions) as input. the model first embeds the words in an embedding space of 64 dimensions. the embeddings are initialized randomly, and only learned during training. the embeddings are passed to a single left-to-right lstm layer with 64 units and followed by a dropout rate of 0.10. the representation built by the lstm layer at the final time step is passed to a fully-connected classification layer with a sigmoid or softmax activation depending on the task. model was trained on 50 epochs with the adam optimizer and cross-entropy loss function. the implementation uses python keras library @xcite . the results for all three tasks were presented in ## conclusion & future work we presented a manually annotated corpus of romanian offensive language on twitter. our annotation scheme is compatible with a number of recent offensive language annotation projects for other languages. as a result, the corpus is suitable for multi-lingual and cross-lingual research on analysis or detection of offensive language. overall, the inter-annotator agreement and initial machine learning experiments on the data yield similar results with earlier studies on offensive language. although similar to the earlier studies, the offensive language detection results can definitely be improved using external resources, such as by using word embeddings, or by fine-tuning large pre-trained language models on this data. furthermore, the uniform annotation scheme with other languages may allow cross-lingual transfer through translation and/or use of cross-lingual, shared word or sentence representations. although the data set created in this study comparable to many of the other data sets presented in the field (see @xcite for a recent review), an obvious direction for future research is to increase the size and diversity of the data. most earlier data sets are either annotated by experts, or through crowd sourcing. in this study, we chose to rely on volunteers. even though this is a common practice for finding participants for experiments in many fields, it is not common in annotation projects. even though we did not use any gamification or other means to attract annotators, we received annotations from over 30 annotators. the present method can be applied where a crowd annotation is possible, and potentially result in a higher quality in comparison to the crowd sourcing. an interesting direction for future research is to compare the quality of the data obtained through different methods of recruiting annotators.