DALC : the D utch Abusive Language Corpus (2021)

Abstract: As socially unacceptable language become pervasive in social media platforms, the need for automatic content moderation become more pressing. This contribution introduces the Dutch Abusive Language Corpus (DALC v1.0), a new dataset with tweets manually an- notated for abusive language. The resource ad- dress a gap in language resources for Dutch and adopts a multi-layer annotation scheme modeling the explicitness and the target of the abusive messages. Baselines experiments on all annotation layers have been conducted, achieving a macro F1 score of 0.748 for binary classification of the explicitness layer and .489 for target classification.

## introduction the growth of online user generated content poses challenges to manual content moderation efforts @xcite . in a 2016 eurobarometer survey, 75% of people who follow or participate in online discussions have witnessed or experienced abuse, threat, or hate speech. 1 the increasing polarization of online debates and conversations, together with the amount of associated toxic and abusive behaviors, call for some form of automatic content moderation. currently, the mainstream approach in automatic content moderation uses reactive interventions, i.e., blocking or deleting 'bad' messages @xcite . there is an open debate on its efficacy @xcite and on the risks of perpetrating bias and discrimination @xcite . alternative, less drastic, and more interactive methods have been proposed, such as the generation of counternarratives @xcite . in either case, the first step towards full or semi-automatic moderation is the detection of potentially abusive lan-guage. such step relies on language-specific resources to train tools to distinguish the "good" messages from the harmful ones. as a contribution in this direction, we have developed the dutch abusive language corpus, or dalc v1.0, a manually annotated corpus of tweets for abusive language detection in dutch. 2 the resource is unique in the dutch-speaking panorama because of the approach used to collect the data, the annotation guidelines, and the final data curation. dalc is compatible with previous work on abusive language in other languages @xcite @xcite @xcite but presents innovations both with respect to the application of the label "abusive" to messages and the adoption of a multi-layered annotation to distinguish the explicitness of the abusive message and its target @xcite . our contributions can be summarized as follows: previous work on abusive language phenomena and behaviors is extensive and varied. however, limitations exist and they mainly concentrate along three dimensions: (i) definitions; (ii) data sources and collection methods; and (iii) language diversity. the development of automatic methods for detecting forms of abusive language has been rapid and has seen a boom of definitions, labels, and phenomena being investigated, including racism @xcite @xcite , hate speech @xcite @xcite , toxicity foot_2 and verbal aggression @xcite , misogyny @xcite @xcite , and offensive language @xcite @xcite . variations in definitions and in annotation guidelines have given rise to isolated datasets, limiting the portability of trained systems and reuse of resources @xcite . comprehensive frameworks that integrate and harmonize the variety of definitions and investigate the interactions across the annotated phenomena are still at early stages @xcite . dalc v1.0 is compatible with existing definitions of abusive language and promotes a multi-layered annotation scheme compatible with previous initiatives, with a special attention to the reusability of datasets. collecting good representative data for abusive language is a challenging task. the majority of existing datasets focuses on messages from social media platforms, with twitter being the most used @xcite . unlike other language phenomena, e.g., named entities, abusive language is less widespread and cannot be easily captured by means of random sampling. schematically, we identify three major methods to collect data: namely: (i) use of communities @xcite @xcite which targets online communities known to be more likely to have abusive behaviors; (ii) use of keywords @xcite @xcite @xcite , where manually compiled lists of words corresponding either to potential targets (e.g, "women", "migrants", a.o.) or profanities are employed; (iii) use of seed users @xcite , which collects messages from users that have been identified to post abusive texts via some heuristics. each of these methods has advantages and disadvantages. for instance, the use of keywords may create denser datasets, but at the same time risks of developing biased data are very high @xcite . furthermore, according to the specific platform used, some of the methods cannot be reliably applied. for instance, in a platform like twitter targeting online communities is not trivial. recently, refinements have been proposed to address limitations of each approach. in some cases controversial posts, videos or keywords are used as proxies for communities @xcite , in other cases hybrid approaches are proposed by combining keywords and seed users @xcite , others exploit platform pre-filtering functionalities @xcite . dalc v1.0 integrates different bottom-up approaches to collect data providing a first crossfertlization attempt across two social media platforms and paying attention to minimize the introduction of biases. vidgen and derczynski (2021) provides a comprehensive survey covering 63 datasets all targeting a specific abusive phenomenon/behavior. the majority of them (25 datasets) is for english, with a long tail of other languages mostly belonging to the indo-european family, although limited in their diversity. the lack of publicly available datasets for any sino-tibetan, niger-congo, or afro-asiatic languages is striking. when it comes to abusive language datasets, dutch is less-resourced. notable previous work has been conducted by @xcite who developed a dataset and systems for detecting racist discourse in dutch social media. dalc v1.0 differentiates because it is a "generic" resource for abusive language where all possible types of abusive phenomena are valid. this leaves room to refinement in the proposed corpus to investigate potential sub-types of abusive phenomena and their associated linguistic devices. ## data collection dalc v1.0 is based on a sample of a large ongoing collection of twitter messages in dutch at the university of @xcite . for its construction, rather than focusing individually on any of the mentioned approaches, we propose a combination of three methods that only partially overlap with previous work. ## annotation and data curation dalc v1.0 has been manually annotated using internally developed guidelines. the guidelines provides the annotators with a definition of abusive language that refines proposals in previous work @xcite @xcite . in particular, abusive language is defined as: impolite, harsh, or hurtful language (that may contain profanities or vulgar language) that result in a debasement, harassment, threat, or aggression of an individual or a (social) group, but not necessarily of an entity, an institution, an organization, or a concept. notably, this definition requires that an identifiable target must be present in the message to qualify as potentially abusive. this is a necessary requirement in our definition and it also helps us to discriminate abusive language from more generic phenomena like offensive language, forms of harsh criticism, and other socially unacceptable language phenomena. we have specifically introduced harsh criticism to restrict the application of the abusive language label. indeed expressing heavy criticisms against an institution (e.g., the e.u. commission, or a government) may result in inappropriate and offensive language but it does not entail being abusive. exceptions, however, hold: cases of synecdoches where an institution, an entity, or a concept are used to attack the members of a social group are considered instances of abusive language. following @xcite and @xcite we perform a multi-layered annotation distinguishing the levels of explicitness of the abusive messages and the targets. explicitness combines three factors: (i) the surface evidence of the message; (ii) the assumed intentions of the user (i.e., is the message debasing someone?); and (iii) its effects on the receiver(s) (i.e., can the message be perceived as debasing by a targeted individual or a community?). while the last two factors (intentions and effects) help to identify the abusiveness nature of the message, the surface forms is essential to distinguish overtly abusive messages from more subtle forms. a distinguishing criterion, in fact, is the presence of profanities, slurs, and offensive terms. we define three values: a further differentiating criteria is that all messages where the author debases or offends him-/herself (e.g., messages that contain the first person singular or plural pronoun) are considered as not abusive the target layer makes explicit to whom the message is directed. we reuse the values and definitions from @xcite . in particular, we have: the annotation has been conducted in two phases. phase 1 (march-may 2020) has seen five annotators, all bachelor students in information science. the students conducted the annotation of the data as part of their bachelor thesis project. phase 2 (november-december 2020) has been conducted by one master student in information science with previous experience in this task. all annotators are native speakers of dutch. more details are reported in the data statement a. during phase 1, we validate the annotation guidelines by means of a pairwise inter-annotator agreement (iaa) study on two independent subsets of 100 messages each. the first sample is obtained using the keyword method and the second using the geolocation. for the keywords sample, cohen's kappa is 0.572 for the explicitness and 0.670 for the target. for the geolocation sample, the kappa for explicitness is comparable (0.522) although that for target is lower (0.466). the results are comparable previous work @xcite indicating substantial agreement. cases of disagreement have been discussed between the annotators and resolved. the data used for the iaa has been integrated in dalc v1.0. no iaa has been computed for the messages collected using seed authors. in phase 2 we further expanded the initial data annotation. the final corpus has been manually curated by one of the authors of this paper. the data curation phase focuses on the creation of the train, dev, test splits in such a way that there is no overlap for time periods and, most importantly, users. ## baselines we present a set of baseline experiments that accompany the release of dalc v1.0 for the two annotation layers. for the explicitness layer, we first experiment a simplified setting by framing the problem as a binary classification task. in this setting the distinction between exp and imp labels is collapsed into a new unique value for all abusive messages (i.e., abu). the follow-up experiment, on the other hand, maintains the fine-grained distinction in the three classes (i.e., exp vs. imp vs. ## conclusions and future work this paper introduces dalc v1.0, the first "generic" resource for abusive language detection in dutch. dalc v1.0 contains more thn 8k twitter messages manually labeled using a multi-layer annotation scheme targeting the explicitness of the message and the targets. a further peculiarity of the dataset is the complete lack of overlap for time periods and users between train and test splits, making the task more challenging. the combination of multiple data collection strategies aims at promoting new bottom-up approaches less prone to additional biases in the data other than those from the manual labeling. dalc v1.0 adopts a definition of abusive language and an annotation philosophy compatible with previous work, paying attention to promote interoperability across language resources, languages, and abusive language phenomena. the baseline experiments and systems that have been developed further indicate the challenges of this dataset. the best results are obtained with a fine tuned transformer-based pre-trained language model, bertje. fine-grained distinction for the explicitness layer is particularly difficult for implicitly abusive messages. furthermore, target classification is a challenging task, with overall macro-f1 below 0.50. future work will focus on an in-depth investigation of the errors to identify easy and complex cases. a. curation rationale the corpus is composed by tweets in dutch extracted using different strategies and covering different time windows.