A Dataset of Offensive G erman Language Tweets Annotated for Speech Acts (2022)

Abstract: We present a dataset consisting of German offensive and non-offensive tweets, annotated for speech acts. These 600 tweets are a subset of the dataset by Struß et al. (2019) and comprises three levels of annotation, i.e., six coarse-grained speech acts, 23 fine-grained speech acts and 14 different sentence types. Furthermore, we provide an evaluation in both qualitative and quantitative terms. The dataset is made publicly available under a CC-BY-4.0 license.

## introduction in recent years, research has invested a considerable amount of effort in offensive language and other phenomena related to online communication. hate speech not only affects individuals or minority groups but it might also threaten social cohesion @xcite . thus, the automatic detection of hate speech and offensive language foot_0 continues to be an important and very relevant research topic. while there is a large body of research in this area, approaches often merely classify text using binary labels @xcite . hate speech classification is in itself a complex task because it is highly subjective what constitutes hate speech. in addition, it is even more difficult to detect hate speech if it is expressed implicitly rather than explicitly @xcite . surprisingly little research exists on the pragmatic characteristics of offensive language. pragmatics is "the study of how utterances have meanings in situations" @xcite . to address this gap and contribute to the improvement of hate speech detection, we analysed the pragmatic characteristics of offensive language. we conducted a speech act analysis of a dataset of german tweets that contain offensive language. we use a subset of the 2019 germeval shared task on the identification of offensive language dataset @xcite . similar studies exist in which speech act theory @xcite is applied to data @xcite @xcite @xcite . however, many of these concentrate on spoken language, often confined to restricted discourse scenarios such as the spaadia trainline corpus @xcite which consists of phone conversations between callcentre agents and customers concerning bookings of train tickets. other approaches apply speech act theory to data from twitter @xcite or reddit @xcite . nevertheless, to our knowledge, only @xcite investigate the distribution of speech acts in a hate speech dataset. consequently, there appears to be a need for more research in this area. our study aims to contribute to automated hate speech detection by providing a pragmatic analysis of offensive language and it also attempts to contribute to speech act theory by testing its applicability on real life data and sharing findings on frequency, syntactical realisation and common sequences of speech acts. we hypothesise the following: (i) there are more directives in offensive than in non-offensive language (excluding address); (ii) there are more expressives of type complain in offensive than in non-offensive language; (iii) speech acts of type assert occur less frequently in offensive than in nonoffensive language; (iv) declarative sentences are the most dominant sentence types overall. hypothesis (i) is based on @xcite ; hypothesis (ii) is motivated due to the assumption that uttering a hateful comment corresponds to the speaker having a negative attitude towards the targeted person or group @xcite . hypothesis (iii) is assumed because of the hypotheses (i) and (ii). if there are more expressives and directives in offensive language, then they might displace a number of assert speech acts. moreover, hypothesis (iv) is expected because we regard declaratives to be the default sentence type @xcite . the remainder of this article is structured as follows. section 2 reports on related work. section 3 provides information about the dataset. section 4 presents our annotation scheme regarding the syntactical and speech act level and in section 5, the results are discussed. section 6 concludes the article. ## related work starting out with the publication of the first hate speech dataset @xcite , a considerable body of research has been carried out in this field of research @xcite , including a number of survey papers @xcite @xcite . while english is the dominant language in current research, datasets in and for other languages also exist, including german @xcite @xcite . with regard to the annotation schemes, we can distinguish three main approaches @xcite , i. e., binary classification @xcite , non-binary classification consisting of more than two labels @xcite and more complex schemes that consist of multiple levels @xcite . several authors examine hate speech for specific linguistic phenomena. neutral adjectives, for example, can acquire a pejorative meaning if they are nominalized @xcite , using the definite plural instead of the bare plural can be used to indicate non-membership @xcite and the use of both including and excluding pronouns may contribute to construct a dichotomy between the in-and the out-group, i. e., distancing or othering @xcite . those findings have contributed to the development of more complex annotation schemes. in addition, @xcite create an annotation scheme based on four questions which include the presence or absence of offensiveness, slurs, adjectival nominalizations and distancing to establish a dataset. regarding speech act theory, nowadays, various studies attempt to classify speech acts automatically. to enable automatic classification, they build speech act taxonomies which are often based on @xcite and @xcite . @xcite , for example, develop a taxonomy based on searle's five speech act classes (assertives, directives, expressives, commissives, declarations) which they validate on a reddit corpus with threads on autoimmune diseases. they also provide sub-classes, resulting in 17 speech acts in total. similar approaches with taxonomies based on searle's classes include @xcite and @xcite . weisser (2018) introduce a more complex scheme with the dialogue annotation and research tool (dart). 2 the current version (version 3.0 foot_2 ) of dart classifies dialogue by syntactic categories (ten in total) and speech acts (162 classes), among others. ## dataset the annotation scheme was applied to a subset of the dataset created for task two of the 2019 germeval shared task on the identification of offensive language @xcite .it consists of offensive and non-offensive german language tweets that do not include any surrounding context in the form of other tweets. evidently, this is not ideal for the application of speech act theory which highly depends on context. this is why we established the category unsure. task two of the germeval shared task included three subtasks. the first subtask uses binary classification (offense, other) and the second subtask uses a more fine-grained classification (profanity, insult, abuse, other). tweets labeled as profanity only contain profane words such as swearwords but do not contain insults or abusive language. if they do, they are labeled as insult or abuse. the category abuse differs from insult insofar as in abusive language, the target functions as a representative of a group and "is ascribed negative qualities that are taken to be universal, omnipresent and unchangeable characteristics of the group" @xcite while tweets labeled insult are only targeted at individuals without being associated with a group. in addition, tweets containing instances of dehumanization are labeled as abuse as well. the third subtask uses binary classification by distinguishing between implicit and explicit offensive language. according to @xcite , offensive language counts as being implicit when the reader needs to infer that the tweet is offensive, as the offense is only implied. moreover, implicit offensive language also entails using figurative language (e. g., sarcasm or irony). the first and second subtask are based on the 2019 data which includes 7,025 tweets while the third subtask is based on the 2018 data only including tweets categorised as abuse or insult. the data for the third subtask consists of 8,541 tweets of which 2,888 are categorised as offensive (393 of these are instances of implicit offensive language) @xcite . the class profanity was excluded from the data used in subtask three as the authors argue that it is "by definition explicit offensive language" @xcite . in both the 2018 and 2019 data, tweets were excluded if they did not exclusively contain german language, were retweets, contained less than five tokens or contained a url @xcite . for our analysis we selected 600 tweets from this dataset. from each of the six classes (implicit, explicit, profanity, insult, abuse, other), 100 tweets were picked randomly. for the classes implicit and explicit, we used the 2019 gold standard files of the test data of subtask 3 @xcite and for the other four classes, we used the 2019 gold standard files of the test data from subtask 1 and 2 @xcite . we randomly shuffled both test datasets using python and for each class, the first 100 occurrences were selected; every tweet was saved as a text file. for the annotation proper we used the open source tool inception @xcite . 4 the tool supports span annotations as well as the creation of new tagsets and annotation layers. the annotator first segmented the tweet (section 4.1) and then decided on the speech act and sentence type labels. if uncertain which label to choose, the issue with the tweet was documented so that a list of challenges could be assembled (section 5.2). after the initial pass through all tweets, all annotations were checked and revised once again. the final dataset consists of 600 xml files. ## annotation scheme our annotation scheme is mainly inspired by @xcite and @xcite . additionally, building upon @xcite , it includes two levels: the syntactical level describes the sentence type of each speech act and the speech act level (consisting of a coarse-grained and a fine-grained level) encodes the type of speech act. ## conclusions and future work the results show that offensive language mainly differs from non-offensive language in the respect that offensive language contains more expressives and less assertives than non-offensive language. the biggest difference is most prominent when we compare tweets with implicit offensive language tweets with explicit offensive language. tweets with implicit offensive language seem to lack the tendency to overtly express emotions, hence they have the lowest frequency of expressives (excluding non-offensive tweets) and the highest frequency of assertives. in contrast, tweets with explicit offensive language show the opposite: they have the lowest frequency of assertives and the highest frequency of expressives. our results suggest that differences exist with regard to the distribution of speech acts in offensive language and non-offensive language. it remains to be seen if an accurate speech act classifier can be developed as one additional component in larger hate speech detection systems. also in terms of future work, we plan to annotate part of the dataset with two more annotators so that we can further improve our annotation approach and also to assess the quality of the dataset and annotations with regard to inter-annotator agreement.