: Tviterasi, tviteraši or twitteraši? Producing and analysing a normalised dataset of Croatian and Serbian tweets. Slovenščina 2.0, 4 (2): 156-188. DOI: http://dx.doi.org/10.4312/slo2.0.2016.2.156-188. In this paper we discuss the parallel manual normalisation of samples extracted from Croatian and Serbian Twitter corpora. We describe the datasets, outline the unified guidelines provided to annotators, and present a series of analyses of standard-to-non-standard transformations found in the Twitter data. The results show that closed part-of-speech classes are transformed more frequently than the open classes, that the most frequently transformed lemmas are auxiliary and modal verbs, interjections, particles and pronouns, that character deletions are more frequent than insertions and replacements, and that more transformations occur at the word end than in other positions. Croatian and Serbian are found to share many, but not all transformation patterns; while some of the discrepancies can be ascribed to the structural differences between the two languages, others appear to be better explained by looking at extralinguistic factors. The produced datasets and their initial analyses can be used for studying the properties of nonstandard language, as well as for developing language technologies for nonstandard data. Key words: computer-mediated communication, CMC corpora, Twitter, normalisation Slovenščina 2.0, 2 (2016) [157] Normalisation has most often been applied to English, with some work also available e.g. for German (Sidarenka et al. 2013) , Spanish (Oliva et al. 2013), and Slovene (Ljubešić et al. 2014a; Ljubešić et al. 2016a). Slovenščina 2.0, 2 (2016) [158] In this paper we adopt the normalisation-based approach, focusing on Twitter messages (tweets) written in Croatian and Serbian. As one of the most widely used CMC platforms, Twitter has already received a lot of attention in NLP. The number of tweets published per day are counted in hundreds of millions (Benhardus, Kalita 2013) , and the content ranges from news broadcasts and official announcements by companies and institutions to personal thoughts and opinions the users share, making Twitter a rich source of data for NLP tasks related to text mining. To enable these tasks to be performed, automatic lowerlevel processing is a must, meaning in turn that the problem of nonstandardness needs to be solved. In the specific case of Twitter, an additional component influencing the structural properties of its language is that messages are constrained by the length restriction of 140 characters. Given the recent availability of basic language tools for standard Croatian and Serbian, a normalisation-based approach was deemed more cost-efficient than an adaptation of standard language tools. Additionally, performing normalisation gives researchers easy access to deflections from standard language occurring in non-standard one. Examples of tweets containing non-standard features in Croatian and Serbian are shown in Table 1 . These features include phenomena typical of CMC in general, such as phonetic spelling of foreign words (e.g. fešn for fashion), abbreviations (e.g. zg for Zagreb), @ name mentions and emoticons, but also phenomena typical of Twitter like hashtags and some terms (e.g. fave), as well as some language-specific features, such as omission of diacritics (which occurs in both Croatian and Serbian, e.g. kauc for kauč -couch), and the use of fully language-specific dialectal and colloquial non-standard forms (e.g. the Ikavian dialectal form isprid for ispred -in front of in Croatian). With the future goal of developing tools for automatic CMC normalisation, we manually normalised a sample of 4000 tweets per language. In the remainder Slovenščina 2.0, 2 (2016) [160] of the paper we first describe the corpus the tweets were sampled from and the samples themselves, moving on to the procedure and the unified Croatian and Serbian guidelines used in the manual normalisation. We then present several initial analyses based on the normalisation outcomes; the analyses were performed starting from the normalised forms and looking towards forms found in the Twitter datasets. Specifically, we look at the distribution of standard -> non-standard transformations across parts of speech and lemmas, as well as the distribution of transformation subtypes (deletions vs. insertions vs. replacements), and we compare Croatian and Serbian. As very little related previous work exists for these languages, our main goals are to give an overview of the key trends, and to compare these trends in the two languages, facilitating the formulation of future specific linguistic hypotheses. The corpus we employ comprises Croatian and Serbian tweets harvested with
For example, the morphosyntactic tagger and lemmatizer for Torlak is an adaptation of the tagger for standard Serbian REF ).
Tviterasi, tviteraši or twitteraši? Producing and analysing a normalised dataset of Croatian and Serbian tweets
Slovenščina 2.0: empirical, applied and interdisciplinary research
Slovenščina 2.0: empirical, applied and interdisciplinary research
Computer Science