Norm It! Lexical Normalization for I talian and Its Downstream Effects for Dependency Parsing (2020)

Abstract: Lexical normalization is the task of translating non-standard social media data to a standard form. Previous work has shown that this is beneficial for many downstream tasks in multiple languages. However, for Italian, there is no benchmark available for lexical normalization, despite the presence of many benchmarks for other tasks involving social media data. In this paper, we discuss the creation of a lexical normalization dataset for Italian. After two rounds of annotation, a Cohen’s kappa score of 78.64 is obtained. During this process, we also analyze the inter-annotator agreement for this task, which is only rarely done on datasets for lexical normalization,and when it is reported, the analysis usually remains shallow. Furthermore, we utilize this dataset to train a lexical normalization model and show that it can be used to improve dependency parsing of social media data. All annotated data and the code to reproduce the results are available at: http://bitbucket.org/robvanderg/normit .

## introduction social media provide a rich source of constant information, which can be used for many purposes. italian is one of the most popular languages on the internet, estimated to be the 9th most popular by w3techs, 1 and was the 13th most popular language on twitter in 2018. 2 however, many online sources are much harder to process automatically, not only because many existing tools are designed with canonical texts in mind, but also because they naturally contain more linguistic variety @xcite . for social media data, this performance drop was observed for multiple tasks; for pos tagging, accuracy dropped from 97% @xcite on news data to 85% @xcite for tweets, whereas for dependency parsing, performance dropped from 91% @xcite to @xcite . even when in-domain training data is available, the performance typically remains much lower compared to newswire texts, as shown by @xcite . one solution to this problem is to translate social media language to canonical language, a task also known as lexical normalization. an example of normalization for the sentence "joker cmq nn e' nnt di ke !" is shown in figure 1 . the example shows that a variety of phenomena is involved and must be properly annotated: (i) vowels omitted in multiple words (e.g. @xmath0; (ii) diacritics not correctly used (e' → è); (iii) orthography based on pronunciation (@xmath1. previous work has shown that lexical normalization can be used to improve performance for a variety of nlp tasks, including pos tagging @xcite , parsing @xcite , machine 1 https://w3techs.com/technologies/ history_overview/content_language 2 estimation based on tweets obtained by the random streaming api. original text joker cmq nn e' nnt di ke ! normalized version joker comunque non è niente di che ! english joker actually not is worthwhile ! joker is actually not worthwhile ! figure 1: example of a normalized sentence in italian. in this work, our main contributions are: ## related work lexical normalization is considered to be beneficial for numerous tasks involving social media texts. indeed, it has been successfully applied to pos tagging in english @xcite , slovenian @xcite , and dutch @xcite . when it comes to more complex tasks such as dependency parsing, the only experiments are for english @xcite . a common shortcoming of lexical normalization datasets is their lack of information for the inter-annotator agreement. in literature, the only two datasets with an interannotator agreement study are from @xcite and @xcite , both for english. both works report kappa scores, with @xcite including the frequencies of how many annotators agree. in both cases, the kappa is calculated on the task of detecting whether a word should be normalized (because the number of classes/words is too large to calculate kappa scores for the full task). 3 baldwin et al. (2015) report a cohen's kappa @xcite of 0.5854, whereas @xcite report a fleiss' kappa @xcite of 0.891. this is a rather large difference, and it is unclear why this is the case. possible explanations include differences in annotators, annotation guidelines, or data collection. to get a clearer view on the agreement for the full task, we took the data from @xcite , which includes annotation from multiple annotators, and we used it to calculate the agreement on the choice of the normalization in cases where annotators agree that the word needs to be replaced. the results show an observed agreement of 98.73%, leading to the conclusion that this last part of the task is easier. like most natural language processing tasks, most previous work on lexical normalization has focused on english. a wide variety of approaches is used; including machine translation @xcite , adapted spelling correction systems @xcite , feature based systems (van der goot, 2019; jin, 2015), sequence to sequence models @xcite , and recently contextualized embeddings @xcite . in this paper, we will make use of monoise (van der goot, 2019), because it currently holds the state-of-art performance for multiple languages; it is described in more detail in section 5.1. the only previous attempt at normalizing italian social media data is from @xcite . however, they have a different scope of the task, mostly focusing on readability, not on normalization on the lexical level. besides spelling correction, their system also aims to remove words that do not contribute to the syntax, e.g. hashtags, emoticons, hyperlinks and other non-words. for this task, they propose a rule-based method, with a specific module for each type of task that they tackle. in this work, we will use an existing twitter dataset for italian @xcite , to which we add a normalization annotation layer. it contains both random tweets and tweets related to politics and it has been annotated with different layers of information such as sentiment (basile et 3 we confirmed this with the authors of @xcite . al., 2014), pos tags @xcite , and dependency structures @xcite . ## data we annotated a subset of the data from @xcite (version 2.1). this dataset consists of 3,510 tweets and is a sub-set of two previously released datasets: sen-tipolc @xcite and sentitut @xcite . most of the tweets are collected in 2011 and 2012, and are filtered based on keywords on politics. there are some tweets from an earlier period (i.e., 2004) and a small sub-set is from the random twitter api stream. to ensure a basic annotation density and save time, we filtered the tweets which contain at least 3 out-of-vocabulary (oov) words foot_0 to mainly focus on tweets containing nonstandard language. a token is considered oov if it does not appear in the aspell foot_1 dictionary for italian, or if it is either a url, a username, an hashtag, or if it only consists of punctuation. these latter elements have been identified by means of regular expressions. furthermore, we created a small list of proper nouns from the most frequent oov words in this dataset, and added them to the vocabulary. we maintained the splits of the original dataset @xcite . because of the small size of the resulting data, we merge the training and development data, and perform experiments in a 10-fold setting. basic statistics of the data after filtering are shown in ## annotation annotation was done by four native speakers of italian. 6 they are all male, between the age of 20 and 38 and from a variety of regions (veneto, tuscany, liguria, and apulia). all annotators have a background in natural language processing and are familiar with the twitter platform. the annotation has been conducted in two different steps and moments in time: first, the annotators were provided with no specific annotation guidelines but a description of the task and some examples in english. each annotator annotated 50 tweets, resulting in 100 tweets which were marked up by two annotators, which will be our test data. for difficult cases, the annotators were encouraged to consult the internet. 7 after this first round, annotation agreement was computed and the results were used to develop language specific annotation guidelines for italian. these guidelines where then used to re-annotate the annotation of the 100 tweets annotated earlier as well as the training data. in the following sections, we present and discuss the guidelines and the agreement scores. ## lexical normalization model in this section, we will evaluate an existing lexical normalization model, and extend it to handle capitalization better. we will first describe the model and data we used, then we evaluate and analyze it on the training data using 10-fold cross-validation. finally, we will evaluate the model on the test data. ## applicability for dependency parsing to evaluate the effect of normalization on dependency parsing, we use the uuparser 2.3 @xcite with default settings. we choose this parser because it is easy and fast to train, and it performs well on tweets. 11 we evaluate two settings, one where we train the dependency on social media data, and a domain adaptation setup where we use canonical data from the isdt treebank @xcite . for the setting with twitter data we only use data where normalization annotation is available (i.e. we use exactly the same data as in section 5) to be able to also run the full experiment with gold-normalized training data. 12 it should be noted that the tokenization of the normalization is different compared to the treebank data; clitics and prepositions are split in the treebank (section 4.1). this potentially has a large effect on the performance, while the splitting is trivial (it is a closed class of cases) so we decided to use the gold splitting during training and testing. similar to section 5 we report average scores of 10 folds on the training data in the twitter setting. for this setup, we will examine three settings: (1) baseline setting which makes no use of normalization, (2) gold normalization, and ## conclusion we introduced a novel benchmark for lexical normalization of italian social media data. we first used this benchmark to get a more detailed understanding of inter-annotator agreement for this task, and learned that the most difficulties are found in the task of error detection, i.e. the decision whether a word requires normalization or not. furthermore, we utilized this dataset to train a lexical normalization model (monoise) and evaluated performance. it became clear that this is a difficult task, and scores are lower than for most other (indo-european) languages, using the same model. to improve performance for capitalization corrections, we added a generation module and a feature, and showed that this leads to a boost in performance. furthermore, we saw that the mistakes of the normalization model are similar to the disagreements of the human annotators (deciding when to normalize). finally, we showed that this normalization model can be used to improve dependency parsing.