The Impact of Text Normalization on Multiword Expressions Discovery in P ersian (2021)

Abstract: This paper evaluates normalization procedures of Persian text for a downstream NLP task - multiword expressions (MWEs) discovery. We discuss the challenges the Persian language poses for NLP and evaluate open-source tools that try to address these difficulties. The best-performing tool is later used in the main task - MWEs discovery. In order to discover MWEs, we use association measures and a subpart of the MirasText corpus. The results show that an F-score is 26% higher in the case of normalized input data.

## introduction the field of computational linguistics (cl) and its engineering domain of natural language processing (nlp) has exploded in recent years. it seems to continue to gain momentum because of a straightforward reason: human civilization is drowning in data. in 2008, google reported that the web had one trillion pages. 1 in 2016, the number was estimated to 130 trillions. 2 international data corporation projects that by 2025, available data may expand to 175 zettabytes. 3 although these estimates include video, image data, and databases, most of it is plain old text. unstructured data (also known as free-form text) comprises 70% -80% of the data available on computer networks. the information content of this resource is unavailable to authorities, businesses, and individuals unless humans read these texts or devise some other means to derive information value from them. and this is where natural language processing comes to the 1 https://www.itpro.co.uk/604911/google-says-the-webhits-a-trillion-pages 2 https://medium.com/@michelkiflen/google-hasindexed-130-trillion-pages-how-would-you-find-the-oneyou-need-d0afa303d6f6 3 https://www.networkworld.com/article/3325397/idcexpect-175-zettabytes-of-data-worldwide-by-2025.html game. nlp procedures can be applied to characterize, interpret, or understand the information content of a free-form text, in other words, to unlock the potential of unstructured data. however, since the quality of the input data influences the quality of the output, in most cases before the nlp pipeline uses it, this unstructured data needs to undergo certain cleaning and normalization tasks, e.g., removal of extra whitespace, substitution of acronyms, transformation of numerical information, accent removal, substitution of special characters and emoji, or normalization of date format. this paper addresses the problem of normalizing texts in the persian language, which is 5 th content language for the web according to w3tech. 4 in particular, we will focus on the impact of persian text normalization on one of the downstream nlp tasks -multiword expressions discovery. mwes are very frequent in language and it has been proved that their proper treatment can make a significant impact on a number of other nlp tasks, e.g. lexicography @xcite , word sense disambiguation @xcite , part-of-speech tagging and parsing @xcite , information retrieval @xcite , language learning (christiansen and arnon, 2017), machine translation @xcite or sentiment analysis @xcite . to the best of our knowledge, there have been no previous attempts to analyze the text normalization impact on the discovery of mwes in the persian language. the following section provides a brief overview of previous work on text normalization problem. then, the specific challenges that the persian language poses for the nlp tasks are described. section 4 focuses on the impact normalization has on the discovery of multiword expressions in persian. it first presents a comparison of different normalization tools and then evaluates the impact of normalizing input data on multiword expressions discovery task. ## related work text normalization focuses on transforming noisy (non-standard, informal) text to a more standard representation. linguistic resources, especially online ones containing slang expressions, acronyms, abbreviations, hashtags, or spelling errors, can deviate a lot from the standard language. text normalization procedures are applied in order to facilitate nlp applications while dealing with such noisy input. one of the first studies to indicate the importance of text normalization was done by @xcite who tried to develop a general normalization process applicable to diverse domains. since then, the impact of normalizing noisy text and its influence on downstream nlp tasks has been analyzed in a few studies. @xcite showed the impact of normalizing social media texts on part-of-speech-tagging. in particular, they focused on tweets and compared original and normalized input texts and different taggers: general stanford pos tagger and domain-specific twitter pos tagger. the influence of normalization on parsing was studied by @xcite who introduced a normalization framework designed with the possibility of domain adaptation. @xcite proposed domain and language independent system based on unsupervised learning for machine translation. since text normalization is, in many cases, a necessary preprocessing step for numerous nlp tasks, there are several normalization steps. however, as noticed by @xcite , it is essential to remember that different normalization tasks would fit different data and downstream nlp applications. moreover, normalization systems as "one size fits all" seem to be less precise than the tailored ones. there have been various tasks that consider text normalization as a crucial preprocessing step, the most popular to be spelling correction @xcite , statistical machine translation @xcite and speech recognition @xcite . some unsupervised studies focused on using probabilistic models @xcite , normalization dictio-naries @xcite , lexicon-based classifiers @xcite or word association graphs @xcite . recently, there has also been an interest in applying deep learning for normalization procedures. @xcite described several systems taking part in shared tasks of twitter lexical normalization and named entity recognition, underlining that deep learning systems based on lexicon-augmented conditional random fields (crfs) achieved the best results. furthermore, a hybrid neural model, which uses word-based encoder-decoder architecture and a character-level sequence-to-sequence model, was introduced for social media text normalization by @xcite al. (2019). mansfield et al. (2019) addressed, on the other hand, text normalization problem by directly normalizing full sentences using subword models. there have also been studies into normalizing less standard or low-resource languages. the impact of normalization was evaluated by @xcite in a study on multilingual projection for parsing low-resource languages. an attempt to normalize dialectal finnish into the normative standard language was presented by @xcite . @xcite studied preprocessing of arabic text on social media. research on preprocessing tools, including normalizer for ainu language, was conducted by @xcite . normalization of six low-resource african languages (afrikaans, amharic, hausa, igbo, malagasy, somali, swahili, and zulu) was presented by @xcite . research on normalizing persian text focused mainly on addressing the specific challenges (described in section 3) this language poses for nlp tasks. this resulted in a number of processing tools. in 2010, @xcite proposed step1, which provides tokenization, morphological analysis, part-of-speech tagging, and spell checking. parsipardaz toolkit, which, apart from providing the same processing steps as step1, also includes additional normalization step, was proposed by @xcite . the first open-source preprocessing tool -hazm -was introduced by @xcite . finally, in 2018 parsivar, another opensource tool, was presented by @xcite . apart from work on preprocessing tools, research in persian normalization focused also on classification tree and support vector machine @xcite , n-gram language model combined with a rule-based method @xcite or sequence labeling models @xcite . ## challenges of persian nlp research in persian nlp faces two significant challenges. the first one arises from the number of resources. although there has been a significant improvement in the number of available nlp resources e.g. hamshahri corpus @xcite , bijankhan corpus @xcite , farsname @xcite , @xcite , persian dependency treebank @xcite , sentipers @xcite , farsnet @xcite or persbert @xcite in recent years, persian is still a heavily unresourced language compared to english or german. the second problem is related to the challenging character of persian itself and inconsistencies in the writing system. the following section discusses the main challenges the persian language poses to nlp applications. ## normalization impact on multiword expressions discovery the challenges presented above: inconsistency in using white-and pseudospace, different encodings, missing short vowels or bidirectionality can pose many difficulties for proper processing of persian for several nlp tasks. therefore, a certain level of text normalization seems necessary. the following section describes the impact normalization procedures have on the discovery of multiword expressions task. ## conclusion and future works in this paper, an impact of text normalization on a downstream nlp task was presented. in particular, we focused on the normalization of persian language data for multiword expression discovery. the experiment results show that the performance of a system without a persian-tailored normalization step is 26% worse (f-score), which is a significant deterioration. to our knowledge, this was the first time when the influence of text normalization on the discovery of multiword expressions in persian was described. since this paper focuses on normalization as a preprocessing step, it would be interesting to compare its impact with post-processing tasks. some further future works include analyzing how normalized data influences other nlp tasks in the persian language, particularly syntactic parsing and sentiment analysis. moreover, we would like to compare the tools described in this paper with a neural network approach to text normalization.