MAN orm: A Normalization Dictionary for M oroccan A rabic Dialect Written in L atin Script (2020)

Abstract: Social media user generated text is actually the main resource for many NLP tasks. This text, however, does not follow the standard rules of writing. Moreover, the use of dialect such as Moroccan Arabic in written communications increases further NLP tasks complexity. A dialect is a verbal language that does not have a standard orthography. The written dialect is based on the phonetic transliteration of spoken words which leads users to improvise spelling while writing. Thus, for the same word we can find multiple forms of transliterations. Subsequently, it is mandatory to normalize these different transliterations to one canonical word form. To reach this goal, we have exploited the powerfulness of word embedding models generated with a corpus of YouTube comments. Besides, using a Moroccan Arabic dialect dictionary that provides the canonical forms, we have built a normalization dictionary that we refer to as MANorm. We have conducted several experiments to demonstrate the efficiency of MANorm, which have shown its usefulness in dialect normalization. We made MANorm freely available online.

## introduction the large part of the world's population is daily connected and very active in social media (sm). this community produces a huge amount of data. this latter, especially textual one is actually very useful for the development of many nlp or text-based applications in general @xcite . however, these texts generated by sm users are of a noisy nature or in other words do not follow the rules of standard communications. another phenomenon, which adds more complexity to this type of content, is the use of dialects, which are non-standard languages used mainly in verbal communication. since the advent of short messaging service (sms), moroccan arabic (ma) dialect has been introduced into users written communications foot_1 and today, in social media, this phenomenon is becoming widespread @xcite . ma dialect is the mother tongue of most moroccan people, it has been used in sm to freely and spontaneously express emotions and thoughts with other peers @xcite . this language does not have a standard spelling since it is not used as formal language. therefore, each social media user writes according to his own. the writing variability is due to the diversity of individual's pronunciation related to their different regional and cultural backgrounds @xcite . thus, for the same word, we find different spellings. for instance, the word 'chkoun' (who) has other five different transliterations @xcite . this problem constitutes a major handicap for many nlp tasks @xcite . to overcome this problem, normalization can be used as a preprocessing in front of the main nlp task. this preprocessing has proved his efficiency in sentiment analysis @xcite , dependency parsing @xcite and also in pos tagging @xcite . in a previous work @xcite , we have introduced machine normalization system for social media text standardization, that can be used as a preprocessing. the current work will be part of this system which will allow us to improve its performance. for standard language, in general the task of normalization aims at mapping each out of vocabulary word to one correct form or standard form among a set of standard words candidates (@xmath0. on the contrary, for dialect, which is a non-standard language. since there is no standard form for dialect words, this task starts from considering a transliteration of word phonemes (the words are written as spoken) as the canonical form. then, we try to capture all its possible transliteration forms (1→n). in this work, we follow this approach for ma dialect normalization. first, we build a ma words dictionary that we consider as the lexicon of canonical word form. we then exploit distributed word representations models trained on a youtube comments corpus to extract the most similar (semantically) words of each dictionary entry, and lexical similarity measures to select all the nearest word forms. the result is a normalization dictionary mapping between each ma word transliteration and its canonical form. we refer to the constructed normalization dictionary for moroccan arabic as manorm. in the next section, we discuss related works particularly closely related ones on standard languages. then, we present the detailed solution with the used resources followed by the evaluation and the discussion of the resulted dictionary and we conclude with future directions and further challenges. ## related works text normalization can be seen as the successor of spelling correction, with the difference that in the first case, the noisy writing style is often intentional @xcite , for example when using abbreviations (e.g. use of 'u' instead of 'you'). whereas in the second case, misspellings are unintentional and are mainly related to cognitive processes (j. @xcite . the first approaches for text normalization were based on rules and the noisy channel model @xcite mainly related to spelling correction techniques. they were used jointly for automatic normalization. among these rules, we find lexical rules using edit distance foot_2 @xcite that can detect misspellings and regular expression patterns used for removing or replacing unnecessary character repetitions or urls, hashtags and logograms foot_3 . they can also be used for the detection of sm special words @xcite . in addition, phonetic rules using soundex algorithm variants, can serve to normalize noisy word related to pronunciation differences @xcite . the noisy channel model normalizes words by selecting the most probable formal ones using probabilities ranking from language model. it was used with supervised and unsupervised training @xcite . in general, these approaches capture the differences of the word's surface forms by detecting the similarity in the lexical level between informal and formal word forms. however, the semantic level remains inaccessible because these techniques are not able to capture the words' context. the problem here is that an informal word can be assigned to a formal one, only based on its lexical form without considering its meaning, which can constitute a source of ambiguity. to mitigate this drawback, supervised learning, machine translation and other techniques were used. for supervised learning, features such as character n-grams, word embedding, pos tag, edit-distances, lookup lists and others are used with labeled data as in @xcite , which is the current state-of-the-art model for most languages. in addition, different architectures of neural networks were adopted such as lstm (long short-term memory) model to predict the word canonical form, using the word itself and its surrounding words as in @xcite . in a very recent work, @xcite have tried to use contextualized embedding with bert (bidirectional encoder representations from transformers) to learn lexical normalization for english. this task has also been approached as statistical machine translation smt-like task @xcite as a means of context-sensitive technique, where the goal has been to translate noisy text into standard one using parallel corpora. csmt or character level smt has also been used for normalization and performed better results than word level smt @xcite . neural machine translation (nmt) was also used, in @xcite a neural encoder-decoder with word and character level language model surpassed csmt performance. nevertheless, these techniques require a large scale of labeled data, which is in itself a complex and costly task. to address these problems, @xcite was the first to introduce the contextualized normalization with a fully unsupervised manner. he has employed the distributed representation of words or word embedding to capture contextual similarity that can match the noisy word to its canonical form if they share the same vector representation. in other words, their vectors are the closest to each other among all the vocabulary. he has used finite state machines (fsm) to represent the resulting lexicon, and the normalization process is carried out by transducing the noisy words from the fsm. the main advantages of this technique are, first, the needless of labeled corpus, he has used twitter and customer care notes as training data therefore it is scalable and adaptive to any language. second, the presence of the contextual dimension, which has been a key factor of its high performance in this task that surpassed microsoft word and aspell accuracies. these positive qualities have inspired other works on normalization. @xcite have performed portuguese normalization using word embedding model trained on products reviews and tweets. they have built a dictionary mapping between noisy and canonical words to represent the lexicon. they have conduct experiments on both internet slang and orthographic error correction. the obtained results outperformed existing tools. @xcite have employed the same approach to build normalization dictionaries for english, french and arabic using twitter corpora to overcome the lack of normalization resources available for these languages. they have also reached high performance in the three languages. all these mentioned works have been done for standard languages normalization. for dialects that suffer from resources scarceness, the related works are very limited. among them we find conventional orthography for dialectal arabic or coda, first proposed in @xcite for the egyptian dialect. it was later improved to coda* in @xcite and extended to include new dialects. @xcite , have introduced a system that generates a list of all possible transliterations for each word in an input sentence using a finite-state transducer trained on character-level alignment from egyptian dialect written in arabizi (latin script) to arabic script. they have learned the transducer on parallel corpus of egyptian arabizi-arabic words. @xcite have used character level nmt to translate dialectal finnish to standard one. they used lstm and transformer models that has been trained on a hand-annotated corpus of transcriptions of different speech records starting from 1950. word embedding has also been used in dialect processing for the construction of a comparable corpus and a lexicon of algerian dialect @xcite by alignment of a corpus extracted from youtube. the built lexicon associates the different transliterations forms of dialect words written in arabic and latin scripts. as a dialect, moroccan arabic is an under-resourced language. there is no available resource or tool for its normalization. the only work done for this purpose was @xcite where the authors have used a corpus from facebook and youtube to analyze spelling inconsistency of ma dialect used in sm text written in arabic and latin scripts. they have compared their corpus with a reference dictionary that has been previously built by the authors and they have found that 35% of this text is noisy. they have concluded that a spell-correction tool is essential to clean up and convert dialectal words into a single standard form of writing. in line with previous works, namely sridhar and htait, we will employ distributed representation for our ma normalization for multiple reasons. first, we only need a corpus of unlabeled data and a dictionary of dialect words to serve as a lexicon for normalized word forms. second, word embedding is able to identify semantically similar words because it constructs word vectors based on the assumption that semantically similar words are surrounded by the same context. therefore, the semantic aspect of each reference word and its associated words is guaranteed. in other words, the reference word and these extracted synonyms have the same meaning in the context in use. the remaining task is then to measure the lexical similarity between these words to identify the different lexical forms of the same canonical word. finally, a normalized form is provided to these words. ## moroccan arabic dialect words forms the ma dialect is mostly derived from arabic foot_4 about 86% and a mixture of other languages namely, french 11.72%, tamazight 0.39% and spanish 0.06% according to @xcite . as previously mentioned, we are interested in ma dialect normalization, specially the dialect written in latin script (also called arabizi). the first time this script appeared goes back to the beginning of sms in early 2000's where mobile phones did not yet have an arabic keypad and some phones cannot display messages written in arabic script, whereas latin script was accessible in all phones. however, until today, there are still people who are keeping this type of writing even if arabic keypads are widely available. the ma dialect written in latin script is the transliteration of phonemes mainly of arabic origin, thus it is more speech like than writing like. this script uses latin consonants that mimic arabic ones and vowels as the equivalent of diacritics. to identify the different phenomena that dialect normalization needs to address. we selected some comments from our collected corpus composed of youtube comments (see sec. 4). then we analyzed the words forms to determine the sources of lexical variation. we identified five categories as listed below: these linguistic features of written dialect are the main source of variations and non-uniformity of ma text in sm. in the next sections, we will present our solution to normalize dialect word forms and hence increase this text uniformity. ## data extraction the used data was gathered from youtube video comments extracted using youtube api. youtube is the most popular sm platform in morocco, used by 44.48% of the population 7 . these videos have been selected using keywords related to various topics such as politics, sport, art, cooking, comedy, and others. for instance, for cooking we use key words like ‫الفياللي'‬ ‫'حليمة‬ ‫'بسطيلة',‬ ‫مغربية',‬ ‫.'شهيوات‬ the goal is to capture a wide range of words from different domains and thus ensure a large coverage of ma vocabulary. the collected corpus contains about 500k sentences. ## data preprocessing before starting the normalization steps, we conduct a set of pre-processing to prepare our corpus for word embedding. the first one is the selection of latin script comments because the raw corpus contains also arabic script comments, then duplicated comments or containing only numbers or just one word are removed and any repetitions of more than three letters are reduced back to two letters. in addition, we removed all punctuations, hashtags, urls, at mentions, emoticons, symbols, and full number strings. finally, we have substituted the numbers within words by their equivalent letters to meet those used in the dialect dictionary (see our system for dialect normalization is based on two steps. namely, transliterations extraction and transliterations selection. starting from three word embedding models and a ma dialect dictionary that will serve as the lexicon of canonical word form. for each canonical word in this dialect dictionary, we first extract the most semantically similar words from the vocabulary produced by the word-embedding models. semantic similarity aims to select the nearest neighbors to each canonical word based on their context. then, from these selected words, we select the most lexically similar ones to the canonical word. the lexical similarity aims to select the most similar transliterations to the canonical word in terms of surface form. we will describe these processing steps in detail in the following sections. ## evaluation the goal of the evaluation of manorm dictionary is to test the quality of the normalization lexicon. since this is the first work on ma dialect normalization, we do not have a gold standard or a reference to evaluate the performance of the produced lexicon. therefore, it is impossible to proceed to an automatic evaluation, thus we have created our own reference. we first combined the outputs dictionaries of the three models, which produced a normalization dictionary of 3057 entries (transliteration, normalization form). we have then, validated each entry based on a human judgment by checking manually the correct association between each word transliteration and its canonical form. this operation resulted in a normalization reference dictionary of 2225 correct entries. examples of result are given in canonical word form (ma dialect dictionary) transliterations (corpus) awel (the first) awl, awwal, awle, aaal, awale choukran (thank you) chokran, chokrane, chkran, khokran, chokrn we have used precision and coverage as evaluation metrics rather than the common information retrieval ones (precision, recall and f-score). because we are not able to measure the recall, which is the ratio between the number of the provided relevant results, and the total of relevant results that we have to provide. since we do not know exactly how many transliterations, we can normalize by each given canonical word form, we cannot define the total value of relevant results. therefore, instead of recall, we measure the coverage of the ma dictionary words. the coverage represents the ratio of useful canonical words or the words that were able to catch some other transliterations over the total of ma dictionary words. to gain more insight into the performance differences of the three word embedding models individually, we calculate their coverages, and we measure their precisions by comparing each model with the created reference normalization dictionary. the results are shown in by looking at we also evaluate the different scoring functions of lexical similarity used in other related works (with a threshold of 70%) and we report the results in to prove our choice of the threshold value for lexical similarity scoring we conduct a series of experiments. the obtained results are reported in ## conclusion written dialect is a phonetic transliteration of spoken words that does not follow any standard orthography. it is mostly used in sm where each user improvises his own spelling. as a result, for each single word we find a mixture of spellings. before performing any nlp task, it is mandatory to transform these dialect words written differently to one normalized form. in this work, we present our solution for ma dialect normalization based on semantic similarity using word-embedding and lexical similarity. we use three word-embedding models that we combine their outputs to form one normalization dictionary manorm. in the resulting dictionary, we find the matching between each word from the corpus and the most similar word form from the ma dictionary. the merge allows us to make a compromise between precision and coverage. the evaluation of the normalization dictionary shows the good performance of this solution. however, other improvements can be done with a larger corpus, especially to increase ma dictionary coverage. as next direction, we will employ the same technique to normalize ma words written in arabic script after building the appropriate dictionary for canonical forms.