Lexical Induction of Morphological and Orthographic Forms for Low-Resourced Languages (2020)

Abstract: In this work we address the issue of high-degree lexical sparsity for non-standard languages under severe circumstance of small resources that are considered insufficient to train recent powerful language models. We proposed a new rule-based approach and utilised word embeddings to connect words with their inflectional and orthographic forms from a given corpus. Our case example is the low-resourced Lebanese dialect Arabizi. Arabizi is the name given to a new social transcription of the spoken Arabic in Latin script. The term comes from the portmanteau of Araby (Arabic) and Englizi (English). It is an informal written language where Arabs transcribe their dialectal mother tongue in text using Latin alphanumeral instead of Arabic script. For example حبيبي Ḥabībī my-love could be transcribed as 7abibi in Arabizi. We induced 175K forms from a list of 1.7K sentiment words. We evaluated this induction extrinsically on a sentiment-annotated dataset pushing its coverage by 13% over the previous version. We named the new lexicon SenZi-Large and released it publicly.

## introduction a language is considered to have a high degree of lexical sparsity if the words of the language could have a high number of forms @xcite , whether morphological (inflections) or orthographic (spellings). as a result the number of possible forms for many words becomes large, a considerable challenge in nlp for low-resourced languages. with the lack of linguistic resources, several nlp tasks become difficult to achieve such as named entity recognition (ner), part of speech (pos) tagging, parsing, question answering (qa), machine translation (mt), and sentiment analysis. stemming, lemmatisation, and normalisation of text are the most intuitive text preprocessing tasks to address a highly-sparsed language foot_0 , with the goal of simplifying a variety of word forms into one standard form, such that the inflected or differently-spelled forms match easily with their lemmas or roots in a lexicon or corpus. on the other hand, it reduces the number of word forms in a dataset to train a machine learning (ml) algorithm. as plausible as this direction sounds, it is not as straightforward for morphologically-rich languages (mrls) as it is for languages with simple morphology. in english for example, the words enjoyed, enjoying, enjoyable, joyful, and unjoyful can easily map with the lexemes enjoy and joy. in arabic however, the inflections are not limited to prefixes and suffixes but also include infixes, proclitics, and enclitics. the affixes consist of objects and pronouns that are tense, number, and gender sensitive as well, that one verb could easily expand to a hundred inflectional forms. arabic is also rich in tri-literal words because of the short vowel diacritic transcript, as such even a slight inflection could map with a list of irrelevant words, for example: mapping words with their base forms becomes even more difficult when the language is purely spoken, lacking a standard orthography. thus, if transcribed, people spell words according to their own perception of how phonemes should be represented in text, resulting in various spellings for each word. in this work, we choose arabizi @xcite , an informal heuristic transcription of the spoken dialectal arabic (da) in latin script. a language that poses both issues, rich morphology and inconsistent orthography. not only can each word be inflected into many forms, but also each form could be written in a variety of ways. hence, the magnitude of its lexical sparsity quickly becomes very large as it is a multiple of its morphology and orthography. unlike the formal modern standard arabic (msa), the spoken vernacular da is quiet esoteric to different regions with different structure, word choice, vocabulary, conjugation, speech tempo, and pronunciation. it is influenced by other languages as well such as french, turkish, spanish, or italian based on the history of the region. we focus on the lebanese dialect arabizi as the use case of this study, a member of the larger levantine dialect family with less resources in the literature than egyptian, north-african, or gulf dialects. the lebanese dialect is also known for its mixing with english and french which is also reflected in its arabizi transcription @xcite , another challenge for word classification. we address the problem of the large lexical sparsity of arabizi by taking a reversed approach; rather than attempting to lemmatise or simplify words, we retrieve a high number of written forms for given input words from a raw corpus. we propose an approach that combines rule-based mapping with word embeddings to automatically expand arabizi words. we test this approach on a lexicon composed of 2k sentiment words that resulted in a new sentiment lexicon of 175k words. we evaluate the approach extrinsically by testing the newly expanded sentiment lexicon against a pre-annotated arabizi dataset. as such, the contributions of this work are mainly an approach to find the written forms of words in highly-sparsed low-resourced languages and the outcome resource, a new lebanese dialect arabizi sentiment lexicon. ## related work in this work we build upon our previous work of @xcite , where we compiled a raw corpus and developed a new sentiment lexicon and datasets for the lebanese dialect arabizi. we started by developing a 2k sentiment words lexicon and followed by expanding it using the cosine word similarity feature, nearest neighbours, from an embedding space trained on the raw corpus. this yielded in a lexicon of 24.6k words, named senzi, an average of 12 forms per word, to cover some inflectional and orthographic variants of the original sentiment words. however, the degree of the lexical sparsity in arabizi is higher than 12 variants per word as can be seen in section 4. we therefore, propose a new approach that expanded the same 2k sentiment words into 175k words using a rule-based technique to match with variants, or forms, from that same corpus. we also utilise the cosine word similarity feature of the embeddings, not to retrieve words, but to associate the new words with sentiment scores and disambiguate them after the expansion. in @xcite , the authors generated morphological inflections computationally by adding prefixes and suffixes to a set of vocabulary. they then measured the cosine similarity of each original word with the generated form to evaluate the morphological generation. although this work is similar in concept to our work, the complexity of arabic morphology is beyond simple prefixes and suffixes as mentioned in section 1, hence we do not transform the vocabulary by adding affixes, rather we map the structure of that vocabulary with words of similar structure in the corpus. similarly, we measure the cosine word similarity to rank the retrieved forms. other works considered the non-standard orthographies, such as the texts found on social media, a noisy or ill forms of the correct standard languages, thus emphasised their efforts on matching the equivalent forms of the noisy text with that of the standard orthography. for example, @xcite developed a classifier to detect the noisy forms, then generated correction candidate words based on the morphophonemic similarity. @xcite replaced the noisy words with a list of possible regular forms in a given set of sentences, then maximised the product of the weighted list and language model scores to generate cleaner versions of these sentences. @xcite used an unsupervised approach to extract word variants, in pairs, from domain specific corpora. they computed the distributional similarity among all word pairs in a domain specific corpus, filtered the common english ones, and scored the relation of the pairs using their semantic and lexical similarities. @xcite also tried to automatically replace noisy words with standard english equivalents. similar works presented in the @xcite shared task on noisy user-generated text, where the participants were specifically asked to convert non-standard words to their standard forms for english tweets. most of these works considered that a correct form of the noisy words exist, therefore perceived the task as finding the best word replacement, similar in concept to text simplification @xcite which aims at transforming original words into simpler variants. this is similar to our problem except that there is no correct or wrong way to transcribe a spoken language that lacks a standard orthography, such as arabizi; for that, our goal is to find all possible morphological and orthographic forms, for any given word, that are naturally written in text. the issue of non-standard orthography is not limited to arabizi, but to many spoken languages and dialects @xcite , we therefore aspire that our contribution spans across other such languages as well. ## background and challenges arabizi scripture contains several linguistic complexities that it is even called hell in @xcite . we now scrutinise the transcription of arabizi to explain how these complexities came to exist. first of all, because dialectal arabic (da) is spoken, in its nature it lacks a consensus on an orthography, for that transcribing arabic phonemes in a script of different languages (latin) heuristically widens the orthographic inconsistency even further for the reasons summed below. additionally, transcribing a high number of phonemes into lower number of letters generates word ambiguity and causes arabizi words to overlap with english words. however, in this work we focus on addressing the sparsity challenge. this large lexical sparsity of arabizi is even faced with a severe lack of resources. the existing corpus of 53mb lebanese dialect arabizi @xcite is considered insufficient to train powerful data-hungry language models such as bert and gpt @xcite ; a fundamental challenge for low-resourced languages. on another note, attempting to convert arabizi to lebanese arabic script, similar to the sophisticated works of @xcite @xcite , serves us no purpose for lexical induction. we simply exploit the existing corpus to maximise the forms of given words using a rule-based combined with word embeddings approach. ## approach we use the datasets that we previously created in @xcite to test and evaluate the proposed approach. briefly, we developed an arabizi sentiment lexicon of 2k words, composed of 600 positive and 1.4k negative words separately, and expanded it to 24.6k words using the cosine word similarity feature of word embeddings trained on a corpus of 1m facebook comments. we evaluated the lexicon using a simple lexicon-based approach against a sentiment-annotated dataset of 1.6k arabizi tweets. we now propose to expand that original sentiment lexicon (2k words) through several steps. we first normalise the text and search the entire facebook corpus for all words matching the structure of the lexicon words under a linguistic constraint of permitted affix letters. we follow by adding cosine word similarity scores obtained from the word embeddings between every matched word and the original lexicon word to use it as a sentiment score and disambiguate forms that are retrieved in both positive and negative lists. we detail each step below. we apply this normalisation to determine which words to match only. if there is a match, we retrieve the word in its original un-normalised format. if more consonant letters precede or follow the sequence, we check if these letters fall within a predefined set of observed prefixes (2 l s w t b y m n) and suffixes (k l t n h w y): fa5r (f5r) matches with all the following inflections benfe5r, mntfekhar, fakhourin, fa5rna, fakhourah because the prefixes b, m, n, t and suffixes n, h are permitted affixes. we minimise the error by matching the lexicon words that contain a cls of length 3 or more, leaving out 126 (6%) short lexicon words. ## evaluation we now evaluate the proposed approach extrinsically by applying a lexicon-based sentiment classification against the small annotated dataset of 1.6k tweets mentioned in section 4. we wish to have had a larger dataset to reveal the true coverage of senzi-large, nevertheless, we settle for showing the value of the lexical induction approach. in the lexicon-based approach, we search every word from the annotated tweets across the sentiment lexicon to assign scores to the matched words. we set the associated score for words that match with the positive list, and set a negated associated score for words that match with the negative list. finally, we sum the scores to classify the tweets as positive or negative. since this is a binary classification task, similar to subtask b from semeval-2017 task 4 @xcite , we present the percentage of the classified tweets and the classification results of these tweets separately in ## discussion spoken languages that lack a standard orthography tend to have a high degree of lexical sparsity, especially if they are morphologically rich as well. if words can be transcribed in a variety of ways without a consensus on an orthography, then they can not be referenced with one correct orthography. this by nature defies word classification tasks such as sentiment analysis, ner, pos tagging, word sense disambiguation (wsd), anaphora resolution, etc. we proposed a new approach to match words with their inflectional and orthographic forms in a lowresourced context. we hand-crafted a rule based approach that worked well with the linguistic nitty gritty complexities of a highly-sparsed scripture, arabizi, and coupled it with word embeddings using minimal resources. the limitation of this approach was adding a manual check to filter out the lexicon words that retrieved irrelevant words. in a resourceful situation however, we would have trained advanced word embedding models to measure the contextual similarity of the words, with the intention to automate this step. such models require large amount of data to pre-train especially for uncommon languages like arabizi. nevertheless, the result was a new sentiment lexicon, senzi-large, composed of 175k forms induced from 1.7k words, 7 times larger than the previous expansion for a 3% trade-off in f1 score. we presented a word-level classification error analysis on the same twitter data to learn that most of the missed words are either orthographic variants of senzi's words or english sentiment words and most of the falsely classified words were false neighbours introduced in the large induction. this consolidates the significance of lexical sparsity in non-standard languages. although the outcome is a new public resource, the main contribution of this work is the proposed approach. the essence of the approach is in normalising and connecting consonant letters which comprise the backbone of most latin script vocabulary. in the case of lebanese dialect arabizi, we observed a set of prefixes and suffixes and used them to minimise the error in the induction. however, given the complexity of arabic morphology described in section 1, a rule-based approach is far from success. we therefore combined it with word embeddings to filter the irrelevant and score the sentiment words. as such, our intuition is that the approach could potentially be useful for at least other arabizi dialects if not other latin script languages that pose similar challenges, such as javanese dialects and alsatian @xcite . the approach might also be catered for other nlp tasks such as lemmatisation or text simplification in morphologically-rich languages. ## conclusion in this work we addressed the issue of high-degree lexical sparsity for a social language under a severe circumstance of small resources that are considered insufficient to train recent powerful language models. we proposed a new rule-based approach that utilises classical word embeddings to connect words with their inflectional and orthographic forms from a given corpus. our case example is the low-resourced lebanese dialect arabizi, hence we used and expanded the only available resources we found in the literature to test and evaluate the approach. we induced 175k forms from a list of 1.7k sentiment words. we evaluated this induction extrinsically on a sentiment-annotated dataset pushing its coverage by 13% over the previous version. we named the new lexicon senzi-large and released it publicly 5 . latinising a non-latin script language heuristically is not a novel phenomenon, as to arabizi, greek, farsi, hindi, bengali, urdu, telugu, tamil and other languages are also transcribed in latin script, known as greeklish @xcite , finglish, hinglish @xcite , binglish, etc. for a future work, we plan to explore cross-lingual approaches @xcite to bridge such low-resourced languages that are written in different scripts.