Developing a Tag-Set and Extracting the Morphological Lexicons to Build a Morphological Analyzer for E gyptian A rabic (2022)

Abstract: This paper sheds light on an in-progress work for building a morphological analyzer for Egyptian Arabic (EGY). To build such a tool, a tag-set schema is developed depending on a corpus of 527,000 EGY words covering different sources and genres. This tag-set schema is used in annotating about 318,940 words, morphologically, according to their contexts. Each annotated word is associated with its suitable prefix(s), original stem, tag, suffix(s), glossary, number, gender, definiteness, and conventional lemma and stem. These morphologically annotated words, in turns, are used in developing the proposed morphological analyzer where the morphological lexicons and the compatibility tables are extracted and tested. The system is compared with one of best EGY morphological analyzers; CALIMA.

## introduction after the emergence of social media networks, and specially, after the arab spring revolutions, the data has become available everywhere. this led to have an increased attention in the field of natural language processing (nlp) for colloquial arabic dialects (cads) where the adopted nlp tools for modern standard arabic (msa) are not suitable to process and understand them @xcite . an important challenge for working on these dialects is to create morphological analyzers or tools that provide all possible analyses for a particular written word out of its context @xcite since it is an essential step in most nlp applications such as machine translation, information retrieval, text to speech, text categorization …etc. @xcite . morphological segmentation is the process of converting the surface form of a given word to its lexical form with additional grammatical information such as parts of speech, gender, and number @xcite . in morphological analyzer (ma) tool, the morphemes along with their morphological information of a given word are provided for all its possible analyses out of its context. this paper presents an in-progress work for building a morphological analyzer for egyptian arabic. to build such a tool, a part-of-speech (pos) tag-set schema is developed depending on different criteria to be used in annotating our corpus morphologically. the annotated data is used in detecting the different analysis solutions of each word, extracting the morphological lexicons and the compatibility tables to allow only valid morphological analysis solutions to be generated by the proposed morphological analyzer. the rest of this paper is organized as follows. in section 2, the related works are reviewed, then the used corpus and the process of developing the tagset schema are discussed in section 3. in section 4, the proposed morphological analyzer and the prosses of the automatic extraction of the used morphological tables and the compatibility tables are discussed. the discussion of the system current status, coverage and evaluation are reviewed in section 5. finally, the discussion of conclusion and future work are listed in section 6. ## related works whereas there are many trials for defining tag-set schemas for msa, for example, khoja's arabic tag-set @xcite , arbtags tag-set @xcite , and penn arabic treebank (patb) part-of-speech tag-set @xcite , only few trials interested in egy; @xcite who present a tag-set schema (arzatb tag-set) that is based on the patb guidelines @xcite . they compare tags for egyptian (arz) with those used in msa. the tags specify the forms of the morphemes used in constructing a word, but do not address discrepancies between morpheme form and functions. for example, the broken plural nouns in this tag set are treated the same as singular nouns: ' ‫'رجالة‬ /riggɑːl+aeh/ 'men' is tagged as noun+nsuff_fem_sg. a new pos tag-set (camel pos) is opted to be used in @xcite . it is inspired by the arzatb tag-set and guidelines. it is designed as single tag-set for both msa and the dialects to facilitate research on adaptation between msa and the dialects, support backward compatibility with previously annotated resources and enforce a functional morphology analysis that is deeper and more compatible with arabic morphosyntactic rules than form-based analysis. the lexicon and rules are the core knowledge base of any morphological analysis/generation system @xcite . the trials for modeling dialectal arabic (da) morphology have followed one of two directions. the first direction interested in extending msa tools to cover dialectal phenomena. some trials built their egyptian colloquial lexicon for morphological analyzer on the top of buckwalter arabic morphological analyzer (bama) version 2.0 @xcite ; @xcite , (abo bakr, @xcite , @xcite , @xcite , @xcite , (al-sabbagh & girju, 2010), @xcite , @xcite and @xcite . the second direction interested in modeling da morphology directly; @xcite , @xcite , @xcite , @xcite , @xcite , @xcite , @xcite , @xcite , @xcite and @xcite . handling the problem of lacking standard orthography for colloquial arabic dialects is very important for building the morphological analyzers. there are few works proposed the egy to offer a set of orthographic rules, standards, and conventions for dialectal arabic varieties; @xcite is an attempt to conventionalize the orthography close to the dialectal pronunciation as much as possible regardless of the way a word is typically written. @xcite provides detailed description of conventional orthography for dialectal arabic (coda) as applied to egy. a unified common set of guidelines and metaguidelines that help in creating dialect specific conventions is presented in @xcite applied to 28 arab city dialects including cairo, alexandria, and aswan. lacking annotated resources considered as the bottleneck for processing and building robust tools and applications. however, low-resource languages still lack datasets, such as the arabic language and its dialects. egy has received a growing attention for building corpora that may be useful for many purposes such as dialect identification or sentiment analysis, for example, but only (abo bakr, @xcite , @xcite , @xcite , @xcite , and @xcite are interested in building multi-dialect, multi-genre, morphologically annotated corpora that include egy. however, these annotated corpora have few shortcomings: none of them are freely available for use; they also do not represent enough variety of resource. moreover, some of them normalize the orthography to msa-like standards which fail to grasp the dialectal orthography differences, e.g., ' ‫'كتير‬ / kitiːr / normalized as ' ‫'كثير‬ /kaɵiːr/. since msa and the colloquials share a large proportion of their lexicon, the msa tags are considered as much as possible as in @xcite and @xcite . nevertheless, we prefer to develop our own tag-set schema since we differ from @xcite in that we detect the tag according to its paradigmatic forms alongside its syntagmatic functions, as in @xcite , as much as possible rather than depending on the morpheme form only. in addition, we opted to add more detailed tags in order to be more suitable to describe egy, such as adverb of time, adverbs of place and adverbs of manner, and combine or split other tags that are described in the previously related-work tag-set schemas. moreover, we feel the need to build a larger and more robust corpus, adding more various resources and genres. consequently, this motivates us for building a new morphologically annotated resource for egy to help in building the proposed morphological analyzer. it provides the conventional orthography guidelines and develops a more suitable pos tag set for the egy. for accessing our corpus, follow the link in 1 . ## the corpus the corpus used in developing the tag-set schema and the morphological analyzer consists of about 527,000 words representing about 82,700 tokens. the texts were selected from different sources such as social media, books and other web articles written in egy (from jan 2011-june 2019). in addition, these selected texts cover more than one genre. lack of the standard orthography form in dialectal arabic is handled by assigning for each word the conventional egy lemma and the conventional stem to be close to the egy pronunciation as much as possible regardless the way a word is typically written. to improve the speed and accuracy of the manual morphological annotation, an interface is developed that allows the annotators to concentrate on the task of providing the best morphological analysis of each word according to its context. six skilled linguistic annotators are trained to morphologically annotate the corpus. the conventional orthography guidelines, the annotation process and the interannotation agreement are reviewed in @xcite . ## the morphological analyzer egy arabic words are rarely written with diacritic marks; consequently, they may have many morphological analyses, and the number of these analyses differs from one word to another. since the morphological analyzer deals with words out of their contexts, it should be able to produce all possible analyses of each form, identify the part-ofspeech of each analysis solution of the word (i.e., noun, verb, and particle) and identify the morphological features (i.e., gender, number, time, and person). it is not an easy task to capture all analysis solutions of each word, but the annotated corpora one of the most important resources that can be helpful in detecting these solutions depending on the different contexts of the same word. we follow a concatenative lexicon-driven approach for the annotation of our morphological corpus. the concatenation can be defined as a sequence of prefix(es), stem and suffix(es) or as a sequence of proclitic(s), word form and enclitic(s), where the morphological segments are recognized and processed as part of the annotation process. we adopt the former scheme, where the plan is to allow for the conversion between the two in our morphological analyzer. the focus in this paper is on the prefix(es), stem and suffix(es) representation. our approach resembles the adopted one in buckwalter version 2.0 @xcite who uses a simple prefixstem-suffix representation where the stem is used as the base form and morphotactics and orthographic rules are built directly into the lexicon itself instead of being specified in terms of general rules that interact to realize the output. it has three components: the lexicon, the compatibility tables, and the analysis engine. ## the current status the extracted morphological stem lexicon contains 39k stems corresponding to about 12,100 egy lemmas and about 18,400 msa lemmas. the extracted prefixes and suffixes lexicons contain 324 complex prefixes and 661 complex suffixes (unique undiacritized form and pos tag combinations). since the annotation process of our corpus is still in progress, the covered stems, prefixes, suffixes and lemmas are still limited compared to calima analyzer @xcite that has 100k stems corresponding to 36k lemmas in addition to 2,421 complex prefixes and 1,179 complex suffixes (unique diacritized form and pos tag combinations). ## conclusion and future work the pos tag-set schema is developed and about 318,940 words are morphologically annotated, and the morphological lexicons and the compatibility tables are automatically extracted. the analyzer output is compared to calima output. we plan to make this tool public so it can be used by other people working on egy nlp tasks, from annotating corpora to building morphological disambiguation tools. to enhance our results, we plan to continue improving the coverage of our correct answer no correct answer oov our system 66.9% 10.3% 22.8% clima 82.1% 9.6% 8.3%