Joint Diacritization, Lemmatization, Normalization, and Fine-Grained Morphological Tagging (2020)

Abstract: The written forms of Semitic languages are both highly ambiguous and morphologically rich: a word can have multiple interpretations and is one of many inflected forms of the same concept or lemma. This is further exacerbated for dialectal content, which is more prone to noise and lacks a standard orthography. The morphological features can be lexicalized, like lemmas and diacritized forms, or non-lexicalized, like gender, number, and part-of-speech tags, among others. Joint modeling of the lexicalized and non-lexicalized features can identify more intricate morphological patterns, which provide better context modeling, and further disambiguate ambiguous lexical choices. However, the different modeling granularity can make joint modeling more difficult. Our approach models the different features jointly, whether lexicalized (on the character-level), or non-lexicalized (on the word-level). We use Arabic as a test case, and achieve state-of-the-art results for Modern Standard Arabic with 20% relative error reduction, and Egyptian Arabic with 11% relative error reduction.

## introduction morphological modeling in semitic languages is challenging. their optional short vowels (diacritics) increase the overall ambiguity of surface forms; and their morphological richness results in large target spaces, which increase model sparsity. the different morphological features can be modeled through combined feature tags, using a single (but very large) target space, or through having separate models for each of the features. the combined features approach models the relationships between the different features explicitly, but the large target spaces for morphologically rich languages further increase sparsity. on the other hand, separate feature modeling guarantees smaller target spaces for the individual features, but the hard separation between the features prevents modeling any interfeature dependencies. the set of morphological features includes lexicalized and non-lexicalized features, which further exacerbates joint modeling. non-lexicalized features, like gender, and number, among others, have limited target spaces, and usually modeled as tagging tasks. lexicalized features, like lemmas and diacritized forms (for semitic languages), are open-ended, with large target vocabularies. moreover, non-lexicalized features are modeled on the word level, whereas lexicalized features are optimally modeled on the character level. this difference in the modeling granularity can be challenging for joint models. in this paper we present a model for handling lexicalized and non-lexicalized features jointly. we use a sequence-to-sequence architecture, with different parameter sharing strategies at the encoder and decoder sides for the different features. the non-lexicalized features are handled with a tagger, which shares several parameters with the encoder, and uses a multitask-learning architecture to model the different non-lexicalized features jointly. the lexicalized features, on the other hand, are handled with a specific decoder for each feature, sharing the same encoder. our architecture models the non-lexicalized features on the word level, with a context representation that spans the entire sentence. the lexicalized features are modeled on the character level, with a fixed character context window. the character level modeling is also suitable for surface form normalization, which is important for noisy texts common in dialectal content. we use modern standard arabic (msa) and egyptian arabic (egy) as test cases. our joint model achieves 20% relative error reduction (1.9% absolute improvement) for msa, and 11% relative error reduction (2.5% absolute improvement) for egy, compared to a baseline that models the different morphological features separately. the rest of the paper is structured as follows. we present a brief background and a survey of related work in section 2. we introduce the approach and various models in section 3, and discuss the experimental setup and results in section 4. we conclude and provide some directions for future work in section 5. ## background and related work in this section we present a brief linguistic overview of the challenges facing morphological modeling in semitic and morphologically rich languages. we then discuss related contributions in literature, and how our model compares to them. ## approach non-lexicalized features are usually modeled on the word level, whereas lexicalized features are better handled through character level models. moreover, the context representation for morphological tagging of the non-lexicalized features usually spans the entire sentence, using lstms for example. the optimal context representation for the lexicalized features, on the other hand, is through a fixed number of characters before and after the target word @xcite . this difference in modeling granularity, in terms of context representation or word/character level modeling, can be very challenging for joint modeling. we use a modified sequence-to-sequence architecture, where some components of the encoder are shared between a tagger, for the non-lexicalized features, and the encoder-decoder architecture, for the lexicalized features. we also use separate decoders for the different lexicalized features, that share the same encoder and trained jointly using a shared loss function. the remainder of this section discusses the architecture in more detail. ## conclusions and future work we presented a joint modeling approach for the lexicalized and non-lexicalized features in morphologically rich and semitic languages. our model achieves a significant improvement over several baselines for arabic, and matches the baseline for msa without having to use an expensive morphological analyzer. the results highlight the benefits of joint modeling, where diacritization seems to have benefitted the most. we observe, however, that further research is needed to enhance the overall consistency of the predicted features, without relying on external morphological analyzers.