Morphosyntactic Tagging with Pre-trained Language Models for A rabic and its Dialects (2022)

Abstract: We present state-of-the-art results on morphosyntactic tagging across different varieties of Arabic using fine-tuned pre-trained transformer language models. Our models consistently outperform existing systems in Modern Standard Arabic and all the Arabic dialects we study, achieving 2.6% absolute improvement over the previous state-of-the-art in Modern Standard Arabic, 2.8% in Gulf, 1.6% in Egyptian, and 8.3% in Levantine. We explore different training setups for fine-tuning pre-trained transformer language models, including training data size, the use of external linguistic resources, and the use of annotated data from other dialects in a low-resource scenario. Our results show that strategic fine-tuning using datasets from other high-resource dialects is beneficial for a low-resource dialect. Additionally, we show that high-quality morphological analyzers as external linguistic resources are beneficial especially in low-resource settings.

## introduction fine-tuning pre-trained language models like bert @xcite has achieved great success in a wide variety of natural language processing (nlp) tasks, e.g., sentiment analysis @xcite ), question answering @xcite , named entity recognition @xcite , and dialect identification @xcite . pre-trained lms have also been used for enabling technologies such as part-of-speech (pos) tagging @xcite @xcite to produce features for downstream processes. previous pos tagging results using pre-trained lms focused on core pos tagsets; however, it is still not clear how these models perform on the full morphosyntactic tagging task of very morphologically rich languages, where the size of the full tagset can be in the thousands. one such language is arabic, where lemmas inflect to a large number of forms through different combinations of morphological features and cliticization. additionally, arabic orthography omits the vast majority of its optional diacritical marks which increases morphosyntactic ambiguity. a third challenge for arabic is its numerous variants. modern standard arabic (msa) is the primarily written variety used in formal settings. dialectal arabic (da), by contrast, is the primarily spoken unstandardized variant. msa and different das, e.g., gulf (glf), egyptian (egy), and levantine (lev), vary in terms of their grammar and lexicon to the point of impeding system usability cross-dialectally @xcite . furthermore, these variants currently differ in the degree of data availability: msa is the highest resourced variant, followed by glf and egy, and then lev. in this paper, we explore different training setups for fine-tuning arabic pre-trained language models in the complex morphosyntactic tagging task for four arabic variants (msa, glf, egy, and lev) under controlled experimental settings. we aim to answer the following questions: our system foot_0 achieves state-of-the-art (sota) performance in full morphosyntactic tagging accuracy in all the variants we study, resulting in 2.6% absolute improvement over previous sota in msa, 2.8% in glf, 1.6% in egy, and 8.3% in lev. hfydk in the different variants of arabic. the table also shows the full range of morphological features: part-of-speech (pos), aspect (asp), mood (mod), voice (vox), person (per), gender (gen), number (num), case (cas), state (stt) and clitics: proclitics (prc3, prc2, prc1, prc0) and enclitic (enc0). in addition to the lemma (lex), fully diacritized form (diac), and english gloss (gloss). ## related work arabic morphological modeling proved to be useful in a number of downstream nlp tasks such as machine translation @xcite speech synthesis (halabi, 2016), dependency parsing @xcite , sentiment analysis @xcite , and gender reinflection @xcite . we expect all of these applications and others to benefit from improvements in morphosyntactic tagging. there have been multiple approaches to morphological modeling for arabic. those approaches differ depending on the target tagset (pos vs full morphology) and the availability of linguistic resources. when it comes to msa and da full morphological tagging, madamira @xcite trained separate svm taggers for each morphological feature (including cliticization) and selected the most probable answer provided by an external morphological analyzer all in one step for both msa and egy. amira @xcite on the other hand used a cascading approach where it performed pos tagging after automatically segmenting the text. a more recent similar approach to madamira was introduced by @xcite but using a neural architecture instead. @xcite presented a multitask neural architecture that jointly models individual morphological features for msa. @xcite extended @xcite 's work using multitask learning and adversarial training for full morphological tagging in msa and egy. similarly, @xcite proposed an approach where they jointly model lemmas, diacritized forms, and morphosyntactic features, providing the current state-of-the-art in msa. the same approach was used in @xcite , where they focused on the effect of the size of the data and the available linguistic resources and the impact on the overall performance on morphosyntactic tagging for glf. zalmout (2020) provides the current state-of-the-art performance in lev by extending @xcite 's work to lev. another line of research that works with da includes @xcite , where they presented a multi-dialectal crf pos tagger, using a small set of 350 manually annotated tweets for each of egy, glf, lev, and maghrebi arabic @xcite . we do not evaluate on their data because their task is defined as shallow morpheme segmentation and tagging; this is quite different from, and not easily mappable to, our task, where we disambiguate morphosyntactic features of the whole word without identifying its morpheme segments. additionally, their tagset includes social media specific tags, such as hash, emot, and mention, which are not in any of the large standard dataset and analyzers we study in this paper. pre-trained lm-based efforts in arabic morphosyntactic tagging are relatively limited and either assume gold segmentation or only produce core pos tags. @xcite leveraged the multilingual bert model with additional wordlevel and character-level lstm layers for lemmatization and morphological tagging, assuming gold segmentation. they reported the results for the sig-morphon 2019 shared task @xcite , which includes msa. @xcite reported pos tagging results in msa, glf, and egy using bert models pre-trained on arabic text with various pre-training configurations. they do not assume pre-segmentation of the text, however, they only consider the core pos tag, rather than the fully specified morphosyntactic tag. @xcite proposed a self-training approach for core pos tagging where they iteratively improve the model by incorporating the predicted examples into the training set used for fine-tuning. in this paper, we work with full morphosyntactic modeling on unsegmented text in four different variants of arabic: msa, glf, egy, and lev. furthermore, we explore the behavior of the pretrained lm with respect to fine-tuning data size under different training setups. given the available resources, we recognize our results' limitations in terms of applicability to different genres and styles, as well as noisy social media text and roman script arabic text @xcite . ## conclusion and future work in this paper, we presented the state-of-the-art results in the morphosyntactic tagging task for modern standard arabic and three arabic dialects that differ in terms of linguistic properties and resource availability. we conducted different experiments to examine the performance of pre-trained lms under different fine-tuning setups. we showed that the factored model outperforms the unfactored model