Improving POS Tagging for A rabic Dialects on Out-of-Domain Texts (2022)

Abstract: We investigate part of speech tagging for four Arabic dialects (Gulf, Levantine, Egyptian, and Maghrebi), in an out-of-domain setting. More specifically, we look at the effectiveness of 1) upsampling the target dialect in the training data of a joint model, 2) increasing the consistency of the annotations, and 3) using word embeddings pre-trained on a large corpus of dialectal Arabic. We increase the accuracy on average by about 20 percentage points.

## introduction although pos tagging has achieved high results across languages and benchmarks @xcite @xcite , there are still challenges, particularly across different domains and for languages with rich morphology, especially in terms of handling rare and unknown words @xcite . for languages such as arabic, their diglossic nature adds additional complexity, as pos tagging models must capture a plethora of lexical and syntactic variation plus orthographic differences. for arabic, the majority of available pos taggers are trained on modern standard arabic (msa), such as madamira @xcite and farasa @xcite . there is however a growing interest in developing tools specifically for dialectal arabic (described in @xcite and @xcite ), given its preferred use in daily communication, especially on social media platforms and integrated into voice systems. our ultimate goal is the computational analysis of syntactic differences across arabic dialects, which requires syntactically annotated parallel data. however, the existing dialectal parallel corpus, ## arabic dialects dialects of arabic show a wide range of linguistic differences, within the dialects themselves and compared to msa. msa is mostly used in formal writing such as books and news articles while dialects are used for most other daily communications. arabic dialects are interesting because much of the variation involves function words, providing strong signals of the presence of syntactic differences. in dialect sentence buckwalter foot_0 msa mrad atzn an alm@xmath0klp rh tnhl hnak egy mrad fakr an alm@xmath1 ktzn alm$klp gady thl tmak eng. do you think (that) this is the solution for the problem murad? ## research questions our main question is how we can improve pos tagging for dialectal arabic when testing on out-ofdomain data. to address this question, we break it down into four sub-questions: 1) does the pos tagger profit more from having access to a large training set even though the majority of training examples are from a different dialect, or is a smaller, dialect specific training set more appropriate? 2) does upsampling help mitigate the data imbalance in a joint dialectal model? 3) can we increase consistency in annotations, using minimal effort? and will increased consistency yield an increased accuracy? 4) can using pre-trained embeddings improve pos tagging performance? a problem concerning dialect specific taggers is that they do not use uniform annotation schemes. thus, they may be ineffective in a cross-dialectal setting. @xcite approach this problem by introducing a multi-dialectal pos tagger for the dialects of gulf, levantine, egyptian and maghrebi by developing a crf tagger, which is extended by @xcite to using bi-lstm layers as input. their system provided state-ofthe-art performance for pos tagging of dialectal arabic. ## addressing the data imbalance one drawback of using a joint model of the four dialects is that it is trained on only 25% examples of the target dialect, which means that dialect specific, correct decisions may be overruled by other dialects. in section 6.2, we showed that creating word original pos new pos negation markers part neg_part interrogatives part adv rel. pronouns part pron fut and prog markers inconsistent fixed unmarked conj part/noun conj adverbs noun adv verbal suffixes pron concatenated verb and suffix nominal suffixes nsuff concatenated noun and suffix this shows that upsampling can successfully combine the advantages of having a large training set and a focused one. 11 11 we also explored tripling the number of samples for the ## annotation changes a closer examination of the pos tagger errors shows that in some cases, the problems derive from the gold annotations of the training data. apart from the expected annotation errors due to lack of attention, which are mostly random, we also find more systematic inconsistencies, partly across dialects. one such inconsistency concerns dialect-specific pos tags, such as negation, progressive, and future markers. for instance, in the glf data, none of the negation markers were annotated with the negationspecific pos tag. systematic inconsistencies can potentially be corrected semi-automatically. to address the annotation inconsistencies, we further experiment with annotation changes on the darwish corpus (our training data; @xcite ). we created a list of annotation inconsistencies, focusing on those which can be found and corrected automatically. we used the camel lab guidelines 12 as a reference since they provide specific and consistent pos tagging guidelines for dialects of arabic. we performed systematic changes on the corpus while maintaining consistency across dialects. a list of the targeted annotations in shown in for the distribution of pos tags in each dialect before and after the corrections, see figure 2 , focusing on the four most frequent pos tags. the plots show that in the original annotations, the ratios per pos tag are similar across dialects; after the corrections, there are more differences, showing that we model differences between dialects better. note that the size of the corpus has changed due to the annotation changes, resulting in differences in pos tag distributions within dialects: we reattached the verb suffixes (previously tagged as pron), for example, the verb (v) and the suffix (pron) are reattached into a single word . we also reattached nominal suffixes (previously tagged as nsuff), such as (noun) and (nsuff) into the single word . as a consequence, the number of words decreases (e.g., the word count for levantine decreases by 6%). reattaching verbal suffixes also causes a decrease in pronouns across all dialects but levantine shown in figure 2 . in lev, relative pronouns which were originally tagged as part are now categorized as pron. we then perform experiments training a joint model on all dialects after annotation modification, and test on each dialect separately to check whether the annotation changes boost the tagging performance. results are reported in ## using pretrained word embeddings next we investigate whether word embeddings can be beneficial and have a positive impact on the quality of pos tagging. the assumption is that the pre-trained word embeddings derived from large corpora of dialectal arabic can help mitigate problems with lexical coverage in the randomly initialized embeddings in the out-of-domain setting. the choice of the pretrained embeddings is important. we use the word embeddings trained on a large corpus of dialectal arabic @xcite . to train the embeddings, erdmann et al. (2018) collected data for four major dialects of arabic: gulf, levantine, egyptian, and maghrebi, which cover the four dialects of our test data. the corpora are a mix of crawled data from a variety of forums and blogs, including comments on posts @xcite @xcite , madar @xcite , news commentary corpus @xcite , tweets from the corpus of palestinian arabic @xcite , with the number of sentences per dialect ranging between 1.1m and 1.7m. the model was trained using fasttext (bojanowski et al., 2017) 13 . since madar is part of the training data for the embeddings, we can expect a higher lexical coverage for the test data. we also had a look at the tagging errors for the model using the pre-trained embeddings. a heatmap of pos tag precision is provided in figure 3 . we see that numbers are still the most difficult pos tag, similar to the results in figure 1 . however, for all dialects but mag, the accuracies are considerably higher. for mag, most of the numbers were mistagged as noun. this seems to be due to inconsistencies in the training data. since the spelling of numbers tends to differ between dialects, the pos tagger cannot learn from the other dialects. the same pattern of gains holds for particles, previously the second most difficult category, except for mag. the current second most difficult pos tag are adjectives. here we see a decrease over all dialects in comparison to figure 1 . this can be explained by the systematic ambiguity between nouns and adjectives. the pos tagger seems to favor annotating these ambiguous words as adjectives, which leads to a high precision for nouns, 13 we do not use bert embeddings since they cannot be easily integrated into the pos tagger architecture. see for instance, the adjective (eng.: sorry) is tagged as noun because of its alternative interpretation 'regret'. ## conclusion and future work we have investigated pos tagging for arabic dialects when the test set is out-of-domain. this setting has proven to be difficult, originally resulting in a low accuracy. our work shows that we can improve the pos tagger's accuracy by upsampling the target dialect in the training data, by increasing consistency of annotations, and by using word embeddings pre-trained on a large corpus of dialectal arabic. on average we have seen improvements of about 20 percent points. our overarching goal is the investigation of morpho-syntactic and syntactic differences between arabic dialects. our next step is to experiment with the granularity of pos tags. the current small pos tagset may not provide enough information for an investigation of syntactic differences. we also plan to develop a parsing model for arabic dialects.