Part-of-Speech and Morphological Tagging of A lgerian J udeo- A rabic (2022)

Abstract: Most linguistic studies of Judeo-Arabic, the ensemble of dialects spoken and written by Jews in Arab lands, are qualitative in nature and rely on laborious manual annotation work, and are therefore limited in scale. In this work, we develop automatic methods for morpho-syntactic tagging of Algerian Judeo-Arabic texts published by Algerian Jews in the 19th–20th centuries, based on a linguistically tagged corpus. First, we describe our semi-automatic approach for preprocessing these texts. Then, we experiment with both an off-the-shelf morphological tagger and several specially designed neural network taggers. Finally, we perform a real-world evaluation of new texts that were never tagged before in comparison with human expert annotators. Our experimental results demonstrate that these methods can dramatically speed up and improve the linguistic research pipeline, enabling linguists to study these dialects on a much greater scale.

## introduction application of natural language processing (nlp) to real-world problems has been the field's goal from its early days. as algorithms advance, the contribution of nlp to real problems has become more evident and more substantial. the present study originates from a real-world challenge faced by linguists of semitic languages, in this case researchers of the judeo-arabic dialects of algeria (aja). their challenge, simply put, is how to scale up linguistic analyses of such dialects. semitic languages in general, and arabic in particular, are characterized by a very rich morphology that uses both templatic and concatenative morphemes, combined with the use of a vowelless script ("abjad"). this makes morphological analysis of arabic very time-consuming even for expert linguists. because speakers of the aja dialects are becoming scarce, the attention of linguists in this field has shifted from fieldwork interviews with native speakers to library-based analysis of texts written in those dialects. fortunately, vast collections of aja texts were preserved in printed books, journals and handwritten manuscripts. analyzing this linguistic treasure-trove, however, is proving to be challenging due to its size. the time-consuming manual annotation does not scale, and requires expertise that is hard to find. we aim to scale up the linguistic analysis of this arabic dialect using nlp tools. in particular, our goal is to develop an nlp tool that will assist aja linguists in their real-world task, in a way that they will find it useful. basing our work on the existing linguistically tagged algerian judeo-arabic (taja) corpus (tirosh-becker and becker, 2022), we set out to develop automatic methods for morpho-syntactic tagging of such texts. several specially designed neural network taggers and an off-the-shelf morphological tagger were experimented with, and assessed for their accuracy and likely usefulness. we also considered a hybrid humanin-the-loop approach. finally, we carried out a realworld evaluation of our best performing part-of-speech (pos) taggers, applying them to untagged texts and assessing their quality via a user study with expert aja linguists. our experimental results demonstrate that these methods can dramatically speed up and improve the linguistic research pipeline, enabling linguists to study this language on a much greater scale. judeo-arabic (ja) lies in the intersection of semitic languages and jewish languages. as a semitic language, and more specifically, an arabic language variety, its words are generally composed of 3-letter roots, with added vowels and consonants according to pattern paradigms, as well as affixes and clitics @xcite . arabic is the most widely spoken semitic language, with 300 million native speakers @xcite . in fact, the term 'arabic' refers both to modern standard arabic (msa) and to the arabic dialects spoken throughout the arab world. the two varieties of arabic coexist in a state of diglossia @xcite or continuuglossia @xcite , meaning the language varieties exist side by side, with writers or speakers shifting between varieties according to circumstance. msa is written using the arabic script, which is a right-to-left alphabet. arabic dialects are usually written in arabic script as well, but there is no standardized spelling for dialectal arabic @xcite . arabic uses both templatic and concatenative morphemes. there are two types of templatic morphemes: roots and templates. roots are usually three consonantal radicals that signify some abstract meaning. roots are inserted into abstract patterns called templates. there are two kinds of concatenative morphemes that attach to the templatic morphemes. clitics are morphemes that have the syntactic characteristics of words, but are phonologically bound to another word @xcite , for example "wa", foot_0 meaning "and". affixes are phonologically and syntactically part of the word, and often represent inflectional features, such as person, gender, number, and more. dialectal arabic (da) is a primarily spoken family of language varieties (and in modern days, widely used in written form on social media as well) that exist alongside the written msa. da diverges from msa on several levels. there are differences in phonology, morphology, lexicon, and orthography @xcite . the regional dialects can be broken down into main groups, with one possible breakdown being egyptian, levantine, gulf, iraqi, and maghrebi. even within dialect groups there can be quite a lot of variance between dialects, although in many cases there is a certain level of intelligibility between speakers of different dialects, with more significant difficulty across dialect groups. maghrebi dialects are influenced by the contact with french and berber languages, and the western-most varieties could be unintelligible by speakers from other regions in the middle east, especially in spoken form @xcite . while ja can be looked at as an ensemble of arabic dialects, it is first and foremost a subgroup of jewish languages. jewish languages are a family of language varieties that developed in jewish communities throughout the diaspora. the original language used by jews in the land of israel was hebrew, followed closely by aramaic. as jews spread across the world, they adopted local languages and developed distinctive varieties of these languages. nonetheless hebrew remained their liturgical language, even as it almost died out as a spoken language until its revival in the late 19th and early 20th centuries. perhaps the most well-known of these jewish languages is yiddish, the judeo-german language developed by ashkenazi jews living in central and eastern europe before the holocaust. jewish languages vary in their distance and divergence from their non-jewish sister languages, some being influenced by multiple languages due to language contact. nonetheless, among the features that tie these languages together are the presence of hebrew and aramaic lexical components @xcite , the use of the hebrew alphabet for writing, and more. algerian ja (aja) is a member of the north african judeo-arabic dialect group, i.e., dialects spoken and written by jews of the maghreb. aja is in contact with moroccan and tunisian arabic dialects (both jewish and muslim), with french and to a lesser extent other trade languages such as spanish and italian, and with hebrew and aramaic, the historical jewish cultural languages. in general aja shares many characteristics with other jewish languages, including the use of hebrew script, presence of hebrew and aramaic components, and a mixture of conservative trends, vernacular features, and heterogeneous elements @xcite . to date, aja has been sparsely studied by linguists. the aja dialect of the city of algiers was studied over a century ago by @xcite , with most of the recent work on aja published by tirosh-becker, focusing on constantine, the third largest city in algeria @xcite @xcite . aja research employs fieldwork interviews of informants and the study of selected written texts (e.g., @xcite @xcite . regretfully, the number of aja speakers has decreased following algeria's independence @xcite and the subsequent dispersion of its jewish communities, making fieldwork today almost impossible. hence, this research is now shifting towards an analysis of the vast textual sources left by many of these jewish communities, in both manuscript and print form. most of the linguistic analyses done thus far on aja texts have been based on single or few texts, as each study requires extended effort of poring over texts, dictionaries, and grammars. given the size of these corpora, this is a perfect match for machine learning and nlp approaches. ## data this project has used the tagged algerian judeo-arabic (taja) corpus developed by tirosh-becker and becker (2022). 3 this aja corpus is a collection of modern aja texts published in algeria in the late 19th and the first half of the 20th century. the texts represent a variety of prose genres written by algerian jews, including: 3 the corpus is available through the authors. these texts were manually typed into computerreadable format and subsequently proofread, as hebrew ocr (optical character recognition) failed on these aja texts. this was due not only to the less-thanfavorable conditions under which the books had been stored, leaving the pages grayed and worn, but also because the fonts used in these books are not identical to standard hebrew, as they have ja-specific adaptations, such as diacritics. each text was manually tokenized and annotated by research assistants (ras, usually ma or phd candidates) in a spreadsheet, according to strict guidelines, and most were verified by a senior expert. the digitization and annotation project spanned several years, with some dozen ras contributing to the annotation efforts. approximately 80% of the time spent on the creation of taja was dedicated to the annotation process, as the digitization is a more straightforward (though non-trivial) task. ## preprocessing in this section, we describe several challenges we faced in the preprocessing stage and the steps we took to address them. ## real-world evaluation the end goal of this project is to provide aja language experts with an automatic tagger to help them annotate large volumes of text, a task which is otherwise laborious and time-consuming when tackled manually. to evaluate such real-world usefulness of the taggers we set out to compare the performance of our two best pos models (the hierarchical char-cnn based model and marmot) with that of manual annotation by two expert aja linguists. ## discussion and conclusion the pressing real-world challenge facing researchers of algerian judeo arabic (aja) dialects is how to scale up their linguistic analyses from individual texts to large textual collections. the rich morphology of arabic (as of other semitic languages) and scarcity of expert linguists makes this complex and time-consuming task impractical unless aided by automation. hence, developing automatic taggers that would support realworld linguistic analysis at scale and prove useful for aja linguists is the challenge we aim to tackle. reflecting the linguists' challenges, we focus on the performance of the morphological tagger in tests that are predictive of the real-world setting. for this reason, we did not limit ourselves to purely automated approaches, but also explored a hybrid human-machine approach, wherein the human expert contributes to the automatic approach. the rich morphology of arabic and its use of morpho-syntactic affixes led us to focus on characterbased models (rather than word-based models), as these can identify key morphemes that are essential for annotating oov words. starting from a wordbased lstm neural network architecture, we integrated character-level information via either an lstm or a cnn. subsequently we explored a two-tier hierarchical approach to morphological tagging with pos tags at its base and the morphology tags building on that. this hierarchy mirrors the underlying character of arabic annotation, where each pos tag has a set of legal morphological tags. the two-tier approach also enables exploring a human-in-the-loop step in between the two tiers. our best performing strategy, denoted ajatag for simplicity, is now available for use by aja linguists. 8 to evaluate the usefulness of the ajatag 8 https://github.com/technion-cs-nlp/ nlp4aja strategy we compared it to the off-the-shelf pos and morphological tagger, marmot, which is based on crf. all models were trained on the annotated taja corpus. for the base task of pos tagging, we found that among the evaluated neural network architectures, representing a word using a cnn run on its characters performed better than an lstm or ignoring the characters altogether. training on the taja corpus, the pos accuracy of the char-cnn model was 87.4±0.58%. this accuracy is only slightly lower than the 89.17% accuracy obtained by marmot for this task. the 1.5% difference suggest essentially similar performance for the two models in a real-world setting. morphology tagging, as indicated above, is the most challenging and time-consuming task that takes up 80% of the expert linguist annotation time. here, too, char-cnn performed better than the other neural network models we explored, especially in a two-tier hierarchical approach. the accuracy of this model, denoted herein as 'hierarchical char-cnn (predicted pos)', ranges from 81% to 91% for the different morphology analysis fields (anal-ysis1, analysis2, additional tags). to further improve the performance, we allowed for human input between the two tiers in the form of manual correction of pos tags. using 'true pos' assignments, instead of the predicted assignments, further improved the performance of the 'hierarchical char-cnn (true pos)' morphology tagger. we denote this hybrid strategy ajatag and have compared its performance on aja to marmot. we use mar-mot as is, without modifications or adaptations to a hybrid setting, because for the linguists it is an off-theshelf tool that is to be used as is. evaluation of the morphological tagging by ajatag demonstrated favorable performance across multiple evaluation metrics: it should be noted that the greatest gain in accuracy is in analysis1, which of the morphological analysis fields is the richest and most difficult to assign. both approaches perform well identifying the enclitic field with an accuracy greater than 96%. however, this important performance indicator is where our hybrid ajatag strategy delivered its most important fruits. the accuracy of ajatag in the challenging task of morphologically tagging oov words is 74.91% and 78.42% for the analysis1 and analysis2 fields, respectively, which is significantly better than marmot's oov tagging for these two fields (55.82% and 59.95%, respectively). ajatag also performs much better in the additional tags field for oov words (85.4% compared to marmot's 75.0%). the justification for the hybrid approach explored herein is in its real-world usefulness, outside of the nlp lab. the 56%-60% accuracy of the off-the-shelf solution for the two most important morphological fields, analysis1 and analysis2, when applied to oov words is not sufficient for real linguistic work. in contrast, the hybrid ajatag strategy achieved an accuracy level of 74.91%-78.42% on morphological tagging of oov words, which is expected to be useful for real-world applications, improving upon marmot by 18%-19% for this task on both analysis fields. it is reassuring that even without the added human input, our fully automated hierarchical char-cnn performed better than marmot on pos and analysis1 tagging of oov words. the value of the ajatag strategy was further confirmed by other performance indicators, including its overall accuracy and its accuracy on words with legal tag combinations, as defined above. to assess the feasibility of the human interface element in ajatag, we performed a real-world evaluation of this process. the first-tier pos output was given to two aja linguists to correct, before moving on to the second-tier morphology tagging. pos tags manually corrected by a senior expert were perceived as the 'true' pos assignment, to which the performance of the automatic taggers as well as the corrections by a junior expert were compared. it is reassuring that both automated taggers, our char-cnn model and marmot, performed well at an almost identical accuracy (~89%) relative to the 'true' pos, an accuracy quite similar to the 91% accuracy by the junior expert, who is a phd candidate with several years of experience in aja linguistics. to conclude, while not perfect, the hybrid ajatag approach provides aja linguists with a working solution that already impacts their real-world workflow in a way that off-the-shelf tools cannot provide. in the future we plan to continue improving these tools by addressing limitations such as tagging words with illegal tag combinations. nonetheless, we believe that even in its current form ajatag could prove useful to linguists as they take on the task of analyzing large untagged aja corpora. we hope that in the future we will be able to expand the utility of these tools to other judeo-arabic dialects.