Cross-Dialectal Transfer and Zero-Shot Learning for A rmenian Varieties: A Comparative Analysis of RNN s, Transformers and LLM s (2024)

Abstract: This paper evaluates lemmatization, POS-tagging, and morphological analysis for four Armenian varieties: Classical Armenian, Modern Eastern Armenian, Modern Western Armenian, and the under-documented Getashen dialect. It compares traditional RNN models, multilingual models like mDeBERTa, and large language models (ChatGPT) using supervised, transfer learning, and zero/few-shot learning approaches. The study finds that RNN models are particularly strong in POS-tagging, while large language models demonstrate high adaptability, especially in handling previously unseen dialect variations. The research highlights the value of cross-variational and in-context learning for enhancing NLP performance in low-resource languages, offering crucial insights into model transferability and supporting the preservation of endangered dialects.

## introduction this research is part of the dalih project foot_0 . the goal of the project is to create a robust digital platform for the exploration of both historical and contemporary forms of the armenian language. the project intends to offer freely accessible and opensource resources, which will include grammatically annotated corpora and a variety of nlp tools such as annotation models, datasets, asr models, among others. this study focuses on four varieties of armenian: classical armenian (ca), modern eastern armenian (mea), modern western armenian (mwa), and the getashen dialect (g). while ca, mea, and mwa have annotated corpora and models included in the universal dependencies (ud) project, the getashen dialect, which is an underdocumented variety 2 linguistic resources like annotated corpora and nlp models for tasks such as lemmatization, postagging, and morphological analysis are essential for both linguists and digital humanities scholars. these tools support tasks like corpus preannotation and the study of historical texts, oral traditions, and regional literature. in this study, we aim to develop models for these tasks across the four varieties, with a particular focus on the under-resourced getashen dialect. our contributions are threefold: ## linguistic background armenian is an indo-european language that constitutes a distinct branch marked by both diachronic and synchronic variation. the historical evolution of armenian comprises: a) classical armenian (5th-10th centuries a.d.), b) middle armenian (11th-16th centuries), and c) modern armenian (17th century to the present). synchronically, modern armenian includes modern eastern armenian (mea), modern western armenian (mwa) and numerous armenian dialects. all the written forms of the armenian language utilize the unique armenian alphabet. while the linguistic divergences in historical variation are considerable, they vary in degree among the two modern standards and dialectal varieties, depending on the areal and genetic distance of each within the armenian linguistic continuum (for more details, see @xcite ; for linguistic variation, see @xcite and @xcite ). this study explores the morphological and morphosyntactic annotation of the following armenian linguistic varieties: classical armenian, modern western armenian, modern eastern armenian and the getashen dialect. classical armenian (ca) is typologically a right-branching language with complex inflectional morphology and syntactic constructions, featuring a tripartite nominativeaccusative-genitive alignment and flexible word order. beyond the bible and other religious texts, numerous original and translated works across various genres (such as historiography, mathematics, medicine, etc.) have been preserved in classical armenian. currently, it is maintained exclusively for liturgical purposes. in contrast, mwa and mea, and the dialect of getashen are typologically left-branching, with nominative-accusative alignment and more agglutinative morphology. they exhibit a richer system of periphrastic constructions and maintain flexible word order. mwa and mea were standardized in the mid-19th century, leading to a rich written heritage. both standards are currently in use, with mwa primarily by the traditional armenian diaspora, and mea used in armenia, armenian community of iran, and armenian communities in ex-soviet countries. the getashen dialect belongs to the -um branch of the karabakh dialect family (for more details on armenian dialects, see @xcite and @xcite ). this dialect is used in oral form. the data utilized in this study were recorded and transcribed as part of the project "migration and complex identities in the republic of armenia (an interdisciplinary study in anthropology and linguistics; migrant groups in armenia : parameters of complex identities)" during fieldworks conducted in armenia in 2014-2015 @xcite . ## related work lemmatization, pos-tagging, and morphological analysis are fundamental tasks in nlp, historically tackled using rnn-based approaches @xcite , or lstm models often combined with pre-trained word embeddings like glove or fasttext for contextual word representations. however, state-of-the-art transformer models like bert @xcite , roberta, and xlm-r have significantly outperformed these traditional methods by capturing long-range dependencies and contextual information more effectively through selfattention mechanisms, which allow them to manage complex linguistic patterns. these transformer-based approaches, though highly effective, generally require large amounts of annotated data, making them less suitable for historical and under-resourced languages due to data scarcity. to address this challenge, recent trends have focused on leveraging attention mechanisms combined with transfer learning from models like deberta @xcite or utilizing large language models (llms) in assisted annotation frameworks for these languages @xcite @xcite . despite these advancements, very underresourced languages like coptic, ancient egyptian, or old french still predominantly rely on seq2seq architectures, often using lstm or gru units with attention mechanisms to handle sequences and generate lemmas or morphological patterns @xcite @xcite . in the case of armenian, most lemmatization, pos-tagging, and morphological analysis experiments have focused on modern eastern armenian @xcite and classical armenian (vidal-gorène and @xcite @xcite , using lstm, joint learning methods with rnns or rule-based approaches @xcite , achieving f1-scores ranging from 0.63 to 0.87 depending on the task and text genre (e.g., gospel, colophon, htr output or historiography). these methods have also been applied to mea with similar results @xcite . experiments in transferring mea models to armenian dialects, including mwa, have reported accuracies around 65% in lemmatization and 80% in pos-tagging @xcite . the application of transformer models or llms to armenian linguistic tasks remains in its early stages, with current usage primarily in classification tasks @xcite . ## armenian datasets this study draws upon four datasets representing different armenian dialects. three of these datasets are sourced from the universal dependencies (ud) project (de marneffe et al., 2021): classical armenian (ca) foot_2 , modern eastern armenian (mea) foot_3 , and modern western armenian (mwa) foot_4 . the fourth dataset, representing the getashen (g) dialect, was compiled and transcribed as part of the project "migration and complex identities in the republic of armenia" @xcite . the ud datasets are designed to provide comprehensive morphological and syntactic annotations following ud guidelines, covering a wide range of armenian language varieties. in contrast, the getashen dataset consists of raw transcribed text, from which a small number of sentences have been manually annotated specifically for this study. ## methodology our approach aims to understand how different models perform on token-level annotation tasks -lemmatization, pos-tagging, and morphological feature tagging -across multiple armenian dialects with varying levels of resources and label sets. we explore a unified sequence labeling framework to handle these tasks, leveraging different model architectures, including rnns, pretrained transformers (mdeberta), and large language models (llms). by comparing these models in supervised, transfer learning, and zero/few-shot learning settings, we study how well they generalize across dialectal variations and whether com-bining data from multiple dialects improves performance, particularly for those with limited training data. codes and raw results are available on github. foot_5 . ## conclusion the results from this study demonstrate the varying strengths of different model architectures in handling token-level annotation tasks across multiple armenian dialects. rnn models consistently performed well, particularly in pos-tagging, where they achieved near-perfect f1 scores of up to 0.98, showcasing their robustness for tasks with known tokens. however, they struggled with less frequent or unseen tokens, where the adaptability of large language models (llms), especially in zero-shot and few-shot learning setups, became evident. for instance, chatgpt-4 in zero-shot learning achieved an f1 score of 0.83 in lemmatization for the getashen dialect. pretrained transformers like mdeberta, while showing potential in handling morphological features with f1 scores reaching 0.73, often lagged behind rnns and llms, particularly in lemmatization tasks, where their performance dropped to as low as 0.36 in the mea dialect. cross-dialect transfer learning revealed that non-specialized models can often generalize better across dialects than specialized ones, suggesting that a cross-dialect approach may be more effective for low-resource languages. in-context learning with llms further highlighted their ability to rapidly adapt and improve performance, particularly in lemmatization, where sequence sampling led to an increase in f1 scores from 0.62 to 0.96 with just 10 samples. in the future, we plan to generalize our approach to include multiple other dialects and to ensure normalization of annotations, facilitating more consistent and comprehensive linguistic analysis.