Joint Learning Model for Low-Resource Agglutinative Language Morphological Tagging (2023)

Abstract: Due to the lack of data resources, rule-based or transfer learning is mainly used in the morphological tagging of low-resource languages. However, these methods require expert knowledge, ignore contextual features, and have error propagation. Therefore, we propose a joint morphological tagger for low-resource agglutinative languages to alleviate the above challenges. First, we represent the contextual input with multi-dimensional features of agglutinative words. Second, joint training reduces the direct impact of part-of-speech errors on morphological features and increases the indirect influence between the two types of labels through a fusion mechanism. Finally, our model separately predicts part-of-speech and morphological features. Part-of-speech tagging is regarded as sequence tagging. When predicting morphological features, two-label adjacency graphs are dynamically reconstructed by integrating multilingual global features and monolingual local features. Then, a graph convolution network is used to learn the higher-order intersection of labels. A series of experiments show that the proposed model in this paper is superior to other comparative models.

## introduction morphological tagging describes the lexical information of a word in a sentence from the part-ofspeech (pos) and morphological features (mfs; case, person, mood, tense, etc.) @xcite and is an essential task in agglutinative language information processing. morphological tagging can analyze semantics @xcite , so some morphological knowledge will be added to many downstream tasks, such as dependency parsing @xcite , named entity recognition @xcite , language models @xcite , and machine translation @xcite , to assist the model in learning semantics and improve interpretability. an example is given in the cats are sleeping . the cat be sleep . det n;pl v;prs;3;pl v;v.ptcp;prs punt in agglutinative languages, a word is formed with a lemma and several suffixes. in the uyghur word, "almidi" (translation: he/she/they did not pick up), "al" is lemma, "mi(a)", "d" and "i" are suffixes, its msd label: 'v;sg/pl;3;neg;pst'. the lemma represents the word's meaning, and the suffix represents the grammatical category @xcite . each suffix represents grammatical information and corresponds to a morphosyntactic description label @xcite . morphological taggers in low-resource agglutinative language mainly focus on rule-based and statistical models @xcite @xcite , while relatively few studies are based on transfer learning or deep learning @xcite @xcite . conventional methods rely on human-designed rules, which are limited to surface rules in the dataset and cannot capture hidden rules and learn or represent deep grammar rules. in this paper, we investigate (1) which features in agglutinative language are related to msd labels, (2) how to reduce the direct impact of error propagation, and (3) whether it is possible to accurately predict more complete msd labels using label relationships and word representation in lowresource languages. therefore, to overcome these issues, we first represent the input word by contextual and word-formation features. second, to reduce error propagation caused by pos, morphological tagging is divided into pos tagging and mf tagging. through literature research and experiments, it has been found that pos can alleviate ambiguity, and pos affects the prediction of morphological feature labels. inspired by the work of @xcite , a fusion mechanism is adopted for the middle layer of the two tasks. finally, the output of the fusion mechanism is input into the conditional random field (crf) layer to predict the pos of each word. we pretrain labels in the mf tagging model and calculate relevant label co-occurrence statistics for the high-resource agglutinative lan-guage to learn the relationship between labels. the co-occurrence of irrelevant labels is calculated for the uyghur (ug), kazakh (kz), tatar (tt), and yakut (yk) datasets. a dynamic adjacency graph is reconstructed by using the above relationships and a gcn to learn the label relationship again to find the hidden relationships between labels. then, the mf labels of each word are predicted by using the word feature and label relationship. we evaluate our model on four low-resource agglutinative languages, uyghur, kazakh, tatar, and yakut, in universal dependencies (ud), and experiments show that the performance of the model proposed in this paper is superior to that of other comparable models. the model's average accuracy in four languages reaches 85.29%, and the average f1 score reaches 92.61%. our contributions are highlighted as follows: ## related work morphological processing is the primary task of natural language processing. relevant tasks include but are not limited to the following: morphological tagging (özateş and çetinoglu, 2021), morphological segmentation @xcite , lemmatization (zalmout and habash, 2020), and morphological analysis @xcite . there is also a close connection between these subtasks. for example, morphological analysis can be split into lemmatization and morphological tagging. similar to other tasks, morphological tagging tasks can also be summarized as rule learning @xcite , statistical learning @xcite , and deep learning @xcite , according to different research methods. recurrent neural networks or pretrained language models have been widely used in morphological tagging for high-resource languages. nine cross-lingual models were submitted for sigmorphon 2019 shared task 2, significantly promoting the development of morphological analysis @xcite . the winning model, udify, was proposed by @xcite and combines a multilingual pretrained language model and several fine-tuning strategies. they trained multilingually over all treebanks in the first stage and then monolingually used saved multilingual weights in the second stage. finally, the model predicts each grammatical category. @xcite proposed a fully neural natural tagging model, combo, for accurate pos tagging, morphological analysis, lemmatization, and (enhanced) dependency parsing. it is a bert-based end-to-end multilingual model. @xcite studied word segmentation and morphological analysis of sanskrit and proposed three models: word segmentation, morphological analysis, and combined segmentation and analysis models. the combined segmentation and analysis model is an end-to-end pipeline model. @xcite proposed a morphological analysis and generation model for more than one thousand languages. they leveraged a parallel corpus to project from english to other low-resource languages and exploited a morphological annotation tool. two separate sequence transduction models, one neural and one nonneural network model, were trained, and each model produced an n-best list. the tagging model achieved better performance in high resources. cotterell and heigold (2017) trained character-level recurrent neural taggers through language transfer to predict the morphological tagger of high and low resource languages. learning joint character representations among multiple related languages successfully enables knowledge transfer from the high-resource languages to the low-resource, improving accuracy by up to 30%. it is difficult to achieve high accuracy using a deep learning or cross-lingual transfer learning model for low-resource languages. because neither fst nor crf requires a large dataset, it is common to achieve morphological tagging by using these two models for low-resource languages. in the fst method (tolegen et al.; @xcite , researchers construct language resources, such as morphological and orthographic rules, and then design a morphological analyzer. @xcite proposed a character-level morphological collaborative analysis model based on the crf. morphological segmentation, annotations, and phonetic changes were combined into a composite label for each character. @xcite proposed a sequence-to-sequence model of morphological disambiguation. it was hypothesized that the vector representation of the correct analysis should be closer to that of the context vector, and the model predicts the correct msd label by calculating the similarity. fst and statistical methods require many morphological rules or highquality annotated data, leading to expensive labor costs. furthermore, the model cannot be combined with contextual features very well, which can easily cause ambiguity problems. in addition, some problems, such as lexical rule conflicts and long running times, may occur. therefore, to avoid ambiguity and improve the model performance of predicting more complete msd labels, the proposed model uses a pretrained model to represent words via contextual features. by jointly training pos and mf tags, the direct impact of pos tags on mf tags is reduced. the related morphological tag relations of the agglutinative language in the treebank are learned and transferred to low-resource languages, and introducing irrelevant tag relations between the low-resource languages and dynamically reconstructing the relations between tags, thus alleviating the problem of data resource scarcity. ## conclusion this paper proposes a joint morphological tagging model based on a neural network for low-resource agglutinative languages. first, to effectively capture the multi-dimensional information of the input words, the model uses the morphological, word and local context features to represent the input words. second, to reduce the impact of the pos label on the mf label, the two models are jointly trained, and a fusion mechanism is used to complete the interac-tion between the two middle layers. finally, pos tagging is regarded as sequence labeling, and the crf model predicts the pos tag. mf tagging is regarded as a classification task, and a new adjacency graph is dynamically reconstructed by using the two relationships between labels and gcns. gcns are used to express the higher-order relationship between labels. this model combines the universal features of agglutinative languages and the characteristic features of ukty languages, learns the relationship between labels, and effectively alleviates the problems caused by data scarcity. the experiments conducted with ud treebanks show that the proposed morphological tagging model outperforms other models. we explore the morphological tagging model based on a neural network under low resources as an example. in future research, we will continue to optimize the model's relationship representation and threshold selection abilities, and further improving the model's performance.