Morphological Analysis Corpus Construction of U yghur (2021)

Abstract: Morphological analysis is a fundamental task in natural language processing and results can beapplied to different downstream tasks such as named entity recognition syntactic analysis andmachine translation. However there are many problems in morphological analysis such as lowaccuracy caused by a lack of resources. In this paper to alleviate the lack of resources in Uyghurmorphological analysis research we construct a Uyghur morphological analysis corpus based onthe analysis of grammatical features and the format of the general morphological analysis corpus. We define morphological tags from 14 dimensions and 53 features manually annotate and correctthe dataset. Finally the corpus provided some informations such as word lemma part of speech morphological analysis tags morphological segmentation and lemmatization. Also this paperanalyzes some basic features of the corpus and we use the models and datasets provided bySIGMORPHON Shared Task organizers to design comparative experiments to verify the corpus’savailability. Results of the experiment are 85.56% 88.29% respectively. The corpus provides areference value for morphological analysis and promotes the research of Uyghur natural language processing.

## introduction morphological analysis is the process of dividing words into different morphologies or morphemes and analyzing their internal structure to obtain grammatical information. it is an essential step in lexical analysis. therefore, @xcite showed that the related research of morphological analysis has also attracted the attention of most researchers. in natural language processing based on deep learning, researchers have found that if modeling to words directly, it is easy to ignore the relationship within the words, which will also bring limitations to the model. thus, the model's input has changed from a single word to a character, subword, and morpheme or morphology. when the word is split into different granularities @xcite , @xcite , @xcite ), the performance of the model is improved. among these words' segmentation methods, morphology or morpheme relies on the morphological analysis tagger or manual annotation. in languages with mature natural language processing technologies such as english, finnish, and chinese, there are many manually annotated morphological analysis corpus, and the lexical analysis (including morphological analysis) technology of these languages has reached a high level. to further promote the lexical analysis of minority languages in xinjiang, many researchers have obtained preliminary results in lexical analysis. @xcite constructed a uyghur lexical tagging corpus of 1.23 million words; @xcite built a uyghur stemming corpus of 10,000 sentence sets; @xcite annotated about 30,000 kazakh sentences and studied a lexical analysis of kazakh; @xcite constructed a character-level uyghur morphological collaborative word corpus and annotated morphological analysis for 3500 sentences; for the first time, @xcite released the uyghur language based on morpheme sequence for morphological segmentation corpus; the corpus includes about 20,000 uyghur words, including wordlevel and sentence-level corpus; based on the statistics of uyghur noun affixes, @xcite proposed a hybrid strategy for the uyghur noun morphological re-inflection model; @xcite constructed a small-scale stemming corpus for uzbek, which includes 7435 words and 568 sentences in total. up until now, no reports have been published about the publicly available uyghur of morphological analysis corpus.in addition, because there is no unified reference standard, the private morphological analysis datasets created by previous studies have fewer or incomplete features.this is not only directly affecting the development of the language's morphological analysis technology but also indirectly affecting the performance of downstream tasks, such as named entity recognition @xcite , syntactic analysis @xcite , text classification @xcite , and machine translation @xcite . therefore, to alleviate the shortage of uyghur language morphological analysis resources, this paper refers to the format of universal morphological feature schema (unimorph schema), analyzes uyghur words from 14 dimensions and 53 features, constructs a uyghur language morphological analysis corpus, and provides lemma, part of speech. morphological analysis tags, morphological segmentation, and lemmatization. this can be used for various lexical analysis tasks, like morphological analysis, stemming, and other downstream tasks of natural language processing. ## construction of uygur morphological analysis corpus this chapter will introduce the construction process of the uyghur morphological analysis corpus. firstly, a more suitable morphological annotation guidelines is proposed based on sylak-glassman et al. ## corpus information statistics and evaluation the statistical distribution of words and sentences in each domain is shown in figure 2 , which including 5014 sentences, 152669 words, 24631 tokens. the average word repetition rate is 1:6, and it can be found that each sentence contains an average of 30 words. compared with other categories, the number of news in finance is most, sport news has the largest number of words, and entertainment news has more tokens. figure 3-(a ) and figure 3-(b ) respectively represent the distribution of sentence length and word length in the corpus. the abscissa represents the length of the statistics, and the ordinate represents the number of statistics. figure 3-(a ) sentence length statistics figure, most of the sentence length is in the (40,140), the shortest sentence length is 3, the longest is 195. figure 3-( (a) sentence length statistics (b) word length statistics figure 3: sentence and word length statistics c c l 2 0 2 1 in general, a sentence contains at least one verb and several nouns, and there are a small number of conjunctions or interjections, etc. each type of word has a different grammatical category, and as the number of words increases, its grammatical features will increase. according to the morphological labels, the pos distribution, morphological feature distribution, and morphological label distribution of some words are respectively counted, as shown in figure 4 from the part of speech distribution in figure 4 -(a), we found that the proportions of noun (n), adjective (adj), pronoun (pro), verb (v), and auxiliary verb (aux) are more than the proportions of classifier (clf), interjection (intj) and imitate (i). from the distribution of lexical features in figure 4-(b ) and the morphological distribution label in figure 5 , we observed that the proportion of grammatical labels that modify nominal words is the largest, such as sg, nom, and pss3s, etc., followed by verbs or auxiliary verbs, such as msdr, ptct, and so on. this statement can be further verified from figure 5 . also, it proves that the morphology of verbal words is more complex and richer than nominal words. to verify the effectiveness of the morphological analysis dataset constructed in this paper, we select part of the agglutinative or morphologically complex languages and use baseline models provided by the sigmorphon shared task. and we design a comparative experiment. evaluation task definition given the word, lemma, and morphological label to train model, so that the model predicts the word that by the lemma and morphological labels, such as given the lemma "shadow" and morphological label "v;sg;3;prs" to predict words "shadows". the experiment uses two modes provided by the task organizers, a non-neural baseline @xcite and a neural baseline @xcite . the neural baseline is a multilingual transformer. the version of this model adopted for character-level tasks currently holds the state-of-the-art on the 2017 sigmorphon shared task data. the transformer takes the lemma and morphological tags as input and outputs the target inflection. the non-neural baseline heuristically extracts lemma-to-form transformations; it assumes that these transformations are suffix-or prefix-based. a simple majority classifier is used to apply the most frequent suitable transformation to an input lemma, given msds, yielding the output form.