Introducing Morphology in U niversal D ependencies J apanese (2023)

Abstract: This paper discusses the need for including morphological features in Japanese Universal Dependencies (UD). In the current version (v2.11) of the Japanese UD treebanks, sentences are tokenized at the morpheme level, and almost no morphological feature annotation is used. However, Japanese is not an isolating language that lacks morphological inflection but is an agglutinative language. Given this situation, we introduce a tentative scheme for retokenization and morphological feature annotation for Japanese UD. Then, we measure and compare the morphological complexity of Japanese with other languages to demonstrate that the proposed tokenizations show similarities to synthetic languages reflecting the linguistic typology.

## introduction this paper introduces morphology-aware tokenization and morphological features to universal dependencies (ud) treebanks for japanese. since its inception in 2015, the ud project has been developed to cover more than 130 languages as of v2.11 @xcite . its crosslinguistically consistent syntactic and morphological annotation has enabled corpusbased multilingual nlp at a greater scale @xcite . however, the japanese treebanks in the current ud v2.11 have divergent policies in terms of tokenization and morphological feature annotation. specifically, sentences are tokenized by morpheme boundaries with almost no morphological feature assigned, despite the linguistic fact that japanese has morphological inflection. given this issue, this paper will propose new tentative schemes for tokenization and morphological annotation that takes into account the synthetic nature of japanese. then, we will demonstrate that the retokenized japanese ud treebanks with these schemes have morphological complexities similar to other synthetic languages. these results agree with the typology of japanese as a synthetic agglutinative language. ## background this section overviews the japanese language and the annotation issues that the current japanese ud treebanks have. it is a typical head-final language with synthetic morphology, where grammatical information is mostly expressed by means of agglutination. ## related work in the nlp literature on japanese, the term "morphological analysis" has been used to refer to the task of morphological segmentation, given the fact that the japanese orthography does not explicitly contain word boundaries @xcite @xcite . since there is no solid linguistic criterion to define what a word is, the smallest meaningful unit (i.e., morpheme) is a stable candidate for tokenizing a language with no orthographic word boundary. this tokenization policy is common in japanese corpora, as is comprehensively defined in the balanced corpus of contemporary written japanese (bccwj) @xcite as suw and luw. existing japanese morphological analyzers such as mecab foot_2 and sudachi foot_3 are based on the same policy, and their main concern has chiefly been morpheme-level tokenization and pos tagging while leaving the analysis of morphological features untouched. the above-mentioned issues of japanese ud have already been raised by multiple researchers. @xcite gives a comprehensive overview of the tokenization of japanese ud from the viewpoint of general linguistics, concluding that the current tokenization scheme is an artifact of decisions made by the corpora on which the ud japanese treebanks were based-decisions which ud japanese should revisit for the sake of the crosslinguistic nature of ud. @xcite provides discussion on defining a word in japanese for ud, and demonstrates that a word (form) in japanese ud does not follow ud's general annotation guideline, which states that "morphological features are encoded as properties of words and there is no attempt at segmenting words into morphemes." foot_4 however, no actual implementation for retokenization has been realized. this situation in fact prevents japanese from being included in crosslinguistic studies with ud data. @xcite investigate various measures of morphological complexity with more than 50 ud treebanks, but they had to exclude japanese and korean treebanks because "no linguistically interesting features were marked despite the fact that both languages are morphologically complex." ## retokenization given the current issues with japanese ud, this section proposes tentative alternative annotation schemes that take into account synthetic aspects of japanese morphology. ## morphological complexity of japanese to confirm the validity of the morphology-aware japanese ud treebanks, this section reports experiments to measure the morphological complexity of japanese, which @xcite could not compare due to the lack of morphological information in current japanese ud. suw 魚 フライ を 食べ た か も しれ ない ペルシャ 猫 sakana hurai wo tabe ta ka mo sire nai perusya neko fish fry acc eat pst q also know neg persia cat noun noun adp verb aux part adp verb aux noun noun luw 魚フライ を 食べ た かもしれない ペルシャ猫 sakanahurai wo tabe ta kamosirenai perusyaneko fried_fish acc eat pst may persian_cat noun adp verb aux aux noun bunsetu 魚フライを 食べたかもしれない ペルシャ猫 sakanahuraiwo tabetakamosirenai perusyaneko proposal (suw1) 魚 フライ を 食べた か も しれない ペルシャ 猫 sakana hurai wo tabeta ka mo sirenai perusya neko fish fry acc eat.pst q also know.neg persia cat noun noun adp verb part adp verb noun noun _ _ _ @xmath0.acc eat.pst q also know.neg persia cat noun noun verb part adp verb noun noun _ @xmath1.pst may persian_cat noun adp verb aux noun _ _ @xmath2.acc eat.pst may persian_cat noun verb aux noun @xmath3 ## concluding remarks this paper has argued for morphology-aware tokenization policies for ud japanese treebanks and conducted an experiment that measures the morphological complexity of japanese based on the retokenized treebanks with morphological features. in doing so, we proposed new annotation schemes for tokenization and morphological features in japanese. the results showed that, although the morphological complexity of the current japanese ud resembled that of an isolating language, our retokenized treebanks have scores more similar to synthetic languages, which reflect the typological reality of japanese. the proposed tokenization will also be suitable for developing ud treebanks for other japanese-ryukyuan languages that syntactically have a similar structure to japanese but can be morphologically more fusional. furthermore, tokenization and morphological annotation conforming to ud's general guidelines enable crosslinguistic comparative studies; therefore, discussions for further cross-treebank consistencies are required.