Long Unit Word Tokenization and Bunsetsu Segmentation of Historical J apanese (2024)

Abstract: In Japanese, the natural minimal phrase of a sentence is the “bunsetsu” and it serves as a natural boundary of a sentence for native speakers rather than words, and thus grammatical analysis in Japanese linguistics commonly operates on the basis of bunsetsu units.In contrast, because Japanese does not have delimiters between words, there are two major categories of word definition, namely, Short Unit Words (SUWs) and Long Unit Words (LUWs).Though a SUW dictionary is available, LUW is not.Hence, this study focuses on providing deep learning-based (or LLM-based) bunsetsu and Long Unit Words analyzer for the Heian period (AD 794-1185) and evaluating its performances.We model the parser as transformer-based joint sequential labels model, which combine bunsetsu BI tag, LUW BI tag, and LUW Part-of-Speech (POS) tag for each SUW token.We train our models on corpora of each period including contemporary and historical Japanese.The results range from 0.976 to 0.996 in f1 value for both bunsetsu and LUW reconstruction indicating that our models achieve comparable performance with models for a contemporary Japanese corpus.Through the statistical analysis and diachronic case study, the estimation of bunsetsu could be influenced by the grammaticalization of morphemes.

## introduction in japanese, "bunsetsu" (base-phrase) is the natural minimal phrase of a sentence. it serves as a natural boundary of a sentence for native speakers rather than words; thus grammatical analysis in japanese linguistics commonly operates on the basis of bunsetsu units. for example, in universal dependencies (ud; @xcite , a framework for the consistent annotation of lexical dependency grammar across different human languages, some japanese corpora have been con-verted from dependency relations between bunsetsu @xcite . in contrast, because japanese does not have delimiters between words, there are many definitions of "words" in japanese. the national institute for japanese language and linguistics defines two hierarchical word tokenization categories: short unit words (suws) and long unit words (luws). suw is a minimal word token in japanese, and is defined by a bottom-up method that consists of at most two morphological units. in contrast, luw is defined by a top-down method that divides a bunsetsu into two parts, and it may contain several suws. for example, the luw "北西大西洋 (northwest atlantic)" consists of two suws "北西 (northwest)" and "大西洋 (atlantic)." dictionaries of suws for historical and contemporary japanese are already publicly available foot_0 , whereas there is no dictionary for luws. hence, a parser that outputs bunsetsu and luws for historical japanese is necessary to analyze the grammatical changes in japanese. for existing historical japanese literature, a sufficient amount of bunsetsu and luw annotated text to train the parser is primarily available from the heian period (ad 794-1185) and later. therefore, this study mainly focuses on the heian period, with the subsequent kamakura (ad 1185-1336) and the muromachi (ad 1336-1573) periods chosen for comparison. the existing bunsetsu parser @xcite for these periods is based on conditional random field (crf), which was used to create the annotated corpus. thus, this study focuses on providing a deep learning-based (or llm-based) bunsetsu and long unit words (luw) parser and evaluating its performances. we model the parser the results indicate that the models trained on historical japanese achieve comparable performance (0.976-0.996 f1 values) to a model for a contemporary japanese corpus. to trace grammatical changes in japanese, we evaluated the zero-shot transfer performance of the heian, kamakura, and muromachi periods for each other. the models trained with a corpus of the heian and kamakura periods performed well on each other, whereas the model trained with a corpus of the muromachi period did not. these results support the consensus among japanese liguists that the large grammatical changes occurred during the muromachi period. furthermore, the analysis focusing on sentence-ending particles revealed that new sentence-ending particle usage has emerged in the muromachi, and they are difficult to predict by the models of the prior periods. foot_1 2 related work parser for historical japanese comainu, a japanese bunsetsu and luw parser, was originally provided for contemporary japanese @xcite , although it can also be applied to historical japanese. comainu takes suw tokens as input, which are tokenized by a crf-based morphological analyzer mecab foot_2 , and then outputs the bunsetsu and luw tokens. as mentioned above, comainu is a crf-based parser; thus, we focused on deep-learning-based methods. parser for contemporary japanese recent japanese corpora of ud contain bunsetsu and luw annotations @xcite ; thus, some parsers trained on these corpora support bunsetsu segmentation and luw tokenization. for example, the spacy-based foot_3 japanese ud parser foot_4 supports luw tokenization @xcite . the parser was trained with a transformerbased language model (tlm) through the spacy pipeline, and it achieved better performance than comainu by adding some rules. ginza (matsuda, 2020), which is also a spacy-based parser, supports bunsetsu output. ## corpus we used the corpus of historical japanese (chj; ninjal 2024), which collects documents from the nara (ad 710-794) to the meiji (ad . bunsetsu and luw annotations were performed on sampled sentences sampled from the chj. we also used ud-japanese-gsd foot_5 , a contemporary japanese corpus, for the model comparison and searching for the best model, because there is a deep-learning-based parser that can output bunsetsu and luw labels @xcite . ## conclusion this study focuses on providing a deep learningbased (or llm-based) bunsetsu, which is a minimal phrase in japanese, and long unit words parser for the heian period (ad 794-1185) to the muromachi period (ad 1336-1573) and evaluating its performances. we model the parser as a joint sequential label that combines the bunsetsu bi tag, luw bi tag, and luw pos tags for each suw token. we used the transformer-based language model to output an suw token representation by taking the appropri-ately pooling of the subword representations for the last layer of the transformer. we trained our models on the corpora of each period, including contemporary and historical japanese. the results ranged from 0.976 to 0.996 in the f1 value for both bunsetsu and luw reconstructions indicating that our models achieved comparable performance to models trained on a contemporary japanese corpus. through the statistical analysis and case studies comparing each period, the bunsetsu estimation can be influenced by the grammaticalization of morphemes. in the future, we will expand the applicable periods. we will build a syntactic parser by annotating the dependencies between bunsetsu segments.