T op WORDS -Poetry: Simultaneous Text Segmentation and Word Discovery for Classical C hinese Poetry via B ayesian Inference (2023)

Abstract: As a precious cultural heritage of human beings, classical Chinese poetry has a very unique writing style and often contains special words that rarely appear in general Chinese texts, posting critical challenges for natural language processing. Little effort has been made in the literature for processing texts from classical Chinese poetry. This study fills in this gap with TopWORDS-Poetry, an unsupervised method that can achieve reliable text segmentation and word discovery for classical Chinese poetry simultaneously without pre-given vocabulary or training corpus. Experimental studies confirm that TopWORDS-Poetry can successfully recognize unique poetry words, such as named entities and literary allusions, from metrical poems of Complete Tang Poetry and segment these poetry lines into sequences of meaningful words with high quality.

## introduction expressing complicated thoughts and feelings in refined language, poetry is the most shining pearl on the crown of human literature. as a special literary style, poetry has unique characteristics in rhythm, metre and language use, making it a critical challenge to process poetry texts via algorithms @xcite @xcite . rising in the spring and autumn period (770-476 b.c.) and flourishing in the tang dynasty (618-907 a.d.), classical chinese poetry has a long history and profound cultural heritage. composed of nearly 50,000 poems by over 2000 poets of the tang dynasty, 《全唐诗》(complete tang poetry, or ctp) displayed in fig 1 classical chinese poetry. among the nearly 50,000 poems in ctp, 34,227 are metrical poems that have recognizable patterns * corresponding author. of syllables. containing 1.3 million chinese characters in total, the collection of metrical poems in ctp (referred to as p hereinafter) is the most essence of tang poetry and our focus in this study. a typical metrical poem in ctp is composed of either 4 or 8 lines, each of which contains either 5 or 7 chinese characters, and can be classified into 4 categories p 4×5 , p 4×7 , p 8×5 and p 8×7 , where p m×n represents the poems with m lines and n chinese characters in each line. in p, there are 3,225 poems in p 4×5 , 9,820 poems in p 4×7 , 13,536 poems in p 8×5 , and 7,646 poems in p 8×7 . fig 1 (d )-(g) show 4 famous poems (each for one of the 4 categories) by the famous poet du fu (杜甫, 712-770 a.d.) of the tang dynasty, with the word boundaries (highlighted by symbol '/') manually labelled by domain experts. for the majority of poems in ctp, however, no word boundaries are labelled yet. the absence of word boundaries posts critical challenges in literary study of classical chinese poetry at a large scale, and leads to a great appeal to develop efficient algorithms to achieve reliable text segmentation and word discovery for poetry texts @xcite . however, such a task is very challenging, because classical chinese poetry is composed of special domain-specific texts with many unique features. first, classical chinese poetry has a very unique writing style and often contains special words that rarely appear in general chinese texts, such as unique named entities and literary allusions. moreover, to express complicated thoughts and feelings with few words, poets were often forced to compress long phrases into shorter "pseudo words" for putting them into short poetry lines. although these "pseudo words" are phrases instead of words from the grammatical point of view, many of them are treated as "poetry words" in practice because they have specific meanings and function as words @xcite . due to these facts, word discovery becomes a critical problem in processing second, to keep proper rhythm, text segmentation of classical chinese poetry lines follows a few major segmentation patterns. previous studies have revealed that 5-character poetry lines mainly follow two segmentation patterns 2-1-2 and 2-2-1, while 7-character poetry lines mainly follow another two segmentation patterns 2-2-1-2 and 2-2-2-1 @xcite @xcite . in this study, we refer to these 4 major segmentation patterns as ϱ 212 , ϱ 221 , ϱ 2212 and ϱ 2221 , respectively. apparently, ϱ 2212 and ϱ 212 correspond to the same segmentation pattern for poetry lines of different lengths, and form a general segmentation pattern ϱ * 12 . similarly, ϱ 2221 and ϱ 221 form another general segmentation pattern ϱ * 21 . to investigate the proportion of these segmentation patterns in p, we randomly sampled 1,000 poems from p (referred to as p 1000 ) by stratification sampling (i.e., 250 poems for each of the 4 poem types in p), and manually segmented them as the demonstrating examples shown in fig 1 (d )-(g). figure 1 (h) summarizes the obtained results, confirming that most poetry lines indeed precisely follow these 4 segmentation patterns. third, on the other hand, however, about 10% of poetry lines violate these major segmentation patterns due to appearance of long words containing more than 2 chinese characters, such as named entities and literary allusions. for example, in the 4 poems shown in fig 1 (d )-(g), there are 5 poetry lines that violate the 4 major segmentation patterns because of long words with 3 chinese characters. due to this phenomenon, text segmentation becomes a non-trivial issue. these unique features of poetry texts make it inappropriate to directly apply popular tools for processing general chinese texts based on supervised learning, such as jieba @xcite , stanfordnlp @xcite , thulac @xcite , pkuseg @xcite , and ltp @xcite , to process classical chinese poetry. as we will show in the result section, due to the lack of training for poetry texts, these methods suffer from serious performance degradation in text segmentation, and tend to miss many poetry words because of the limited ability in discovering unregistered words. and, it is not realistic to tune these tools for this specific task, because no qualified training corpus that contains enough well-segmented classical chinese poetry lines is publicly available yet, to the best of our knowledge. some methods based on unsupervised learning, e.g., topwords @xcite , gts @xcite and others @xcite @xcite , are strong in discovering unregistered words, but suffer from poor performance on text segmentation in the current scenario because they do not take the special features of poetry texts into consideration. as pointed out by @xcite , when processing domain-specific chinese texts like poetry that potentially contain many unregistered words, word discovery and text segmentation become two critically important tasks entangled with each other. most existing methods for chinese text segmentation work under the support of a pre-loaded vocabulary, either explicitly given apriori or implicitly encoded in a pre-trained large language model. if the pre-loaded vocabulary covers the target corpus well, these methods typically perform reasonably well; once the underlying vocabulary of the target corpus exceeds the pre-given vocabulary seriously, however, we often observe significant performance degeneration of these methods, as demonstrated in @xcite . because classical chinese poetry contains many unique named entities and literary allusions that never appear in general texts and follow specific segmentation templates, methods without considering these features tend to miss many poetry words and segment poetry lines in an inappropriate manner. in the literature, few methods have been pro-posed for processing classical chinese poetry. @xcite proposed to recognize meaningful words in classical chinese poetry by filtering out fake words from word candidates via a modified support vector machine trained by manually labelled training samples. this method relies heavily on manually labelled samples to capture features of poetry words, and completely ignores the segmentation patterns of poetry lines. @xcite proposed to formulate the text segmentation problem of a poetry line into a pattern selection problem under the assumption that every poetry line must precisely follow one of the two general segmentation patterns ϱ * 21 or ϱ * 12 , and achieve text segmentation by selecting the plausible pattern based on only the last three characters of each poetry line via weighted point-wise mutual information. although such a method takes good advantage of the segmentation patterns and performs reasonably well, it oversimplifies the problem via a restrictive assumption and tends to miss all named entities and literary allusions with more than 2 chinese characters. to better resolve this challenging problem, we need an algorithm that can wisely utilize the segmentation patterns of poetry lines, is flexible enough to deal with situations where the regular segmentation patterns are violated, and has strong word discovery ability to recognize special poetry words with little training information. in this study, we fill in this gap with topwords-poetry, an algorithm with all these desired features. extending the bayesian framework of topwords-seg proposed by @xcite to a more sophisticated hierarchical bayesian model to reflect the constraints on text segmentation due to the major segmentation patterns, topwords-poetry is able to achieve effective text segmentation and word discovery simultaneously for classical chinese poems with metrical patterns. an experiment study on metrical poems in ctp confirms that topwords-poetry can correctly segment these poetry texts, with various named entities, literary allusions and special poetry words effectively recognized. compared to existing methods in the literature, topwords-poetry is superior on processing classical chinese poetry with a significant performance margin. the outputs of topwords-poetry may promote the applications of ai approaches in the study of classical chinese literature and inspire more research efforts to study classical chinese poetry via the paradigm of digital proposed by @xcite , topwords-seg is an effective method for processing open-domain chinese texts that can achieve high-quality text segmentation and word discovery simultaneously. modelling the target texts by the uni-gram language model with an over-complete initial dictionary d obtained by enumerating all qualified word candidates, topwords-seg utilizes the word boundaries suggested by a pre-given text segmenter (e.g., pkuseg) as prior information to guide usage frequency estimation and pruning of word candidates in d under a bayesian framework, and finally achieves text segmentation based on the pruned word dictionary with respect to the prior word boundaries. as demonstrated by @xcite , the bayesian framework of topwords-seg integrates the advantages of a strong word discoverer topwords and an effective text segmenter pkuseg, leading to a stronger word discoverer and a more robust text segmenter for open-domain chinese texts. in this section, we extend the bayesian framework of topwords-seg with a more flexible way to utilize prior information on text segmentation. instead of using the word boundaries suggested by a pre-given text segmenter, e.g., pkuseg, as prior information to guide learning, we encode the constraints on text segmentation due to various segmentation patterns (i.e., ϱ * 12 and ϱ * 21 ) into a more sophisticated prior distribution, coming up with a hierarchical bayesian model to infer the unknown segmentation pattern of each poetry line. ## experimental study we applied topwords-poetry on the poetry texts in p, the collection of all metrical poems in ctp, under the default setting of τ @xmath0, τ @xmath1, ρ = 0.5, κ @xmath2.5, κ @xmath3.001, τ ψ = 0.05 and τ @xmath4.5. for comparison purpose, we also applied 6 competing methods to the same data set, including jieba @xcite , stanfordnlp @xcite , thulac @xcite , pkuseg @xcite , ltp @xcite , topwords @xcite and topwords-seg under their default settings. unfortunately, however, no executable program is available for either the modified svm method (referred to as modsvm) for discovering poetry words by @xcite or the pattern selection method (referred to as selectpattern) for segmenting poetry texts by @xcite . although it is straightforward to implement selectpattern by ourselves, it is non-trivial to implement modsvm due to many subtle details. therefore, we only include selectpattern in the experimental study. moreover, liu (2019) furthermore, we also evaluate the performance of gpt-style methods on segmenting classical chinese poetry by asking chatgpt (model version: gpt-35-turbo) to segment all poetry lines in the testing dataset p 1000 . appendix f reports the prompt designing for calling chatgpt for this task in detail. ## conclusions and discussions in this paper, we proposed topwords-poetry, a powerful unsupervised method for processing classical chinese poetry that can segment poetry lines wisely with respect to a few major segmentation patterns, and recognize unique poetry words such as name entities and literary allusions successfully with little training information. relying on a hierarchical bayesian model to utilize prior information about known segmentation patterns of poetry lines, and rigorous statistical principles to achieve parameter estimation and model selection, topwords-poetry leads to an elegant solution to challenging nlp problems in an interpretable way. we hope this proposed method can serve as a useful tool for scholars in the field of chinese literary or digital humanity to conduct in-depth study on classical chinese poetry via distant reading. for example, the literary allusions discovered from ctp by topwords-poetry may lead to very exciting research in near future, which may extend the research efforts discussed by @xcite ; @xcite ; @xcite . we also hope this work can promote more and more interactions between literary scholars and nlpers, and applications of nlp technology in cultural education and dissemination, helping more people to better enjoy the charm of chinese traditional culture.