Introducing a Large Corpus of Tokenized Classical C hinese Poems of Tang and Song Dynasties (2022)

Abstract: Classical Chinese poems of Tang and Song dynasties are an important part for the studies of Chinese literature. To thoroughly understand the poems, properly segmenting the verses is an important step for human readers and software agents. Yet, due to the availability of data and the costs of annotation, there are still no known large and useful sources that offer classical Chinese poems with annotated word boundaries. In this project, annotators with Chinese literature background labeled 32399 poems. We analyzed the annotated patterns and conducted inter-rater agreement studies about the annotations. The distributions of the annotated patterns for poem lines are very close to some well-known professional heuristics, i.e., that the 2-2-1, 2-1-2, 2-2-1-2, and 2-2-2-1 patterns are very frequent. The annotators agreed well at the line level, but agreed on the segmentations of a whole poem only 43% of the time. We applied a traditional machine-learning approach to segment the poems, and achieved promising results at the line level as well. Using the annotated data as the ground truth, these methods could segment only about 18% of the poems completely right under favorable conditions. Switching to deep-learning methods helped us achieved better than 30%.

## introduction word segmentation is an important step for understanding chinese texts because the chinese language do not include explicit word delimiters, like the spaces in english, in the texts. different segmentations of the same statements can lead to different interpretations, so segmenting chinese strings into correct word sequences is crucial for understanding and processing chinese in computer systems. classical chinese poems typically consist of sequences of short verses, so the quality of word segmentation influences the reading of poems significantly. the segmented poems can facilitate further analysis and applications, e.g., poem styles @xcite . the literature has seen a wide variety of approaches to the problem of word segmentation for vernacular chinese in the past many years, e.g., @xcite @xcite . annotated corpora have been created for research and competition as well @xcite @xcite . in contrast, relatively few researchers of chinese linguistics and literature discussed word segmentation for classical chinese poems. @xcite examined the problem from both syntactic and semantic perspectives, while @xcite argued that the perspective of semantic interpretation should be more natural for native speakers. @xcite inherited and emphasized more on the semantic viewpoints. relying on modern databases of tang poems, @xcite and @xcite can access and compare more poems conveniently, and they adopt the observations discussed in the previous literature for the word segmentation task. beyond conceptual discussion, it is harder to segment words in corpora of classical poems in large scale. lee and colleagues discussed the topics of annotating part-of-speech tags (2012) and of creating dependency trees (2012) for classical chinese poems. when they analyzed some interesting syntactic patterns in classical poems, they mentioned around one thousand poems @xcite . in this paper, we report a relatively larger scale of work for annotating word boundaries in two collections of classical chinese poems. at the time of writing, we have annotated 32399 classical chinese poems of the tang and song dynasties. 1 we evaluated our annotations in some different ways. first, we conducted inter-rater agreement (ira) analysis, and the results are convincing. we applied machine learning methods for segmenting words in classical chinese poems, and have achieved and published some preliminary results when we annotated only thousands of poems @xcite . we have improved the quality of our word segmenters significantly by using more annotated data and embracing the technology of deep learning. in addition, we compared our annotations with relevant information in a wellknown website, and found that our annotations have a reasonable consensus. we provide information about data sources, define the task of word segmentation, and discuss some domain-dependent heuristics in section 2. we explain methods for measuring the quality of word segmentation in section 3. we introduce our annotation team and their annotations, and report a basic statistical analysis of the annotated poems in section 4. we explored different perspectives for ira analysis in section 5. we introduce the probabilistic classifiers for word segmentation in section 6. we compared the performances of two different designs of the probabilistic classifier in section 7, and wrap up this paper in section 8. ## data sources and problem definition we provide a brief introduction to the forms of classical chinese poems in section 2.1, and define the task of marking word boundaries in section 2.2. ## inter-rater agreement analysis we report results of our inter-rater agreement analysis in this section, and argue that the observed agreements are not just results of the annotators' accepting the heuristics that were explained in section 2.3. ## simple probabilistic classifiers recall that the task of word segmentation can be viewed as classifying characters as a terminal or non-terminal character for a word. ## empirical evaluations since we discussed the differences between dpmi and traditional pmi, and we claimed the superiority of weighted dpmi (wdpmi) against dpmi. we conducted a wide variety of experiments to verify this projection. since we will use the ctp1 and csp1 as the test data, we will remove the poems in ctp1 and csp1 from ctp2 and csp2, respectively, at training time. we do not indicate this exclusion in we list 14 such experiments and their results in recall the definition for wdpmi and our discussion in section 6.2. we set β, σ, and α to 0. we could verify that using wdpmi indeed led to better performances than using dpmi, if we compare the corresponding statistics in we can compare the performances of wdpmi and dpmi from other perspectives, and we can include more domain knowledge about the classical poems to improve the performances of our probabilistic classifiers in an extended report of our work. of course, with the annotated poems, we could apply deep learning @xcite and other machine learning methods to train and test classifiers that may further enhance the quality of word segmentation. ## concluding remarks the main purpose of this paper is to report the annotation of word boundaries for 32399 classical chinese poems. seven annotators of chinese literature background carried out the task. to investigate the quality of these human annotation, we conducted inter-rater agreement studies. in fact, we have also compared the annotations with some relevant information extracted from the sou-yun website 9 , which is a highly recommended website for learning classical chinese poems, but we cannot provide the details here. based on these further analyses, we gained confidence on the quality of our annotations. we have used the annotated data to train classifiers for algorithmically segmenting classical chinese poems. it was relatively easy to segment the lines in poems correctly, but remained challenging to segment poems completely correct. we understand that there may not be "the" correct answer to segment a poem. "the" correct answer