We present a stochastic finite-state model for segmenting Chinese text into dictionary entries and productively derived words, and providing pronunciations for these words; the method incorporates a class-based model in its treatment of personal names. We also evaluate the system's performance, taking into account the fact that people often do not agree on a single segmentation. The initial step of any text analysis task is the tokenization of the input into words. For many writing systems, using whitespace as a delimiter for words yields reasonable results. However, for Chinese and other systems where whitespace is not used to delimit words, such trivial schemes will not work. Chinese writing is morphosyllabic (DeFrancis, 1984) , meaning that each hanzi-'Chinese character' -(nearly always) represents a single syllable that is (usually) also a single morpheme. Since in Chinese, as in English, words may be polysyllabic, and since hanzi are written with no intervening spaces, it is not trivial to reconstruct which hanzi to group into words. While for some applications it may be possible to bypass the word-segmentation problem and work straight from hanzi, there are several reasons why this approach will not work in a text-to-speech (TI'S) system for Mandarin Chinese --the primary intended application of our segmenter. These reasons include: 1. Many hanzi are homographs whose pronunciation depends upon word affiliation. So, ~ is pronounced deO ~ when it is a prenominal modification marker, but di4 in the word [] ~ mu4di4 'goal'; ~ is normally ganl 'dry',but qian2 in a person's given name. We present a stochastic finite-state model for segmenting Chinese text into dictionary entries and words derived via the above-mentioned productive processes; as part of the treatment of personal names, we discuss a class-based model which uses the Good-Turing method to estimate costs of previously unseen personal names. The segmenter handles the grouping of hanzi into words and outputs word pronunciations, with default pronunciations for hanzi it cannot group; we focus here primarily on the system's ability to segment text appropriately (rather than on its pronunciation abilities). We evaluate various specific aspects of the segmentation, and provide an evaluation of the overall segmentation performance: this latter evaluation compares the performance of the system with that of several human judges, since even people do not agree on a single correct way to segment a text, There is a sizable literature on Chinese word segmentation: recent reviews include (Wang et al., 1990; Wu and Tseng, 1993) . Roughly, previous work can be classified into purely statistical approaches (Sproat and Shih, 1990) , statistical approaches which incorporate lexical knowledge (Fan and Tsai, 1988; Lin et al., 1993) , and approaches that include lexical knowledge combined with heuristics .
For Chinese, REF used the word unigram model in their word segmenter based on weighted finite-state transducer.
A Stochastic Finite-State Word-Segmentation Algorithm For Chinese
Annual Meeting Of The Association For Computational Linguistics
Computer Science