This paper presents a Chinese word segmentation system which can adapt to different domains and standards. We first present a statistical framework where domain-specific words are identified in a unified approach to word segmentation based on linear models. We explore several features and describe how to create training data by sampling. We then describe a transformation-based learning method used to adapt our system to different word segmentation standards. Evaluation of the proposed system on five test sets with different standards shows that the system achieves state-of-the-art performance on all of them. Chinese word segmentation has been a longstanding research topic in Chinese language processing. Recent development in this field shows that, in addition to ambiguity resolution and unknown word detection, the usefulness of a Chinese word segmenter also depends crucially on its ability to adapt to different domains of texts and different segmentation standards. The need of adaptation involves two research issues that we will address in this paper. The first is new word detection. Different domains/applications may have different vocabularies which contain new words/terms that are not available in a general dictionary. In this paper, new words refer to OOV words other than named entities, factoids and morphologically derived words. These words are mostly domain specific terms (e.g. 蜂窝式 'cellular') and time-sensitive political, social or cultural terms (e.g. 三通'Three Links', 非典 'SARS'). The second issue concerns the customizable display of word segmentation. Different Chinese NLP-enabled applications may have different requirements that call for different granularities of word segmentation. For example, speech recognition systems prefer "longer words" to achieve higher accuracy whereas information retrieval systems prefer "shorter words" to obtain higher recall rates, etc. (Wu, 2003) . Given a word segmentation specification (or standard) and/or some application data used as training data, a segmenter with customizable display should be able to provide alternative segmentation units according to the specification which is either pre-defined or implied in the data. In this paper, we first present a statistical framework for Chinese word segmentation, where various problems of word segmentation are solved simultaneously in a unified approach. Our approach is based on linear models where component models are inspired by the source-channel models of Chinese sentence generation. We then describe in detail how the new word identification (NWI) problem is handled in this framework. We explore several features and describe how to create training data by sampling. We evaluate the performance of our segmentation system using an annotated test set, where new words are simulated by sampling. We then describe a transformation-based learning (TBL, Brill, 1995) method that is used to adapt our system to different segmentation standards. We compare the adaptive system to other state-of-the-art systems using four test sets in the SIGHAN's First International Chinese Word Segmentation Bakeoff, each of which is constructed according to a different segmentation standard. The performance of our system is comparable to the best systems reported on all four test sets. It demonstrates the possibility of having a single adaptive Chinese word segmenter that is capable of supporting multiple user applications.
REF described a transformation-based converter to transfer a certain word segmentation result to another annotation guideline.
Adaptive Chinese Word Segmentation
Annual Meeting Of The Association For Computational Linguistics
Computer Science