In this paper, we present a hybrid method for Chinese and Japanese word segmentation. Word-level information is useful for analysis of known words, while character-level information is useful for analysis of unknown words, and the method utilizes both these two types of information in order to effectively handle known and unknown words. Experimental results show that this method achieves high overall accuracy in Chinese and Japanese word segmentation.
For instance, Nakagawa REF proposed a hybrid method for Chinese and Japanese word segmentation, using word-level information for known words, and character-level information for unknown words.
Chinese And Japanese Word Segmentation Using Word-Level And Character-Level Information
International Conference On Computational Linguistics
Computer Science