We propose an unsupervised segmentation method based on an assumption about language data: that the increasing point of entropy of successive characters is the location of a word boundary. A large-scale experiment was conducted by using 200 MB of unsegmented training data and 1 MB of test data, and precision of 90% was attained with recall being around 80%. Moreover, we found that the precision was stable at around 90% independently of the learning data size.
Jin and Tanaka-Ishii REF propose an unsupervised word segmentation algorithm that is based on the entropy of successive characters at word boundaries.
Unsupervised Segmentation Of Chinese Text By Use Of Branching Entropy
International Conference On Computational Linguistics And Annual Meeting Of The Association For Computational Linguistics - Poster Sessions
Computer Science