Chinese word segmentation is a difficult, important and widely-studied sequence modeling problem. This paper demonstrates the ability of linear-chain conditional random fields (CRFs) to perform robust and accurate Chinese word segmentation by providing a principled framework that easily supports the integration of domain knowledge in the form of multiple lexicons of characters and words. We also present a probabilistic new word detection method, which further improves performance. Our system is evaluated on four datasets used in a recent comprehensive Chinese word segmentation competition. State-of-the-art performance is obtained.
This approach was extended in REF to the sequence modeling task, and linear conditional random fields were used to attempt it and receive state of the art results.
Chinese Segmentation And New Word Detection Using Conditional Random Fields
International Conference On Computational Linguistics
Computer Science