T op WORDS -Seg: Simultaneous Text Segmentation and Word Discovery for Open-Domain C hinese Texts via B ayesian Inference (2022)

Abstract: Processing open-domain Chinese texts has been a critical bottleneck in computational linguistics for decades, partially because text segmentation and word discovery often entangle with each other in this challenging scenario. No existing methods yet can achieve effective text segmentation and word discovery simultaneously in open domain. This study fills in this gap by proposing a novel method called TopWORDS-Seg based on Bayesian inference, which enjoys robust performance and transparent interpretation when no training corpus and domain vocabulary are available. Advantages of TopWORDS-Seg are demonstrated by a series of experimental studies.

## introduction due to absence of word boundaries in chinese, chinese natural language processing (cnlp) faces a few unique challenges, including text segmentation and word discovery. when processing opendomain chinese corpus containing many unregistered words and named entities, these challenges become more critical as they often entangle with each other: we usually cannot segment chinese texts correctly without knowing the underlying vocabulary; on the other hand, it is often difficult to precisely discover unregistered words and named entities from open-domain corpus without guidance on text segmentation. most methods for cnlp in the literature assume that the underlying vocabulary is known and focus on improving performance of text segmentation in closed test. the first category of methods along this research line are simple methods based on word matching @xcite @xcite , which segment a chinese sentence by matching sub-strings in the sentence to a pre-given vocabulary in a forward or * corresponding author. reserve order. the second category of methods utilize manually segmented corpus or large-scale pretraining corpus to train statistical models such as maximum entropy @xcite @xcite , hmm @xcite and crf @xcite @xcite , or deep learning models including cnn (wang and xu), lstm @xcite , bi-lstm @xcite and @xcite , or hybrid models like bi-lstm-crf @xcite and lstm-cnns-crf @xcite , to achieve text segmentation directly or indirectly. methods of this category have led to popular toolkits for processing chinese texts, including jieba @xcite , stanfordnlp @xcite , thulac @xcite , pkuseg @xcite , and ltp @xcite . a popular strategy adopted by some of these toolkits is to segment the target texts into sequences of basic words first, and capture unregistered words and named entities, which are often word compounds consisting of basic words, later via chunking and syntactic analysis. although such a strategy can equip these toolkits with some ability on word discovery, it is apparently sub-optimal, because we may mis-segment basic words at the first place without realizing the existence of potential technical words, making it impossible to discover technical word compounds correctly in post analysis such as chunking and syntactic analysis. on the other hand, unsupervised methods are also developed to achieve text segmentation when no pre-given vocabulary and manually segmented training corpus are available. some methods of this research line segment texts based on local statistics of the target texts, including description length gain @xcite , mutual information @xcite , accessor variety @xcite , evaluation-selection-adjustment process @xcite , and normalized variation of branching entropy @xcite . the others, however, rely on generative statistical models whose parameters can be estimated from the target texts only, including hierarchical dirichlet process @xcite , nested pitman-yor process @xcite , bayesian hmm @xcite , topwords @xcite and @xcite . in general, methods based on word matching and unsupervised learning cannot produce high-quality text segmentation (zhao and @xcite , although some unsupervised methods are successful on word discovery @xcite . methods based on supervised learning can achieve excellent performance in closed test @xcite , but often suffer from dramatic performance degradation when applied to open-domain chinese corpus containing many unregistered words and named entities @xcite . methods based on deep learning are usually more robust under the "pre-training and fine-tuning" framework, but still suffer from unstable performance and often fail to correctly segment technical words, which play a key role in deciphering the meaning of domain-specific texts, when applied to opendomain texts @xcite . there are also some efforts in the literature to integrate supervised and unsupervised methods for improved performance (zhao and @xcite @xcite @xcite . but, these methods either heavily depend on manually labelled corpus for model training, or suffer from unbalanced emphasis on text segmentation and word discovery, resulting in limited improvement for cnlp in open domain. these facts make processing open-domain chinese texts a critical bottleneck in computational linguistics even for today. many factors contribute to the stagnation on development of efficient tools for processing opendomain chinese texts. from the methodology point of view, we do not have a proper learning framework yet to connect the text segmentation problem to the word discovery problem and deal with them at the same time effectively. from the practical point of view, the lack of proper evaluation criterion in open domain places a critical barrier for fair comparison of different methods and discourages researchers from looking for potential solutions. this study tries to provide solutions to these critical issues. first, we propose a novel bayesian framework to integrate topwords, an effective word discoverer @xcite , and pkuseg, a strong text segmenter, leading to a more efficient text segmenter called topwords-seg, which can achieve effective text segmentation and word discovery simultaneously in open domain. next, we design a cocktail strategy for method evaluation and comparison by measuring the overall performance of a target method on both text segmentation in benchmark corpus and technical word discovery and segmentation in open-domain corpus. experimental studies demonstrate that the proposed topwords-seg outperforms existing methods with a significant margin for cnlp in open domain. ## topwords-seg proposed by @xcite , topwords is a general approach for offline natural language processing based on unsupervised statistical learning. assuming that sentences are generated by randomly sampling and concatenating words from an underlying word dictionary (i.e., unigram language model), topwords starts with an over-complete initial word dictionary d containing all plausible word candidates in the target texts, and gradually simplifies the model by removing non-significant word candidates from d based on statistical model selection principles, with the unknown word usage frequencies estimated by em algorithm @xcite . topwords is closely related to methods widely used in neural machine translation for constructing sub-word dictionary, and can be viewed as an advanced version of wordpiece @xcite , byte pair encoding @xcite and unigram language model @xcite . in practice, topwords is particularly effective on discovering words, technical terms and phrases from open-domain chinese texts, but tends to segment texts with coarser granularity at phrase instead of word level. in this section, we upgrade topwords from a weak text segmenter with strong ability on word discovery to a more powerful tool enjoying balanced ability on both dimensions via bayesian inference. ## experimental study on wikipedia composed of over 10 billion chinese character tokens from 3.6 million webpages, chinese wikipedia ( https://dumps.wikimedia.org/ ) is one of the largest open-source chinese corpus. containing rich contents of various domains and millions of technical terms highlighted by hyperlinks, the chinese wikipedia is an ideal corpus for studying cnlp in open domain. considering that it's computationally expensive to processing all webpages in chinese wikipedia, we randomly picked up 1,500 webpages involving 8 million chinese character tokens (referred to as chinese wiki-rand, or t w -r ) as the representative samples of the general texts in chinese wikipedia. moreover, we selected two collections of special webpages from chinese wikipedia with label "电 影" (referred to as chinese wiki-film, or t w -f ) or "物理" (referred to as chinese wiki-physics, or t w -p ), involving ∼5 million chinese character tokens for each, as the representatives of the domain-specific texts in chinese wikipedia. in this section, we apply topwords-seg to process these wikipedia corpora separately, and compare its performance to 6 existing methods, including jieba (sun, 2012), stanfordnlp @xcite , thulac @xcite , pkuseg @xcite , ltp @xcite , and topwords @xcite itself, from various aspects. ## processing the book of deep learning written by @xcite , the book deep learning has become a classic tutorial for deep learning. in 2017, its chinese version was published in china (see figure 3 (a) for the book's cover), which is composed of more than 400,000 chinese character tokens (referred to as t d ). covering rich technical contents in the domain of machine learning, including over 800 technical terms as listed in the index table at the end of the book, such a book is an ideal target for testing the performance of the proposed topwords-seg in real application. feeding full text of the book to topwords-seg and competing methods respectively, we obtained results as summarized in figure 3 . figure 3 (b) shows that topwords-seg discovers 84.1% technical terms listed in the index table of the book with a vocabulary of 10.7k discovered words. top-words achieves a slightly higher r @xmath0.0% at the price of a larger vocabulary with 12.8k discovered words. other methods based on supervised learning result in much lower r d with the vocabulary size varying between 6.8k to 12.2k. figure 3 (d) shows the most frequent words discovered by topwords-seg. figure 3 (e ) displays part of the technical terms captured by topwords-seg but missed by all supervised methods, which are all meaningful technical terms like unsupervised learning (无监督学习) and stochastic gradient decent (随机梯度下降). figure 3 (f) summarizes typical pseudo words and phrases reported by topwords but eliminated by topwords-seg, which are all common collocations widely used but usually not treated as words in chinese, e.g., in the model (模 型中) and it is because of (是因为). these results suggest that topwords-seg is indeed more effective than competing methods on word discovery. in terms of text segmentation, the heatmap in figure 3 (c ) visualizes the similarity between topwords-seg and other approaches on this corpus in a similar fashion as in figure 2 (d ). again, the performance of topwords-seg is very similar to the supervised methods, and demonstrates significant difference from topwords, suggesting that topwords-seg is a robust tool with balanced ability on processing open-domain chinese texts. ## conclusions and discussions in this paper, we proposed topwords-seg, a powerful tool for processing open-domain chi-nese texts based on bayesian inference with balanced ability on text segmentation and word discovery. a series of experimental studies confirm that topwords-seg can discover unregistered technical terms in open-domain texts effectively, and achieve high-quality text segmentation on both benchmark and open-domain corpora. taking advantage of the bayesian framework, topwords-seg is ready to process large scale open-domain chinese texts without extra training corpus or pregiven domain vocabulary, leading to an ideal solution to a critical bottleneck existing in computational linguistics for decades. moreover, combing the strong points of pkuseg and topwords via bayesian inference, topwords-seg enjoys transparent reasoning process, and is fully interpretable to most people. in practical applications, such a property is very attractive to many researchers and practicers. meanwhile, topwords-seg also suffers from a few obvious limitations. for example, although the current learning framework is effective to discover frequent words, it tends to miss many rare words that appear only a few times in the texts. for another instance, because pkuseg is more reliable on segmenting general texts, but less reliable on segmenting technical texts, in the ideal case we should adopt prior information provided by pkuseg adaptively when processing texts of different types. unfortunately, topwords-seg does not take such a natural idea into consideration yet, and simply use the pkuseg prior at the same intensity everywhere. these deficiencies partially explain why topwords-seg still misses about 15% technical terms in both experimental studies reported in this paper. more research efforts are needed to fill in these gaps in future.