Multi-grained C hinese Word Segmentation with Weakly Labeled Data (2020)

Abstract: In contrast with the traditional single-grained word segmentation (SWS), where a sentence corresponds to a single word sequence, multi-grained Chinese word segmentation (MWS) aims to segment a sentence into multiple word sequences to preserve all words of different granularities. Due to the lack of manually annotated MWS data, previous work train and tune MWS models only on automatically generated pseudo MWS data. In this work, we further take advantage of the rich word boundary information in existing SWS data and naturally annotated data from dictionary example (DictEx) sentences, to advance the state-of-the-art MWS model based on the idea of weak supervision. Particularly, we propose to accommodate two types of weakly labeled data for MWS, i.e., SWS data and DictEx data by employing a simple yet competitive graph-based parser with local loss. Besides, we manually annotate a high-quality MWS dataset according to our newly compiled annotation guideline, consisting of over 9,000 sentences from two types of texts, i.e., canonical newswire (NEWS) and non-canonical web (BAIKE) data for better evaluation. Detailed evaluation shows that our proposed model with weakly labeled data significantly outperforms the state-of-the-art MWS model by 1.12 and 5.97 on NEWS and BAIKE data in F1.

## introduction as a preliminary but critical processing step for chinese language processing, word segmentation (ws) has been extensively studied for decades and made great progress @xcite @xcite @xcite . however, most of previous works adopt the single-grained word segmentation (sws) formulation, where a sentence corresponds to a single word sequence according to some pre-defined annotation guidelines. as shown in figure 1 (left), the sws annotations of the sentence are different according to the guidelines of penn chinese treebank (ctb) @xcite , the people daily corpus of the peking university (ppd) @xcite , and the microsoft research ws corpus (msr) @xcite . this is largely due to the fact that the boundary between words is usually subtle and vague @xcite and there are various underlying linguistic theories. @xcite show that the consensus ratio over word boundaries is only 76% even among chinese native speakers without training on any guideline. in order to guarantee the annotation consistency, people turn to detailed annotation guidelines according to specific tasks or applications. for example, ctb usually prefers fine-grained words over coarse-grained words to facilitate further annotation of syntax and semantics, while ppd accommodates more coarse-grained words with information extraction and retrieval tasks in mind. this poses a strong challenge in chinese ws since words of different granularities are necessary for a variety of tasks and applications at the same time. @xcite and @xcite adopt heuristic rules based on lexicon dictionaries to accommodate the necessity. similar functions are provided by publicly available chinese ws tools such as jieba and pullword. however, the effectiveness of such tools are far below satisfactory without tackling segmentation ambiguity problem. it is worth emphasizing that even for the same task or application, words of different granularities can be useful due to its potential complementarity: fine-grained words capture local features and help reduce data sparseness, whereas coarse-grained words reserve more semantics to perform exacter matching and analysis. this facilitates researchers to employ multiple sws outputs at the same time in information retrieval (ir) @xcite and machine translation (mt) @xcite . motivated by above perspectives, multi-grained word segmentation (mws) is formally proposed by @xcite as a useful and challenging direction for research on word segmentation. given an input sentence, mws aims to accommodate all words of different granularities with a hierarchical tree structure. figure 1 (right) presents an example, where "w" means the spanning characters compose a word. in this example, "二 (two)", "老 (elder)", "二老 (two elders)", "都 (both)", "是 (are)", "都是 (both are)", "令 (make)", "人 (people)", "令人 (make people)", "尊敬 (respect)", "令人尊敬 (respectable)", " 的 (of)", "科学 (science)", "家 (expert)", "科学家 (scientist)" are all the words of different granularities. to solve the issue of lacking labeled mws data, they construct a large-scale pseudo mws data for model training and tuning. they propose several mws approaches and justify the superiority of treating mws as constituent parsing. however, their approaches only learn from pseudo mws data and do not fully exploit word boundary information from other available sources which are helpful and easy to obtain. this paper advances the state-of-the-art mws model with weakly labeled data. particularly, we propose to accommodate two types of weakly labeled data, i.e., sws and naturally annotated dictionary example (dictex) sentences, as extra training data, by employing a simple but competitive graphbased parsing model with local span-wise loss. besides, we develop a unified annotation guideline for mws and manually annotate a large-scale high-quality mws dataset containing over 9,000 sentences from both canonical newswire texts (news) and non-canonical web texts (baike) for better evaluation. detailed evaluation shows that our proposed model with weakly labeled data significantly improves the state-of-the-art mws model by 1.12 on news and by 5.97 on baike in f1. we release all the newly annotated data and the codes at https://github.com/gloria0108/ multi-grained-word-seg . ## graph-based model with local loss given an input sentence, the task of mws is to retrieve all words of different granularities, which can be naturally organized as a hierarchical tree structure as shown in figure 1 (right). @xcite propose several mws approaches and show that treating mws as constituent parsing leads to the best performance. they adopt the transition-based parser of @xcite , which greedily searches an optimal shift-reduce action sequence to build a tree. in this work, instead of adopting the transition-based parser as @xcite , we employ the graph-based parser of @xcite and replace the original global max-margin loss with local span-wise loss @xcite as our basic mws model due to two considerations: 1) the graph-based parser with local loss gains more efficiency without hurting the performance compared with the transitionbased parser and the graph-based parser with global loss, which will be discussed in section 5.3; 2) more importantly, this work aims to conduct in-depth study on a simple, efficient, and effective way to incorporate weakly labeled data for mws. the graph-based parser with local loss trains the model directly on individual labeled spans, and thus can accommodate weakly labeled data naturally. ## mws with weakly labeled data due to the lack of manually labeled high-quality data, @xcite construct a large-scale pseudo labeled mws data by automatically converting existing heterogeneous sws data in a pairwise way. they train their model with the pseudo labeled data and obtain promising performance on a small scale manually labeled mws data. however, they observe that the pseudo labeled data inevitably has a lot of noise due to the pairwise conversion model and severly suffer from the under-representation phenomena of multi-grained words. in order to alleviate above issues, we propose to accommodate two types of weakly labeled data (i.e., sws data and dictex data) as extra data for mws training, inspired by previous work on utilizing naturally annotated data for sws @xcite . ## experiments we conduct various experiments to show the effectiveness of our proposed mws approaches. data. data statistics are shown in evaluation metrics. given an input sentence, mws aims to precisely segment the sentence into all words of different granularities. we adopt the same evaluation metircs as sws. precision ( #word gold∩pred #word pred ), recall ( #word gold∩pred #word gold ) and f1 ( 2p r p +r ) score are used to measure the mws performance. we adopt dan bikel's randomized parsing evaluation comparator for significance test @xcite . model setting. we preserve most of the hyperparameter settings in @xcite , and our preliminary experiments show that the performance of our approach is quite stable. each model is trained for at most 1000 iterations, and early stopping is triggered when the peak performance does not increase in 50 consecutive iterations. ## conclusions this work advances the state-of-the-art mws research from three perspectives. first, we manually annotate over 9,000 sentences for better evaluation, consisting of both canonical news and non-canonical baike texts. second, we employ a simple graph-based parsing model with local loss to facilitate the use of weakly labeled data. finally, we propose to accomodate two types of weakly labeled data as extra training data, i.e., the sws data and the dictex data. detailed analysis show that 1) the simple graph-based parsing model with local loss achieves highly competitive performance; 2) both types of weakly labeled data can provide consistent and substantial gains; 3) our proposed approach outperforms the state-of-the-art mws model by 1.12 on news and by 5.97 on baike in f1.