Improving C hinese Word Segmentation with Wordhood Memory Networks (2020)

Abstract: Contextual features always play an important role in Chinese word segmentation (CWS). Wordhood information, being one of the contextual features, is proved to be useful in many conventional character-based segmenters. However, this feature receives less attention in recent neural models and it is also challenging to design a framework that can properly integrate wordhood information from different wordhood measures to existing neural frameworks. In this paper, we therefore propose a neural framework, WMSeg, which uses memory networks to incorporate wordhood information with several popular encoder-decoder combinations for CWS. Experimental results on five benchmark datasets indicate the memory mechanism successfully models wordhood information for neural segmenters and helps WMSeg achieve state-of-the-art performance on all those datasets. Further experiments and analyses also demonstrate the robustness of our proposed framework with respect to different wordhood measures and the efficiency of wordhood information in cross-domain experiments.

## introduction unlike most written languages in the world, the chinese writing system does not use explicit delimiters (e.g., white space) to separate words in written text. therefore, chinese word segmentation (cws) conventionally serves as the first step in chinese language processing, especially for many downstream tasks such as text classification @xcite , question answering @xcite , machine translation @xcite , etc. in the past two decades, the mainstream methodology of cws treated cws as a character-based sequence labeling task @xcite @xcite @xcite @xcite @xcite , where various studies were proposed to effectively extract contextual features to help better predicting segmentation labels for each character @xcite @xcite . among all the contextual features, the ones measuring wordhood for n-grams illustrate their helpfulness in many nonneural cws models @xcite @xcite . later, following the track of the sequence labeling methodology, recent approaches with neural networks are proved to be powerful in this task @xcite @xcite . however, since neural networks (e.g., lstm) is considered to be able to provide a good modeling of contextual dependencies, less attention is paid to the idea of explicitly leveraging wordhood information of n-grams in the context as what had previously been done in non-neural models. although some studies sidestepped the idea by incorporating contextual n-grams @xcite or word attention @xcite into the sequence labeling process, they are limited in either concatenating word and character embeddings or requiring a well-defined word lexicon. therefore, it has not been fully explored what would be the best way of representing contextual information such as wordhood features in neural cws models. moreover, consider there are various choices of wordhood measures, it is also a challenge to design a framework that can incorporate different wordhood features so that the entire cws approach can be general while being effective in accommodating the input from any measures. in this paper, we propose wmseg, a neural framework with a memory mechanism, to improve figure 1 : the architecture of wmseg. "n " denotes a lexicon constructed by wordhood measures. n-grams (keys) appearing in the input sentence "部分居民生活水平" (some residents' living standard) and the wordhood information (values) of those n-grams are extracted from the lexicon. then, together with the output from the text encoder, n-grams (keys) and their wordhood information (values) are fed into the memory module, whose output passes through a decoder to get final predictions of segmentation labels for every character in the input sentence. cws by leveraging wordhood information. in detail, we utilize key-value memory networks @xcite to incorporate character n-grams with their wordhood measurements in a general sequence labeling paradigm, where the memory module can be incorporated with different prevailing encoders (e.g., bilstm and bert) and decoders (e.g., softmax and crf). for the memory, we map n-grams and their wordhood information to keys and values in it, respectively, and one can use different wordhood measures to generate such information. then for each input character, the memory module addresses all the n-grams in the key list that contain the character and uses their corresponding values to generate an output vector to enhance the decoder for assigning a segmentation label to the character. experimental results from five widely used benchmark datasets confirm that wmseg with wordhood information can improve cws over powerful baseline segmenters and ourperform previous studies, where state-of-the-art performance is observed on all the datasets. further experiments and analyses are also performed to investigate different factors affecting wmseg's performance. ## the proposed framework following previous studies, we regard cws as a character-based sequence labeling task. the architecture of wmseg is illustrated in figure 1 , where the general sequence labeling paradigm is the top part with a memory module inserted between the encoder and the decoder. the model predicts a tag (e.g., tag b for the 1st character in a word) for each character, and the predicted tag sequence is then converted to word boundary in the system output. the bottom part of the figure starts with a lexicon n , which is simply a list of n-grams and can be built by various methods (see section 2.1). given an input sentence @xmath0...x i ...x l , for each character x i in x , our approach uses the lexicon n to generate (keys, values) for x i and send it to the memory module. in all, the process of wmseg to perform cws can be formalized as in the rest of this section, we describe the construction of the n-gram lexicon, the proposed wordhood memory networks, and how it is integrated with different encoders and decoders, respectively. ## results and analyses in this section, we firstly report the results of wm-seg with different configurations on five benchmark datasets and its comparison with existing models. then we explore the effect of using different lexicon n and different wordhood measures in wmseg. we also use a cross-domain experiment to illustrate the effectiveness of wmseg when more oovs are in the test set. lastly, a case study is performed to visualize how the wordhood information used in wmseg helps cws. ## related work as one of the most fundamental nlp tasks for chinese language processing, cws has been studied for decades, with two steams of methods, i.e., word-based and character-based ones @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . among these studies, most of them follow the character-based paradigm to predict segmentation labels for each character in an input sentence; n-grams are used in some of these studies to enhance model performance, which is also observed in many other nlp tasks @xcite @xcite @xcite . recently, cws benefits from neural networks and further progress are made with embeddings @xcite @xcite @xcite , recurrent neural models @xcite @xcite and even adversarial learning @xcite . to enhance cws with neural models, there were studies leverage external information, such as vocabularies from auto-segmented external corpus @xcite , where @xcite introduced a word attention mechanism to learn from large granular texts during the cws process. in addition, the studies from @xcite and @xcite try to improve cws by learning from data annotated through different segmentation criteria. moreover, there is a study leveraging auto-analyzed syntactic knowledge obtained from off-the-shelf toolkits to help cws and part-of-speech tagging @xcite . compare to these studies, wmseg offers an alternative solution to robustly enhancing neural cws models without requiring external resources. ## conclusion in this paper, we propose wmseg, a neural framework for cws using wordhood memory networks, which maps n-grams and their wordhood information to keys and values in it and appropriately models the values according to the importance of keys in a specific context. the framework follows the sequence labeling paradigm, and the encoders and decoders in it can be implemented by various prevailing models. to the best of our knowledge, this is the first work using key-value memory networks and utilizing wordhood information for neural models in cws. experimental results on various widely used benchmark datasets illustrate the effectiveness of wmseg, where state-of-the-art performance is achieved on all datasets. further experiments and analyses also demonstrate the robustness of wm-seg in the cross-domain scenario as well as when using different lexicons and wordhood measures.