Lexicon-Based Graph Convolutional Network for C hinese Word Segmentation (2021)

Abstract: Precise information of word boundary can alleviate the problem of lexical ambiguity to improve the performance of natural language processing (NLP) tasks. Thus, Chinese word segmentation (CWS) is a fundamental task in NLP. Due to the development of pre-trained language models (PLM), pre-trained knowledge can help neural methods solve the main problems of the CWS in significant measure. Existing methods have already achieved high performance on several benchmarks (e.g., Bakeoff-2005). However, recent outstanding studies are limited by the small-scale annotated corpus. To further improve the performance of CWS methods based on fine-tuning the PLMs, we propose a novel neural framework, LBGCN, which incorporates a lexicon-based graph convolutional network into the Transformer encoder. Experimental results on five benchmarks and four cross-domain datasets show the lexicon-based graph convolutional network successfully captures the information of candidate words and helps to improve performance on the benchmarks (Bakeoff-2005 and CTB6) and the cross-domain datasets (SIGHAN-2010). Further experiments and analyses demonstrate that our proposed framework effectively models the lexicon to enhance the ability of basic neural frameworks and strengthens the robustness in the cross-domain scenario.

## introduction neural methods often leverage word-level information to improve the performance of many downstream natural language processing (nlp) tasks such as text classification and machine translation @xcite , etc. therefore, in determining the word boundary, word segmentation is regarded as a prerequisite for most downstream nlp tasks. unlike most written languages, the chinese written language has no explicit delimiters to separate words in the written text. thus, chinese word segmentation (cws) is an essential and pre-processing step for many chinese nlp tasks. with the development of deep learning techniques, recent neural cws approaches that do not heavily rely on the hand-craft feature engineering have already achieved high performance on several benchmark datasets @xcite @xcite . in particular, recent outstanding studies have also exploited the learning paradigm in applying pre-trained language models (plm) for many nlp tasks. various methods that fine-tune plms have achieved progress on indomain and cross-domain cws without much manual effort @xcite @xcite . prior research has shown that the problems of cws are segmentation ambiguity and out-ofvocabulary (oov) words @xcite . with the help of the pre-trained knowledge @xcite , the fine-tuning cws methods can effectively alleviate these two issues and outperform other neural network architectures. the methods fine-tuning plms become the mainstream approach for cws. however, the performance of fine-tuning cws methods is limited by the scale and quality of annotated cws corpus. the dependencies between neighboring chinese characters are diverse and it is hard to build a large-scale annotated corpus because of the characteristics of linguistics in chinese. the difficulty of manual annotation restricts the scale and quality of cws datasets. besides, directly fine-tuning methods do not utilize contextual n-grams or other contextual information, which is important for previous model architectures (e.g., bilstm and transformer) @xcite @xcite . the methods that fine-tune plms may generate segmentation errors because of ambigu-ous contextual information. thus, it is a challenge to design a framework that can effectively transfer pre-trained knowledge into the cws. in this paper, we propose the lbgcn, a neural framework with a lexicon-based graph convolutional network (gcn), to improve the performance of the cws by leveraging lexicon knowledge. in detail, we utilize the gcn to extract contextual features of candidate words and the information of word boundary from the pre-defined lexicon. the neural framework incorporates the gcn into the transformer encoder @xcite which is a part of plm (e.g., bert @xcite ). the additional lexicon-based gcn can supply a gap of fine-tuning paradigm and better transfer pre-trained knowledge into the in-domain and cross-domain cws tasks. besides, through multi-feature interaction, the disambiguation and oov word recognition are effectively carried out. to sum up, the contributions of this work are as follows: ## related work chinese word segmentation since @xcite formalizes the cws as a sequence labeling problem, most studies follow the character-based paradigm to predict segmentation labels for each character in the sentence. in particular, the adopted methods fall into two categories, including 1) statistical machine learning methods @xcite @xcite and 2) neural network methods @xcite @xcite @xcite . as the studies of deep learning techniques develop in-depth, the neural cws methods achieve better performance compared with statistical learning methods @xcite @xcite @xcite . and neural network architectures gradually replace statistical machine learning methods as the mainstream approaches for cws. cross-domain cws however, there is an obvious gap in the cross-domain cws scenario. neural cws methods still suffer from the oov problems. to alleviate this problem, many kinds of research utilize external resources (e.g., pre-trained embeddings, unlabeled data, and lexicons) to improve the performance of the cross-domain cws @xcite @xcite . for example, @xcite try to transfer pre-trained knowledge into the crossdomain cws in full by leveraging more annotated datasets with different segmentation criteria @xcite . @xcite utilize lexicons and wordhood measures to enhance the robustness in the cross-domain cws scenario. graph neural network in recent years, the graph neural network has been fully explored and achieved significant progress in several kinds of nlp tasks @xcite . when dealing with text scenarios, graphs can extract the features from non-structural data by modeling a set of objects (nodes) and their relationships (edges). in particular, we can consider each variable in the text as a node and the dependencies as edges for the sequence labeling task. @xcite present a syntactic gcn to solve the problem of semantic role labeling. @xcite utilize a multi-graph structure to capture the information that the gazetteers offer. in addition, the graph neural network based on the domain lexicon is used to learn the local composition features for medical domain cws @xcite . ## proposed framework the framework of lbgcn is illustrated in figure 1 . it mainly consists of two parts: an encoder-decoder layer and a gcn. in the first part, we utilize the transformer as the encoder and the dense as the decoder. the transformer encoder adopts the plms @xcite figure 1: the illustration of the proposed framework. continuous nodes with the same color denote a chinese word in the pre-defined lexicon. the gray nodes of "b", "m" and "e" indicate the three additional nodes, named begin, middle, and end, respectively. ## conclusion to make up for the insufficiency of previous methods that fine-tune plms, in this paper, we propose a lexicon-based graph convolutional network to better transfer pre-trained knowledge from plms into the cws. our proposed framework lbgcn provides baseline models with the information of word boundary and contextual information, in addition to preserving the merits of baseline models in applying plms. in summary, the advantages of lbgcn are threefold. first, the novel framework does not rely on a particular plm, and it can get further promotion on all baseline methods based on three mainstream plms for cws. second, the results on extensive experiments show that lbgcn achieves competitive performance on the cws benchmarks, compared with previous methods. third, further experiments and analyses demonstrate the effectiveness of lbgcn in the cross-domain scenario as well as when using different lexicons and components. overall, this paper presents an elegant way to use a graph neural network for cws and enhance fine-tuning cws methods. for future work, we plan to investigate other sequence labeling tasks using the same methodology.