A Graph-based Model for Joint C hinese Word Segmentation and Dependency Parsing (2020)

Abstract: Chinese word segmentation and dependency parsing are two fundamental tasks for Chinese natural language processing. The dependency parsing is defined at the word-level. Therefore word segmentation is the precondition of dependency parsing, which makes dependency parsing suffer from error propagation and unable to directly make use of character-level pre-trained language models (such as BERT). In this paper, we propose a graph-based model to integrate Chinese word segmentation and dependency parsing. Different from previous transition-based joint models, our proposed model is more concise, which results in fewer efforts of feature engineering. Our graph-based joint model achieves better performance than previous joint models and state-of-the-art results in both Chinese word segmentation and dependency parsing. Additionally, when BERT is combined, our model can substantially reduce the performance gap of dependency parsing between joint models and gold-segmented word-based models. Our code is publicly available at https://github.com/fastnlp/JointCwsParser

## introduction unlike english, chinese sentences consist of continuous characters and lack obvious boundaries between chinese words. words are usually regarded as the minimum semantic unit, therefore chinese word segmentation (cws) becomes a preliminary pre-process step for downstream chinese natural language processing (nlp). for example, the fundamental nlp task, dependency * corresponding author. parsing, is usually defined at the word-level. to parse a chinese sentence, the process is usually performed in the following pipeline method: word segmentation, part-of-speech (pos) tagging, and dependency parsing. however, the pipeline method always suffers from the following limitations: a traditional solution to this error propagation problem is to use joint models @xcite @xcite . these previous joint models mainly adopted a transitionbased parsing framework to integrate the word segmentation, pos tagging, and dependency parsing. based on standard sequential shift-reduce transitions, they design some extra actions for word segmentation and pos tagging. although these joint models achieved better performance than the pipeline model, they still suffer from two limitations: recently, graph-based models have made significant progress for dependency parsing @xcite , which fully exploit the ability of the bidirectional long short-term memory network (bilstm) @xcite and attention mechanism @xcite to capture the interactions of words in a sentence. different from the transition-based models, the graph-based models assign a score or probability to each possible arc and then construct a maximum spanning tree from these weighted arcs. in this paper, we propose a joint model for cws and dependency parsing that integrates these two tasks into a unified graph-based parsing framework. because the segmentation is a characterlevel task and dependency parsing is a word-level task, we first formulate these two tasks into a character-level graph-based parsing framework. in detail, our model contains (1) a deep neural network encoder, which can capture the longterm contextual features for each characterit can be a multi-layer bilstm or pre-trained bert, (2) a biaffine attentional scorer @xcite , which unifies segmentation and dependency relations at the character level. besides, unlike the previous joint models @xcite @xcite , our joint model does not depend on the pos tagging task. in experiments on three popular datasets, we obtain state-of-the-art performance on cws and dependency parsing. in this paper, we claim four contributions: ## related work to reduce the problem of error propagation and improve the low-level tasks by incorporating the knowledge from the high-level tasks, many successful joint methods have been proposed to simultaneously solve related tasks, which can be categorized into three types. ## proposed model previous joint methods are mainly based on the transition-based model, which modifies the standard ''shift-reduce'' operations by adding some extra operations, such as ''app'' and ''tag''. different from previous methods, we integrate word segmentation and dependency parsing into a graph-based parsing framework, which is simpler and easily implemented. first, we transform the word segmentation to a special arc prediction problem. for example, a chinese word '' second, we transform the word-level dependency arcs to character-level dependency arcs. assuming that there is a dependency arc between words w 1 = x i:j and w 2 = x u:v , where x i:j denotes the continuous characters from i to j in a sentence, we make this arc to connect the last characters x j and x v of each word. for example, the arc '' (develop)→ (financial sector)'' is translated to '' → ''. figure 1 illustrates the framework for joint cws and dependency parsing. thus, we can use a graph-based parsing model to conduct these two tasks. our model contains two main components: (1) a deep neural network encoder to extract the contextual features, which converts discrete characters into dense vectors, and (2) a biaffine attentional scorer @xcite , which takes the hidden vectors for the given character pair as input and predicts a label score vector. figure 2 illustrates the model structure for joint cws and dependency parsing. the detailed description is as follows. ## conclusion and future work in this paper, we propose a graph-based model for joint chinese word segmentation and dependency parsing. different from the previous joint models, our proposed model is a graph-based model and is more concise, which results in fewer efforts of feature engineering. although no explicit handcrafted parsing features are applied, our joint model outperforms the previous feature-riched joint models by a large margin. the empirical results in ctb-5, ctb-7, and ctb-9 show that the dependency parsing task is also beneficial to chinese word segmentation. additionally, labeled dependency parsing not only is good for chinese word segmentation, but also avails the dependency parsing head prediction. apart from good performance, the comparison between our joint model and the pipeline model shows great potentialities for character-based chinese dependency parsing. and owing to the joint decoding between chinese word segmentation and dependency parsing, our model can use a pre-trained character-level language model (such as bert) to enhance the performance further. after the incorporation of bert, the performance of our joint model increases substantially, resulting in the character-based dependency parsing performing near the gold-segmented wordbased dependency parsing. our proposed method not merely outpaces the pipeline model, but also avoids the preparation for pre-trained word embeddings that depends on a good chinese word segmentation model. in order to fully explore the possibility of graphbased chinese dependency parsing, future work should be done to incorporate pos tagging into this framework. additionally, as illustrated in @xcite , a more reasonable intra-word dependent structure might further boost the performance of all tasks.