A Coarse-to-Fine Labeling Framework for Joint Word Segmentation, POS Tagging, and Constituent Parsing (2021)

Abstract: The most straightforward approach to joint word segmentation (WS), part-of-speech (POS) tagging, and constituent parsing is converting a word-level tree into a char-level tree, which, however, leads to two severe challenges. First, a larger label set (e.g., ≥ 600) and longer inputs both increase computational costs. Second, it is difficult to rule out illegal trees containing conflicting production rules, which is important for reliable model evaluation. If a POS tag (like VV) is above a phrase tag (like VP) in the output tree, it becomes quite complex to decide word boundaries. To deal with both challenges, this work proposes a two-stage coarse-to-fine labeling framework for joint WS-POS-PAR. In the coarse labeling stage, the joint model outputs a bracketed tree, in which each node corresponds to one of four labels (i.e., phrase, subphrase, word, subword). The tree is guaranteed to be legal via constrained CKY decoding. In the fine labeling stage, the model expands each coarse label into a final label (such as VP, VP*, VV, VV*). Experiments on Chinese Penn Treebank 5.1 and 7.0 show that our joint model consistently outperforms the pipeline approach on both settings of w/o and w/ BERT, and achieves new state-of-the-art performance.

## introduction as shown in figure 1 (a), the results of word segmentation (ws), part-of-speech (pos) tagging, and constituent parsing (par) can be organized in a unified hierarchical tree, where leaf nodes include words and their pos tags, and non-terminal nodes correspond to phrases (or constituents) with their syntactic label. before the deep learning (dl) era, there had been intensive research interest in jointly modeling of the three tasks, i.e., ws-pos-par @xcite @xcite word-level tree 2013; @xcite . the motivations are three-fold: 1) alleviating error propagation, 2) promoting knowledge sharing and interaction during inference, 3) simplifying system architecture by training a single model. in contrast, there have been few work on joint ws-pos-par in the dl era, with the exception of @xcite . similar to this work, they treated the joint problem as a char-level parsing task, which is obviously the most straightforward way for chinese, as illustrated in figure 1 @xcite . however, they showed that the joint model fails to outperform the pipeline approach. in the past few years, constituent parsing has achieved significant progress. compared with pre-dl parsers based on discrete features @xcite @xcite , the major characteristic of graphbased parsers @xcite @xcite is that the score of a constituent tree is decomposed into scores of labeled spans (i.e., (2, 3, vp) or vp 2,3 ), without any consideration on production rules (i.e., vp 2,3 → advp 2,2 vp 3,3 ). for char-level graph-based ws-pos-par, there exist two severe challenges. (1) the first challenge is high model complexity. on the one hand, a char sequence is almost twice longer than its word sequence. on the other hand, the size of the la-bel set is very large (e.g., ≥ 600) after transforming char-level trees into chomsky normal form (cnf), as illustrated in figure 2 (a). both factors greatly increase time and memory complexity. in this work, we propose a two-stage coarse-tofine labeling framework for joint ws-pos-par, which can deal with both challenges. in the coarse labeling stage, the joint model outputs a bracketed tree with coarse labels (i.e., phrase, subphrase, word, subword). the constrained cky algorithm is used to guarantee that the predicted tree contains no illegal production rules. in the fine labeling stage, the model expands each coarse label into a final fine-grained label (such as vp, vp * , vv, vv * ). experiments on three chinese treebank (ctb) benchmark datasets show the joint framework is superior to the pipeline framework on both settings of without and with bert @xcite , and achieves new state-of-the-art performance. we will release our code at https://github.com/ ironsword666/jointparser . ## proposed two-stage coarse-to-fine labeling framework as shown in figure 3 , our proposed framework consists of two independent decoders and a shared encoder. during training, two parts of losses are directly added, i.e., the weights of two tasks are both one 2 , and the whole model is trained via standard multi-task learning (mtl). given a sentence consisting of n characters @xmath0, . . . , x n , we use (i, j, c) to denote a candidate span x i ...x j with a coarse label c, and (i, j, f ) to denote one with a fine-grained label f . section 3.3 illustrates how to obtain their corresponding scores, i.e., s(i, j, c) and s(i, j, f ). ## the pipeline framework we adopt the typical cascaded pipeline framework as our baseline. in the training phase, three separate models (ws/pos/par) are separately trained. in the evaluation phase, the first step is word segmentation; then the word sequence is fed into the pos tagger; finally the word sequence is fed into the constituent parser. constituent parsing. we directly adopt the competitive two-stage crf parser of @xcite , which is also backbone of our joint ws-pos-par model. as a word-level parser, its input is composed of two parts: 1) word embedding and 2) charlstm word representation @xcite . other components, such as encoding and span scoring, are the same with those in section 3.3. ## conclusions in this work, we propose a two-stage coarse-to-fine labeling framework of joint ws-pos-par, which is shown to be able to handle both challenges for char-level parsing, i.e., high model complexity, and word-vs-phrase label conflicts. in particular, we believe that it is a novel and elegant way to remove illegal trees using constrained cky decoding in the coarse-labeling stage. it is also very interesting that under a treecrf loss, performance is consistently improved by ruling out illegal trees in computing the normalization factor via the constrained inside algorithm. experiments and analysis on three chinese benchmark datasets (ctb5 and ctb7) show that 1) the joint framework is clearly superior to the pipeline framework, and achieves new state-of-theart performance; 2) our joint framework has very similar parsing speed compared with the pipeline counterpart, but only requires a half number of parameters.