Joint C hinese Word Segmentation and Part-of-speech Tagging via Multi-channel Attention of Character N-grams (2020)

Abstract: Chinese word segmentation (CWS) and part-of-speech (POS) tagging are two fundamental tasks for Chinese language processing. Previous studies have demonstrated that jointly performing them can be an effective one-step solution to both tasks and this joint task can benefit from a good modeling of contextual features such as n-grams. However, their work on modeling such contextual features is limited to concatenating the features or their embeddings directly with the input embeddings without distinguishing whether the contextual features are important for the joint task in the specific context. Therefore, their models for the joint task could be misled by unimportant contextual information. In this paper, we propose a character-based neural model for the joint task enhanced by multi-channel attention of n-grams. In the attention module, n-gram features are categorized into different groups according to several criteria, and n-grams in each group are weighted and distinguished according to their importance for the joint task in the specific context. To categorize n-grams, we try two criteria in this study, i.e., n-gram frequency and length, so that n-grams having different capabilities of carrying contextual information are discriminatively learned by our proposed attention module. Experimental results on five benchmark datasets for CWS and POS tagging demonstrate that our approach outperforms strong baseline models and achieves state-of-the-art performance on all five datasets.

## introduction chinese word segmentation (cws) and part-of-speech (pos) tagging are two fundamental tasks in chinese natural language processing (nlp). although they can be treated as two separate tasks in a sequential order, it has been demonstrated by previous studies that processing them jointly in a unified sequence labeling framework could be more effective, where cws and pos tags are predicted in a single step @xcite @xcite @xcite @xcite @xcite . in doing so, existing studies mainly focused on incorporating contextual information (e.g., n-grams) as features into their joint taggers, which had been widely used as an effective way to improve model performance especially before neural models were widely used. although neural models are powerful in modeling long text sequences, external features from larger granular texts are still demonstrated to be useful in existing neural models @xcite @xcite . in these models, contextual features are leveraged by directly concatenating their embeddings with the character embeddings in the embedding layer, where all contextual features are treated equally without distinguishing their importance to the joint tagging process in the specific context. however, this concatenation approach to incorporating contextual features into a joint tagger fails to consider that different contextual features could have different contributions to the joint task in a specific context, especially when there are ambiguities in the input sentence. for example, in an example sentence figure 1 : an illustration of the different roles of different n-gram features for the character "大" (highlighted in yellow) in an example sentence. in this case, "放大" (enlarge) and "大道" (avenue) are two n-gram features associated with "大", where the former one (in red color) suggests incorrect joint cws and pos labels while the latter one (in green color) suggests the correct labels. in figure 1 , there are two n-gram features associated with the character "大" (highlighted in yellow), i.e., "放大" (enlarge) and "大道" (avenue). in this case, both n-grams are common chinese words; the former (in red color) might suggest incorrect cws and pos tagging results, while the latter (in green color) suggests correct results. 2 if a model treats both n-gram features equally, it could be misled by the former (i.e., "放大" (enlarge) in red color). therefore, it is important to distinguish the contributions of different n-grams to the joint task in a specific context; extra efforts are needed to effectively and smartly leverage such n-grams. in addition, considering that n-grams with different properties could also contribute differently for the joint task, it can be helpful to categorize the n-gram features into groups according to these properties and then model them separately. in this paper, we propose a neural character-based joint cws and pos tagger with multi-channel attention (mcatt) of character n-grams to improve the joint task. specifically, to tag each character in an input sentence, the proposed mcatt first extracts the n-grams associated with the character from a pre-constructed lexicon and next categorizes such n-grams according to a specific metric (i.e., frequency or length). then, we feed all n-grams within the same category into each channel, where those n-grams are compared and weighted according to their contribution to the joint label prediction in a specific context. afterwards, the attentions from different channels are combined to help with the tagging process for each corresponding character. compared to normal attention, where all associated n-grams are compared and weighted together without categorization, multi-channels provide an alternative approach to discriminatively leverage n-grams with different properties. therefore, the weights for n-grams with similar properties are computed in their own channel rather than computed globally with all other n-grams. this multi-channel mechanism could be helpful to leverage the infrequent yet important n-grams, because the parameters for those n-grams are updated infrequently during training so that models with normal attentions may fail to distinguish these infrequent important n-grams in a specific context. we experiment our proposed model on five widely used benchmark datasets. our model with multi-channel attentions outperforms strong baselines and achieves state-of-the-art results on all datasets. ## our approach the architecture of our approach is shown in figure 2 . the left side illustrates the backbone model following the sequence labeling paradigm; the right side elaborates the multi-channel attention module used to incorporate contextual n-gram information into the backbone model. formally, given an input sentence figure 2 : the overall architecture of our character-based model for the joint cws and pos tagging with an example input and output. on the left is the backbone model following the sequence labeling paradigm; on the right is the multi-channel attention module with n-grams categorized by their length. different attention channels for n-grams associated with "大" (big) are highlighted with distinct colors. ## related work there are basically two approaches to cws and pos tagging: to perform the two tasks it in a pipeline framework; or to treat them as a joint task where the two tasks are conducted simultaneously, which is known as joint cws and pos tagging. @xcite provided a comprehensive study to compare the two approaches and found that the joint approach outperform the pipeline one. therefore, in the past two decades, the majority of studies on cws and pos tagging applied the joint approach to these tasks @xcite @xcite @xcite , where n-grams are widely used as features carrying contextual information to improve model performance. recently, neural methods, especially the recurrent neural networks (e.g., bi-lstm) have demonstrated their effectiveness to encode contextual information, and thus significantly improve the model performance in joint cws and pos tagging. even though, improvements can still be obtained when n-grams are incorporated into the neural taggers @xcite @xcite @xcite . for example, @xcite used a stacked bi-lstm model to incorporate n-grams and achieved state-of-the-art results on ctb5 and ctb7. in addition to n-grams, approaches leveraging external resources are also used to improve joint cws and pos tagging: @xcite leveraged n-grams and radical information of chinese characters to enhance model performance; @xcite pre-trained their character embeddings on large data where both segmentation and pos lebels are auto-tagged by an existing model. compared to these studies, our model provide a way to leverage n-grams through a multi-channel attention mechanism, where n-grams are categorized by their frequencies or lengths and the n-grams in the same category are compared and weighted. therefore, the n-grams that contribute more to the joint task in a specific context are highlighted and the model will not be dominated by the frequent or short n-grams (short n-grams also tend to be frequent) because their parameters are intensively updated during training. ## conclusion in this paper, we propose a neural character-based tagger for joint cws and pos tagging, where a multi-channel attention mechanism is used to leverage context information carried by n-grams. in detail, for multi-channel attention, we categorize n-grams according to a specific metric, such as their frequencies or lengths, and model the n-grams separately in each attention channel. in doing so, n-grams with different properties (frequencies or lengths) can be weighted and distinguished separately under a specific context, so that they can be reasonably treated for the joint cws and pos tagging task. experimental results on five chinese benchmark datasets shows that our approach with the multi-channel attentions can work well with n-grams extracted by different methods and provides consistent improvements over strong baseline taggers (i.e., bert and zen) without using it. particularly, our model with zen achieves the state-of-the-art performance for joint cws and pos on all datasets.