Joint C hinese Word Segmentation and Part-of-speech Tagging via Two-way Attentions of Auto-analyzed Knowledge (2020)

Abstract: Chinese word segmentation (CWS) and part-of-speech (POS) tagging are important fundamental tasks for Chinese language processing, where joint learning of them is an effective one-step solution for both tasks. Previous studies for joint CWS and POS tagging mainly follow the character-based tagging paradigm with introducing contextual information such as n-gram features or sentential representations from recurrent neural models. However, for many cases, the joint tagging needs not only modeling from context features but also knowledge attached to them (e.g., syntactic relations among words); limited efforts have been made by existing research to meet such needs. In this paper, we propose a neural model named TwASP for joint CWS and POS tagging following the character-based sequence labeling paradigm, where a two-way attention mechanism is used to incorporate both context feature and their corresponding syntactic knowledge for each input character. Particularly, we use existing language processing toolkits to obtain the auto-analyzed syntactic knowledge for the context, and the proposed attention module can learn and benefit from them although their quality may not be perfect. Our experiments illustrate the effectiveness of the two-way attentions for joint CWS and POS tagging, where state-of-the-art performance is achieved on five benchmark datasets.

## introduction chinese word segmentation (cws) and part-ofspeech (pos) tagging are two fundamental and crucial tasks in natural language processing (nlp) for chinese. the former one aims to find word boundaries in a sentence and the latter, on the top of segmentation results, assigns a pos tag to each word to indicate its syntactical property in the sentence. to effectively perform cws and pos tagging, combining them into a joint task is proved to have better performance than separately conducting the two tasks in a sequence @xcite . therefore, many studies were proposed in the past decade for joint cws and pos tagging @xcite @xcite @xcite @xcite . these studies, regardless of whether they used conventional approaches @xcite @xcite or deep learning based approaches @xcite @xcite , focused on incorporating contextual information into their joint tagger. in addition, it is well known that syntactic structure is also able to capture and provide the information of long-distance dependencies among words. for example, figure 1 shows an example of local ambiguity, where the green highlighted part has two possible interpretations -"报告 vv/书 nn" (report a book) and "报告书 nn" (the report). the ambiguity can be resolved with syntactic analysis; for instance, the dependency structure, if available, would prefer the first interpretation. while the subject and the object of the sentence (highlighted in yellow) are far away from the ambiguous part in figure 2 : the architecture of twasp for the joint cws and pos tagging with the two-way attention mechanism, which is presented with example context features and their dependency knowledge (highlighted in yellow) from auto-analyzed results for a character (i.e., "分" (split) highlighted in green) in the given sentence. the surface word order, they are much closer in the dependency structure (the subject depends on "报告 vv" and "书 nn" depends on the the object). this example shows that syntactic structure provides useful cues for cws and pos tagging. syntactic knowledge can be obtained from manually constructed resources such as treebanks and grammars, but such resources require considerate efforts to create and might not be available for a particular language or a particular domain. a more practical alternative is to use syntactic structures automatically generated by off-the-shelf toolkits. some previous studies @xcite @xcite verified the idea for this task by learning from autoprocessed corpora. however, their studies treat auto-processed corpora as gold reference and thus are unable to distinguishingly use it according to its quality (the resulted knowledge is not accurate in most cases). therefore, the way to effectively leverage such auto-generated knowledge for the joint cws and pos tagging task is not fully explored. in this paper, we propose a neural model named twasp with a two-way attention mechanism to improve joint cws and pos tagging by learning from auto-analyzed syntactic knowledge, which are generated by existing nlp toolkits and provide necessary (although not perfect) information for the task. in detail, for each input character, the proposed attention module extracts the context features associated with the character and their corresponding knowledge instances according to the auto-analyzed results, then computes the attentions separately for features and knowledge in each attention way, and finally concatenates the attentions from two ways to guide the tagging process. in doing so, our model can distinguish the important auto-analyzed knowledge based on their contributions to the task and thus avoid being influenced by some inferior knowledge instances. compared to another prevailing model, i.e., key-value memory networks @xcite , which can learn from pair-wisely organized information, the twoway attentions not only are able to do so, but also fully leverage features and their knowledge rather than using one to weight the other. 2 we experiment with three types of knowledge, namely, pos labels, syntactic constituents, and dependency relations, in our experiments. the experimental results on five benchmark datasets illustrate the effectiveness of our model, where state-of-the-art performance for the joint task is achieved on all datasets. we also perform several analyses, which confirm the validity of using two-way attentions and demonstrate that our model can be further improved by synchronously using multiple types of knowledge. ## the model the architecture of twasp is illustrated in figure 2 . the left part shows the backbone of the model for the joint cws and pos tagging following 2 ) takes the syntactic knowledge produced from the input sentence, analyzes it and then feeds it to the tagging process. in this section, we firstly introduce the auto-analyzed knowledge, then explain how the two-way attentions consume such knowledge, and finally describe how the joint cws and pos tagging works with the resulted attentions. ## related work there are basically two approaches to cws and pos tagging: to perform pos tagging right after word segmentation in a pipeline, or to conduct the two tasks simultaneously, known as joint cws and pos tagging. in the past two decades, many studies have shown that joint tagging outperforms the pipeline approach @xcite @xcite @xcite . in recent years, neural methods started to play a dominant role for this task @xcite @xcite , where some of them tried to incorporate extra knowledge in their studies. for example, @xcite exploited to model n-grams to improve the task; @xcite extended the idea by incorporating pre-trained n-gram embeddings, as well as radical embeddings, into character representations. @xcite tried to leverage the knowledge from character embeddings, trained on an automatically tagged corpus by a baseline tagger. compared to these previous studies, twasp provides a simple, yet effective, neural model for joint tagging, without requiring a complicated mechanism of incorporating different features or pre-processing a corpus. ## conclusion in this paper, we propose neural approach with a two-way attention mechanism to incorporate autoanalyzed knowledge for joint cws and pos tagging, following a character-based sequence labeling paradigm. our proposed attention module learns and weights context features and their corresponding knowledge instances in two separate ways, and use the combined attentions from the two ways to enhance the joint tagging. experimental results on five benchmark datasets illustrate the validity and effectiveness of our model, where the two-way attentions can be integrated with different encoders and provide consistent improvements over baseline taggers. our model achieves stateof-the-art performance on all the datasets. overall, this work presents an elegant way to use autoanalyzed knowledge and enhance neural models with existing nlp tools. for future work, we plan to apply the same methodology to other nlp tasks.