A Concise Model for Multi-Criteria C hinese Word Segmentation with Transformer Encoder (2020)

Abstract: Multi-criteria Chinese word segmentation (MCCWS) aims to exploit the relations among the multiple heterogeneous segmentation criteria and further improve the performance of each single criterion. Previous work usually regards MCCWS as different tasks, which are learned together under the multi-task learning framework. In this paper, we propose a concise but effective unified model for MCCWS, which is fully-shared for all the criteria. By leveraging the powerful ability of the Transformer encoder, the proposed unified model can segment Chinese text according to a unique criterion-token indicating the output criterion. Besides, the proposed unified model can segment both simplified and traditional Chinese and has an excellent transfer capability. Experiments on eight datasets with different criteria show that our model outperforms our single-criterion baseline model and other multi-criteria models. Source codes of this paper are available on Github.

## introduction chinese word segmentation (cws) is a preliminary step to process chinese text. the mainstream cws methods regard cws as a character-based sequence labeling problem, in which each character is assigned a label to indicate its boundary information. recently, various neural models have been explored to reduce efforts of the feature engineering @xcite @xcite @xcite . recently, @xcite proposed multicriteria chinese word segmentation (mccws) to effectively utilize the heterogeneous resources with different segmentation criteria. specifically, they regard each segmentation criterion as a single task under the framework of multi-task learning, where a shared layer is used to extract the criteriainvariant features, and a private layer is used to extract the criteria-specific features. however, it is unnecessary to use a specific private layer for each criterion. these different criteria often have partial overlaps. for the example in in this work, we propose a concise unified model for mccws task by integrating shared knowledge from multiple segmentation criteria. inspired by the success of the transformer @xcite , we design a fully shared architecture for mccws, where a shared transformer encoder is used to extract the criteria-aware contextual features, and a shared decoder is used to predict the criteria-specific labels. an artificial token is added at the beginning of the input sentence to determine the output criterion. the similar idea is also used in the field of machine translation, @xcite used a single model to translate between multiple languages. figure 1 illustrates our model. there are two reasons to use the transformer encoder for mccws. the primary reason is its neatness and ingenious simplicity to model the criterion-aware context representation for each character. since the transformer encoder uses selfattention mechanism to capture the interaction each two tokens in a sentence, each character can immediately perceive the information of the criteriontoken as well as the context information. the secondary reason is that the transformer encoder has potential advantages in capturing the long-range context information and having a better parallel efficiency than the popular lstm-based encoders. finally, we exploit the eight segmentation criteria on the five simplified chinese and three traditional chinese corpora. experiments show that the proposed model is effective in improving the performance of mccws. the contributions of this paper could be summarized as follows. ## background in this section, we first briefly describe the background knowledge of our work. ## proposed unified model in this work, we propose a more concise architecture for mccws, which adopts the transformer encoder @xcite to extract the contextual features for each input character. in our proposed architecture, both the encoder and decoder are shared by all the criteria. the only difference for each criterion is that a unique token is taken as input to specify the target criterion, which makes the shared encoder to capture the criterion-aware representation. figure 2 illustrates the difference between our proposed model and the previous models. a more detailed architecture for mccws is shown in figure 3 . ## experiments datasets we use eight cws datasets from sighan2005 @xcite and sighan2008 @xcite . among them, the as, cityu, and ckip datasets are in traditional chinese, while the msra, pku, ctb, ncc, and sxu datasets are in simplified chinese. except where otherwise stated, we follow the setting of @xcite , and translate the as, cityu and ckip datasets into simplified chinese. we do not balance the datasets and randomly pick 10% examples from the training set as the development set for all datasets. similar to the previous work @xcite , we preprocess all the datasets by replacing the continuous latin characters and digits with a unique token, and converting all digits, punctuation and latin letters to half-width to deal with the full/half-width mismatch between training and test set. we have checked the annotation schemes of different datasets, which are just partially shared and no two datasets have the same scheme. according to our statistic, the averaged overlap is about 20.5% for 3-gram and 4.4% for 5-gram. pre-trained embedding based on on @xcite @xcite , n-gram features are of great benefit to chinese word segmentation and pos tagging tasks. thus we use unigram and bigram embeddings for our models. we first pre-train unigram and bigram embeddings on chinese wikipedia corpus by the method proposed in @xcite , which improves standard word2vec by incorporating token order information. hyper-parameters we use adam optimizer @xcite with the same warmup strategy as @xcite . the development set is used for parameter tuning. all the models are trained for 100 epochs. pre-trained embeddings are fixed for the first 80 epochs and then updated during the following epochs. after ## related work the previous work on the mccws can be categorized into two lines. one line is multi-task based mccws. @xcite proposed a multi-criteria learning framework for cws, which uses a shared layer to extract the common underlying features and a private layer for each criterion to extract criteria-specific features. @xcite proposed a domain adaptive segmenter to capture diverse criteria based on bidirectional encoder representations from transformer (bert) @xcite . another line is unified mccws. @xcite presented switch-lstms to segment sentences, which consists of several lstm layers, and uses a criterion switcher at every position to change the routing among these lstms automatically. however, the complexity of the model makes switch-lstms hard to be applied in practice. @xcite used a shared bilstm by adding two artificial tokens at the beginning and end of an input sentence to specify the output criterion. however, due to the long-range dependency problem, bil-stm is hard to carry the criterion information to each character in a long sentence. compared to the above two unified models, we use the transformer encoder in our unified model, which can elegantly model the criterion-aware context representation for each character. with the transformer, we just need a special criterion-token to specify the output criterion. each character can directly attend the criterion-token to be aware of the target criterion. thus, we can use a single model to produce different segmented results for different criteria. different from @xcite , which uses the pre-trained transformer bert and several extra projection layers for different criteria, our model is a fully-shared and more concise. ## conclusion and future work we propose a concise unified model for mccws, which uses the transformer encoder to extract the criterion-aware representation according to a unique criterion-token. experiments on eight corpora show that our proposed model outperforms the previous models and has a stronger transfer capability. the conciseness of our model makes it easy to be applied in practice. in this work, we only adopt the vanilla transformer encoder since we just want to utilize its selfattention mechanism to model the criterion-aware context representation for each character neatly. therefore, it is promising for future work to look for the more effective adapted transformer encoder for cws task or to utilize the pre-trained models @xcite , such as bert-based mccws @xcite . besides, we are also planning to incorporate other sequence labeling tasks into the unified model, such as pos tagging and named entity recognition.