Towards Fast and Accurate Neural C hinese Word Segmentation with Multi-Criteria Learning (2020)

Abstract: The ambiguous annotation criteria lead to divergence of Chinese Word Segmentation (CWS) datasets in various granularities. Multi-criteria Chinese word segmentation aims to capture various annotation criteria among datasets and leverage their common underlying knowledge. In this paper, we propose a domain adaptive segmenter to exploit diverse criteria of various datasets. Our model is based on Bidirectional Encoder Representations from Transformers (BERT), which is responsible for introducing open-domain knowledge. Private and shared projection layers are proposed to capture domain-specific knowledge and common knowledge, respectively. We also optimize computational efficiency via distillation, quantization, and compiler optimization. Experiments show that our segmenter outperforms the previous state of the art (SOTA) models on 10 CWS datasets with superior efficiency.

## introduction chinese word segmentation (cws) is typically regarded as a low-level nlp task. unlike english and french that uses the space token to separate the words, chinese is a kind of polysynthetic languages where compounds are developed from indigenous morphemes @xcite . the ambiguous distinction between morphemes and compound words leads to the cognitive divergence of word concepts. consequently, the labeled datasets seriously diverge due to the annotation inconsistency, resulting in multi-grained compounds. as shown in in practice, a segmenter usually provides multiple configures with different granularities to better serve various downstream tasks. fine-grained criterion is able to reduce the vocabulary, thereby relieves the sparseness issue. on the other hand, coarse-grained words provide more specific meanings, which may benefit the domain-specific tasks. in recent years, several multi-criteria learning methods for cws have been proposed to explore the common knowledge of heterogeneous datasets. by utilizing the information across all corpora, multicriteria learning methods can boost the out-of-vocabulary (oov) recalls as well as practical performance @xcite @xcite . despite its effectiveness, there still are three unresolved issues. (1) even with multiple datasets, the data is still limited to provide adequate linguistic knowledge. (2) learning from a dataset is likely to hurt the others as the segmentation follows inconsistent criteria. (3) the advanced models, e.g., bi-lstm-crf @xcite , are computationally expensive. they are based on the recurrent neural networks (rnns). since rnns are auto-regressive and the computation cannot be completed in parallel, their applications are usually limited due to the poor computational efficiency. in fact, the inference speed is heavily required for the cws system as it serves as a fundamental module of nlp pipelines. for example, search engines generally can only afford to spend tens of milliseconds or even milliseconds in cws. to alleviate the limitations of existing methods, we propose a multi-criteria method for cws. recent studies @xcite @xcite pointed out that exploiting external knowledge can improve the cws accuracy. based on this observation, we adopt bert @xcite as the backbone to extract the open-domain knowledge. on the top of bert, private projection layers and shared projection layers are used to capture domain-specific knowledge and common underlying knowledge respectively. to make it more practicable, three techniques, i.e., knowledge distillation, numeric quantization and compiler optimization, are adopted to accelerate our segmenter. the bertology analysis @xcite @xcite indicated that the representations from different layers of bert capture specific meanings. it is sufficient to use the representations from a middle layer for the cws task (see section 4.5.2 for detailed analysis). to make the best use of bert, the knowledge distillation method proposed by @xcite is utilized. simultaneously, we also adopt quantization and compiler optimization techniques to improve the scalability. experiments show that our method not only significantly outperforms the best known results on 10 cws datasets with better efficiency. the contributions could be summarized as follows. ## model description current neural cws models usually consist of three components: a character embedding layer, a feature extraction layer and a crf tag inference layer. to equip our model with the ability of multi-criteria learning, we insert an extra domain projection layer before the inference layer, as shown in figure 1 . in this section, we describe the proposed model architecture and the objective function in detail. ## model acceleration neural cws models improve the performance by increasing the model complexity, which however harms the decoding speed and limits their real-world application. to bridge this gap, we apply model acceleration techniques as follows. ## related work recently, multi-criteria learning of neural cws has drawn great attention of scholars. @xcite adopted the stack-based model to take advantage of annotated data from multiple sources. @xcite utilize multiple corpora using coupled sequence labeling model to learn and infer two heterogeneous annotations directly. @xcite proposed switch-lstm to improve the performance of every single criterion by exploiting the underlying shared sub-criteria across multiple heterogeneous criteria. @xcite have proposed a multi-criteria learning framework for cws. they proposed three shared-private models to integrate multiple segmentation criteria. an adversarial strategy is used to force the shared layer to learn criteria-invariant features. all these works utilize heterogeneous annotation data and show that they can indeed help improve each other. model compression and acceleration in deep networks are vital in practice, which makes it possible to deploy deep models on mobile, embedded, and iot devices. techniques like parameter pruning, lowrank factorization, quantization and knowledge distillation had been widely used in visual tasks @xcite @xcite . however, model compression and acceleration are rarely investigated in nlp tasks, especially neural cws task. to the best of our knowledge, we are the first to compress the neural cws model to accelerate the segmentation speed, by three model acceleration techniques, knowledge distillation, quantization and compiler optimization. we emphasize the segmentation speed is very important in industrial application, e.g., search engine. ## conclusion in this paper, we propose an effective chinese word segmentation method that employs bert and adds a domain projection layer on the top with multi-criteria learning. they both serve to capture heterogeneous segmentation criteria and common underlying knowledge. and we visualize the attention score to illustrate linguistic within cws. to be practicability, acceleration techniques are applied to improve the word segmentation speed. it consists of knowledge distillation, quantization and compiler optimization. experiments show that our proposed model achieves higher performance on the word segmentation accuracy and faster prediction speed than the state-of-the-art methods.