R ethink CWS : Is C hinese Word Segmentation a Solved Task? (2020)

Abstract: The performance of the Chinese Word Segmentation (CWS) systems has gradually reached a plateau with the rapid development of deep neural networks, especially the successful use of large pre-trained models. In this paper, we take stock of what we have achieved and rethink what’s left in the CWS task. Methodologically, we propose a fine-grained evaluation for existing CWS systems, which not only allows us to diagnose the strengths and weaknesses of existing models (under the in-dataset setting), but enables us to quantify the discrepancy between different criterion and alleviate the negative transfer problem when doing multi-criteria learning. Strategically, despite not aiming to propose a novel model in this paper, our comprehensive experiments on eight models and seven datasets, as well as thorough analysis, could search for some promising direction for future research. We make all codes publicly available and release an interface that can quickly evaluate and diagnose user’s models: https://github.com/neulab/InterpretEval

## introduction chinese word segmentation (cws), as a crucial first step in chinese language processing, has drawn a large body of research @xcite @xcite . recent years have seen remarkable success in the use of deep neural networks on cws @xcite @xcite @xcite @xcite , and the large unsupervised pre-trained models drive the state-of-the-art results to a new level @xcite . however, the performance of cws systems gradually reaches a plateau and the development of this * these two authors contributed equally. q1: does current excellent performance (e.g. more than 98.0 f 1-score on the msr dataset) indicate a perfect cws system, or are there still some limitations? existing cws systems are mainly evaluated by a corpus-level metric. the holistic measure fails to provide a fine-grained analysis. as a result, we are not clear about what the strengths and weaknesses of a specific model are. to address this problem, we shift the traditional trend of holistic evaluation to fine-grained evaluation, in which the notion of the attribute (i.e., word length) has been introduced to describe a property of each word. then test words will be partitioned into different buckets, in which we can observe the system's performances under different aspects based on word's attributes (e.g. long words will obtain lower f 1-score). q2: is there a one-size-fits-all system (i.e., bestperforming systems on different datasets are the same)? if no, how can we make different choices of model architectures in different datasets? insights are still missing for how the choices of different datasets influence architecture design. to answer this question, we make use of our proposed fine-grained evaluation methodology and present two types of diagnostic methods for existing cws systems, which not only helps us to identify the strengths and weaknesses of current approaches but provides us with more insight about how different choices of datasets influence the model design. q3: now that existing works show cws sys- settings measures application model spearman s ρ (eq. 2) in-variance s σ (eq. 3) model diagnosis dataset data sys-indep α µ (eq. 4) sec. 3.6 (q1,q2) sys-dep α ρ (eq. 5) cross-model generali. u (eq. 6) multi-source tra-dataset data criterion ψ (eq. 7) nsfer sec. 4.4 (q3) to answer this question, we extend the in-dataset evaluation (i.e., a system is trained and tested on the same dataset) to the setting of cross-dataset, in which a cws model trained on one corpus would be evaluated on a range of out-of-domain corpora. on the other hand, it's the above in-dataset analysis (in q1 & q2) that helps us to design a measure to quantify the discrepancies of cross-dataset criterion. empirical results not only show that the measure, calculated solely based on statistics of two datasets, has a higher correlation with cross-dataset performances but also helps us avoid the negative transfer (i.e., selecting the useful parts of source domains as training sets and achieve better results based on fewer training samples) our contributes can be summarized as follows: 1) instead of using a holistic metric, we proposed an attribute-aided evaluation methodology for cws systems. this allows us to diagnose the weakness of existing cws systems (e.g., bert-based models are not impeccable and limited in dealing with words with high label inconsistency). 2) we show that best-performing systems on different datasets are diverse. based on some proposed quantified measures, we can make good choices of model architectures in different datasets. 3) we quantify the criterion discrepancy between different datasets, which can alleviate the negative transfer problem when performing multi-criteria learning for cws. ## investigation on cross-dataset setting the above in-dataset analysis aims to interpret model bias and dataset bias based on individual datasets. in many real-world scenarios, we need to transfer a trained model to a new dataset or domain, which requires us to understand the crossdataset generalization behavior of current systems. in this section, our investigation on cross-dataset generalization is driven by two questions: 1) how different architectures (i.e. cw2vbavglstmcrf ) of cws systems influence their cross-dataset generalization ability? 2) now that we have found the common factor (label consistency) that affects model performance across different datasets in the previous section, can we design a measure based on it and use it to interpret cross-data generalization? we will detail our exploration below. ## discussion we summarize the main observations from our experiments and try to give preliminary answers to our proposed research questions: does existing excellent performance imply a perfect cws system? no. beyond giving this unsurprising answer, we present an interpretable evaluation method to help us diagnose the weaknesses of existing top-performing systems and relative merits between two systems. for example, we find even top-scoring bert-based models still cannot deal with the words with low label consistency or long words well, and bert is inferior to elmo as an encoder in dealing with long sentences. is there a one-size-fits-all system? no (bestperforming systems on different datasets frequently involve diverse neural architectures). although this question can be answered relatively easily by simply looking at the overall results of different systems in diverse data sets (sec.2), we take a step further to how to make choices of them (bert v.s elmo, lstm v.s cnn)) by conducting dataset bias-aware aided-diagnosis (sec.3.6). can we design a measure to quantify the discrepancies among different criteria? yes. we first verify that the label consistency of words and word length have a more consistent impact on cws performance. based on this, we design a measure to quantify the distance between different datasets, which correlates well with the cross-dataset performance and can be used for multi-source transfer learning, help us avoid the negative transfer. s s s l l l l -10 0 10 s s s s l l l 0 10 20 l s s s m l l 0 2 4 l s s s l l l 0 5 10 l s l s l m s -5 0 l s m l l l s -2 -1 0 l s l l l m l 0 10 20 30 a: cw2vbavglstmcrf b: cw2vbavglstmmlp aided-diagnosis overall f1 a:96.46; b:96.21 a:94.10; b:94.22 a:95.08; b:95.32 a:92.81; b:92.81 a:93.67; b:93.54 a:92.04; b:92.01 a:94.71; b:94.87