Improving Multi-Criteria C hinese Word Segmentation through Learning Sentence Representation (2023)

Abstract: Recent Chinese word segmentation (CWS) models have shown competitive performance with pre-trained language models’ knowledge. However, these models tend to learn the segmentation knowledge through in-vocabulary words rather than understanding the meaning of the entire context. To address this issue, we introduce a context-aware approach that incorporates unsupervised sentence representation learning over different dropout masks into the multi-criteria training framework. We demonstrate that our approach reaches state-of-the-art (SoTA) performance on F1 scores for six of the nine CWS benchmark datasets and out-of-vocabulary (OOV) recalls for eight of nine. Further experiments discover that substantial improvements can be brought with various sentence representation objectives.

## introduction chinese word segmentation (cws) is a fundamental step for chinese natural language processing (nlp) tasks. researchers have publicized various labeled datasets for evaluating cws models. however, due to the varied properties among the cws datasets, different segmentation criteria exist in different datasets @xcite . a straightforward solution is to create a model for each segmentation criterion @xcite , but this constrains the model from learning cross-dataset segmentation instances. in order to facilitate the differentiation of various segmentation criteria, researchers started to work on building multi-criteria cws (mccws) models. common mccws models employ either a single encoder with multiple decoders @xcite or a single model with extra special tokens @xcite @xcite @xcite . the former assigns distinct decoders to different criteria, sharing other model parts. the previous works sentence a table tennis balls are sold out in the auction. where can i buy limited table tennis balls? sentence b table tennis rackets are sold out. where can i buy table tennis rackets? 乒乓球/拍卖完/了/, /要/去/哪里/买到/限量/的/ 乒乓球/? 乒乓球/拍卖完/了/ , /要/去/哪里/买/新/的/球 拍/? 乒乓球/拍卖/完/了/ , /要/去/哪里/买到/限量/的/ 乒乓球/? 乒乓球拍/卖/完/了/ , /要/去/哪里/买新/的/球拍/? learning in-vocabulary words learning context semantics our approach 乒乓球拍卖完了, 要去哪里买新的球拍? 乒乓球拍卖完了, 要去哪里买到限量的乒乓球? input sequence segment result segment result latter uses special tokens at each input's start, serving the same purpose as private decoders, for a compact model that still differentiates segmentation criteria. however, both of the approaches tend to overfit the in-domain majority criteria for each dataset in use and therefore fail to provide correct segmentations for the minority words, especially the context-dependent ones. we show an example in figure 1 . in this paper, we present a context-aware approach to improve mccws. to enhance the model's understanding of context, inspired by @xcite , we leverage the randomness of dropout @xcite and introduce an auxiliary task for minimizing the difference of sentence representations under the multi-criteria training framework. our contributions lie in (1) the proposed approach sets a new state-of-the-art in the evaluations of f1-score and oov recall on several cws datasets. (2) various objective designs for sentence representation learning can also be effective for improving cws. ## related work after @xcite first treated the cws task as a character tagging problem, most successive researchers followed the approach and performed well. @xcite incorporated multiple cws datasets by matching a specific decoder to different criteria during training. inspired by the idea of a decoder for each criterion, @xcite used a pre-trained language model as their encoder, further enhancing cws performance. but the cost of maintaining decoders increases when the mccws model addresses more datasets. to reduce the model parameters, he et al. (2019); ke et al. (2020); qiu et al. (2020); huang et al. (2020a) ; @xcite utilized extra special tokens to represent criteria respectively, such that the mccws model can interpret the extra special token as a hint and segment text differently. learning sentence representation enhances the pre-trained language model's contextual understanding. in recent years, contrastive learning @xcite @xcite has been the most popular method for learning sentence representation without additional annotated data. maximizing the similarity of the hidden representation of the same text masked by different masks and minimizing the similarity of the hidden representation of different text help the model further understand the text. their model performs better on natural language understanding (nlu) tasks after training on the sentence representation task. as a result, we use an extra special token as a hint to control the criterion and add a sentence representation task to produce the sota performance. ## conclusion in this paper, we introduce a novel training task to cws, achieving the sota f1 score and oov recall on several datasets. we then demonstrate that learning sentence representation is more crucial than adding criterion classification objective function because our model can learn the criteria differences during training in most cases. after realizing each context, our model can also deliver different results for a sequence of words with different contexts (more examples can be found in