Weighted self Distillation for C hinese word segmentation (2022)

Abstract: Recent researches show that multi-criteria resources and n-gram features are beneficial to Chinese Word Segmentation (CWS). However, these methods rely heavily on such additional information mentioned above and focus less on the model itself. We thus propose a novel neural framework, named Weighted self Distillation for Chinese word segmentation (WeiDC). The framework, which only requires unigram features, adopts self-distillation technology with four hand-crafted weight modules and two teacher models configurations. Experiment results show that WeiDC can make use of character features to learn contextual knowledge and successfully achieve state-of-the-art or competitive performance in terms of strictly closed test settings on SIGHAN Bakeoff benchmark datasets. Moreover, further experiments and analyses also demonstrate the robustness of WeiDC. Source codes of this paper are available on Github.

## introduction chinese is written without explicit word delimiters, while numerous natural language processing (nlp) applications are word-based. moreover, cws is always a fundamental and essential step for processing most language tasks. following the pace of many researchers @xcite @xcite , we also choose [b, i/m, e, s] tags (beginning, inside/middle, end, single character), which represent the precise position of a character in one word. generally, a cws task usually consists of three important parts: embedding, encoder and decoder. google published two papers, @xcite and @xcite , and distributed representation has been widely used in nlp due to its low dimensions and efficiency in semantic similarity. most researchers keep a close eye to the encoder part which includes maximum entropy (me) @xcite , feed-forward neural network @xcite , recursive neural network @xcite , long-short-term memory (lstm) @xcite , pre-training of deep bidirectional transformers such as bert @xcite and other models. as for the decoder part, in addition to softmax, conditional random fields (crf) @xcite usually plays a vital role because it can use the rich contextual feature in the annotation process. with the prevalence of pre-training and finetuning, transformer-based pre-trained models have dominated the field of cws in recent years. given sufficient training data, the pre-trained models @xcite have achieved remarkable results. however, these works may suffer from poor predicting accuracy when rare words or oov (out-of-vocab) words exist. what's more, @xcite confirm that the loss of word segmentation accuracy, caused by oov words, is at least 5 times greater than word segmentation ambiguity. we believe that improving the accuracy of the oov words is worthy of further exploration. unlike traditional knowledge distillation (kd) methods, self distillation teaches a student network by itself instead of a separate teacher @xcite . specifically, during one training epoch, the best student model or the student model from the last iteration will be saved as the teacher model for the next training epoch to teach the student itself. moreover, we believe that the student model should study knowledge selectively according to the importance of information, so it is a practical solution to add an weight matrix to the training process. different from the temperature distillation technology proposed by @xcite , we subtly utilize the information gap between pseudo labels, predicted by the teacher model or student model, and real labels to obtain the hand-crafted weight matrix. from another perspective, the process of acquiring weight matrices can also be seen as a kind of communication between teachers and students. finally, to more precisely demonstrate the impact of weidc, we will temporarily ignore all external information. our contributions are summarized below. we proposed weidc, which only requires unigram features and adopts self-distillation technology with four hand-crafted weight modules and two teacher models configurations. considering there are few choices of weight measures, it is also a challenge to design a feasible method to obtain a rational weight value. we also performed various experiments, such as testing its robustness in some low-resource settings, and explored the efficiency of our framework by combining different encoders and decoders. experimental results from four widely used benchmark datasets confirm that weidc can achieve state-of-the-art or competitive performance, especially in oov recall. ## related work xue and converse (2002) first treat cws as a sequence labeling task and use a maximum entropy tagger to train the data set. @xcite shows a unique charm of the sequential labeling method in the cws bakeoffs @xcite , especially its results on r oov (recall of out of vocabulary). people thus turn their attention to the research of sequence labeling method @xcite @xcite . and huang and zhao (2007) conclude that treating the word segmentation process as a character labeling problem can balance the recognition of vocabulary words and unregistered words, because all words are realized through one unified character marking process. in general, our research is related to the following works. pre-trained frameworks transformer-based pretrained models, such as bert @xcite , roberta @xcite , and zen @xcite , have demonstrated excellent performance in cws tasks. @xcite propose one unified model for multi-criteria cws by leveraging the powerful ability of the transformer encoder. huang et al. (2020) tian et al. (2020) and liu et al. (2021) consider the combination of lexicon features and bert for cws. huang et al. (2021) propose a semi-supervised neural method based on roberta encoder through pseudo labels. knowledge distillation hinton et al. (2015) first propose knowledge distillation, using a larger network to teach a smaller network. tang et al. (2019) choose to distill knowledge from bert, a stateof-the-art language representation model, into a simple heterogeneous model. huang et al. (2020) also extract knowledge from bert to a truncated (3 or 6 layers) bert to balance computational cost and segmentation accuracy on cws tasks. jiao et al. (2020) adopt multiple distilling strategies to reduce the number of parameters of the pre-trained language models. huang et al. (2021) collect massive unlabeled data and distill knowledge from the teacher model to the student model by generating pseudo labels. zhang et al. (2019) put forward self-distillation, which has recently been used in computer vision, but not commonly used in nlp. to summarize, for further improving word segmentation accuracy, many researchers make use of lexicon information @xcite , multi-criteria label data @xcite @xcite and even unlabeled data @xcite @xcite . ## results and analysis in this section, we firstly report the results of weidc and its comparison with the state-of-the-art works available. then we explore the robustness of weidc through lots of experiments in different low-resource settings. we also analyze the impact of oov words on the model. finally, we perform various ner tasks to test weidc's effectiveness. ## case study for cws tasks, it is very hard to get the right segmentations if two adjacent words, such as "天外" ("outside the sky") and "客" ("guest"), both appear for the first time, as shown in although in some cases both "天外客" ("a visitor from outside the sky") and "天外 客" ("outside the sky" and "guest") are rational representations, here we assume that "天外 客" ("outside the sky" and "guest") is correct one and let these models learn it by enhancing the semantic environment. first, according to "replace 1" and "replace 4", if only "天外" ("outside the sky") appears in the previous text, bert obtains "天外 客" ("outside the sky" and "guest") at the cost of inconsistent segmentation criteria in "天外" ("outside the sky"). for weidc, "天外客" ("a visitor from outside the sky") is regarded as a derivative of "天外" ("outside the sky"), as shown in "replace 1". after semantic information gets enriched, the possibility of "天外" ("outside the sky") becoming an independent word increases, so the correct result is obtained. we also notice that when text content is rich, weidc will get desired results even if there is interference information such as "外来" ("outside") in the added semantic knowledge. second, from "replace 2", when "的" ("of") locates between "天外" ("outside the sky") and "客" ("guest"), both bert and weidc learn the right segmentation position by treating "的" ("of") as a single word. we analyzed the pku training set for further exploration and found that "的" ("of") is a high-frequency single-character word. when we blur the semantic information, as shown in "replace 3", weidc treats "天外客" ("a visitor from outside the sky") as a word, while bert can still obtain the correct segmentation. we speculate that the added interference information hurts the small text content. from another perspective, weidc has a strong ability to learn contextual knowledge from different semantic environments to assist cws tasks. last but not least, we make heatmaps to visualize the word segmentation process in figure 3 . ## conclusion in this paper, we proposed a novel framework named weidc, which could make good use of the knowledge in teacher models through selfdistillation. the framework also follows the sequence labeling paradigm but first applies self distillation and weight mechanism to cws, combining four hand-crafted weight modules and two types of teacher models. experimental results show that weidc could achieve higher performance on four cws datasets, with the average f-score ranking second and the average r oov score ranking first. however, for non-sequential labeling problems, such as text classification, a paragraph only corresponds to one tag, so the number of labels is too small, which may render the method in this paper ineffective. how to solve such a dilemma deserves more exploration. besides, it is also promising to consider whether more efficient weight methods exist.