Pre-training with Meta Learning for C hinese Word Segmentation (2021)

Abstract: Recent researches show that pre-trained models (PTMs) are beneficial to Chinese Word Segmentation (CWS). However, PTMs used in previous works usually adopt language modeling as pre-training tasks, lacking task-specific prior segmentation knowledge and ignoring the discrepancy between pre-training tasks and downstream CWS tasks. In this paper, we propose a CWS-specific pre-trained model MetaSeg, which employs a unified architecture and incorporates meta learning algorithm into a multi-criteria pre-training task. Empirical results show that MetaSeg could utilize common prior segmentation knowledge from different existing criteria and alleviate the discrepancy between pre-trained models and downstream CWS tasks. Besides, MetaSeg can achieve new state-of-the-art performance on twelve widely-used CWS datasets and significantly improve model performance in low-resource settings.

## introduction chinese word segmentation (cws) is a fundamental task for chinese natural language processing (nlp), which aims at identifying word boundaries in a sentence composed of continuous chinese characters. it provides a basic component for other nlp tasks like named entity recognition @xcite , dependency parsing @xcite , and semantic role labeling @xcite , etc. generally, most previous studies model the cws task as a character-based sequence labeling task @xcite @xcite @xcite . recently, pretrained models (ptms) such as bert @xcite have been introduced into cws tasks, which could provide prior semantic knowledge and boost the performance of cws systems. @xcite directly fine-tunes bert on several cws benchmark datasets. @xcite multi-criteria learning framework, where each criterion shares a common bert-based feature extraction layer and has separate projection layer. @xcite combines chinese character glyph features with pre-trained bert representations. @xcite proposes a neural cws framework wmseg, which utilizes memory networks to incorporate wordhood information into the pre-trained model zen @xcite . ptms have been proved quite effective by finetuning on downstream cws tasks. however, ptms used in previous works usually adopt language modeling as pre-training tasks. thus, they usually lack task-specific prior knowledge for cws and ignore the discrepancy between pre-training tasks and downstream cws tasks. to deal with aforementioned problems of ptms, we consider introducing a cws-specific pretrained model based on existing cws corpora, to leverage the prior segmentation knowledge. however, there are multiple inconsistent segmentation criteria for cws, where each criterion represents a unique style of segmenting chinese sentence into words, as shown in in this paper, we propose a cws-specific pretrained model metaseg. to leverage shared segmentation knowledge of different criteria, metaseg utilizes a unified architecture and introduces a multi-criteria pre-training task. moreover, to alleviate the discrepancy between pretrained models and downstream unseen criteria, meta learning algorithm @xcite is incorporated into the multi-criteria pre-training task of metaseg. experiments show that metaseg could outperform previous works significantly, and achieve new state-of-the-art results on twelve cws datasets. further experiments show that metaseg has better generalization performance on downstream unseen cws tasks in low-resource settings, and improve recalls for out-of-vocabulary (oov) words. to the best of our knowledge, metaseg is the first task-specific pre-trained model especially designed for cws. ## related work recently, ptms have been used for cws and achieve good performance @xcite . these ptms usually exploit fine-tuning as the main way of transferring prior knowledge to downstream cws tasks. specifically, some methods directly fine-tune ptms on cws tasks @xcite , while others fine-tune them in a multi-task framework @xcite . besides, other features are also incorporated into ptms and fine-tuned jointly, including chinese glyph features @xcite , wordhood features @xcite , and so on. although ptms improve cws systems significantly, their pre-training tasks like language modeling still have a wide discrepancy with downstream cws tasks and lack cws-specific prior knowledge. task-specific pre-trained models are lately studied to introduce task-specific prior knowledge into multiple nlp tasks. specifically designed pretraining tasks are introduced to obtain the taskspecific pre-trained models, and then these models are fine-tuned on corresponding downstream nlp tasks, such as named entity recognition @xcite , sentiment analysis @xcite ) and text summarization @xcite . in this paper, we propose a cws-specific pre-trained model metaseg. ## approach as other task-specific pre-trained models @xcite , the pipeline of metaseg is divided into two phases: pre-training phase and fine-tuning phase. in pre-training phase, we design a unified architecture and incorporate meta learning algorithm into a multi-criteria pre-training task, to obtain the cws-specific pre-trained model which has less discrepancy with downstream cws tasks. in finetuning phase, we fine-tune the pre-trained model on downstream cws tasks, to leverage the prior knowledge learned in pre-training phase. in this section, we will describe metaseg in three parts. first, we introduce the transformerbased unified architecture. second, we elaborate on the multi-criteria pre-training task with meta learning algorithm. finally, we give a brief description of the downstream fine-tuning phase. ## conclusion in this paper, we propose a cws-specific pretrained model metaseg, which employs a unified architecture and incorporates meta learning algorithm into a multi-criteria pre-training task. experiments show that metaseg could make good use of common prior segmentation knowledge from different existing criteria, and alleviate the discrepancy between pre-trained models and downstream cws tasks. metaseg also gives better generalization ability in low-resource settings, and achieves new state-of-the-art performance on twelve cws datasets. since the discrepancy between pre-training tasks and downstream tasks also exists in other nlp tasks and other languages, in the future we will explore whether the approach of pre-training with