Recent researches show that pre-trained models such as BERT (Devlin et al., 2019 ) are beneficial for Chinese Word Segmentation tasks . However, existing approaches usually finetune pre-trained models directly on a separate downstream Chinese Word Segmentation corpus. These recent methods don't fully utilize the prior knowledge of existing segmentation corpora, and don't regard the discrepancy between the pre-training tasks and the downstream Chinese Word Segmentation tasks. In this work , we propose a Pre-Trained Model for Chinese Word Segmentation, which can be abbreviated as PTM-CWS. PTM-CWS model employs a unified architecture for different segmentation criteria, and is pre-trained on a joint multi-criteria corpus with meta learning algorithm  . Empirical results show that our PTM-CWS model can utilize the existing prior segmentation knowledge , reduce the discrepancy between the pre-training tasks and the downstream Chinese Word Segmentation tasks, and achieve new state-of-the-art performance on twelve Chinese Word Segmentation corpora .
Recent researches show that pre-trained models (PTMs ) are beneficial to Chinese Word Segmentation (CWS) . However, PTMs used in previous works usually adopt language modeling as pre-training tasks, lacking task-specific prior segmentation knowledge and ignoring the discrepancy between pre-training tasks and downstream CWS tasks. In this paper , we propose a CWS-specific pre-trained model METASEG, which employs a unified architecture and incorporates meta learning algorithm into a multi-criteria pre-training task . Empirical results show that METASEG could utilize common prior segmentation knowledge from different existing criteria and alleviate the discrepancy between pre-trained models and downstream CWS tasks. Besides, METASEG can achieve new state-of-the-art performance on twelve widely-used CWS datasets and significantly improve model performance in low-resource settings .
R
such as BERT (Devlin et al., 2019
(PTMs
R
for
to
R
tasks
(CWS)
R
existing approaches usually finetune pre-trained models directly on a separate downstream Chinese Word Segmentation corpus. These recent methods don't fully utilize the prior knowledge of existing segmentation corpora, and don't regard
PTMs used in previous works usually adopt language modeling as pre-training tasks, lacking task-specific prior segmentation knowledge and ignoring
D
the
R
the downstream Chinese Word Segmentation
downstream CWS
R
work
paper
R
Pre-Trained Model for Chinese Word Segmentation, which can be abbreviated as PTM-CWS. PTM-CWS model
CWS-specific pre-trained model METASEG, which
R
for different segmentation criteria, and is pre-trained on a joint multi-criteria corpus with
and incorporates
A
into a multi-criteria pre-training task
R
our PTM-CWS model can utilize the existing
METASEG could utilize common
R
, reduce
from different existing criteria and alleviate
R
the pre-training tasks and the downstream Chinese Word Segmentation tasks, and
pre-trained models and downstream CWS tasks. Besides, METASEG can
R
Chinese Word Segmentation corpora
widely-used CWS datasets and significantly improve model performance in low-resource settings