Revisiting Pre-Trained Models for C hinese Natural Language Processing (2020)

Abstract: Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and consecutive variants have been proposed to further improve the performance of the pre-trained language models. In this paper, we target on revisiting Chinese pre-trained language models to examine their effectiveness in a non-English language and release the Chinese pre-trained language model series to the community. We also propose a simple but effective model called MacBERT, which improves upon RoBERTa in several ways, especially the masking strategy that adopts MLM as correction (Mac). We carried out extensive experiments on eight Chinese NLP tasks to revisit the existing pre-trained language models as well as the proposed MacBERT. Experimental results show that MacBERT could achieve state-of-the-art performances on many NLP tasks, and we also ablate details with several findings that may help future research. https://github.com/ymcui/MacBERT

## introduction bidirectional encoder representations from transformers (bert) @xcite has become enormously popular and has proven to be effective in recent natural language processing studies, which utilizes large-scale unlabeled training data and generates enriched contextual representations. as we traverse several popular machine reading comprehension benchmarks, such as squad @xcite , coqa @xcite , quac @xcite , naturalquestions @xcite , race @xcite , we can see that most of the top-performing models are based on bert and its variants @xcite @xcite , demonstrating that the pre-trained language models have become new fundamental components in natural language processing field. starting from bert, the community have made great and rapid progress on optimizing the pretrained language models, such as ernie @xcite , xlnet @xcite , roberta @xcite , spanbert @xcite , al-bert @xcite , electra @xcite , etc. however, training transformer-based @xcite pre-trained language models are not as easy as we used to train word embeddings or other traditional neural networks. typically, training a powerful bert-large model, which has 24-layer transformer with 330 million parameters, to convergence needs high-memory computing devices, such as tpu, which is very expensive. on the other hand, though various pre-trained language models have been released, most of them are based on english, and there are few efforts on building powerful pre-trained language models on other languages. in this paper, we aim to build chinese pre-trained language model series and release them to the public for facilitating the research community, as chinese and english are among the most spoken languages in the world. we revisit the existing popular pre-trained language models and adjust them to the chinese language to see if these models generalize well in the language other than english. besides, we also propose a new pre-trained language model called macbert, which replaces the original mlm task into mlm as correction (mac) task and mitigates the discrepancy of the pretraining and fine-tuning stage. extensive experiments are conducted on eight popular chinese nlp datasets, ranging from sentence-level to documentlevel, such as machine reading comprehension, text classification, etc. the results show that the proposed macbert could give significant gains in most of the tasks against other pre-trained language bert ernie xlnet roberta albert electra macbert type ae ae ar ae ae ae ae embeddings t/s/p t/s/p t/s/p t/s/p t/s/p t/s/p t/s/p masking t t/e/ph -t t t wwm/nm lm task mlm mlm plm mlm mlm gen-dis mac paired task nsp nsp --sop -sop ## related work in this section, we revisit the techniques of the representative pre-trained language models in the recent natural language processing field. the overall comparisons of these models, as well as the proposed macbert, are depicted in ## chinese pre-trained language models while we believe most of the conclusions in the previous works are true in english condition, we wonder if these techniques still generalize well in other languages. in this section, we illustrate how the existing pre-trained language models are adapted for the chinese language. furthermore, we also propose a new model called macbert, which adopts the advantages of previous models as well as newly designed components. note that, as these models are all originated from bert without changing the nature of the input, no modification should be made to adapt to these models in the fine-tuning stage, which is very flexible for replacing one another. ## discussion while our models achieve significant improvements on various chinese tasks, we wonder where the essential components of the improvements from. to this end, we carried out detailed ablations on macbert to demonstrate their effectiveness, and we also compare the claims of the existing pretrained language models in english to see if their modification still holds true in another language. ## conclusion in this paper, we revisit pre-trained language models in chinese to see if the techniques in these state-of-the-art models generalize well in a different language other than english only. we created chinese pre-trained language model series and proposed a new model called macbert, which modifies the masked language model (mlm) task as a language correction manner and mitigates the discrepancy of the pre-training and fine-tuning stage. extensive experiments are conducted on various chinese nlp datasets, and the results show that the proposed macbert could give significant gains in most of the tasks, and detailed ablations show that more focus should be made on the mlm task rather than the nsp task and its variants, as we found that nsp-like task does not show a landslide advantage over one another. with the release of the chinese pre-trained language model series, we hope it will further accelerate the natural language processing in the chinese research community. in the future, we would like to investigate an effective way to determine the masking ratios instead of heuristic ones to further improve the performance of the pre-trained language models.