A Robustly Optimized BERT Pre-training Approach with Post-training (2021)

Abstract: In the paper we present a ‘pre-training’+‘post-training’+‘fine-tuning’ three-stage paradigm which is a supplementary framework for the standard ‘pre-training’+‘fine-tuning’ languagemodel approach. Furthermore based on three-stage paradigm we present a language modelnamed PPBERT. Compared with original BERT architecture that is based on the standard two-stage paradigm we do not fine-tune pre-trained model directly but rather post-train it on the domain or task related dataset first which helps to better incorporate task-awareness knowl-edge and domain-awareness knowledge within pre-trained model also from the training datasetreduce bias. Extensive experimental results indicate that proposed model improves the perfor-mance of the baselines on 24 NLP tasks which includes eight GLUE benchmarks eight Su-perGLUE benchmarks six extractive question answering benchmarks. More remarkably our proposed model is a more flexible and pluggable model where post-training approach is able to be plugged into other PLMs that are based on BERT. Extensive ablations further validate the effectiveness and its state-of-the-art (SOTA) performance. The open source code pre-trained models and post-trained models are available publicly.

## introduction recently, the introduction of pre-trained language models (plms), including gpt @xcite , bert @xcite , and elmo @xcite , among many others, has achieved tremendous success to the natural language processing (nlp) research. typically, the basic structure of such a model consists of two successive stages, one step during the pre-training phase and another step during the fine-tuning phase. during the pre-training phase it pre-trains on unsupervised dataset firstly, then during the fine-tuning phase it fine-tunes on downstream supervised nlp tasks. up to now, these models obtained the best performance on various nlp tasks. some of the most prominent examples are bert, and bert based spanbert @xcite , albert @xcite . these plms are trained on the large unsupervised corpus through some unsupervised training objectives. however, it is not obvious that the model parameters which is obtained during unsupervised pre-training phase can be well-suited to support the this kind of transfer learning. especially during the fine-tuning phase, for the target nlp task only a small amount of supervised text data is available, fine-tuning the pre-trained model are potentially brittle. and for the pre-trained model, supervised fine-tuning requires substantial amounts of task-specific supervised training dataset, not always available. for example, in glue benchmark @xcite , winograd schema dataset @xcite have only 634 training data, too small for fine-tuning natural language inference (nli) task. moreover, although plms, such bert, can learn contextualized representations across many nlp tasks (to be task-agnostic), which leverages plms alone still leaves the domain-specific challenges unresolved (bert are trained on general domain corpora only, and capture a general language knowledge from training dataset, but lack domain or task-specific data severely). for example, in financial domain, they often contain unique vocabulary information, such as stock, bond type, and the sizes of labeled data are also very small (even only few hundreds of samples). in the paper, to overcome the aforementioned issues, we proposed a novel three-stage bert (called pp-bert) architecture, in which we add a second stage of training, that is 'post-training', to improving the original bert architecture model. typically there are two directions to pursue new state-of-art in the post pre-trained plms era. one is to construct novel neural network architecture model based on plms, like bertserini @xcite and bertcmc @xcite . other approach is to optimize pre-training, like gp-t 2.0 @xcite , mt-dnn @xcite , spanbert @xcite , and al-bert @xcite . in the paper, we present another novel method to improve the plms. we present a 'pre-training'+'post-training'+'fine-tuning' three-stage paradigm and further present a language model named ppbert. compared with original bert architecture that is based on the standard 'pre-training'+'fine-tuning' plms approach, we do not fine-tune pre-trained models directly, but rather post-train them on the domain or task related training dataset first, which helps to better incorporate taskawareness knowledge and domain-awareness knowledge within pre-trained model, also in the training dataset can reduce bias. more specifically, our framework involves three sequential stages: pre-training stage using on large-scale corpora (see subsection 2.1), post-training stage using the task or domain related datasets via multi-task continual learning method (see subsection 2.2), and fine-tuning stage using target datasets, even with little labeled samples or without labeled samples (see subsection 2.3). thus, ppbert can benefits from the regularization effect since it leverages cross-domain or cross-task data, which helps model generalize better with limited data and adapt to new domains or tasks better. sum up, on a wide variety of tasks our proposed post-training process outperforms existing bert benchmark, and achieved better performance on small dataset and domain-specific tasks in particular substantially. specifically, we compared our model with bert baselines on glue and superglue benchmark tasks and consistently significantly outperform bert on all of 16 tasks (8 glue tasks and 8 superglue tasks), increasing by the glue average score of 87.02, showing an absolute improvement of 2.97 over bert; showing an absolute improvement of 5.55, pushing the superglue to 74.55. more remarkably, our model is a more flexible and pluggable. the post-training appoach can be straight plugged into other plms based on bert. in our ablation studies, we plug the post-training strategy into original bert (i.e., ppbert) and its variant, albert (called ppalbert), respectively. our approaches advanced the sota results for five popular question answering datasets, surpassing the previous pre-trained models by at least 1 point in absolute accuracy. moreover, through further ablation studies, the best model obtains sota results on small datasets (1/20 training set). all of these clearly demonstrate our proposed three-stage paradigms exceptional generalization capability via post-training learning. figure 1 : an illustration of the architecture for our ppbert, which is a 'pre-training'-'post-training'-then-'fine-tuning' three-stage bert. compared with standard bert architecture that has the two-stage 'pre-training'-then-'fine-tuning', we do not directly fine-tune pre-trained models, but rather add a second stage of training (called 'post-training'). more specifically, during the pre-training stage, we first on the large-scale dataset conduct unsupervised pre-training, and then during the posttraining stage post-train pre-trained models on the task or domain related dataset, and last during the fine-tuning stage conduct fine-tuning on downstream supervised nlp tasks. ## conclusion in the paper, we present a 'pre-training'+'post-training'+'fine-tuning' three-stage paradigm and a language model named ppbert based on the three-stage paradigm, which is a supplementary framework for the standard 'pre-training'+'fine-tuning' two-stage architecture. our proposed three-stage paradigm helps to incorporate task-awareness knowledge and domain knowledge within pre-trained model, also reduce the bias in the training corpus. ppbert can benefits from the regularization effect since it leverages cross-domain or cross-task data, which helps model generalize better with limited data and adapt to new domains or tasks better. with the latest plms as baseline and encoder backbone, ppbert is evaluated on 24 well-known benchmarks, which outperforms strong baseline models and obtains new sota results. we hope this work can encourage further research into the language models training, and the future works involve the choice of other transfer learning sources such as cv etc.