  There have been various types of pretraining architectures including
autoregressive models (e.g., GPT), autoencoding models (e.g., BERT), and
encoder-decoder models (e.g., T5). On the other hand, NLP tasks are different
in nature, with three main categories being classification, unconditional
generation, and conditional generation. However, none of the pretraining
frameworks performs the best for all tasks, which introduces inconvenience for
model development and selection. We propose a novel pretraining framework GLM
(General Language Model) to address this challenge. Compared to previous work,
our architecture has three major benefits: (1) it performs well on
classification, unconditional generation, and conditional generation tasks with
one single pretrained model; (2) it outperforms BERT-like models on
classification due to improved pretrain-finetune consistency; (3) it naturally
handles variable-length blank filling which is crucial for many downstream
tasks. Empirically, GLM substantially outperforms BERT on the SuperGLUE natural
language understanding benchmark with the same amount of pre-training data.
Moreover, GLM with 1.25x parameters of BERT-Large achieves the best performance
in NLU, conditional and unconditional generation at the same time, which
demonstrates its generalizability to different downstream tasks.
