GLM : General Language Model Pretraining with Autoregressive Blank Infilling (2022)

Abstract: There have been various types of pretraining architectures including autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and encoder-decoder models (e.g., T5). However, none of the pretraining frameworks performs the best for all tasks of three main categories including natural language understanding (NLU), unconditional generation, and conditional generation. We propose a General Language Model (GLM) based on autoregressive blank infilling to address this challenge. GLM improves blank filling pretraining by adding 2D positional encodings and allowing an arbitrary order to predict spans, which results in performance gains over BERT and T5 on NLU tasks. Meanwhile, GLM can be pretrained for different types of tasks by varying the number and lengths of blanks. On a wide range of tasks across NLU, conditional and unconditional generation, GLM outperforms BERT, T5, and GPT given the same model sizes and data, and achieves the best performance from a single pretrained model with 1.25× parameters of BERT Large , demonstrating its generalizability to different downstream tasks.

## introduction language models pretrained on unlabeled texts have substantially advanced the state of the art in various nlp tasks, ranging from natural language understanding (nlu) to text generation @xcite @xcite @xcite @xcite . downstream task performance as well as the scale of the parameters have also constantly increased in the past few years. in general, existing pretraining frameworks can be categorized into three families: autoregressive, autoencoding, and encoder-decoder models. autoregressive models, such as gpt @xcite , learn left-to-right language models. while they succeed in long-text generation and show fewshot learning ability when scaled to billions of parameters @xcite , the inherent disadvantage is the unidirectional attention mechanism, which cannot fully capture the dependencies between the context words in nlu tasks. autoencoding models, such as bert @xcite , learn bidirectional context encoders via denoising objectives, e.g. masked language model (mlm). the encoders produce contextualized representations that suit natural language understanding tasks, but could not be directly applied for text generation. encoder-decoder models adopt bidirectional attention for the encoder, unidirectional attention for the decoder, and cross attention between them @xcite @xcite . they are typically deployed in conditional generation tasks, such as text summarization and response generation. foot_1 . t5 @xcite unifies nlu and conditional generation via encoder-decoder models but requires more parameters to match the performance of bret-based models such as roberta @xcite and deberta @xcite . none of these pretraining frameworks is flexible enough to perform competitively across all nlp tasks. previous works have tried to unify different frameworks by combining their objectives via multi-task learning @xcite . however, since the autoencoding and autoregressive objectives differ by nature, a simple unification cannot fully inherit the advantages of both frameworks. in this paper, we propose a pretraining framework named glm (general language model), based on autoregressive blank infilling. we randomly blank out continuous spans of tokens from the input text, following the idea of autoencoding, and train the model to sequentially reconstruct the spans, following the idea of autoregressive pretraining (see figure 1 ). while blanking filling has been used in t5 @xcite ) for text-to-text pretraining, we propose two improvements, namely span shuffling and 2d positional encoding. empirically, we show that with the same amount of parameters and computational cost, glm significantly outperforms bert on the superglue benchmark by a large margin of 4.6% -5.0% and outperforms roberta and bart when pretrained on a corpus of similar size (158gb). glm also significantly outperforms t5 on nlu and generation tasks with fewer parameters and data. inspired by pattern-exploiting training (pet) @xcite , we reformulate nlu tasks as manually-crafted cloze questions that mimic human language. different from the bertbased models used by pet, glm can naturally handle multi-token answers to the cloze question via autoregressive blank filling. furthermore, we show that by varying the number and lengths of missing spans, the autoregressive blank filling objective can pretrain language models for conditional and unconditional generation. through multi-task learning of different pretraining objectives, a single glm can excel in both nlu and (conditional and unconditional) text generation. empirically, compared with standalone baselines, glm with multi-task pretraining achieves improvements in nlu, conditional text generation, and language modeling tasks altogether by sharing the parameters. ## glm pretraining framework we propose a general pretraining framework glm based on a novel autoregressive blank infilling objective. glm formulates nlu tasks as cloze questions that contain task descriptions, which can be answered by autoregressive generation. ## experiments we now describe our pretraining setup and the evaluation of downstream tasks. ## conclusions glm is a general pretraining framework for natural language understanding and generation. we show that the nlu tasks can be formulated as conditional generation tasks, and therefore solvable by autoregressive models. glm unifies the pretraining objectives for different tasks as autoregressive blank infilling, with mixed attention masks and the novel 2d position encodings. empirically we show that glm outperforms previous methods for nlu tasks and can effectively share parameters for different tasks. towards story-like visual explanations by watching movies and reading books. in iccv 2015, pages 19-27.