On the Role of Bidirectionality in Language Model Pre-Training (2022)

Abstract: Prior work on language model pre-training has explored different architectures and learning objectives, but differences in data, hyperparameters and evaluation make a principled comparison difficult. In this work, we focus on bidirectionality as a key factor that differentiates existing approaches, and present a comprehensive study of its role in next token prediction, text infilling, zero-shot priming and fine-tuning. We propose a new framework that generalizes prior approaches, including fully unidirectional models like GPT, fully bidirectional models like BERT, and hybrid models like CM3 and prefix LM. Our framework distinguishes between two notions of bidirectionality (bidirectional context and bidirectional attention) and allows us to control each of them separately. We find that the optimal configuration is largely application-dependent (e.g., bidirectional attention is beneficial for fine-tuning and infilling, but harmful for next token prediction and zero-shot priming). We train models with up to 6.7B parameters, and find differences to remain consistent at scale. While prior work on scaling has focused on left-to-right autoregressive models, our results suggest that this approach comes with some trade-offs, and it might be worthwhile to develop very large bidirectional models.

## introduction nlp has undergone a paradigm shift driven by pretrained models like gpt and bert @xcite . these models are trained on unlabeled corpora in a self-supervised fashion, and can be effectively adapted to downstream tasks either through conventional fine-tuning @xcite or few-shot priming @xcite . despite their widespread use, there is not a universal formula to pre-train language models: prior work has explored different architectures and learning objectives, often focusing on different applications. for instance, bert @xcite pre-trained masked language models for nlu finetuning, bart @xcite pre-trained seq2seq models on denoising for both nlu and generation tasks, and @xcite scaled autoregressive language models focusing on zero-and few-shot priming. however, such models differ on many factors in addition to their architecture and learning objective (e.g., the pretraining data, compute and hyperparameters), making a principled comparison difficult. motivated by that, @xcite presented a comprehensive study exploring various pre-training objective and architecture variants in a controlled environment. however, they conducted most of the exploration using small models, while recent work has found that different approaches behave differently at scale @xcite , and their evaluation was limited to fine-tuning. in this paper, we focus on a key factor that differentiates many pre-training approachesbidirectionality-and study it in different settings as a function of scale. we propose a new framework that distinguishes between two notions of bidirectionality: bidirectional context (whether the prediction of a given token is conditioned on both the right and the left context, or only on either of them), and bidirectional attention (whether there are blocks of tokens that can all attend to each other, contrasting with triangular attention masking). our framework offers knobs to control each of them separately, generalizing several previous approaches (e.g. bert leverages both types of bidirectionality, gpt does not use any, prefix lms only leverage bidirectional attention, and cm3 only leverages bidirectional context). we train a total of 24 models covering 6 variants of our framework and 5 model sizes with up to 6.7b parameters, and evaluate them on 4 settings: language modeling, text infilling, zero-shot priming, and fine-tuning. we find that bidirectional attention and context have a different impact depending @xmath0 starting from the original document, we mask n mask tokens at random and move them-along with their positional embeddings-to the end. we define our loss over the last n predict tokens, predicting the masked token for the last n mask , and the next token for the remaining n predictn mask . we use bidirectional attention over the first n bidir tokens, and unidirectional attention over the rest. refer to appendix a for a more detailed description. name n mask n bidir n predict related models nxtuni 0 0 n gpt (radford et al., 2018, 2019; brown et al., 2020) nxtpre † 0 u(1, n) nn bidir prefix lm (raffel et al., 2020; wu et al., 2021) mskuni b(n, 0.15) 0 n mask -mskbi b(n, 0.15) n n mask bert (devlin et al., 2019), roberta (liu et al., 2019) hybuni † b(n, 0.15) 0 n cm3 (aghajanyan et al., 2022) hybpre † b(n, 0.15) u(1, n) max(nn bidir , n mask ) - ## proposed framework as illustrated in figure 1 , we propose a generalized framework to pre-train transformer models on unlabeled corpora. our framework supports both unidirectional and bidirectional attention, as well as next token prediction and single-token infilling, using the following parameters to balance them: as such, our framework allows us to vary the two notions of bidirectionality discussed above: n bidir controls the weight of bidirectional attention, whereas n mask and n predict control the weight of bidirectional context. in addition, larger values of n predict result in more tokens of supervision. our implementation is based in fairseq @xcite . we apply the procedure described in §2 to each document separately, and combine multiple documents into a single sequence to speed up train-ing. 3 as such, we move the masked tokens to the end of each document (as opposed to the end of the whole sequence), and apply a bidirectional attention prefix to each document rather than the sequence as a whole. foot_3 ## results we visualize our main results in figure 2 , and discuss each setting in more detail next. ## related work while it was once common to use random initialization for supervised learning, a series of works showed substantial improvements from pretraining autoregressive models on next token prediction @xcite @xcite . the pre-train/fine-tune paradigm was further popularized by bert @xcite and its derivatives like roberta @xcite , which obtained further gains from pre-training bidirectional encoders on masked language modeling. subsequent work explored masking spans instead of individual tokens, using either bidirectional encoderonly models @xcite or encoder-decoder models @xcite . more recently, there has been a reborn interest on scaling left-to-right autoregressive language models with a focus on few-shot priming @xcite @xcite @xcite @xcite . while unidirectional and bidirectional models have largely been developed as separate strains of work serving a different purpose, there have also been some attempts to combine the best of both worlds. xlnet @xcite pre-trained autoregressive models over all permutations of the factorization order, enabling the model to use bidirectional context with strong results on fine-tuning. similarly, cm3 @xcite trained left-to-right autoregressive models, masking some spans that are predicted at the end of the sequence. ernie 3.0 @xcite proposed a modular architecture, combining a shared unidirectional module with either another unidirectional module for nlg or a bidirectional module for nlu. finally, @xcite and @xcite explored splitting documents in two halves and predicting the second one conditioned on the first one, using unidirectional attention for the former and bidirectional attention for the latter. despite the large body of work on language @xcite conduct a comprehensive study with a focus on zero-shot learning and multi-task fine-tuning. in contrast, we focus on the specific role of bidirectionality, and compare models of different sizes. ## conclusions in this work, we study the role of bidirectionality in language model pre-training through a new framework that generalizes previous approaches. our main findings are as follows: all in all, our results show that there is not a single configuration that is optimal for all use cases, and this remains generally consistent within the scale range explored in this work. while prior work on scaling has focused on left-to-right autoregressive models, this suggests that there might be other objectives and architectures that are better suited for other applications like fine-tuning. given the cost of pre-training several models, we would like to explore modular @xcite or adaptation @xcite approaches in the future, where one would either have a single model with modular components specialized for different use cases, or efficiently adapt an existing model by changing the parameters in our framework instead of training several models from scratch.