Pre-Training Transformers as Energy-Based Cloze Models (2020)

Abstract: We introduce Electric, an energy-based cloze model for representation learning over text. Like BERT, it is a conditional generative model of tokens given their contexts. However, Electric does not use masking or output a full distribution over tokens that could occur in a context. Instead, it assigns a scalar energy score to each input token indicating how likely it is given its context. We train Electric using an algorithm based on noise-contrastive estimation and elucidate how this learning objective is closely related to the recently proposed ELECTRA pre-training method. Electric performs well when transferred to downstream tasks and is particularly effective at producing likelihood scores for text: it re-ranks speech recognition n-best lists better than language models and much faster than masked language models. Furthermore, it offers a clearer and more principled view of what ELECTRA learns during pre-training.

## introduction the cloze task @xcite of predicting the identity of a token given its surrounding context has proven highly effective for representation learning over text. bert @xcite implements the cloze task by replacing input tokens with @xcite , but this approach incurs drawbacks in efficiency (only 15% of tokens are masked out at a time) and introduces a pre-train/fine-tune mismatch where bert sees @xcite tokens in training but not in fine-tuning. electra @xcite uses a different pre-training task that alleviates these disadvantages. instead of masking tokens, electra replaces some input tokens with fakes sampled from a small generator network. the pre-training task is then to distinguish the original vs. replaced tokens. while on the surface it appears quite different from bert, in this paper we elucidate a close connection between electra and cloze modeling. in particular, we develop a new way of implementing the cloze task using an energy-based model (ebm). then we show the resulting model, which we call electric, is closely related to electra, as well as being useful in its own right for some applications. 1 ebms learn an energy function that assigns low energy values to inputs in the data distribution and high energy values to other inputs. they are flexible because they do not have to compute normalized probabilities. for example, electric does not use masking or an output softmax, instead producing a scalar energy score for each token where a low energy indicates the token is likely given its context. unlike with bert, these likelihood scores can be computed simultaneously for all input tokens rather than only for a small masked-out subset. we propose a training algorithm for electric that efficiently approximates a loss based on noise-contrastive estimation @xcite . then we show that this training algorithm is closely related to electra; in fact, electra can be viewed as a variant of electric using negative sampling instead of noise-contrastive estimation. we evaluate electric on glue @xcite and squad @xcite , where electric substantially outperforms bert but slightly under-performs electra. however, electric is particularly useful in its ability to efficiently produce pseudo-likelihood scores @xcite ) for text: electric is better at re-ranking the outputs of a speech recognition system than gpt-2 @xcite and is much faster at re-ranking than bert because it scores all input tokens simultaneously rather than having to be run multiple times with different tokens masked out. in total, investigating electric leads to a more principled understanding of electra and our results ... suggest that ebms are a promising alternative to the standard generative models currently used for language representation learning. ## method bert and related pre-training methods @xcite @xcite train a large neural network to perform the cloze task. these models learn the probability p data (x t |x \t ) of a token x t occurring in the surrounding context typically the context is represented as the input sequence with x t replaced by a special [mask]placeholder token. this masked sequence is encoded into vector representations by a transformer network @xcite . then the representation at position t is passed into a softmax layer to produce a distribution over tokens p θ (x t |x \t ) for the position. ## experiments we train two electric models the same size as bert-base (110m parameters): one on wikipedia and bookscorpus @xcite for comparison with bert and one on openwebtextcorpus @xcite for comparison foot_1 with gpt-2. the noise distribution transformers t ltr and t rtl are 1/4 the hidden size of electric. we do no hyperparameter tuning, using the same hyperparameter values as electra. further details on training are in the appendix. ## related work language modeling @xcite @xcite and cloze modeling @xcite @xcite have proven to be effective pre-training tasks for nlp. unlike electric, these methods follow the standard recipe of estimating token probabilities with an output softmax and using maximumlikelihood training. energy-based models have been widely explored in machine learning @xcite . while many training methods involve sampling from the ebm using gradientbased mcmc @xcite or gibbs sampling @xcite , we considered these approaches too slow for pre-training because they require multiple passes through the model per sample. we instead use noise-contrastive estimation @xcite , which has widely been used in nlp for learning word vectors @xcite ) and text generation models @xcite . while ebms have previously been applied to leftto-right @xcite or globally normalized @xcite text generation, they have not previously been applied to cloze models or for pre-training nlp models. several papers have pointed out the connection between ebms and gans @xcite , which is similar to the electric/electra connection. ## conclusion we have developed an energy-based cloze model we call electric and designed an efficient training algorithm for electric based on noise-contrastive estimation. although electric can be derived solely from the cloze task, the resulting pre-training method is closely related to electra's ganlike pre-training algorithm. while slightly underperforming electra on downstream tasks, electric is useful for its ability to quickly produce pseudo-log-likelihood scores for text. furthermore, it offers a clearer and more principled view of the electra objective as a "negative sampling" version of cloze pre-training. as electra's @xcite , which adds some additional ideas from @xcite on top of the bert codebase, such as dynamic masking and removing the next-sentence prediction task. we use the weight sharing trick from electra, where the transformers producing the proposal distribution and the main transformer share token embeddings. we do not use whole-word or n-gram masking, although we believe it would improve results too. we did no hyperparameter tuning, directly using the hyperparameters from electra-base for electric and our baselines. these hyperparameters are slightly modified from the ones used in bert; for completeness, we show these values in