SCRIPT : Self-Critic P re T raining of Transformers (2021)

Abstract: We introduce Self-CRItic Pretraining Transformers (SCRIPT) for representation learning of text. The popular masked language modeling (MLM) pretraining methods like BERT replace some tokens with [MASK] and an encoder is trained to recover them, while ELECTRA trains a discriminator to detect replaced tokens proposed by a generator. In contrast, we train a language model as in MLM and further derive a discriminator or critic on top of the encoder without using any additional parameters. That is, the model itself is a critic. SCRIPT combines MLM training and discriminative training for learning rich representations and compute- and sample-efficiency. We demonstrate improved sample-efficiency in pretraining and enhanced representations evidenced by improved downstream task performance on GLUE and SQuAD over strong baselines. Also, the self-critic scores can be directly used as pseudo-log-likelihood for efficient scoring.

## introduction in natural language processing, the landscape of unsupervised learning methods is dominated by masked language modeling (mlm) for bidirectional encoders, such as bert @xcite @xcite @xcite @xcite , and causal masking for uni-directional autoregressive decoders @xcite @xcite @xcite such as gpt. in mlm an encoder is pretrained on a generic corpus of text with the hope of learning universal contextual embeddings, which, then, are fine-tuned on a specific down-stream task. whereas recent developments in causal masking aim to learn a large-scale model once and define the down-stream task as an auto-regressive manner in the form of few-shot evaluation @xcite . in practice, while an universal autoregressive neural backbone model without the need for fine-tuning such as gpt-3 is desirable, the computational complexity at inference time remains an open problem. while the two-stage approach of mlm of smaller models is computationally convenient, the pretraining still incurs a substantial computational cost. hence, in this work, we focus on learning contextual bi-directional representations with the goal of improving upon sample efficiency. in mlm, the input sequence of tokens is perturbed by randomly masking out a small subset of the identities of tokens @xcite or attention scores to those tokens @xcite . then, the generative model is learned as a denoising auto-encoder @xcite which recovers the masked out tokens. while the learned contextual representations achieve remarkable performance on down-stream tasks, the pretraining requires substantial compute. this is mainly due to learning from gradients from the restricted subset of tokens @xcite . in electra @xcite , the input sequence is perturbed by replacing a subset of tokens by sampled tokens drawn from an auxiliary generator model in the form of a bi-directional encoder, which itself is learned by mlm. then, the discriminative model is learned by a binary classification task which detects whether a token is unperturbed or has been replaced. this approach enjoys remarkable sample efficiency, which, we believe, stems primarily from reducing the complexity of the classification task from masked token prediction over a large set of classes (i.e., a typical vocabulary size of 30, 522 classes) to replaced token detection (i.e., 2 classes). despite it being less efficient, mlm training guides the model to learn rich representations. electra uses mlm only in learning the auxiliary generator which is discarded after pretraining. we propose to combine mlm and discriminative figure 1 : an overview of script. we combine mlm and discriminative training in a single transformer encoder, exploiting the rich representations extracted through mlm training and the compute-and sample-efficiency though discriminative training, resulting in a simple yet effective pretraining approach for representation learning. pretraining starts with replacing a small portion of tokens (e.g., 15%) in a text sequence x x x with [mask], yielding x contributions. (1) we propose a novel pretraining approach in which the model acts as a self-critic. (2) we demonstrated improved downstream task performance over state-of-the-art under computational constraints. (3) we show the selfcritic scores may serve as computationally efficient pseudo-log-likelihood for scoring tasks. ## method we propose a pretraining approach which combines masked token recovery and replaced token detection and does not introduce any additional parameters compared to a regular bert. in the following sections, we first introduce mlm training which is the same as that in bert, and then present selfcritic training. suppose x x @xmath0, ..., x t , ..., x t ] is a text sequence where x t is the tth token. in mlm training, a portion of tokens (e.g., 15%) are replaced with a special token @xcite . let x besides defining the log-likelihood for mlm training, p θ (x t |x x x) naturally provides a conditional distribution of x t with which we can construct a sampled sequence, x x @xmath1, ..., xt ], by replacing x t with xt , a token sampled from p θ (x t |x x x). x t is replaced only if it is masked in x for notational clarity. then the probability of sampling the vth token in the vocabulary for x t is, to apply discriminative training to the model, we derive a discriminator from the existing model and parameters. xt is considered as a positive token if @xmath2, while deemed a negative token if @xmath3. in the mlm training, the last layer defines a v -class classifier with the parameters w . we can augment w with an extra row for computing the score or logit for the negative token class, making it classify v + 1 classes. denote the augmented weight matrix as w + . then the classification logits are s + @xmath4. however, it is unnecessary to bring in new parameters and over-parameterization since subtracting an arbitrary function f (e t ) ∈ r from all the logits, s + tv -f (e t ) ∀@xmath5, ..., v + 1, does not change the softmax output. thus we fix the last row of w + to all zeros 0 0 0 ∈ r 1×d . then we have the logit for the tth token, then the probability of the tth token in x x x being a negative token is, the overall loss function of script combines mlm and discriminative training, l θ = l m lm (θ) + αl disc (θ), where α is an coefficient determining the strength of discriminative training. the learning of script involves two forward passes through a single model, one for mlm with x x x as input, one for discriminative training with x x x as input, and a single backward pass. figure 1 gives an overview of our model. ## experiments in the subsequent empirical evaluations, we shall address the following questions: (1) does the learning as self-critic lead to competitive down-stream task performance? (2) can we treat the self-critic scores as pseudo-log-likelihoods? (3) is the sample efficiency improved over state-of-the-art baselines? hence, we train and evaluate two script models "small" and "base" with an encoder of the 14m and 110m parameters, respectively. for a direct comparison, the models are trained on the open-webtext corpus @xcite with identical pre-processing and optimization procedures as in @xcite and @xcite . we refer to the appendix for details. ## conclusion this work presents script for representation learning. it is a transformer encoder like bert. in pretraining, it recovers masked tokens, proposes negative samples, and acts as a self-critic, discriminating between sampled and original tokens. the joint mlm and discriminative learning improves sample efficiency in pretraining and enhances representation learning, leading to improved performance over strong baselines on various downstream tasks. it also provides an efficient way for computing pseudo-log-likelihood for scoring tasks and achieves competitive performance.