BERTAC : Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks (2021)

Abstract: Transformer-based language models (TLMs), such as BERT, ALBERT and GPT-3, have shown strong performance in a wide range of NLP tasks and currently dominate the field of NLP. However, many researchers wonder whether these models can maintain their dominance forever. Of course, we do not have answers now, but, as an attempt to find better neural architectures and training schemes, we pretrain a simple CNN using a GAN-style learning scheme and Wikipedia data, and then integrate it with standard TLMs. We show that on the GLUE tasks, the combination of our pretrained CNN with ALBERT outperforms the original ALBERT and achieves a similar performance to that of SOTA. Furthermore, on open-domain QA (Quasar-T and SearchQA), the combination of the CNN with ALBERT or RoBERTa achieved stronger performance than SOTA and the original TLMs. We hope that this work provides a hint for developing a novel strong network architecture along with its training scheme. Our source code and models are available at https://github.com/nict-wisdom/bertac .

## introduction transformer-based language models (tlms) such as bert @xcite , albert @xcite , and @xcite have shown that large-scale self-supervised pretraining leads to strong performance on various nlp tasks. many researchers have used tlms for various downstream tasks, possibly as subcomponents of their methods, and/or they have focused on scaling up tlms or improving their pretraining schemes. as a result, other architectures like recurrent neural networks (rnn) @xcite and convolutional neural networks (cnn) @xcite are fading away. in this work, we propose a method for improving tlms by integrating a simple conventional cnn to them. we pretrained this cnn on wikipedia using a generative adversarial network (gan) style training scheme @xcite , and then combined it with tlms. @xcite similarly used gan-style training to improve a qa model using a cnn, but their training scheme was applicable only to qaspecific datasets. on the other hand, similarly to tlm, our proposed method for training the cnn is independent of specific tasks. we show that the combination of this cnn with tlms can achieve higher performance than that of the original tlms on publicly available datasets for several distinct tasks. we hope that this gives an insight into how to develop novel strong network architectures and training schemes. we call our combination of a tlm and a cnn bertac (bert-style tlm with an adversarially pretrained convolutional neural network). its architecture is illustrated in fig. 1 . we do not impose any particular restriction on the tlm in bertac, so any tlm, albert @xcite !"#@xmath0+0 #10+')"), 2%+(340*%025(.+4 6+(3-+')"),-%+7%+#+')()"*'0 8+'+%()*%0" 9(.+-+')"),-%+7%+#+')()"*'0 8+'+%()*%0# "!@xmath1%+(30%+7%+#+')()"*'0 *50):+0+')"),0! #!%"#@xmath2%&'()'*+%!,-.,(/0(1" ![em] ,2-3+',4')562-!',)-,)1#()'1,0)'4-',(-+%*7" figure 2 : gan-style pretraining of cnns. the discriminator d takes either a real representation generated by r or a fake representation generated by f as its input and then it predicts whether the input is a real or fake representation. suvarnabhumi ::::::: airporte 1 is thailand's main international air hub. m1 [em] is thailand's main international air hub. e1 we pretrained the cnn using a gan-style training scheme in order to generate representations of sentences rather freely without the constraint of token embedding prediction in the masked language modeling used for tlms, as we explain later. for the training, we used masked sentences autogenerated from wikipedia. as in the masked language modeling, neither human intervention nor downstream task-specific hacking is required. as illustrated in fig. 2 , the ganstyle training requires three networks, namely, a discriminator d and two cnn-based generators r and f . once the training is done, we use the generator f as cnn in bertac. the training data consists of pairs of an entity mention and a sentence in which the entity mention is masked with a special token @xcite . for example, the entitymasked sentence m 1 in in the original gan, a generator learns to generate an artificial image from random noise so that the resulting artificial image is indistinguishable from given real images. by analogy, we used an entity-masked sentence as "random noise" and a masked entity as a "real image." in our gan-style training, we regard the vector representation of a masked entity given by generator r as a real representation of the entity (or the representation of the "real image" in the above analogy). on the other hand, we regard the representation of the masked sentence, generated by f , as a fake representation of the entity (or the representation of the "artificial image" generated from the "random noise" in the above analogy). this representation is deemed fake because the entity is masked in the masked sentence, and f does not know what the entity is exactly. during the training, f should try to deceive the discriminator d by mimicking the real representation and generating a fake representation that is indistinguishable from the real representation of the entity generated by r. on the other hand, r and d, as a team, try to avoid being mimicked by f and also to make the mimic problem harder for f . if everything goes well, once the training is over, f should be able to generate a fake representation of the entity that is similar to its real representation. an interesting point is that f 's output can be interpreted in two ways: it is a representation of a masked sentence because it is computed from the sentence, and at the same time it is a representation of the masked entity because it is indistinguishable from r's representation of the entity. this duality suggests that f 's output can be seen as a representation of the entire sentence. we exploit f as a cnn in bertac as follows: first, we use f to compute a representation of a masked version of the sentence originally given as input to a tlm. the entity mention to be masked is chosen by simple rules and, if the input consists of multiple sentences, we generate a representation of each (masked) input sentence and concatenate these together into a single one. then, this representation is integrated to the output of the tlm through multiple tier layers. our gan-style pretraining is conceptually similar to tlm pretraining with masked language modeling (predicting what a masked word in a sentence should be). however, it was designed to pretrain a model that is able to rather freely generate entity representations without strongly sticking to the prediction of token embeddings. our hypothesis is that such freely generated representations may be useful for improving the performance of downstream tasks. moreover, we assumed that using multiple text representations computed from different perspectives (i.e., predicting token embeddings and freely generating entity representations) would help to improve the performance of downstream tasks. in our experiments, we show that for the glue tasks @xcite ), bertac's average performance on the development set was 0.7% higher than that of albert, which was used as a subcomponent of bertac, leading to a performance on the test set comparable to that of sota (90.3% vs 90.8% (sota)). it also outperformed the sota method of open-domain qa @xcite on quasar-t @xcite and searchqa @xcite using either albert or roberta. we also compared our method with alternative models using a cnn pretrained in a self-supervised (non gan-style) manner to directly predict embeddings of the entity mentions. consequently, we confirmed that our method worked better: only the cnn trained by our gan-style pretraining gave significant performance improvement over base tlms. note that the computational overhead of bertac is reasonably small. it took 20 hours with 16 gpus to pretrain a single cnn model and 180 hours for the nine models tested with different parameter settings in this work (cf., 480 hours with 96 gpus for pretraining deberta @xcite , for example). moreover, once pretrained, the cnn models can be re-used for various down-stream tasks and combined with various tlms, including potentially future ones. as for the parameter number, bertac had just a 14% increase in parameters when albert-xxlarge was used as its base tlm (268 m parameters for bertac vs. 235 m for albert-xxlarge). we confirmed from these results that bertac could improve pretrained tlms with reasonably small computational overhead. the code and models of bertac are available at https://github.com/nict-wisdom/bertac . ## related work pretraining tlms with entity information: there have been attempts to explicitly learn entity representation from text corpora using tlms @xcite @xcite @xcite . our proposed method is a complementary alternative to these existing methods in the sense that entity representations are integrated into tlms via cnns and not directly produced by the tlms. fine-tuning tlms with external resources or other nns: yang et al. (2019a) and liu et al. (2020) have used knowledge graphs for augmenting tlms with entity representations during finetuning. unlike these approaches, bertac uses unstructured texts rather than clean structured knowledge, such as knowledge graphs, to adversarially train a cnn. other previous works have proposed combining cnns or rnns with bert for nlp tasks @xcite @xcite , but their use of cnns/rnns was task-specific, so their models were not directly applicable to other tasks. adversarial learning for improving tlms @xcite proposed a cnn-based answer representation generator for qa that can guess the vector representation of answers from given whytype questions and answer passages. the generator was trained in a gan-style manner using qa datasets. we took inspiration from their adversarial training scheme to train task-independent representation generators from unsupervised texts (i.e., wikipedia sentences in which an entity was masked in a cloze-test style). electra @xcite also employed an adversarial technique (not a gan) to pretrain two tlms: a generator was trained to perform masked language modeling and a discriminator was trained to distinguish tokens in the training data from tokens replaced by the generator. on downstream tasks, only the discriminator was finetuned. in bertac, the gan-style pretraining was applied only to the cnn, thus reducing the training cost. furthermore, the cnn can be combined easily with any available tlm, even potentially future ones, without having to re-do the pretraining. in this work, we show that bertac outperformed electra on the glue task. @xcite proposed a method that used an adversarial objective and an adversarial classifier for regularizing the fine-tuning process of tlms, inspired by adversarial learning for domain adaptation @xcite . our work uses a gan-style training scheme only for pretraining cnns, not for fine-tuning tlms. ## pretraining of cnns this section describes the training data and training algorithm for our cnn. ## bertac as illustrated in fig. 1 , bertac (bert-style tlm with an adversarially pretrained convolutional neural network) incorporates the representation provided by the adversarially pretrained cnn to the representation generated by a tlm. for the integration, we use several layers of tiers (transformers for integrating external representation) stacked on top of the tlm. ## experiments we tested our model on glue and on opendomain qa. in this section, we report the results.