Are Pretrained Convolutions Better than Pretrained Transformers? (2021)

Abstract: In the era of pre-trained language models, Transformers are the de facto choice of model architectures. While recent research has shown promise in entirely convolutional, or CNN, architectures, they have not been explored using the pre-train-fine-tune paradigm. In the context of language models, are convolutional models competitive to Transformers when pre-trained? This paper investigates this research question and presents several interesting findings. Across an extensive set of experiments on 8 datasets/tasks, we find that CNN-based pre-trained models are competitive and outperform their Transformer counterpart in certain scenarios, albeit with caveats. Overall, the findings outlined in this paper suggest that conflating pre-training and architectural advances is misguided and that both advances should be considered independently. We believe our research paves the way for a healthy amount of optimism in alternative architectures.

## introduction in the modern era of pre-training, there appears to be an unbreakable tie between transformer architectures @xcite and pre-trained language models. models such as bert @xcite , roberta @xcite , and t5 @xcite have all adopted transformers as their underlying architecture. as a matter of fact, there are barely any recent pre-trained models not based on transformers. while the contextual representation learning has a rich history @xcite ; dai and le, * google ai resident 2015; @xcite @xcite , modern pre-trained language modeling started with models like elmo @xcite and @xcite which are based on recurrent (e.g. lstm @xcite ) architectures. although they were successful, research using these architectures dwindled as transformers stole the hearts of the nlp community, having, possibly implicitly, been perceived as a unequivocal advancement over its predecessors. recent work demonstrates the promise of entirely convolution-based models @xcite and questions the necessity of self-attentive architectures like transformers. for example, in @xcite , the proposed convolutional seq2seq models outperform transformers on a series of canonical benchmarks such as machine translation and language modeling. from these findings emerge a rather natural line of questioning -should we consider pre-trained models beyond transformers? despite early success, the relevance of convolutional models in the era of pre-trained language models remains an open question. to the best of our knowledge, convolutional architectures have not yet been rigorously evaluated under the pretrain-fine-tune paradigm. this is the primary purpose of this work. concretely, this paper seeks to empirically validate whether pre-trained convolutions are competitive with pre-trained transformers across a range of tasks. the interaction between pre-training schemes and model architectures is an under-studied topic. are only transformers able to capitalize on the benefits of pre-training? if we use a different architectural inductive bias, would there also be a substantial gain unlocked by pre-training? are pretrained convolutions better in particular scenarios? this paper investigates these questions. there are a number of obvious benefits of convolution-based models. firstly, convolutions do not suffer from the quadratic memory complexity of self-attention -a problem significant enough that it spawned the creation of the entirely new category of "efficient" transformer architectures @xcite . secondly, convolutions operate locally and do not rely on positional encodings as an order signal to the model. that said, convolutions also come with a slew of downsides. for example, being unable to access global information means such models are unable to perform a form of cross-attention across multiple sequences. we dive into the details of this more in subsequent sections. in this paper, we present a pre-trained convolutional sequence-to-sequence, or seq2seq, model. we train our convolutional model using span-based sequence-to-sequence denoising objectives similar to those employed in t5 @xcite . we evaluate a variety of convolutional variants (e.g., dilated, lightweight, dynamic @xcite , etc.) under both raw (no pre-training) and pre-train-finetune paradigms. our goal is to understand the true competitiveness of convolutional architectures in the era of pre-training. we show that pre-trained convolutions are competitive against pre-trained transformers via a set of experiments on a potpourri of nlp tasks, like toxicity detection, sentiment classification, news classification, query understanding and semantic parsing/compositional generalization @xcite . moreover, we find that pretrained convolutions can outperform, in terms of model quality and training speed, state-of-the-art pre-trained transformers @xcite in certain scenarios. however, to provide a balanced perspective, we also describe scenarios where pretrained convolutions do not perform well and may be deemed unsuitable. contributions overall, the main contributions of this paper can be summarized as follows: ## related work pre-training on a large corpus has become the primary method of learning universal language representations to solve different downstream nlp tasks. the first generation of pre-trained models aimed at learning embedding for words, like skip-gram @xcite and glove @xcite , and quickly developed to learning contextualized representation for words, like @xcite , gpt @xcite , and bert @xcite . this, however, is not the only axis in which pre-trained models have evolved. different objective functions and various tasks, both supervised and unsupervised, have been explored for pre-training. for instance, cove @xcite uses machine translation as the pre-training task, @xcite and gpt @xcite use language modeling objectives, bert @xcite uses masked language modeling, t5 @xcite and mass @xcite use seq2seq masked language modeling, and xlnet @xcite utilizes permuted language modeling. in addition to this, bart @xcite uses a denoising autoencoder setup during pre-training, where the model takes a partially corrupted input and is trained to recover the original, undistorted input. some models use a contrastive learning setup during pertaining, like replaced token detection, used by electra @xcite , and sentence order prediction, used by albert @xcite and structbert @xcite . another axis where pre-trained models in nlp explored different ideas is model architecture. @xcite and @xcite used lstms as the base model. later, transformers @xcite became the de facto architecture of pre-trained nlp models. bert @xcite , xlnet @xcite and roberta @xcite use the transformer encoder, while gpt @xcite , gpt-2 @xcite , and gpt-3 (brown et al., 2020) use the transformer decoder as the backbone. some pre-trained models are also are based on the encoder-decoder transformer architecture, like t5 @xcite , mass @xcite , and bart @xcite . in this paper, we investigate another model architecture variation by studying the power of convolutional neural network as the backbone of pre-trained models for nlp. convolutions have always been an interesting choice for sequence modeling and nlp applications @xcite @xcite . convolutions are lightweight and fast and have many interesting use-cases, notably for lightweight classification. in the era when lstms were the workhorses of nlp applications, convolutions were positioned nicely on the pareto frontier of the compute-performance curve. they are fast and lightweight, and unlike transformers, they do not suffer from quadratic complexity. our work is also well-aligned with the resurgence of interest in convolutions where @xcite showed that convolutions can outperform self-attention on several sequence transduction tasks. moreover, the necessity of the self-attention inductive bias in transformers have been also a subject of recent interest. synthesizer models @xcite showed that transformers can still do pretty well without token-token dot product self-attention and a random attention matrix can perform competitively on certain tasks. ## pre-trained convolution models this section describes the pre-trained convolution model. for most of our experiments, we adopt depthwise separable convolutions @xcite @xcite which have shown to be fast and efficient variants of the standard convolution. ## research questions and discussion before we delve into our experiments, we establish a set of research questions and agenda we hope this work aims to bring clarity to. ## experiments and analysis this section presents our analysis and results. ## discussion and analysis this section expands on the results via a detailed analysis and discussion. we discuss the pros/cons of pretrained convolutions, the impact of pretraining on performance and also recommendations to the broader community. in our experimental section, we observed the potential upsides of convolutional models over wellestablished pre-trained transformers and observe that we are able to get quality improvements in certain cases. however, it might be good to further understand the drawbacks of convolutions. one obvious weakness of pre-trained convolutions are their lack of cross-attention inductive bias that comes for free with self-attention in the transformer encoder. for this reason, it is not a that said, we leave it to the practitioner to decide whether the cross-attention inductive bias is actually important for the problem at hand. we also like to emphasize that the pattern of concatenating sentence pairs is not necessary practical when scaling up since this requires inference on every permutation of sentence pairs. for this reason, dual encoder setups that do fast embedding space look-ups are more practical and feasible in practice @xcite . given the strong performance of convolutions in a series of encoding tasks, we can expect pre-trained convolutions to do well in a dual encoder setup. ## conclusion in this paper, we conducted an extensive study of the viability and feasibility of pre-trained convolu-tions. our experimental results show that convolutions can outperform transformers in both pretrain and non-pre-trained setups. our extensive experiments across 8 datasets spanning a diverse range of tasks, show that convolutions are able to benefit from pre-training to the same (or sometimes greater) extent than transformers. while pre-trained transformers are the de-facto choice of architecture, our results show that they might not be the best in certain scenarios. additionally, we discussed the caveats, trade-offs pertaining with runtime, scalability, number of flops and model quality. finally, we discussed the situations or data types that convolutions are not well equipped to handle and make an empirically informed recommendation for practitioners.