Reservoir Transformers (2021)

Abstract: We demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. Inspired by old and well-established ideas in machine learning, we explore a variety of non-linear “reservoir” layers interspersed with regular transformer layers, and show improvements in wall-clock compute time until convergence, as well as overall performance, on various machine translation and (masked) language modelling tasks.

## introduction transformers @xcite have dominated natural language processing (nlp) in recent years, from large scale machine translation @xcite to pre-trained (masked) language modeling @xcite , and are becoming more popular in other fields as well, from reinforcement learning @xcite to speech recognition @xcite and computer vision @xcite . their success is enabled in part by ever increasing computational demands, which has naturally led to an increased interest in improving their efficiency. scalability gains in transformers could facilitate bigger, deeper networks with longer contexts @xcite @xcite @xcite . conversely, improved efficiency could reduce environmental costs @xcite and hopefully help democratize the technology. in this work, we explore a simple question: if some layers of the transformer are kept frozeni.e., never updated after random initializationcan we match the performance of fully learned transformers, while being more efficient? surprisingly, the answer is resoundingly yes; and what is more, we find that freezing layers may actually improve performance. beyond desirable efficiency gains, random layers are interesting for several additional reasons. fixed randomly initialized networks @xcite converge to gaussian processes in the limit of infinite width @xcite , have intriguing interpretations in metric learning @xcite , and have been shown to provide excellent "priors" either for subsequent learning @xcite or pruning @xcite . fixed layers allow for efficient low-cost hardware implementations @xcite and can be characterized using only a random number generator and its seed. this could facilitate distributed training and enables highly efficient deployment to edge devices, since it only requires transmission of a single number. the strong performance of networks with fixed layers also sheds new light on the inner workings of bert @xcite , and layer-wise interpretations of such models @xcite . it appears that "not all layers are created equal" @xcite is true to such an extent that some layers can simply remain random and fixed. random projections have a long history in machine learning. by cover's theorem @xcite , any high-dimensional non-linear transformation is more likely to be linearly separable than its lower-or-equal-dimensional input space. by @xcite , random projections distort euclidean distances very little under mild assumptions, which is useful e.g. for dimensionality reduction and random indexing @xcite . fixed random layers in neural networks pre-date deep learning by far @xcite . indeed, random kernel methods have long been influential in machine learning @xcite . one way to think of such layers is as "reservoirs" @xcite , where a highly non-linear high-dimensional black box representation is provided to a lightweight "readout" network, as in echo state networks @xcite and liquid state machines @xcite . the benefit of such an approach is that the reservoir has fixed parameters and is computationally efficient, as it can be pre-computed and does not (necessarily) require backpropagation. in nlp, @xcite showed that random sentence encoders present a strong baseline for text classification, with subsequent work showing applications in a variety of tasks from summarization to machine translation @xcite @xcite . to our knowledge, this work is the first to examine this phenomenon in transformers, and the first to recursively alternate reservoirs with subsequent transformer layers acting as readout functions. we introduce "reservoir transformers", wherein fixed random reservoir layers are interspersed with regular updateable transformer layers. the goal of this work is to put our understanding of transformer models on a more solid footing by providing empirical evidence of their capabilities even when some of their parameters are fixed. our contributions are as follows: ## approach this paper is based on a very simple idea. neural networks are trained via backpropagation, which involves consecutive steps of matrix addition and multiplication, i.e., for some objective j, parameterization θ and learning rate η, with the gradient computed via the chain rule, where l i is the i-th layer of the neural network and x is the input. let @xmath0, i.e., now, during every "backward pass", we compute the jacobian for parameters θ l at layer l, which are used to update the parameters of l, θ l t , as well as to compute the next layer's jacobian, thus back-propagating the gradients. in this work however, for some of the layers, we still backpropagate through them to compute gradients for earlier layers, but we never apply the parameter update. as a result, these layers stay fixed at their initialization, saving computational resources. ## evaluation we evaluate the proposed approach on a variety of well-known tasks in natural language processing, namely: machine translation, language modelling and masked language model pre-training. we set out to do this work with the main objective of examining any potential efficiency gains, i.e. the relationship between compute time and task performance. this is closely related to efforts in green ai, which are concerned with the trade-offs between compute, data, and performance @xcite . we propose to measure this trade-off via the area under the convergence curve (aucc): similarly to how the area under the receiver operating characteristic (bradley, 1997, auc-roc) measures a classifier's performance independent of the classification threshold, aucc measures a model's performance independent of the specific compute bud-get. specifically, aucc is computed as follows: one potential downside of this approach is that the aucc metric could lead to higher scores for a model that converges quickly but to ultimately worse performance, if measured in a small window. this can be solved by making sure that t is set sufficiently high. we include the raw validation curves in the appendix to demonstrate that the chosen window sizes are sufficient and the results are not a influenced by this limitation. in addition, we report the number of trainable parameters and the wall-clock training time until maximum performance (plus 95% and 99% convergence results in the appendix). finally, we show test set generalization in each experiment. overall, this gives us a wide set of axes along which to examine models. ## experiments in what follows, we first show our main result, on a variety of tasks: reservoir transformers mostly have better aucc metrics; less training time per epoch; less convergence time until the best validation performance is achieved; and even improved test set generalization metrics. as a strong baseline method, we compare to layerdrop @xcite . layerdrop can also be seen as a method that dynamically bypasses parts of the computation during transformer training in an attempt to improve efficiency, and making it a strong comparison to examine our methods. then, we examine whether we can minimize the expectation over the gradients of upstream layers in the network such that we do not at all have to pass gradients through the reservoir layers, skipping their backward pass. ## related work recent work has shown that modern nlp models are able to function with different numbers of layers for different examples @xcite @xcite ; that different layers specialize for different purposes @xcite ; that layers can be compressed @xcite @xcite ; and, that layers can be reordered @xcite . there is a growing body of work in efficient self-attention networks @xcite , such as linear attention @xcite , on how to process long context information @xcite and on approximations to make transformers more scalable @xcite . bigbird @xcite provides random keys as additional inputs to its attention mechanism. locality sensitive hashing (lsh) as employed e.g. in reformer @xcite utilizes a fixed random projection. random feature attention @xcite uses random fea-ture methods to approximate the softmax function. performer @xcite computes the transformer's multi-head attention weights as a fixed orthogonal random projection. closely related to this work, @xcite showed that randomized alignment matrices in their "synthesizer" architecture are sufficient for many nlp tasks. while these works focus on random attention, we show that entire layers can be random and fixed. we also show that entire layers can be replaced by fixed random projections that do not have any attention whatsoever. beyond transformers, random features have been extensively explored. examples of this include freezeout @xcite , deep reservoir computing networks @xcite , as well as applications in domains as varied as text classification @xcite @xcite or music classification @xcite . it is well known that randomly initialized networks can display impressive performance on their own @xcite @xcite , which underlies, for example, the recently popularized lottery ticket hypothesis @xcite . we know that learning deep overparameterized networks appears to help in general @xcite . our method constitutes a way to add both depth and parameters to transformer networks without much computational cost. ## conclusion this work demonstrated that state-of-the-art transformer architectures can be trained without updating all of the layers. this complements a long history in machine learning of harnessing the power of random features. we use the "area under the convergence curve" (aucc) metric to demonstrate that on a variety of tasks, and in a variety of settings, "reservoir transformers" achieve better performance-efficiency trade-offs. we show that such reservoir transformers show better convergence rates and test-set generalization. we demonstrated that the backward pass can be skipped altogether, opening up exciting vanues for future research. future work includes further investigating hybrid networks and backskipping strategies, as well as utilizing pruning.