H yper M ixer: An MLP -based Low Cost Alternative to Transformers (2023)

Abstract: Transformer-based architectures are the model of choice for natural language understanding, but they come at a significant cost, as they have quadratic complexity in the input length, require a lot of training data, and can be difficult to tune. In the pursuit of lower costs, we investigate simple MLP-based architectures. We find that existing architectures such as MLPMixer, which achieves token mixing through a static MLP applied to each feature independently, are too detached from the inductive biases required for natural language understanding. In this paper, we propose a simple variant, HyperMixer, which forms the token mixing MLP dynamically using hypernetworks. Empirically, we demonstrate that our model performs better than alternative MLP-based models, and on par with Transformers. In contrast to Transformers, HyperMixer achieves these results at substantially lower costs in terms of processing time, training data, and hyperparameter tuning.

## introduction attention-based architectures, such as the transformer @xcite , have accelerated the progress in many natural language understanding tasks. part of their success is a result of a parallelizable training scheme over the input length. this improves training times and allows for larger volumes of data which makes these models amenable to pretraining @xcite . therefore, many current state-of-the-art models are fine-tuned extensions of large pretrained transformers @xcite . however, these models come at a significant computational cost. they require considerable resources for pretraining and fine-tuning, which induces high energy consumption @xcite and limits access to research @xcite . subsequently, @xcite argue the need for "green ai". they propose a cost evaluation of a result r as following: to achieve a cost reduction, this paper proposes a simpler alternative to transformers. we take inspiration from the computer vision community, which has recently seen a surge of research on multi-layer perceptrons (mlps). most prominently, mlpmixer @xcite , which is a simple architecture based on two mlps: one for token mixing and one for feature mixing. however, the token mixing mlp learns a fixed-size set of position-specific mappings, arguably making mlpmixer's architecture too detached from the inductive biases needed for natural language understanding, in contrast to transformers @xcite . in this paper, we propose a simple variant, hy-permixer (figure 1 ), which creates a token mixing mlp dynamically using hypernetworks @xcite . this variant is more appropriate, as it learns to generate a variable-size set of mappings in a position-invariant way, similar to the attention mechanism in transformers @xcite . in contrast to transformer's quadratic complexity, hypermixer's complexity is linear in the input length. this makes it a competitive alternative for training on longer inputs. empirically, we demonstrate that hypermixer works substantially better on natural language understanding tasks than the original mlpmixer and related alternatives. in comparison to transformers, hypermixer achieves competitive or improved results at a substantially lower cost cost(r) ∝ e • d • h: improved inference speeds (e), especially for long inputs; favorable performance in the token mixing ## related work research on all-mlp models like mlpmixer @xcite is widespread in the computer vision community @xcite @xcite , among many others). however, they lack some desirable inductive biases for nlp, which we discuss in length in appendix a.2. specifically, in contrast to hypermixer, none of the previously proposed methods simultaneously provide i) position invariance, which is important for generalization, ii) adaptive size for variable-length inputs, iii) a global receptive field, which allows interactions to not be limited to small token neighborhoods, iv) learnabilty allowing for universal applicablility to various tasks, and v) dynamicity, which means that token mixing is a function of the input. consequently, only a few works have used mlp-based models as their backbone in nlp tasks. gmlp @xcite serves as one of our baselines and pnlp-mixer @xcite employs standard mlpmixer on top of a novel token embedding method. apart from all-mlp models, there is an abundance of research on efficient alternatives to standard attention layers @xcite . while they don't qualify as all-mlp models, they have close connections to our work (see appendix e) and aim at lowering the cost of ai, albeit it on fewer dimensions than our work (appendix a.1). we employ fnet (lee-thorp et al., 2021) and linear transformers (katharopoulos et al., 2020) as representatives of these as a baseline. ## experiments our experiments are designed to test the following three hypotheses. h1 (section 4.3): since hy-permixer reflects more inductive biases that are adequate for nlp, our hypothesis is that hyper-mixer performs better at nlp tasks than mlp-mixer and similar mlp-based alternatives, specifically at those tasks that require to model the interactions between tokens. h2: since hypermixer has similar inductive biases as transformers but is considerably simpler conceptually and in terms of computational complexity, it can be seen as a low cost alternative to transformers, reducing the cost in terms of single example processing time (section 4.4), required dataset size (section 4.5), and hyperparameter tuning (section 4.6). h3 (section 4.7): due to its inductive biases mirroring those of transformers, hypermixer also learns similar patterns as the attention mechanism. ## discussion in the following, we first discuss the merits of our proposed model, which are the core contributions of our paper. we then discuss the scope of our analysis. ## conclusion while large pretrained transformer language models have led to impressive progress, they require so much resources that many research labs are excluded from participation, leading to calls for green ai. we have proposed an mlp-based method, hypermixer, that, in contrast to previous mlp-based methods, is equipped with the same inductive biases that made transformers so successful for natural language understanding. while it performs on par with transformers, it incurs substantially lower cost in terms of processing time, training data, and hyperparameter tuning. hence, we believe our study demonstrates the merits of mlp-based models for natural language understanding as an alternative to attention-based models, and we hope that the community pursues this direction further. avenues for future work include large-scale pretraining, evaluation on a wider range of tasks and domains, and the model's adaptation to text generation.