p NLP -Mixer: an Efficient all- MLP Architecture for Language (2023)

Abstract: Large pre-trained language models based on transformer architectureƒhave drastically changed the natural language processing (NLP) landscape. However, deploying those models for on-device applications in constrained devices such as smart watches is completely impractical due to their size and inference cost. As an alternative to transformer-based architectures, recent work on efficient NLP has shown that weight-efficient models can attain competitive performance for simple tasks, such as slot filling and intent classification, with model sizes in the order of the megabyte. This work introduces the pNLP-Mixer architecture, an embedding-free MLP-Mixer model for on-device NLP that achieves high weight-efficiency thanks to a novel projection layer. We evaluate a pNLP-Mixer model of only one megabyte in size on two multi-lingual semantic parsing datasets, MTOP and multiATIS. Our quantized model achieves 99.4% and 97.8% the performance of mBERT on MTOP and multiATIS, while using 170x less parameters. Our model consistently beats the state-of-the-art of tiny models (pQRNN), which is twice as large, by a margin up to 7.8% on MTOP.

## introduction large language models based on transformer architecture have been fueling the latest successes in natural language processing (nlp). nowadays, fine-tuning pre-trained models represents the defacto framework to tackle diverse nlp tasks, even those with limited amounts of annotations. while the merit of large pre-trained language models is undeniable, using models of several gigabytes and billions of parameters is not always practical or even possible due to computational and memory requirements. in addition, there are many simple and yet important tasks, such as slot filling * work done during a research stay at ibm research. in home assistants, which do not require the complex linguistic knowledge encoded by large pretrained models and for which smaller models may reach competitive performance at a much lower cost. reducing model sizes to the order of the megabyte is a necessity for resource constrained devices, such as smart watches, and in general it is attractive for edge use cases as i) updating models at the edge requires pushing updates to potentially millions of devices, and ii) multiple models solving different tasks can be deployed even on embedded devices with limited memory capacity. transformer-based architectures are not suitable to downscale to such ultra small model sizes, mostly due to the space required to store embedding tables @xcite . projection-based models @xcite have shown that the dense representations learned as part of the training process and stored in the embedding tables can be replaced by non-trainable representations computed on-thefly over the text, hence the name embedding-free. in this work, we introduce the pnlp-mixer, a novel embedding-free architecture for ultrasmall nlp models targeting on-device applications. our architecture relies on a novel projection layer which creates text representations for individual tokens by combining the minhash fingerprints @xcite corresponding to each subword unit. the projected features are given as input to a mlp-mixer @xcite , which grants our model architecture linear scalability in the sequence length and seamless hardware acceleration. to the best of our knowledge, this is the first work combining subword-unit tokenization and minhash fingerprints in projection networks. our evaluation on two semantic parsing datasets representative of on-device applications, mtop and multiatis, showcases that the pnlp-mixer beats the current state-of-the-art for ultra-small models, pqrnn @xcite , by up to 7.8% on sequence tagging tasks. on mtop, a pnlp-mixer model with only one million parameters achieves 99.4% of the performance of mbert, which has 170x more parameters. ## related work since the introduction of transformer-based language models such as bert @xcite , model sizes have been increasing at unprecedented pace @xcite @xcite . using current large language models for on-device applications is simply not feasible due to the size and computational requirements, especially in resource constrained devices such as smart watches. transformer-based models optimized for smartphone use cases, such as distilbert @xcite , tinybert @xcite , and mobilebert @xcite have shown that by combining knowledge distillation @xcite and quantization @xcite , one can achieve model sizes in the order of tens to hundreds of megabytes in size. embedded devices, such as wearables, require instead model sizes in the order of the megabyte, a target that is very challenging to achieve with transformerbased architecture, mostly because of the size of the embedding tables @xcite . embedding-free model architectures have been introduced to completely eliminate the dependency on large embedding tables from models. instead of learning embeddings at training time, text representation are computed on-the-fly using solely the surface forms of the tokens by means of localitysensitive hashing (lsh) @xcite techniques. this way tokens that are similar at the surface level have similar representations. the idea of replacing trainable parameters stored in embedding tables with lsh-based projections has been introduced in @xcite and @xcite . follow up research work on model architectures targeting ultra-small model sizes has resulted in several model architectures including sgnn @xcite , sgnn++ @xcite , prado @xcite , and pqrnn @xcite . our model architecture belongs to the same line of research, but introduces a linguistically informed projection layer which combines subword-unit tokenization @xcite with lsh principles. in our work, we evaluate and compare multiple lsh techniques, including simhash @xcite and minhash @xcite . in our projec- projection layer bottleneck pnlp-mixer non-trainable minhash the 𝐹 !"# minhash sun 𝐹 @xmath0 minhash hot 𝐹 "() mlp-mixer output @xmath1, by exploiting the associativity property of minhash, fingerprints of individual tokens can be efficiently computed from the fingerprints of their subword units. our model does not use attention mechanisms, as it feeds the representations to a mlp-mixer (tolstikhin et al., 2021) model. while using mlp only architectures is not new in the nlp landscape @xcite , this work is the first proposing an all-mlp architecture for ultra-small models. there are numerous studies around efficient transformer-based models @xcite and solutions to make them scale linearly with the sequence length. however, none of those work targets models of the size of the single megabyte. ## pnlp-mixer: a projection mlp-mixer the pnlp-mixer has been designed from the ground up as an efficient architecture suitable for both edge cases, memory and latency constrained, and as a backbone for complex nlp pipelines. figure 1 depicts the model architecture at high level. the pnlp-mixer falls into the category of projection-based models: instead of storing large embedding tables, like transformer-based models do, our model uses a projection layer which captures morphological knowledge from individual tokens using non-trainable hash functions. this projection layer can be seen as a feature extractor that produces a representation from input text. once the input features are computed, they are passed through a trainable linear layer called bottleneck layer. the output of the bottleneck layer is the input of a series of mlp blocks of a standard mlp-mixer architecture @xcite . there are several advantages of using an all-mlp architecture for language processing. in contrast to attention-based models, the mlp-mixer captures long-range dependencies without introducing a quadratic cost on the sequence length. further, by using only mlps, the model becomes simple to implement and has out-of-the-box hardware acceleration in devices ranging from mobile phones to server-grade inferencing accelerators. the main contribution of our work is to show that a simple model like the mlp-mixer represents a valid alternative to transformer-based models in nlp, even in setups where large embedding tables are replaced with projections computed on the fly. the key to achieve competitive performance with such small and computationally efficient models is to feed them with high-quality input features. ## experimental setup our architecture is designed as an alternative to transformer-based models for ultra-small models (i.e., one megabyte) targeting on-device applications. in the league of extremely small models, common evaluation datasets used by research and industry are not the same as the ones used for evaluating the generalizability of large pre-trained language models (e.g., glue @xcite ), but datasets for simpler tasks, that are more realistic applications for tiny models. thus, we align to prior works on tiny models for on-device applications @xcite ) that assess models on two multilingual semantic parsing datasets. mtop @xcite . it covers six languages, english, spanish, french, german, hindi, and thai. it was created by translating from english to the other languages. the train, dev, and test set for each language contain 10k, 1.5k, and 3k samples. we assess the models on slot parsing (named entity recognition) on 78 different slot labels. we report the exact match accuracy score, computed as the number of instances whose all tokens have been correctly labeled over the number of instances. multiatis @xcite . it is a multilingual version of the atis dataset @xcite , that contains queries related to air travel in nine languages: english, spanish, french, german, hindi, japanese, portuguese, turkish, and chinese. each language except hindi and turkish consists of 4, 488/490/893 samples for train, dev, and test sets; for hindi and turkish the splits are 1, 440/160/893 and 578/60/715. we evaluate on intent classification: determining the intent of a query from 18 labels, and we report the accuracy. ## model investigation we provide detailed insights on the impact of different projection layers and other architectural components as well as a comparison to alternative architectures. we perform the experiments on the english variant of the mtop and multiatis datasets. ## evaluation finally, we run a complete evaluation on the test sets of mtop and multiatis. we compare our pnlp-mixer with three very large models: xlu @xcite , which is a bi-lstm model with pretrained xlu embeddings, and two pretrained multilingual models: xlm-r @xcite and multilingual bert (mbert) @xcite . we also include two small models: pqrnn @xcite and a simple transformer using the same projection as pqrnn. pqrnn is an embedding-free quasi-rnn @xcite model that shares the same philosophy of our proposed pnlp-mixer: a small and task-specific model that learns directly from the text. for a fair comparison against pqrnn, we quantize our pnlp-mixer models and report the performance on the 8-bit version. finally, we include pqrnn distilled with mbert on mtop (the original study did not distill pqrnn on multiatis). the performance values of all the baselines are taken from @xcite . ## conclusion we introduce pnlp-mixer, the first embeddingfree model based on the mlp-mixer architecture. our main contribution is an efficient and yet effective projection layer which combines minhash fingerprints and subword-unit tokenization to create rich token representations. our evaluation shows that the pnlp-mixer beats the state-of-the-art of tiny nlp models, pqrrn, and offers sequence tagging performances that are up to 7.8% higher while using half of the parameters. the results are remarkable: a pnlp-mixer model of only 1 million parameters provides a performance of 99.4% and 97.8% on mtop and multiatis, respectively, compared to mbert which is a a pretrained model with 170x more parameters. our pnlp-mixer model is simple to implement and accelerate, and provides competitive performance even without pre-training or distillation. our work demonstrates the importance of projection methods and embedding-free architectures to advance the field of ultra-small models.