SHARCS : Efficient Transformers Through Routing with Dynamic Width Sub-networks (2023)

Abstract: We introduce SHARCS for adaptive inference that takes into account the hardness of input samples. SHARCS can train a router on any transformer network, enabling the model to direct different samples to sub-networks with varying widths. Our experiments demonstrate that: (1) SHARCS outperforms or complements existing per-sample adaptive inference methods across various classification tasks in terms of accuracy vs. FLOPs; (2) SHARCS generalizes across different architectures and can be even applied to compressed and efficient transformer encoders to further improve their efficiency; (3) SHARCS can provide a 2 times inference speed up at an insignificant drop in accuracy.

## introduction web-scale pretrained models, including large language models (llms), are widely used in various applications @xcite @xcite . however, their computational resource requirements can be problematic, especially in environments with limited resources. to address this issue, more efficient methods are needed, particularly those that can run on-device with efficient inference @xcite . several methods (e.g., knowledge distillation @xcite , pruning @xcite , and quantization @xcite ) have been proposed to improve the inference efficiency of transformer-based models. while these methods are promising, one drawback is that the resulting model is static. this raises concerns about whether the model is too complex for simple samples and not complex enough for more challenging ones. to tackle this problem, previous work have investigated sample-adaptive inference methods that use varying amount of compute to process different input samples @xcite . two predominant approaches exist in the field of sample adaptive inference: early-exiting and token dropping. the former incorporates internal classifiers into intermediate layers of the model. various techniques have been explored for early exiting, including using confidence scores or entropy of internal classifier predictions @xcite , using a module that predicts whether a layer should exit early @xcite , or implementing a patience-based method to adjust these internal predictions @xcite . the other category of sample-adaptive inference methods, token dropping, enhances efficiency by progressively decreasing sequence length as the forward pass proceeds layer by layer @xcite . in this paper, we propose sample hardness aware routing based on confidence scores (sharcs foot_0 ), which is a novel category within the efficient sample adaptive inference domain. our approach introduces training a light-weight router. the router dynamically assigns input samples, based on their hardness @xcite , to one of the sub-networks with varying widths. due to the lack of ground truth notion of hardness, we estimate sample hardness heuristically based on the network's prediction history during training for each sample. these estimates are then used as labels to train the router. we make the following contributions: ## method we introduce sharcs for adaptive inference that takes into account the hardness of input samples. our approach has three main steps: (1) obtaining labels that represent sample hardness ( §2.1); (2) training a router that can predict sample hardness ( §2.2); (3) adjusting the network's inference capacity according to the predicted hardness ( §2.3). ## experimental setup datasets. we evaluate sharcs on 8 classification tasks in the standard glue benchmark @xcite : mnli-m @xcite , qnli @xcite , qqp @xcite , sst-2 @xcite , mrpc @xcite , rte @xcite , cola @xcite , wnli @xcite . evaluation metrics. following previous work, we report the accuracy on the validation set, with an exception to cola for which we report matthews correlation. we measure the total flops on the validation set as it is invariant to the run time environment @xcite . training details. we train network with adamw optimizer @xcite . we choose the number of epochs in {5, 10, 15} and use learning rate in {2e -5, 5e -5} in our experiments. please see appendix c for more details. ## conclusion we presented sharcs as a new sample adaptive inference approach that can improve any network's inference efficiency. sharcs incorporated a lightweight router which is trained with a novel approach using the confidence of the network predictions during training. our experiments showed the superiority or complementary role of sharcs compared to other sample adaptive inference methods across various datasets and backbones.