Dynamic Transformers Provide a False Sense of Efficiency (2023)

Abstract: Despite much success in natural language processing (NLP), pre-trained language models typically lead to a high computational cost during inference. Multi-exit is a mainstream approach to address this issue by making a trade-off between efficiency and accuracy, where the saving of computation comes from an early exit. However, whether such saving from early-exiting is robust remains unknown. Motivated by this, we first show that directly adapting existing adversarial attack approaches targeting model accuracy cannot significantly reduce inference efficiency. To this end, we propose a simple yet effective attacking framework, SAME, a novel slowdown attack framework on multi-exit models, which is specially tailored to reduce the efficiency of the multi-exit models. By leveraging the multi-exit models’ design characteristics, we utilize all internal predictions to guide the adversarial sample generation instead of merely considering the final prediction. Experiments on the GLUE benchmark show that SAME can effectively diminish the efficiency gain of various multi-exit models by 80% on average, convincingly validating its effectiveness and generalization ability.

## introduction pre-trained language models @xcite @xcite @xcite have shown great potential in a wide range of nlp tasks. while large language models offer unparalleled performance, their high computation during inference limits the scope of applications. more studies recently concentrate on efficient nlp, which aims to speed up the inference of deep language models without significant performance degradation @xcite @xcite . among these, the multi-exit models @xcite ) attract widespread attention. the idea of the multi-exit models stems from the observation that inputs with varying semantics demand distinct computational resources. by automatically adjusting different computational resources according to input semantics, one can effectively speed up the inference of a multi-exit model with minimum performance loss. furthermore, such multi-exit model can be easily combined with other static speedup approaches, e.g., distillation @xcite , by replacing the backbone model. in addition to higher efficiency, previous studies also show that the multiexit models are more robust to correctness-based adversarial samples @xcite . the study of nlp attacks has mostly focused on harming models' accuracy, and taken static transformers as victim models @xcite . there exists another type of attack on the model efficiency, i.e., to make the models computationally slow. considering this type of attack, the intrinsic dynamic nature of the multiexit models might be vulnerable to such attacks. it remains unexplored, however, how significantly the efficiency or speedup from early exiting will be affected by the attacks. motivated by this, we first analyze the efficiency robustness of dynamic nlp transformers. we find that previous accuracyoriented approaches cannot significantly slow down the dynamic transformers and sometimes even lead to shorter inference time. to this end, we propose a novel slowdown attack framework on multi-exit language models: same. unlike accuracy-oriented adversarial attacks, there are several unique challenges for effective efficiency attacks. first, existing accuracy-oriented attacks aim to mislead neural networks to generate wrong predictions, which is not suitable for efficiency-oriented attacks. therefore, we develop a new objective function to guide the generation of efficiency-oriented adversarial samples. in ad-dition, our objective function must be general to handle various exit mechanisms in multi-exit transformers. second, multi-exit transformers are not static during inference, so the "static" search strategies used in adversarial attacks are not suitable. to overcome the challenges, we propose a dynamic importance adjustment strategy that assigns different importance to each exit layer, allowing the adversarial example search process to focus on the layers that contribute to model efficiency. we evaluate our same using two widely-used multi-exit strategies (entropy-based @xcite and patience-based @xcite ) with various pre-trained language models @xcite @xcite as the backbone on eight tasks from the glue benchmark. experimental results show that our same can effectively reduce the computational saving by 80% on average, which significantly outperforms previous accuracy-oriented approaches by a large margin. further experiments on the multi-goal attack, attacking transferability, and adversarial training convincingly validate the effectiveness and generalization ability of our proposed same. the contributions of this work are summarised as follows: (1) new problem: we identify a new vulnerability of the multi-exit nlp models, namely, the network efficiency. (2) novel approach: we propose the first efficiency-oriented attacking framework to measure the efficiency robustness of the multi-exit nlp models. (3) comprehensive evaluation: we conduct a systematic evaluation of various dynamic transformers, which shows that future studies on improving and protecting the efficient robustness of the multi-exit nlp models are necessary. ## conclusion and future works in this paper, we systematically evaluate the efficiency robustness of dynamic transformers. we also propose same, a novel white-box slowdown attack framework that effectively degrade the efficient performance of dynamic multi-exit language models. specifically, same generates adversarial examples that could delay the exit of dynamic multi-exit language models with the guidance of heuristic and mess loss. extensive experimental demonstrate the superior effectiveness of same across various dynamic multi-exit language models. future works include the development of efficient robust dynamic transformers and the extension to other nlp models with dynamic inference time.