S low BERT : Slow-down Attacks on Input-adaptive Multi-exit BERT (2023)

Abstract: For pretrained language models such as Google’s BERT, recent research designs several input-adaptive inference mechanisms to improve the efficiency on cloud and edge devices. In this paper, we reveal a new attack surface on input-adaptive multi-exit BERT, where the adversary imperceptibly modifies the input texts to drastically increase the average inference cost. Our proposed slow-down attack called SlowBERT integrates a new rank-and-substitute adversarial text generation algorithm to efficiently search for the perturbation which maximally delays the exiting time. With no direct access to the model internals, we further devise a time-based approximation algorithm to infer the exit position as the loss oracle. Our extensive evaluation on two popular instances of multi-exit BERT for GLUE classification tasks validates the effectiveness of SlowBERT. In the worst case, SlowBERT increases the inference cost by 4.57\times , which would strongly hurt the service quality of multi-exit BERT in practice, e.g., increasing the real-time cloud services’ response times for online users.

## introduction in natural language processing, pre-trained language models such as bert @xcite , gpt @xcite , xl-net @xcite have brought significant improvements to various down-stream applications. despite the success on the effective performances, the enormous training and inference cost (i.e., processing time and gpu consumption) severely hinders its practice on real-time applications and hardwareconstrained edge devices. to accelerate inference for bert, recent works from ai2 @xcite and microsoft @xcite propose the design of multiexit bert, which implements the general idea that simple texts only requires simple models to make * corresponding authors. accurate predictions. technically, multi-exit bert couples an internal classifier with each layer of the feature extractor and dynamically stops inference when meeting certain early-exit criterion. as shown in fig. 1 , the early-exit criterions are divided into confidence-based @xcite @xcite and patience-based @xcite inferences. input-adaptive multi-exit bert can be deployed on real-time cloud services for fast inferences (e.g., to process million of queries per second) and hardware-constrained edge devices for low inference costs. other than the common adversarial threat @xcite against large language models' utility, we focus on the vulnerability of the multi-exit bert from the perspective of efficiency, where the exit positions can be deliberately changed by the imperceptible changes on texts. if such an attack is possible, it will pose serious challenges to the practical deployments. for real-time cloud services, it increases the inference response time, which is not acceptable for online users. for hardware-constrained edge devices, it increases the average inference cost, where the resources (e.g., battery life) are limited and valuable to the edge devices. despite the successful attempt against the efficiency of the multi-exit mechanisms in the vision domain @xcite , there are still challenges to directly apply this algorithm. firstly, the attack goal is optimized through gradients in the continuous space, which is not suitable for the typical search-based adversarial attack @xcite in the discrete space. secondly, the attack only provide attack against confidence-based inference, which limits it effectiveness on multi-exit bert, especially the patience-based inference. lastly, the attack can't perform under the practical scenario with cloud services that have no the direct access to the model internals. the non-negligible challenges call for a more careful design specifically targeted at the multi-exit bert. in our work, we propose an effective and novel slow-down attack against multi-exit bert called slowbert to address the mentioned challenges. our attack generates malicious text samples to make the internal classifiers not aligned with the early-exit criterions, which ultimately delays the exit positions of the input-adaptive multi-exit bert. we propose a new rank-and-substitute algorithm to generate malicious texts to slow down the inference of multi-exit bert. to achieve better attack performance, we propose the exit status containing the exit position information and its corresponding output to guide the generation process, which works for both early-exit criterions. to solve the challenge where no model internals are available, we propose a time-based approximation algorithm to accurately infer the exit position using inference time, making our attack applicable to various deployment scenarios. the contributions of our paper can be summarized as follows: confidence-based inferences @xcite exit when the predictions are confident enough. patient-based inferences @xcite exit when consecutive internal classifiers make unanimous decisions. ## conclusion in our work, we systematically study the potential vulnerability of the input-adaptive bert from the perspective of large language model efficiency. we propose slowbert, which drastically increases the average inference cost of the multi-exit bert. this new attack surface poses serious challenges to the deployments of the multi-exit bert on real-time cloud services or local hardware-constrained edge devices. as a security problem of the large language model, our work welcomes future research to devise strong defense against our attacks.