IM - BERT : Enhancing Robustness of BERT through the Implicit Euler Method (2024)

Abstract: Pre-trained Language Models (PLMs) have achieved remarkable performance on diverse NLP tasks through pre-training and fine-tuning. However, fine-tuning the model with a large number of parameters on limited downstream datasets often leads to vulnerability to adversarial attacks, causing overfitting of the model on standard datasets. To address these issues, we propose IM-BERT from the perspective of a dynamic system by conceptualizing a layer of BERT as a solution of Ordinary Differential Equations (ODEs). Under the situation of initial value perturbation, we analyze the numerical stability of two main numerical ODE solvers: *the explicit and implicit Euler approaches.* Based on these analyses, we introduce a numerically robust IM-connection incorporating BERT’s layers. This strategy enhances the robustness of PLMs against adversarial attacks, even in low-resource scenarios, without introducing additional parameters or adversarial training strategies. Experimental results on the adversarial GLUE (AdvGLUE) dataset validate the robustness of IM-BERT under various conditions. Compared to the original BERT, IM-BERT exhibits a performance improvement of approximately 8.3%p on the AdvGLUE dataset. Furthermore, in low-resource scenarios, IM-BERT outperforms BERT by achieving 5.9%p higher accuracy.

## introduction many pre-trained language models (plms) @xcite @xcite have shown remarkable performances on a wide range of downstream tasks. the high performance of plms in domain-specific downstream tasks is achieved through a two-stage training process. in the first stage, plms are pre-trained on a high-resource corpus for general representation learning. subsequently, plms are fine-tuned on low-resource data for specific downstream tasks. on the other hand, this two-stage training process makes plms prone to overfitting on the standard dataset, rendering them vulnerable to adversarial attacks through perturbations @xcite @xcite @xcite . these issues arise from aggressively fine-tuning plms with a vast number of parameters on lowresource datasets to enhance performance @xcite . given that plms require pre-training on high-resource data to obtain high-quality representations, maintaining the number of parameters is essential. the dilemma suggests addressing the issue during the second step, the fine-tuning stage. numerous studies @xcite @xcite have been conducted in the field of adversarial training, focusing on the fine-tuning stage. they focus on enhancing robustness using various training strategies, including regularization and optimization techniques in backward propagation with dataset-based hyperparameters. unlike previous approaches, there is a largely unexplored area that enhances robustness against adversarial attacks in forward propagation on standard training. we propose a robust network architecture for plms inherently more resistant to adversarial attacks on standard fine-tuning. from a dynamic system perspective, a plethora of studies @xcite @xcite have explored the relationship between neural networks and ordinary differential equations (odes), interpreting the residual connection within neural networks as an euler discretization of odes. this insight facilitates the integration of various numerical ode solvers into neural networks. the numerical solvers can be categorized into two approaches: explicit and implicit approaches. among the two approaches, the im-plicit method itself exhibits high stability against perturbations, such as adversarial attacks @xcite . by adopting the implicit method, several studies @xcite @xcite have demonstrated stable performance against both white-box and black-box adversarial attack methods in vision tasks. inspired by these previous works, we analyze the theoretical stability of two kinds of numerical odes solvers. we consider adversarial attacks as a kind of perturbation on inputs and compare the numerical stability of the two approaches. based on theoretical analysis, we propose the im-connection, utilizing the implicit euler method, to construct a robust architecture against perturbations. by incorporating im-connection within bert, we develop im-bert to mitigate vulnerabilities to adversarial attacks. this analysis enables us to establish a theoretical basis to construct robust neural architectures without necessitating specialized training strategies or an increased number of parameters. to validate that our method effectively defends against a wide range of adversarial attacks, we conduct experiments using the adversarial glue (ad-vglue) dataset @xcite . our model outperforms the baseline by up to 4.5%p on average across all tasks in advglue. moreover, under stringent conditions with sampled instances, im-bert exhibits higher accuracy than bert, with an improvement of up to 5.9%p on average. to mitigate the time-cost problem, we demonstrate that an im-bert model, applying the im-connection between several layers, is comparable to one with full implicit connections among layers. our main contributions are: ## method in this section, we first describe the network as an ode solver for the initial value problem and compare the numerical stability of the explicit and implicit euler methods. while the explicit method offers simplicity and speed advantages, it diverges the hidden states when an input is perturbed. conversely, the implicit method shows absolute stability compared to the explicit method. with this insight, we opt for the implicit method as an ode solver. leveraging the gradient descent method to implement the implicit method, we introduce imconnection and im-bert, a more stable approach for plms. ## experiment in this section, we conduct experiments in various adversarial attacks and resource situations to demonstrate the inherent robustness of im-bert. ## conclusion in this paper, we propose an approach to enhance plms with a more robust architecture by interpreting them as an ode solver. through our analysis of the numerical stability, we find that the implicit method exhibits absolute stability against