Bridging Robustness and Generalization Against Word Substitution Attacks in NLP via the Growth Bound Matrix Approach (2025)

Abstract: Despite advancements in Natural Language Processing (NLP), models remain vulnerable to adversarial attacks, such as synonym substitutions. While prior work has focused on improving robustness for feed-forward and convolutional architectures, the robustness of recurrent networks and modern state space models (SSMs), such as S4, remains understudied. These architectures pose unique challenges due to their sequential processing and complex parameter dynamics. In this paper, we introduce a novel regularization technique based on Growth Bound Matrices (GBM) to improve NLP model robustness by reducing the impact of input perturbations on model outputs. We focus on computing the GBM for three architectures: Long Short-Term Memory (LSTM), State Space models (S4), and Convolutional Neural Networks (CNN). Our method aims to (1) enhance resilience against word substitution attacks, (2) improve generalization on clean text, and (3) providing the first systematic analysis of SSM (S4) robustness. Extensive experiments across multiple architectures and benchmark datasets demonstrate that our method improves adversarial robustness by up to (8.8%) over existing baselines. These results highlight the effectiveness of our approach, outperforming several state-of-the-art methods in adversarial defense. Codes are available at https://github.com/BouriMohammed/GBM

## introduction deep learning models have demonstrated impressive results across various machine learning tasks, particularly in language modeling (lm), which has provided powerful tools for natural language processing (nlp). however, recent studies have shown that these models are vulnerable to adversarial examples. initially, adversarial examples were identified in image classification tasks @xcite , leading to a surge of research focused on adversarial attacks in nlp i from s(w i ). despite preserving meaning, these changes can alter the model's predicted sentiment (e.g., positive to negative) due to its sensitivity to specific word choices. ## related work spelling and grammar checkers have been shown to be robust against character-level and sentence-level attacks, which often violate grammatical requirements @xcite . however, these methods are ineffective against wordlevel attacks. existing defense methods against word-level attacks can be categorized as follows: adversarial training (at) is one of the most popular defense strategies @xcite @xcite @xcite @xcite @xcite . based on the fast gradient projection method (fgpm), an adversarial text attack technique, adversarial training with fgpm enhanced by logit pairing (atfl) @xcite generates adversarial examples and injects them into the training set. region-based adversarial training @xcite enhances model robustness by optimizing performance within the convex hull of a word's embedding and its synonyms. certified defense methods aim to ensure robustness against all adversarial perturbations @xcite @xcite . interval bound propagation (ibp)-based methods @xcite certify robustness by iteratively computing upper and lower bounds of the output for an interval-constrained input, minimizing the worstcase loss from word substitutions. however, ibp methods struggle with scalability due to high computational costs and strict constraints. randomized smoothing-based methods @xcite @xcite are structure-free, constructing stochastic ensembles of input texts and leveraging their statistical properties to certify robustness. moreover, most certified defense methods assume access to the attacker's synonym set, which is unrealistic, as adversaries are not restricted in their choice of synonyms. ## preliminaries in this paper, we address the text classification problem. consider a model f : @xmath0, where @xmath1, w 2 , . . . , w n ⟩ is a sequence of n words, where each word w @xmath2, and w denotes the vocabulary set. the output space @xmath3, y 2 , . . . , y c } denotes all classification labels. standard nlp models embed the input x into a sequence of vectors v @xmath4, v w 2 , . . . , v w n ⟩ and classify via p(y|v x ), parameterized using a neural network. adversarial robustness against synonym substitutions has been extensively studied in recent literature @xcite @xcite @xcite . the synonym set for a given word w i , denoted as s(w i ), is constructed by selecting the k nearest words to w i within an euclidean distance d e in the embedding space. to ensure semantic consistency, the perturbed input space is defined as follows: our adversarial defense aims to certify the robustness of the model by ensuring that: the objective of this work is to ensure that for all possible combinations of words in a sentence, when replaced by their synonyms, the classification outcome remains consistent, as shown in figure 1 . ## methodology in this section, we present our main contribution. we first introduce the concept of growth bound matrices (gbm) and explore how integrating these matrices into the training process can enhance model robustness. we then provide a detailed computation of gbms for multiple architectures, including lstm networks, state space model s4 and cnn models. ## experiments this section provides an experimental evaluation of our proposed gbm approach, comparing its performance with four defense baselines against three distinct adversarial attacks on the bilstm and cnn models, and with eleven defense baselines against the textfooler attack on the bert model. we also study the robustness of the s4 model against three distinct adversarial attacks. additionally, we emphasize the scalability of the gbm approach in terms of time efficiency, and we provide visual evidence of its resilience to adversarial perturbations. ## discussion our results demonstrate that minimizing the gbm yields two advantages : (1) enhanced robustness against adversarial attacks and (2) improved generalization on clean data. this is achieved by introducing a regularization term, which encourages the model to learn smoother decision boundaries. adversarial methods (e.g., ga, pwws, pso, textfooler) exploit small input perturbations (e.g., synonym substitutions) to provoke large output deviations. by reducing sensitivity (via low gbm), gradient-based or search-driven attack strategies face significant difficulty in identifying effective perturbations under practical modification limits. crucially, a controlled gbm penalty preserves the model's ability to learn discriminative features, retaining high accuracy on clean data. thus, reducing sensitivity by minimizing the gbm ensures a cru-cial balance between adversarial robustness and performance on clean inputs. further analysis can be found in the appendix e. ## conclusion in this work, we introduce a novel method, growth bound matrices (gbm), which provides tight bounds on the input-output variations of diverse models. by minimizing the change in a model's predictions when substituting a word with one of its synonyms, gbm ensures robustness against such perturbations. empirical evaluations demonstrate the effectiveness of our approach, revealing significant improvements in model robustness. for future work, we plan to investigate the applicability of gbm across additional domains, including speech recognition @xcite and control systems @xcite . furthermore, we aim to explore gbm's potential for state space model s6 (mamba) @xcite , thereby extending its contribution to robust model design.