Enhancing adversarial robustness in Natural Language Inference using explanations (2024)

Abstract: The surge of state-of-the-art transformer-based models has undoubtedly pushed the limits of NLP model performance, excelling in a variety of tasks. We cast the spotlight on the underexplored task of Natural Language Inference (NLI), since models trained on popular well-suited datasets are susceptible to adversarial attacks, allowing subtle input interventions to mislead the model. In this work, we validate the usage of natural language explanation as a model-agnostic defence strategy through extensive experimentation: only by fine-tuning a classifier on the explanation rather than premise-hypothesis inputs, robustness under various adversarial attacks is achieved in comparison to explanation-free baselines. Moreover, since there is no standard strategy for testing the semantic validity of the generated explanations, we research the correlation of widely used language generation metrics with human perception, in order for them to serve as a proxy towards robust NLI models. Our approach is resource-efficient and reproducible without significant computational limitations.

## introduction natural language inference (nli) is a fundamental nlp task, aiming to define whether a hypothesis is entailed by, contradicts or remains neutral with respect to a given premise @xcite . despite primarily being a classification task, the subtle intricacies related to the semantic relationship of premise-hypothesis inputs with respect to the final label pose inherent challenges even for humans @xcite , causing annotation difficulties and thus data scarcity. within the rapidly evolving nlp landscape, there are still several unsolved challenges, especially concerning the usage of large language models (llms) for nli, which are yet unable to fully capture the semantic sophistications of the task @xcite . at the same time, explainability remains a point of reference for state-of-the-art (sota) nlp @xcite ; however, it holds an even more crucial position for nli, as stated in the seminal work of @xcite , where authors hint that generating an intermediate explanation before predicting the final label is adequate for robustness enhancement. this is a fundamental claim, as nli models are widely susceptible to adversarial attacks @xcite @xcite ). yet, to the best of our knowledge, there is no prior work attempting to solely harness explanations for adversarial defence, in order to answer whether this claim holds or not. the additional power of intermediate explanations is that they shed some light on the black-box nature of nli models, providing information regarding the semantic relationship between the premise and the hypothesis. under this breakdown of the nli process, the weight is shifted towards producing a semantically valuable and correct explanation, which is highly associated with the final label, as we will demonstrate later in this paper. therefore, without exploiting any other mechanism rather than the intermediate explanations, we are able to open the black-box while simultaneously rendering it more robust. overall, in this work, we propose a very simple, yet effective approach to tackle adversarial brittleness of nli: we leverage the explainthenpredict framework proposed in @xcite to acquire explanations derived from given premisehypothesis input pairs, based on which we predict the final label. to further promote the simplicity of our method, we only exploit smaller language models for explanation generation, as well as for classification in the entailment/neutral/contradiction classes, proving that despite not being the most powerful learners, they are adequate in proving the robustness-enhancing power of explanations. specifically, our contributions are: adversarial robustness is a major concern in nlp @xcite calling for successful detection and creation of defence strategies @xcite @xcite . crafting adversarial attacks @xcite @xcite reveals weak spots of models in cases they return unreasonably deviating outputs with respect to the semantic minimality of input perturbations. in general, the quest for robustness may require some sacrifice in accuracy @xcite , at least under certain scenarios that cannot be satisfied by theoretical guarantees @xcite @xcite , such as in black-box settings where adequate engineering regarding training details and hyperparameters is not feasible. this trade-off has not been extensively studied in nlp or at least in various black-box cases, therefore it is unknown if it holds when studying them in conjunction. despite the suggested incorporation of explanations for robust nli @xcite , this topic has not received much attention yet, with only a few notable exceptions @xcite , while the utilized explanations benefit other robustness-related research questions, such as the robustness of in-context learning in llms @xcite . we highly acknowledge this research gap, promoting the exploitation of explanations as a model-agnostic strategy to enhance nli robustness under adversarial attacks. ## on the use of intermediate explanations in the core of our approach lies the explainthen-predict framework @xcite that instead of predicting the final entailment (e)/neutral (n)/contradiction (c) label using the input premise and hypothesis, it generates an intermediate explanation in natural language, which serves as an input to a classifier to decide the final label. as a first step, a seq2seq model receives the premise p and the hypothesis h and outputs a free-form explanation e, which aims to justify the semantic relationship between them under an entailment/neutral/contradiction format. for example, given p: "a land rover is being driven across a river" and h: "a vehicle is crossing a river", the seq2seq stage generates an explanation e: "land rover is a vehicle". in the second step, an expl2label classifier defines the output label @xmath0, n, c}, leveraging the "hints" provided in the explanation. in the aforementioned example, given e: "land rover is a vehicle", the expl2label classifier outputs entailment as the final label. notably, seq2seq and expl2label are fine-tuned independently and are only joined during inference, acting as a black-box model overall. ## adversarial attacks we stress the robustness of the explainthenpredict pipeline by performing targeted adversarial attacks either on p or h independently. the outline of our proposed approach is illustrated in figure 1 . we focus on applying minimal interventions that influence the semantics of the inputs, resulting in adversarial @xmath1. such interventions consequently lead to @xmath2, which finally affect the outcome of the bert classifier, resulting in a @xmath3. given the semantic minimality of the intervention, a @xmath4, indicating a vulnerable behaviour in terms of classification robustness. the intervention needs to be targeted, since altering the predicted nli label is significant to view an attack as "successful": a negligible semantic intervention erroneously leads to a change of the nli prediction; in the explainthenpredict case this change happens because the attack on p/h affected the intermediate explanation e (if no change had occurred on the e, no outcome change could be possible). therefore, we materialize the desired attacks using attack recipes from bert-attack @xcite and textfooler @xcite , which serve as sota word-level editors, providing the requested determinism that guarantees the minimality of edits, while preserving the meaning and syntax of the attacked sentence. by attacking the explanation-free baseline, as well as the ex-plainthenpredict variants we are able to measure the attack success rate, i.e. the ratio of attempted attacks that successfully produce adversarial examples in each case. therefore, the higher the attack success rate, the more vulnerable the model is against such interventions. moreover, we calculate the after-attack accuracy, corresponding to the percentage of inputs that were unsuccessfully attacked and correctly classified, with higher values denoting more robust models. the attack success rate and after-attack accuracy accuracy metrics hold an inverse relationships, with more robust models presenting lower attack success rate and higher afterattack accuracy. for the sake of completeness, we further report the average number of queries, which denotes the attack efficiency, corresponding to the number of attacks that the attacker needs to perform in order to change the outcome. higher values indicate that the attacker needs to place more effort in order to achieve the alternative outcome, implying comparatively advanced resistance from the side of the attacked model. this experimentation aims to conclude under which circumstances the explainthenpredict framework leads to more robust nli classification and how we can obtain some guarantees regarding advanced robustness. ## conclusion in this work, we delve into the underexplored field of nli robustness. we experimentally prove that the robustness of nli models against adversarial attacks can be boosted by solely generating intermediate explanations. furthermore, we demonstrate that linguistic quality and human perception of faithfulness are strongly correlated with advanced robustness of the final model, drawing the attention to explanation evaluation as the natural next step in advancing trustworthy and interpretable nli.