BBAEG : Towards BERT -based Biomedical Adversarial Example Generation for Text Classification (2021)

Abstract: Healthcare predictive analytics aids medical decision-making, diagnosis prediction and drug review analysis. Therefore, prediction accuracy is an important criteria which also necessitates robust predictive language models. However, the models using deep learning have been proven vulnerable towards insignificantly perturbed input instances which are less likely to be misclassified by humans. Recent efforts of generating adversaries using rule-based synonyms and BERT-MLMs have been witnessed in general domain, but the ever-increasing biomedical literature poses unique challenges. We propose BBAEG (Biomedical BERT-based Adversarial Example Generation), a black-box attack algorithm for biomedical text classification, leveraging the strengths of both domain-specific synonym replacement for biomedical named entities and BERT-MLM predictions, spelling variation and number replacement. Through automatic and human evaluation on two datasets, we demonstrate that BBAEG performs stronger attack with better language fluency, semantic coherence as compared to prior work.

## introduction recent studies have exposed the importance of biomedical nlp in the well-being of human-beings, analyzing the critical process of medical decisionmaking. however, the dialogue managing tools targeted for medical conversations @xcite , @xcite , @xcite between patients and healthcare providers in assisting diagnosis may generate certain insignificant perturbations (spelling errors, paraphrasing), which when fed to the classifier to determine the type of diagnosis required/detecting adverse drug effects/drug recommendation, might provide unreasonable performance. insignificant perturbations might also creep in from the casual language expressed in the tweets @xcite . thus, the classifier needs to be robust towards these perturbations. generating adversarial examples in text is challenging compared to computer vision tasks because of (i) discrete nature of input space and (ii) preservation of semantic coherence with original text. initial works for attacking text models relied on introducing errors at the character level or manipulating words @xcite to generate adversarial examples. but due to grammatical disfluency, these seem very unnatural. some rule-based synonym replacement strategies @xcite , @xcite have lead to more natural looking examples. @xcite proposed textfooler, as a baseline to generate adversaries for text classification models. but, the adversarial examples created by textfooler rely heavily on word-embedding based word similarity replacement technique, and not overall sentence semantics. recently, @xcite proposed bert-mlm-based @xcite word replacements to create adversaries to better fit the overall context. despite these advancements, there is much less attention towards making robust predictions in critical domains like biomedical, which comes with its unique challenges. @xcite has proposed two types of rule-based adversarial attacks inspired by natural spelling errors and typos made by humans and synonym replacement in the biomedical domain. some challenges include: 1) biomedical named entities are usually multi-word phrases such as colorectal adenoma. during token replacement, we need the entire entity to be replaced, but the mlm model (token-level replacement) fails to generate correct synonym of entity fitting in the context. so, we need a bioner+entity linker @xcite , @xcite to link entity to ontology for generating correct synonyms. in this paper, we present bbaeg (biomedical bert-based adversarial example generation) 1 , a novel black-box attack algorithm for biomedical text classification task leveraging both the bert-mlm model for non-named entity replacements combined with ner linked synonyms for named entities to better fit the overall context. in addition to replacing words with synonyms, we explore the mechanism of generating adversarial examples using typographical variations and numeric entity modification. our bbaeg attack beats the existing baselines by a wide margin on both automatic and human evaluation across datasets and models. to the best of our knowledge, we are the first to introduce a novel algorithm for generating adversarial examples for biomedical text whose success attack is higher than the existing baselines like textfooler and bae (garg and ramakrishnan, 2020), @xcite . the overall contributions of the paper include: 1) we explore several challenges of biomedical adversarial example generation. 2) we propose bbaeg, a biomedical adversarial example generation technique for text classification combining the power of several perturbation techniques. 3) we introduce 3 type of attacks for this purpose on two biomedical text classification datasets. 4) through human evaluation, we show that bbaeg yields adversarial examples with improved naturalness. ## methodology problem definition: given a set of n inputs (d, y ) = [(d 1 , y 1 ), . . .(d n , y n )] and a trained classifier m : @xmath0, we assume the soft-label black-box setting where the attacker can only query the classifier for output probabilities on a given input, and has no access to the model parameters, gradients or training data. for an input of length l consisting of words w i , where 1 ≤ @xmath1, (d @xmath2, ..., w l ], y), we want to generate an adversarial example d adv such that m (d adv ) = y. we would like d adv to be grammatically correct, tag the entities in d, named entities are in sne and the rest in snne ; 2 compute token importance ii ∀ @xmath3; 3 for i in descending order of ii do 4 @xmath4; 5 if (wi in sne and (wi-t..wi+t) is a ne) then 6 @xmath5; 7 for @xmath6:i-t-1] [s]d adv[i+t+1:l] 9 end for; 10 else if (wi in snne) then 11 d @xmath7:i-1] [m]d adv[i+1:l] ; 12 @xmath8; 13 for @xmath9:i-1] [t]d adv[i+1:l] 15 end for; 16 end if; 17 if ∃ @xmath10: d adv ← l[t ] where m (l[t]) = y and l[t ] has maximum similarity with d 19 else 20 @xmath11; 21 @xmath12; 22 @xmath13; 23 for @xmath14:i-1] [t]d adv[i+1:l] 25 end for; 26 if ∃ @xmath15: d adv ← l[t ] where m (l[t]) = y and l[t ] has maximum similarity with d 28 else if wi contains numeric entity then 29 @xmath16; 30 l[t] = d adv[1:i-1] [t]d adv[i+1:l] ; 31 return: d adv ← l[t] if m (l[t]) = y 32 else 33 return: d adv ← l[t ] where l[t ] causes max reduction in y probability 34 end if; 35 end if; 36 end for; 37 return d adv ← none semantically similar to d (sim(d, d adv ) ≥ α), where α denotes the similarity threshold. ## experimental setup datasets and experimental details: we evaluate bbaeg on two different biomedical text classification datasets: 1) adverse drug event (ade) detection @xcite and 2) twitter ade dataset @xcite for the task of classifying whether the sentence contains mention of ade (binary). we use 6 classification models as m : hierarchical attention model @xcite , bert @xcite , roberta @xcite , biobert @xcite , clinical-bert @xcite , scibert @xcite . we fine-tune these models on the training data (of each corpus) using adam optimizer (kingma and ba, 2015) with learning rate of 0.00002, 10 epochs and perform adversarial attack on the test data. for the bbaeg non-ner synonym attacks, we use bert-base-uncased mlm to predict the masked tokens. we consider top @xmath17.75 for cosine similarity between @xcite embeddings of the adversarial and input text, we set @xmath18. for more details refer to the appendix. effectiveness: ## conclusion and future work in this paper, we propose a new technique for generating adversarial examples combining contextual perturbations based on bert-mlm, synonym replacement of biomedical entities, typographical errors and numeric entity expansion. we explore several classification models to demonstrate the efficacy of our method. experiments conducted on two benchmark biomedical datasets demonstrate the strength and effectiveness of our attack. as a future work, we would like to explore more about retraining the models with the perturbed samples in order to improve model robustness.