TABS : Efficient Textual Adversarial Attack for Pre-trained NL Code Model Using Semantic Beam Search (2022)

Abstract: As pre-trained models have shown successful performance in program language processing as well as natural language processing, adversarial attacks on these models also attract attention.However, previous works on black-box adversarial attacks generated adversarial examples in a very inefficient way with simple greedy search. They also failed to find out better adversarial examples because it was hard to reduce the search space without performance loss.In this paper, we propose TABS, an efficient beam search black-box adversarial attack method. We adopt beam search to find out better adversarial examples, and contextual semantic filtering to effectively reduce the search space. Contextual semantic filtering reduces the number of candidate adversarial words considering the surrounding context and the semantic similarity.Our proposed method shows good performance in terms of attack success rate, the number of queries, and semantic similarity in attacking models for two tasks: NL code search classification and retrieval tasks.

## introduction many pre-trained models have been proposed and achieved success not only in natural language processing @xcite @xcite , but also in program language processing @xcite @xcite . as pre-trained models attract attention, so do the adversarial attacks on those models @xcite @xcite @xcite . if pre-trained code models are attacked, they may produce incorrect results for slightly perturbed inputs in the sentence of natural language. however, most of these black-box adversarial attack methods generate adversarial examples with inefficient greedy search. the method simply tries every adversarial word without considering the sur-rounding context and the contextual semantic word similarity. moreover, it simply replaces vulnerable words in a sentence one by one without considering the global context. their attack success rates are low, and a number of queries are required for attack success. it is not guaranteed that the adversarial examples are generated semantically similar to the original examples even if the attack was successful. in this paper, we propose the semantic beam search method that can efficiently generate semantically similar adversarial examples with fewer queries while having higher performance. our proposed method selects adversarial words considering the contextual semantic similarity with the original word as well as the word probability predicted by the bert masked language model (mlm). with this contextual semantic filtering, the number of queries is significantly reduced compared to the existing methods without performance reduction. to increase the success rate, we adopt beam search @xcite to generate better adversarial examples. our approach keeps top-k candidate adversarial sentences with beam search while attacking. it helps choose a better adversarial sentence considering the global context. beam search may expand the search space, but we can effectively reduce the search space with semantic filtering. finally, we can achieve the highest success rate with a minimum number of queries. we apply our approach to the nl-code search task that searches for codes with the same meaning for a given natural language sentence for the first time. the nl code search task provided by codexglue consists of two sub-tasks: classification task and retrieval task @xcite . the classification task is a task to classify whether or not the text-code pair represents the same meaning, and the retrieval task is a task that finds the most relevant code when a text is given. we show successful attack results for two tasks: classification and retrieval task of nl code search task. ## methodology in this section, we describe tabs: efficient textual adversarial attack using semantic beam search, that requires a very small number of queries to generate adversarial text with very high quality for the nl code search tasks. the proposed approach, tabs, is based on bert-attack. we determine the order of words to be replaced by output differences as the previous method did. suppose that there is a pair of text @xmath0, w 2 , ..., w n ] and code c, the corresponding label y , and the output logit of the classification victim model or the output rank of the retrieval victim model, f (t, c). the word importance by the output difference is defined as follows: to address this issue, we adopt beam search which keeps k candidate adversarial sentences while attacking. however, if beam search is simply applied, the number of queries may increase about k times. in order to solve this effectively, we do not try all the synonym words chosen by probability of the masked language model for the current position word, but we sample m important words among the n synonyms by considering the contextual semantic similarity to the original word. contextual semantic filtering is based on the similarity between word embeddings from bert-mlm. first, we replace the current word with each algorithm 1 tabs input: text t ; code c; victim model f (•) output: adversarial example t adv 1: for wi in t do 2: compute the importance score iw i using eq. 1 3: end for 4: initialization t adv ← t ; b ← ϕ; @xmath1;b ′ ← ϕ 5: create a word list v sorted by descending order of importance score iw i 6: for vi in v do 7: for b in b do 8: predict n synonyms for b\vi based on bert-mlm 9: calculate each context embedding for n synonyms 10: sort by similarity scores based on context embedd-11: ings between the original word and n synonyms 12: select top-m words 13: for m in m do 14: b[vi] = m 15: b ′ ← b ′ ∪ b 16: end for 17: end for 18: sort b ′ by the logit values of the victim model f (•) 19: if satisfy eq. 2 or eq. 3 in b ′ then 20: t adv ← adversarial example in b ′ with the maximum sentence similarity to the original sentence t 21: return t adv 22: else 23: b ← top-k(b ′ ), b ′ ← ϕ 24: end if 25: end for during attacking with beam search, we sort k candidate adversarial sentences by the logit values in ascending order, and check if there are adversarial sentences that satisfy the attack success condition. if there are, the sentence most similar to the original sentence is selected as the final adversarial example. the attack success conditions are defined as fol we summarize the proposed approach, tabs, in algorithm 1. ## conclusion in this paper, we proposed an efficient beam search black-box adversarial attack method, tabs, that not only replaces words by considering the context of surrounding words but also reduces queries by considering the similarity of generated sentences. to the best of our knowledge, we are the first to apply an adversarial attack to the nl code search task. our approach showed good performance in terms of attack success rate, number of queries, and semantic similarity in the classification task and attack success rate, number of queries, semantic similarity, rank change, and ndcg in the retrieval task.