Adversarial Textual Robustness on Visual Dialog (2023)

Abstract: Adversarial robustness evaluates the worst-case performance scenario of a machine learning model to ensure its safety and reliability. For example, cases where the user input contains a minimal change, e.g. a synonym, which causes the previously correct model to return a wrong answer. Using this scenario, this study is the first to investigate the robustness of visually grounded dialog models towards textual attacks. We first aim to understand how multimodal input components contribute to model robustness. Our results show that models which encode dialog history are more robust by providing redundant information. This is in contrast to prior work which finds that dialog history is negligible for model performance on this task. We also evaluate how to generate adversarial test examples which successfully fool the model but remain undetected by the user/software designer. Our analysis shows that the textual, as well as the visual context are important to generate plausible attacks.

## introduction adversarial robustness has recently gained increased attention within the nlp community (e.g. @xcite @xcite . in contrast to this previous wok, which focuses on text-only models, we evaluate the adversarial robustness of visual dialog (visdial) models with the aim to understand how different input components contribute to robustness. for example, it has previously been established that multiple input modalities increase robustness of pre-neural conversational interfaces, e.g. @xcite . here, we want to know which modalities can mitigate input attacks on neural visual dialog systems, and to what extent. this is important, since worst- prediction answers relevance yes. (gt answer) no. i think so. i can not tell. ... a fork resting on the plate. 0.8 0 0.2 0.2 ... 0 image question after attack are any of them retailers? prediction answers after attack relevance no. yes. i can not tell. not that i can see. ... a fork resting on the plate. 0 0.8 0.2 0.2 ... 0 before attack after attack inputs figure 1 : a visdial agent aims to answer an imagerelated question by ranking a list of candidate answers, given the dialog history. the attacker permutes the text by replacing a word with its synonym so that the ranking of the predicted answers changes. a successful attack in our setting is a perturbation which changes the model prediction, but, at the same time, remains unnoticed by the user/ software developer. while this is relatively simple for pixel-level visual attacks, textual perturbations are substantially harder to conceal. in the following we thus investigate the trade-off between effectiveness and linguistic quality for text-based attacks. the focus on text-only attacks is further motivated by our preliminary experiments, which show that encoding visual information has very little gain for performance and robustness (around 5%, cf. to the best of our knowledge, we are the first to explore adversarial attacks on visdial, which was introduced as a shared task by @xcite . a visual dialog system consists of three components: an image (with a caption), a question and the dialog history, i.e. previous user and system turns. in order to answer the question accurately, the ai agent has to ground the question in the image and infer the context from history, see fig. 1 . most existing research has focused on improving the modelling performance on this task, (e.g. @xcite @xcite @xcite @xcite @xcite @xcite , whereas our aim is to evaluate adversarial robustness. in addition, we use these attacks to improve our understanding of how the model works (i.e. interpretability). our main contributions are: ## related work adversarial attacks have been widely investigated within uni-modal applications, foremost for com-puter vision @xcite @xcite . adversarial attacks on text are more challenging due to its discrete nature, which makes it harder to stay undetected. adversarial textual attack methods can be divided into three levels of granularity @xcite : character-level, word-level and sentence-level attacks. while the former two are relatively easy to detect, recent word-level attack methods (e.g. @xcite @xcite @xcite , on the other hand, are more subtle: they are targeted towards 'vulnerable' words, which are substituted via their synonyms in order to preserve semantic meaning. in our paper, we explore word-level attack methods on visdial. research on adversarial attacks for multi-modal tasks is relatively scarce, including studies for optical character recognition @xcite , scene text recognition @xcite , image captioning @xcite and vqa @xcite . most of these works utilise white box attack, where the parameters, gradient and architecture of the model are available, e.g. by attacking attention @xcite . whereas we follow a more realistic black-box setting which assumes that the attacker only has access to the model's prediction on test data. @xcite is closest related to our work: they generate adversarial textual attacks for the vqa task using contrastive examples and thus don't pay attention to semantic similarity. in contrast, we are interested in generating adversarial attacks which follow three desiderata, as outlined by @xcite : an adversarial text should (1) keep the same semantic meaning (semantic similarity); (2) guarantee fluency and grammar (grammaticality); (3) stay unnoticed by humans, i.e. the human still assigns the correct label, while the model prediction changes (label consistency). ## detailed analysis of linguistic quality constraints next, we analyze the impact of the linguistic quality constraints. we are interested in the trade-off between using these constraints to produce high quality text (which increases the chance of the attack to remain unnoticed by humans) versus an effective attack (which increases the chance of the model changing its prediction). effect of selecting vulnerable words first, we compare the results of choosing a random word in text to attack and our vulnerable word attack. the results in effect of stop words next, we compare the results with/without stop words. the results in ## human evaluation study we evaluate the quality of our generated adversarial question attack by asking human judges on amazon mechanical turk (amt) to rate three aspects: if the generated question preserve the semantic similarity (semantic similarity with/without given image); if the generated question is natural and grammatical (grammaticality); if the human's prediction is unchanged for the generated question (label consistency). we evaluate a total of 198 generated attacks, randomly sampled from the development set, where three users are asked to rate each instance, following best practices for human evaluation in nlp (van der @xcite . 5 see appendix for further details. ## conclusions and limitations we present a detailed study investigating adversarial robustness of visual dialog models. we find that multiple inputs increase robustness, and in particular dialog history contributes to robustness, despite previous results which suggest that history has negligible effect on model performance, e.g. @xcite . we investigate the trade-off between effectiveness and linguistic quality, where we show limitations of current synonym-based textual attack models, and stress the importance of context (both textual as well as multi-modal) to generate semantically coherent and grammatically fluent adversarial attacks, which are likely remain undetected. while the observed effects of visually-grounded interpretations in our human evaluation were relatively small, we do believe that it is an important future direction. for example, we expect improved results by using synonym substitution methods based on visuallygrounded word embeddings.