Exploring Weaknesses of VQA Models through Attribution Driven Insights (2020)

Abstract: Deep Neural Networks have been successfully used for the task of Visual Question Answering for the past few years owing to the availability of relevant large scale datasets. However these datasets are created in artificial settings and rarely reflect the real world scenario. Recent research effectively applies these VQA models for answering visual questions for the blind. Despite achieving high accuracy these models appear to be susceptible to variation in input questions. We analyze popular VQA models through the lens of attribution (input’s influence on predictions) to gain valuable insights. Further, We use these insights to craft adversarial attacks which inflict significant damage to these systems with negligible change in meaning of the input questions. We believe this will enhance development of systems more robust to the possible variations in inputs when deployed to assist the visually impaired.

## introduction visual question answering (vqa) is a semantic task, where a model attempts to answer a natural language question based on the visual context. with the emergence of large scale datasets @xcite @xcite @xcite , there has been outstanding progress in vqa systems in terms of accuracy obtained on the associated test sets. however these systems are seen to somewhat fail when applied in real-world situations @xcite majorly due to a significant domain shift and an inherent language/image bias. a direct application of vqa is to answer the questions for images captured by blind people. the vizwiz @xcite is a first of its kind goal oriented dataset which reflects the challenges conventional vqa models might face when applied to assist the blind. the questions in this dataset are not straightforward and are often conversational which is natural knowing that they have been asked by visually impaired people for assistance. due to unsuitable images or irrelevant questions most of these questions are unanswerable. these questions differ from those in other datasets mainly in the type of answer they are expecting. the questions are often subjective and require the algorithm to actually read (ocr)/ detect/ count, moreover understand the image before answering. we believe models trained on such a challenging dataset must be interpretable and should be analyzed for robustness to ensure they are accurate for the right reasons. ## model interpretability deep neural networks often lack interpretability but are widely used owing to their high accuracy on the representative test sets. in most applications a high test-set accuracy is sufficient, but in certain sensitive areas, understanding causality is crucial. when deploying such vqa models to aid the blind, utmost care needs to be taken to prevent the model from answering wrongly to avoid possible accidents. in the past, various saliency methods have been used to interpret models which have textual inputs. vanilla gradient method @xcite visualizes the gradients of the loss with respect to each input token(word in this case). smoothgrad @xcite averages the gradient by adding gaussian noise to the input. layerwise relevance propagation (lrp) @xcite , deeplift @xcite are similar methods used for this purpose. gradients (ig) @xcite satisfies the necessary axioms, we use it for the purpose of interpretability. ig computes attributions for the input features based on the network's predictions. these attributions assign credit/blame to the input features (pixels in case of an image and words in case of a question) which are responsible for the output of the model. these attributions can help identify when a model is accurate for the wrong reasons like over-reliance on images or possible language priors. these attributions are computed with respect to a baseline input. in this paper, we use an empty question as the baseline. we use these attributions which specify word importance in the input question to design adversarial questions, which the model fails to answer correctly. while doing so, we try to preserve the original meaning of the question and ensure the simplicity of the same. we design these questions manually by incorporating highly attributed content-free words in the original question,taking into consideration the free-formed conversational nature of the questions that any user of such a system might ask. by content-free, we refer to words that are context independent like prepositions (e.g., "on", "in"), determiners (e.g., "this", "that") and certain qualifiers (e.g., "much", "many") among others. ## related work the main idea of adversarial attacks is to carefully perturb the input without making perceivable changes, in order to affect the prediction of the model. there has been significant research on adversarial attacks concerning images @xcite . these attacks exploit the oversensitivity of models towards changes in the input image. sharma et al. 2018 study attention guided implementations of popular imagebased attacks on vqa models. @xcite discuss methods to generate targeted attacks to perturb input images in a multimodal setting. ramakrishnan et al. 2018 observe that vqa models heavily rely on certain language priors to directly arrive at the answer irrespective of the image. they further develop a bias-reducing approach to improve performance. kafle and kanan 2017 study the response of vqa models towards various question categories to indicate the deficiencies in the datasets. @xcite analyze the robustness of vqa models on basic questions ranked on the basis of similarity by lasso based optimization method. finally, mudrakarta et al. 2018 use attributions to determine word importance and leverage them to craft adversarial questions. we adapt their ideas to the conversational aspect of questions in vizwiz to better suit our task. in this paper we restrict ourselves to attacks in the language domain, i.e. we only perturb the input questions and analyze the network's response. ## conclusion we analyzed two popular vqa models trained under different circumstances for robustness. our analysis was driven by textual attributions, which helped identify shortcomings of the current approaches to solve a real world problem. the attacks discussed in this paper, illuminate the need for achieving robustness to scale up better to the task of visual assistance. to improve accessibility for the visually impaired, these vqa systems must be interpretable and safe for operation even under adverse conditions arising out of conversational variations. we believe these insights can be useful to surmount this challenging task.