UKP - SQ u ARE v2: Explainability and Adversarial Attacks for Trustworthy QA (2022)

Abstract: Question Answering (QA) systems are increasingly deployed in applications where they support real-world decisions. However, state-of-the-art models rely on deep neural networks, which are difficult to interpret by humans. Inherently interpretable models or post hoc explainability methods can help users to comprehend how a model arrives at its prediction and, if successful, increase their trust in the system. Furthermore, researchers can leverage these insights to develop new methods that are more accurate and less biased. In this paper, we introduce SQuARE v2, the new version of SQuARE, to provide an explainability infrastructure for comparing models based on methods such as saliency maps and graph-based explanations. While saliency maps are useful to inspect the importance of each input token for the model’s prediction, graph-based explanations from external Knowledge Graphs enable the users to verify the reasoning behind the model prediction. In addition, we provide multiple adversarial attacks to compare the robustness of QA models. With these explainability methods and adversarial attacks, we aim to ease the research on trustworthy QA models. SQuARE is available on https://square.ukp-lab.de .

## introduction the recent explosion of question answering datasets and models is pushing the boundaries of qa systems and making them widely used by the general public in virtual assistants or chatbots @xcite . this ubiquitous adoption is making regulators start preparing policies for artificial intelligence with special emphasis on explainability and robustness to adversarial attacks. 2 figure 1 : visualization of two saliency maps computed using integrated gradients. the darker the highlighting color, the higher its importance to get the prediction. hovering on a word shows its importance value. there are multiple methods to explain the predictions of ai models @xcite and analyze their robustness @xcite . some explainability methods focus on specific input attributions such as attention-and gradient-based saliency maps @xcite . others design interpretable models instead of using post hoc methods @xcite . lastly, most approaches that analyze the robustness of ai systems are based on adversarial attacks, i.e., the use of inputs such as questions with minor modifications that change the system's output. however, exploring and comparing these methods is not straightforward for most models. researchers usually need to manipulate libraries and create interfaces to compare them in a satisfactory manner, which is a slow and complicated process that hinders the research in trustworthy qa. the square platform @xcite simplifies the process of comparing qa models by empowering nlp researchers with an online platform to deploy, run, and compare the most common qa pipelines while removing technical barriers such as model and infrastructure configu-rations. it includes dozens of models of multiple types, namely open-domain, extractive, multiple choice, and abstractive qa. however, the only explainability method currently implemented is behavioral test (appendix 3), limiting the comparison between qa models based solely on the models' final predictions. in this work, we propose square v2, a new online platform for trustworthy qa research implementing various explainability, interpretability, and robustness methods and interfaces to facilitate research in trustworthy qa models. specifically, we make the following contributions: 1) square v2 supports the comparison of models based on different post hoc explainability methods. we create interactive saliency maps that illustrate the importance of each input token for the model's prediction @xcite . 2) we extend the datastores to include support for knowledge graphs (kg), deploy qa-gnn @xcite , an interpretable graph-based model, and create an interactive visualization graph. 3) square v2 further provides various adversarial attacks, which change the prediction by modifying the input but keeping its semantics in order to evaluate the robustness of qa models @xcite . ## related work allennlp demo foot_0 is the closest system to square v2. they provide a web interface to interact with their library, where users can explore explainability functionalities @xcite . however, only two non-transformer models include saliency maps and attack methods. in addition, users cannot deploy their models on this web demo, and instead, they would need to install their library and create their own interface. among the explainability libraries, captum @xcite is of special relevance. it is a model interpretability library for pytorch that includes multiple saliency maps and provides built-in visualizations. however, it does not provide a user interface to run all their methods and compare them at a glance. on the other hand, it provides an adversarial attack method, fast gradient sign method @xcite , and some variants; however, these are not designed for nlp. lastly, there are some efforts to ease the study of adversarial attacks on nlp models. textattack @xcite is a library that supports several attacks and is model agnostic. however, they do not provide a web interface, so users must therefore create their own visualizations in order to be able to easily compare attacks on multiple models. in summary, square is a single entry-point for nlp practitioners to analyze, compare, and teach qa through models' outputs, explainability, and robustness with a user-friendly interface. ## ukp-square square @xcite is an opensource, online platform for nlp researchers to share, run, compare, and analyze their qa models. the platform implements a flexible and scaleable microservice architecture containing four highlevel services: square is designed to ease the comparison and analysis of models. users can deploy their models using a simple interface without the need of any code and then, they can compare outputs of different models side-by-side. this paper describes a new major update of square. modern neural networks have significantly improved in performance in recent years; however, their explainability have not followed the same improvement @xcite . additionally, despite their impressive performance, the models are vulnerable to adversarial attacks. the goal of square v2 is to provide the research community with a set of tools to facilitate the research on trustworthy qa. square simplifies and provides visualizations for saliency maps, graph-based interpretable models, and adversarial attacks. the following sections briefly describe the methods provided in square. ## datastores for knowledge graphs to best re-use the existing datastore while being efficient and robust, we rely on an elasticsearch instance to store kgs. in particular, we represent nodes and edges as documents and include information to recreate the graph structure, such as their connectivity. we show the schema of these documents in appendix a. in addition, we implement two main functionalities: firstly, users can dynamically add and update new kgs as long as the structure of the kg can be converted to the schema shown in appendix a. this allows supporting any kg that is requested by the community. for demonstration purposes, we provide conceptnet @xcite as a built-in kg. more information is available on the datastores documentation foot_1 . secondly, we implement a subgraph extraction method. given a list of root nodes (e.g., the entities in a question), it extracts all the nodes and edges in the vicinity of k hops to the roots. since conceptnet is densely connected, we limit the maximum number of hops to 3. however, this is a parameter that can be adjusted for any kg. lastly, after the extraction, we prune the disconnected nodes. ## conclusion and future work we present square v2, a web platform that unifies three families of methods for analyzing qa models: saliency maps, adversarial attacks, and interpretable models. firstly, we offer an interactive interface that allows users to compare multiple saliency map methods for all the skills deployed in square. secondly, we provide an interface to conduct adversarial attacks. this interface allows the community to study the robustness of qa models. lastly, we deploy an interpretable graph-based model and provide an interface to visualize the reasoning paths that the model may conduct. to deploy this skill, we extend the datastores module to support both text documents and kgs. these contributions give square a set of tools to compare, analyze, and explain the behavior of qa models. since square allows the deployment of almost any transformer-based model effortlessly, our new explainability interface empowers the community with tools for trustworthy qa research. square is actively under development. future updates will include new kgs such as wikidata @xcite , automated skill selection @xcite , and skill collaboration @xcite .