Robust Question Answering Through Sub-part Alignment (2021)

Abstract: Current textual question answering (QA) models achieve strong performance on in-domain test sets, but often do so by fitting surface-level patterns, so they fail to generalize to out-of-distribution settings. To make a more robust and understandable QA system, we model question answering as an alignment problem. We decompose both the question and context into smaller units based on off-the-shelf semantic representations (here, semantic roles), and align the question to a subgraph of the context in order to find the answer. We formulate our model as a structured SVM, with alignment scores computed via BERT, and we can train end-to-end despite using beam search for approximate inference. Our use of explicit alignments allows us to explore a set of constraints with which we can prohibit certain types of bad model behavior arising in cross-domain settings. Furthermore, by investigating differences in scores across different potential answers, we can seek to understand what particular aspects of the input lead the model to choose the answer without relying on post-hoc explanation techniques. We train our model on SQuAD v1.1 and test it on several adversarial and out-of-domain datasets. The results show that our model is more robust than the standard BERT QA model, and constraints derived from alignment scores allow us to effectively trade off coverage and accuracy.

## introduction current text-based question answering models learned end-to-end often rely on spurious patterns between the question and context rather than learning the desired behavior. they may ignore the question entirely @xcite , focus primarily on the answer type @xcite , or otherwise bypass the "intended" mode of reasoning for the task @xcite . thus, these models are not robust to adversarial attacks @xcite what day was super bowl 50 played on? the game was played on february 7, 2016 … super bowl 50 was an american football game to determine the champion … ## qa as graph alignment our approach critically relies on the ability to decompose questions and answers into a graph over text spans. our model can in principle work for a range of syntactic and semantic structures, including dependency parsing, srl @xcite , and amr @xcite . we use srl in this work and augment it with coreference links, due to the high performance and flexibility of current srl systems @xcite . throughout this work, we use the bert-based srl system from @xcite and the spanbert-based coreference system from @xcite . an example graph we construct is shown in figure 2 . both the question and context are represented as graphs where the nodes consist of predicates and arguments. edges are undirected and connect each predicate and its corresponding arguments. since srl only captures the predicateargument relations within one sentence, we add coreference edges as well: if two arguments are in the same coreference cluster, we add an edge between them. finally, in certain cases involving verbal or clausal arguments, there might exist nested structures where an argument to one predicate contains a separate predicate-argument structure. in this case, we remove the larger argument and add an edge directly between the two predicates. this is shown by the edge from was to determine (labeled as nested structure) in figure 2 ). breaking down such large arguments helps avoid ambiguity during alignment. aligning questions and contexts has proven useful for question answering in previous work @xcite @xcite . our framework differs from theirs in that it incorporates a much stronger alignment model (bert), allowing us to relax the alignment constraints and build a more flexible, highercoverage model. alignment constraints once we have the constructed graph, we can align each node in the question to its counterpart in the context graph. in this work, we control the alignment behavior by placing explicit constraints on this process. we place a locality constraint on the alignment: adjacent pairs of question nodes must align no more than k nodes apart in the context. @xmath0, @xmath1. in our experiments, we set @xmath2. in the following sections, we will discuss more constraints. altogether, these constraints define a set a of possible alignments. ## experiments: adversarial and cross-domain robustness our focus in this work is primarily robustness, interpretability, and controllability of our model. we focus on adapting to challenging settings in order to "stress test" our approach. ## generalizing by alignment constraints one advantage of our explicit alignments is that we can understand and inspect the model's behavior more deeply. this structure also allows us to add constraints to our model to prohibit certain behaviors, which can be used to adapt our model to adversarial settings. in this section, we explore how two types of constraints enable us to reject examples the model is less confident about. hard constraints can enable us to reject questions where the model finds no admissible answers. soft constraints allow us to set a calibration threshold for when to return our answer. we focus on evaluating our model's accuracy at various coverage points, the so-called selective question answering setting @xcite . constraints on entity matches by examining addsent and addonesent, we find the model is typically fooled when the nodes containing entities in the question align to "adversarial" entity nodes. an intuitive constraint we can place on the alignment is that we require a hard entity matchfor each argument in the question, if it contains normal addsent addonesent overall adv ∆ overall adv ∆ r.m-reader @xcite 86.6 58.5 -31.1 67.0 -19.6 kar @xcite 83.5 60.1 -23.4 72.3 -11.2 bert + adv @xcite 92.4 63.5 -28.9 72.5 -19.9 ## discussion and conclusion we note a few limitations and some possible future directions of our approach. first, errors from srl and coreference resolution systems can propagate through our system. however, because our graph alignment is looser than those in past work, we did not observe this to be a major performance bottleneck. the main issue here is the inflexibility of the srl spans. for example, not every srl span in the question can be appropriately aligned to a single srl span in the context. future works focusing on the automatic span identification and alignment like recent work on end-to-end coreference systems @xcite , would be promising. second, from the error analysis we see that our proposed model is good at performing noun phrase alignment but not predicate alignment, which calls attention to the better modeling of the predicate alignment process. for example, we can decompose the whole alignment procedure into separate noun phrase and predicate alignment modules, in which predicate alignment could be learned using different models or datasets. finally, because our bert layer looks at the entire question and answer, our model can still leverage uninterpretable interactions in the text. we believe that modifying the training objective to more strictly enforce piecewise comparisons could improve interpretability further while maintaining strong performance. in this work, we presented a model for question answering through sub-part alignment. by structuring our model around explicit alignment scoring, we show that our approach can generalize better to other domains. having alignments also makes it possible to filter out bad model predictions (through score constraints) and interpret the model's behavior (by inspecting the scores).