Retrieval-guided Counterfactual Generation for QA (2022)

Abstract: Deep NLP models have been shown to be brittle to input perturbations. Recent work has shown that data augmentation using counterfactuals — i.e. minimally perturbed inputs — can help ameliorate this weakness. We focus on the task of creating counterfactuals for question answering, which presents unique challenges related to world knowledge, semantic diversity, and answerability. To address these challenges, we develop a Retrieve-Generate-Filter(RGF) technique to create counterfactual evaluation and training data with minimal human supervision. Using an open-domain QA framework and question generation model trained on original task data, we create counterfactuals that are fluent, semantically diverse, and automatically labeled. Data augmentation with RGF counterfactuals improves performance on out-of-domain and challenging evaluation sets over and above existing methods, in both the reading comprehension and open-domain QA settings. Moreover, we find that RGF data leads to significant improvements in a model’s robustness to local perturbations.

## introduction models for natural language understanding (nlu) may outperform humans on standard benchmarks, yet still often perform poorly under a multitude of distributional shifts @xcite ; @xcite ; @xcite , inter alia) due to over-reliance on spurious correlations or dataset artifacts. this behavior can be probed using counterfactual data @xcite designed to simulate interventions on specific attributes: for example, perturbing the movie review "a real stinker, one out of ten!" to "a real classic, ten out of ten!" allows us to discern the who won the inaugural best in vfl 2018 season? who captained richmond football club's women's team? figure 1 : retrieve-generate-filter to generate counterfactual queries for natural question @xcite using an open-domain retrieval system, question generation and post-hoc filtering. in this paper, we focus on counterfactual data for question answering, in both the reading comprehension and open-domain settings (e.g. @xcite . model inputs consist of a question and optionally a context passage, and the target a is a short answer span. counterfactuals are often considered in the context of a specific causal model @xcite , but in this work we follow @xcite and @xcite and seek a method to generate counterfactuals that may be use-ful in many different settings. in qa, the set of possible causal features is large and difficult to specify a priori; relevant factors are often instance-specific and exploring them may require world knowledge. for example, going from "who is the captain of the richmond football club" to a perturbed question "who captained richmond's women's team?" as in figure 1 requires knowledge about the club's alternate teams, and the perturbation "who was the captain of rfc in 1998?" requires knowledge about the time-sensitive nature of the original question. in the absence of such knowledge, otherwise reasonable edits -such as "who captained the club in 2050?" -can result in false premises or unanswerable questions. we develop a simple yet effective technique to address these challenges: retrieve, generate, and filter (rgf; figure 1 ). we use the near-misses of a retrieve-and-read qa model to propose alternate contexts and answers which are closely related to -but semantically distinct from -the original question. we then use a sequence-to-sequence question generation model @xcite to generate corresponding questions to these passages and answers. this results in fully-labeled examples, which can be used directly to augment training data or filtered post-hoc for analysis. while our method requires no supervised inputs besides the original task training data, it is able to generate highly diverse counterfactuals covering a range of semantic phenomena ( §4), including many transformation types which existing methods generate through heuristics @xcite , meaning representations @xcite or human generation @xcite . compared to alternative sources of synthetic data ( §5.1), training augmented with rgf data improves performance on a variety of settings ( §5.2, §5.3), including out-ofdomain @xcite and contrast evaluation sets @xcite , while maintaining in-domain accuracy. additionally, we introduce a measure of pairwise consistency, and show that rgf significantly improves robustness to a range of local perturbations ( §6). ## rgf: counterfactuals for information-seeking queries we define a counterfactual example as an alternative input x which differs in some meaningful, controlled way from the original x, which in turn allows us to reason -or teach the model -about changes in the label (the outcome). for questionanswering, we take as input triples (q, c, a) consisting of the question, context passage, and short answer, and produce counterfactual triples (q , c , a ) where @xmath0. this setting poses some unique challenges, such as the need for background knowledge to identify relevant semantic variables to alter, ensuring sufficient semantic diversity in question edits, and avoiding questions with false premises or no viable answers. ensuring (or characterizing) minimality can also be a challenge, as small changes to surface form can lead to significant semantic changes, and vice-versa. we introduce a general paradigm for data generation -retrieve, generate and filter -to tackle these challenges. ## data augmentation unlike many counterfactual generation methods, rgf natively creates fully-labeled (q , c , a ) examples which can be used directly for counterfactual data augmentation (cda). we augment the original nq training set with additional examples from rgf, shuffling all examples in training. we explore two experimental settings, reading comprehension ( §5.2) and open-domain qa ( §5.3), and compare rgf-augmented models to those trained only on nq, as well as to alternative baselines for synthetic data generation. as described in section 3.4, we use edit-distance based filtering to choose one generated (q , c , a ) triple to augment for every original example, (q, c, a). 3 additional training details for all models and baselines are included in appendix a. ## analysis to better understand how cda improves the model, we introduce a measure of local consistency ( §6.1) to measure model robustness, and perform a stratified analysis ( §6.2) to show the benefits of the semantic diversity available from rgf. ## conclusion retrieve-generate-filter (rgf) creates counterfactual examples for qa which are semantically diverse, using knowledge from the passage context and a retrieval model to capture semantic changes that would be difficult to specify a priori with a global schema. the resulting examples are fullylabeled, and can be used directly for training or filtered using meaning representations for analysis. we show that training with this data leads to improvements on open-domain qa, as well as on challenge sets, and leads to significant improvements in local robustness. while we focus on question answering, for which retrieval components are readily available, we note that the rgf paradigm is quite general and could potentially be applied to other tasks with a suitable retrieval system. creation of paired data for counterfactual evaluation once again, authors annotate for correctness of counterfactual rgf instances that are paired by reference or predicate, as described in §3.5. filtering is done until 1000 examples are available under each category.