SCENE : Self-Labeled Counterfactuals for Extrapolating to Negative Examples (2023)

Abstract: Detecting negatives (such as non-entailment relationships, unanswerable questions, and false claims) is an important and challenging aspect of many natural language understanding tasks. Though manually collecting challenging negative examples can help models detect them, it is both costly and domain-specific. In this work, we propose Self-labeled Counterfactuals for Extrapolating to Negative Examples (SCENE), an automatic method for synthesizing training data that greatly improves models’ ability to detect challenging negative examples. In contrast with standard data augmentation, which synthesizes new examples for existing labels, SCENE can synthesize negative examples zero-shot from only positive ones. Given a positive example, SCENE perturbs it with a mask infilling model, then determines whether the resulting example is negative based on a self-training heuristic. With access to only answerable training examples, SCENE can close 69.6% of the performance gap on SQuAD 2.0, a dataset where half of the evaluation examples are unanswerable, compared to a model trained on SQuAD 2.0. Our method also extends to boolean question answering and recognizing textual entailment, and improves generalization from SQuAD to ACE-whQA, an out-of-domain extractive QA benchmark.

## introduction many natural language understanding tasks require a model to distinguish claims that are supported by available evidence (i.e., positive instances) from ones that are not (i.e., negative instances). in question answering, unanswerable questions (negative) can be subtly different from answerable ones (positive)-for instance, inserting an unmentioned entity into an answerable question can make it unanswerable. in recognizing textual entailment (rte) or fact verification, a hypothesis or claim that is not entailed by a given premise (negative) can be very ## method we first present our method for extractive qa, then show how it applies to a broader set of tasks in §3.4. our synthetic data generation process consists of two steps: the first is dedicated to generate easy unanswerable question-passage pairs; and the second is to generate harder unanswerable examples foot_0 through question perturbation and self-labelling. ## conclusion in this work, we have developed a counterfactual generation pipeline, scene, which synthesizes negative examples (as well as some positive examples) from real positive ones. the counterfactual generation is performed through text perturbation, filtering and relabelling. scene enables extrapolation to negative examples given only real positive examples, closing 69.6% of the performance gap on extractive qa, 89.3% on boolean qa, and 56.1% on rte, compared with models trained on both real positive and negative ones. in the future, we hope to combine our automatic pipeline with human annotation effort. for example, we could use adversarial data collection to collect a small number of perturbations that create challenging negative examples, and use these to guide generation of self-labeled counterfactuals. we would also like to explore ways to backpropagate signal from the filtering process into the generator, so the generator learns to generate useful perturbations. overall, we hope that our work can inspire more work on how synthetic data can enable new types of model extrapolation.