Logic-Guided Data Augmentation and Regularization for Consistent Question Answering (2020)

Abstract: Many natural language questions require qualitative, quantitative or logical comparisons between two entities or events. This paper addresses the problem of improving the accuracy and consistency of responses to comparison questions by integrating logic rules and neural models. Our method leverages logical and linguistic knowledge to augment labeled training data and then uses a consistency-based regularizer to train the model. Improving the global consistency of predictions, our approach achieves large improvements over previous methods in a variety of question answering (QA) tasks, including multiple-choice qualitative reasoning, cause-effect reasoning, and extractive machine reading comprehension. In particular, our method significantly improves the performance of RoBERTa-based models by 1-5% across datasets. We advance state of the art by around 5-8% on WIQA and QuaRel and reduce consistency violations by 58% on HotpotQA. We further demonstrate that our approach can learn effectively from limited data.

## introduction comparison-type questions @xcite @xcite ask about relationships between properties of entities or events such as cause-effect, qualitative or quantitative reasoning. to create comparison questions that require inferential knowledge and reasoning ability, annotators need to understand context presented in multiple paragraphs or carefully ground a question to the given situation. this makes it challenging to annotate a large number of comparison questions. most current datasets on comparison questions are much smaller than standard machine reading comprehension (mrc) datasets @xcite figure 1 : inconsistent predictions by roberta. top row shows an example of symmetric inconsistency and the second row shows an example of transitive inconsistency. the examples are partially modified. @xcite . this poses new challenges to standard models, which are known to exploit statistical patterns or annotation artifacts in these datasets @xcite . importantly, state-of-the-art models show inconsistent comparison predictions as shown in figure 1 . improving the consistency of predictions has been previously studied in natural language inference (nli) tasks @xcite , but has not been addressed in qa. in this paper, we address the task of producing globally consistent and accurate predictions for comparison questions leveraging logical and symbolic knowledge for data augmentation and training regularization. our data augmentation uses a set of logical and linguistic knowledge to develop additional consistent labeled training data. subsequently, our method uses symbolic logic to incorporate consistency regularization for additional supervision signal beyond inductive bias given by data augmentation. our method generalizes previous consistency-promoting methods for nli tasks @xcite to adapt to substantially different question formats. our experiments show significant improvement over the state of the art on a variety of qa tasks: a classification-based causal reasoning qa, a multiple choice qa for qualitative reasoning and an extractive mrc task with comparisons between entities. notably, our data augmentation and consistency constrained training regularization improves performance of roberta-based models @xcite by 1.0%, 5.0% and 2.5% on wiqa, quarel and hotpotqa. our approach advances the stateof-the-art results on wiqa and quarel with 4.7 and 8.4% absolute accuracy improvement, respectively, reducing inconsistent predictions. we further demonstrate that our approach can learn effectively from limited labeled data: given only 20% of the original labeled data, our method achieves performance on par with a competitive baseline learned with the full labeled data. ## related work data augmentation has been explored in a variety of tasks and domains @xcite @xcite . in nlp, using backtranslation @xcite or dictionary based word replacement @xcite has been studied. most relevant to our work, @xcite study nli-specific logic and knowledgebased data augmentation. concurrent to our work, @xcite study visual qa models' ability to answer logically composed questions, and show the effectiveness of logic-guided data augmentation. our data augmentation does not rely on task-specific assumptions, and can be adapted to different formats of qa task. we further leverage consistency-promoting regularization, which gives improvements in accuracy and consistency. improving prediction consistency via training regularization has been studied in nli tasks. @xcite present model-dependent first-order logic guided adversarial example generation and regularization. @xcite introduce consistency-based regularization incorporating the first-order logic rules. previous approach is modeldependent or relies on nli-specific rules, while our method is model-agnostic and is more generally applicable by combining it with data augmentation. regularizing loss to penalize violations of structural constraints in models' output has been also studied in previous work on constraint satisfaction in structured learning @xcite . our work regularizes models to produce globally consistent predictions among augmented data following logical constraints, while those studies incorporates structured prediction models following linguistics rules. ## method we present the components of our qa method: first-order logic guided data augmentation (section 3.1 and section 3.2), and consistency-based regularization (section 3.3). ## conclusion we introduce a logic guided data augmentation and consistency-based regularization framework for accurate and globally consistent qa, especially under limited training data setting. our approach significantly improves the state-of-the-art models across three substantially different qa datasets. notably, our approach advances the state-of-the-art on quarel and wiqa, two standard benchmarks requiring rich logical and language understanding. we further show that our approach can effectively learn from extremely limited training data.