Flexible Generation of Natural Language Deductions (2021)

Abstract: An interpretable system for open-domain reasoning needs to express its reasoning process in a transparent form. Natural language is an attractive representation for this purpose — it is both highly expressive and easy for humans to understand. However, manipulating natural language statements in logically consistent ways is hard: models must cope with variation in how meaning is expressed while remaining precise. In this paper, we describe ParaPattern, a method for building models to generate deductive inferences from diverse natural language inputs without direct human supervision. We train BART-based models (Lewis et al., 2020) to generate the result of applying a particular logical operation to one or more premise statements. Crucially, we develop a largely automated pipeline for constructing suitable training examples from Wikipedia. We evaluate our models using out-of-domain sentence compositions from the QASC (Khot et al., 2020) and EntailmentBank (Dalvi et al., 2021) datasets as well as targeted perturbation sets. Our results show that our models are substantially more accurate and flexible than baseline systems. ParaPattern achieves 85% validity on examples of the ‘substitution’ operation from EntailmentBank without the use of any in-domain training data, matching the performance of a model fine-tuned for EntailmentBank. The full source code for our method is publicly available.

## introduction developing models that can make useful inferences from natural language premises has been a core goal in artificial intelligence since the field's early days @xcite . since then, there has been massive progress in automated formal reasoning @xcite ; in contrast, progress in automated 1 https://github.com/alephic/parapattern the green bay packers play in the nfl. nfl teams play from september to january. ## methods we consider an operation g, like our substitution example (fig. 1 ), to be analogous to a proof rule, allowing one or more premise statements to be combined and transformed to yield a valid conclusion statement. a model for g places a distribution p g (y | x 0 , . . . , x n ) over conclusions y conditioned on premises x i . we would like models to satisfy three criteria: consistency: predicted outputs should be valid deductions from the model's inputs. robustness: models should be robust to linguistic variation present in their inputs. ## conclusion building systems that use natural language as a medium for reasoning will require operations to logically combine and transform natural language statements. in this work, we present parapattern, a method for creating such models with minimal manual effort by finetuning pretrained sequence-to-sequence language models on data generated through a three-step process of syntactic retrieval, template expansion, and automatic paraphrasing. our experimental results show that parapattern yields operation models capable of generating consistent logical transformations over a diverse range of natural language inputs, matching the performance of models trained with in-domain human supervision.