A bduction R ules: Training Transformers to Explain Unexpected Inputs (2022)

Abstract: Transformers have recently been shown to be capable of reliably performing logical reasoning over facts and rules expressed in natural language, but abductive reasoning - inference to the best explanation of an unexpected observation - has been underexplored despite significant applications to scientific discovery, common-sense reasoning, and model interpretability. This paper presents AbductionRules, a group of natural language datasets designed to train and test generalisable abduction over natural-language knowledge bases. We use these datasets to finetune pretrained Transformers and discuss their performance, finding that our models learned generalisable abductive techniques but also learned to exploit the structure of our data. Finally, we discuss the viability of this approach to abductive reasoning and ways in which it may be improved in future work.

## introduction since its introduction, models based on the transformer @xcite have, due to their learning ability and turing-completeness @xcite , sparked research into their use in many applications beyond their original purpose of natural language processing (nlp), including image processing and generation @xcite , theorem proving @xcite , and chess @xcite . one such task is logical inference -reasoning over first-order logic (fol) knowledge bases (collections of facts and rules). given a knowledge base, one may attempt to find logical implications (deduction), discover rules that extrapolate patterns in known facts (induction), or infer facts that would explain surprising observations (abduction). more specifically, if a newly observed fact p cannot be deduced from an existing knowledge base, abduction is the process of finding one or more facts that, if added to the knowledge base, would allow p to be deduced from existing rules. figure 1 demonstrates the difference between these three kinds of inference. traditionally, fol is represented using a formal mathematical syntax, with facts resembling human(socrates) and rules resembling ∀x : human(x) =⇒ mortal(x). @xcite recently pioneered an alternative approach we call natural-language logic, which might represent these as "socrates is human" and "humans are mortal". this approach, properly followed, retains the precision of the mathematical syntax while also taking advantage of transformers' nlp aptitude and pretraining. this approach also allows reasoning over texts not written in formal representations. @xcite examined their models' potential for deduction only. @xcite extended this work to explore abduction but retained a focus on deduction. our goal is to train transformers to perform abductive reasoning with the following properties: deduction: socrates is @xmath0: socrates is @xmath1: ? → humans are @xmath2. we present the following contributions: ## methodology prior to our work, there existed no dataset capable of training or testing the kind of abductive reasoning we seek. we therefore present abduction-rules, a natural-language logic dataset designed for this task, and use it to train and test several models based on a pretrained text-to-text transfer transformer, or t5 @xcite . ## results ## discussion our results show that models trained on our simplest datasets struggle to generalise to new complexity levels and domains, while those trained on our more complex datasets are better able to generalise but still perform suboptimally. meanwhile, those trained on combined multi-domain datasets achieve performance superior to the sum of models trained on their parts and easily apply skills outside domains in which they were learned. it is also clear that models trained in the animal-domain achieve better intra-domain and inter-domain performance than person-domain models. the untrained t5 model performs abysmally and merits little discussion, indicating that this abduction task is non-trivial (at least in the way we present it here). ## conclusion we have presented the abductionrules datasets and shown that pretrained t5 models finetuned on them exhibit generalised abductive reasoning. our more complex datasets train abduction more generally and reliably than our less complex datasets. further, training in multiple domains is superior to training in only one domain, and we have clear evidence of generalisation of techniques from one domain to another. we have presented an innovation in naturallanguage logic dataset generation, presenting a new middle-ground between the template-based pararule plus @xcite and the manually rephrased pararules @xcite . we believe our results are promising and demonstrate the viability of transformer-based abduction (and logical reasoning in general), but also indicate opportunities for improvement.