Language Models as Inductive Reasoners (2024)

Abstract: Inductive reasoning is a core component of human intelligence. In the past research of inductive reasoning within computer science, formal language is used as representations of knowledge (facts and rules, more specifically). However, formal language can cause systematic problems for inductive reasoning such as disability of handling raw input such as natural language, sensitiveness to mislabeled data, and incapacity to handle ambiguous input. To this end, we propose a new paradigm (task) for inductive reasoning, which is to induce natural language rules from natural language facts, and create a dataset termed DEER containing 1.2k rule-fact pairs for the task, where rules and facts are written in natural language. New automatic metrics are also proposed and analysed for the evaluation of this task. With DEER, we investigate a modern approach for inductive reasoning where we use natural language as representation for knowledge instead of formal language and use pretrained language models as ”reasoners”. Moreover, we provide the first and comprehensive analysis of how well pretrained language models can induce natural language rules from natural language facts. We also propose a new framework drawing insights from philosophy literature for this task, which we show in the experiment section that surpasses baselines in both automatic and human evaluations. We discuss about our future perspectives for inductive reasoning in Section 7. Dataset and code are available at https://github.com/ZonglinY/Inductive_Reasoning.

## introduction inductive reasoning is to reach to a hypothesis (usually a rule that explains an aspect of the law of nature) based on pieces of evidence (usually observed facts of the world), where the observations can not provide conclusive support to the hypothesis @xcite . it is ampliative, which means * contribution during internship at microsoft research. past research works on inductive reasoning within computer science are investigated by inductive logic programming (ilp) @xcite . ilp investigates the inductive construction of first-order logic (fol) @xcite rules from examples and background knowledge @xcite . however, ilp uses formal language as representation and uses symbolic reasoner, which results in systematic disadvantages @xcite . specifically, ilp systems heavily rely on human effort, since it typically assumes that the input has already been preprocessed into symbolic declarative form, otherwise ilp systems cannot handle raw inputs such as natural language and images. in addition, ilp systems are very sensitive to label error and ambiguity in data, since the final induced rules are required to satisfy all input facts, and symbolic systems can not recognize different symbols with the same meaning (e.g. be capable of, be able to). to overcome the challenges above, we present a novel paradigm for inductive reasoning based entirely on natural language, i.e., inducing natural language rules from natural language facts. in particular, we create a first-of-its-kind natural language inductive reasoning dataset named deer containing 1.2k rule-fact pairs (more details illustrated in §3.1). with this dataset, we investigate short fact 1 short fact 2 short fact 3 rule the venus flytrap is a carnivorous plant native to subtropical wetlands on the east coast of the united states in north carolina and south carolina. it catches its prey-chiefly insects and arachnids-with a trapping structure formed by the terminal portion of each of the plant's leaves, which is triggered by tiny hairs on their inner surfaces. pitcher plants are several different carnivorous plants which have modified leaves known as pitfall traps-a prey -trapping mechanism featuring a deep cavity filled with digestive liquid. the traps of what are considered to be "true" pitcher plants are formed by specialized leaves. the plants attract and drown their prey with nectar. drosera, which is commonly known as the sundews, is one of the largest genera of carnivorous plants, with at least 194 species. the trapping and digestion mechanism of drosera usually employs two types of glands: stalked glands that secrete sweet mucilage to attract and ensnare insects and enzymes to digest them, and sessile glands that absorb the resulting nutrient soup. if a plant is carnivorous , then it probably has a trapping structure. a modern approach to inductive reasoning where both facts and rules are in natural language, and pretrained language models (plms) are used as the inductive reasoner. note that the inductive reasoning considered in this paper has several distinctions considered by other reasoning tasks over text @xcite @xcite . we defer a more detailed discussion to §2. with natural language as representation and plms as the reasoner, such an inductive reasoning system can avoid the systematic disadvantages of formal language and symbolic reasoners. specifically, with natural language as representation, it can naturally handle raw input as natural language text. in addition, different from symbolic methods, plms contain knowledge via pretraining @xcite and use embedding for concepts @xcite , making it less affected by input errors @xcite and more robust to paraphrasing. based on the proposed dataset, we study the plm's ability to induce (generate) natural language rules from natural language facts under different settings, such as different fol rule types and topics with varying input facts and plm model sizes. we also propose a new framework for this task, named chain-of-language-models (colm) which is shown in figure 1 . it draws insights from the requirements of rule induction in philosophy literature @xcite . specifically, colm consists of five modules all based on plms, where one model proposes rules (rule proposer m1), and the other four models (m2, m3, m4, m5) each classify whether a generated rule satisfies one particular requirement of induction. in our experiments, we find that our framework surpasses the baselines in terms of both automatic and human evaluations. to sum up, our contributions are three-fold: in §7 we discuss our future perspectives on inductive reasoning in detail. ## related work definition of inductive reasoning it is still under debate on the definition of inductive reasoning in philosophy research @xcite . here we adopt @xcite 's view that an inductive argument should satisfy (1) its premise cannot provide conclusive support to its conclusion since its conclusion amplify or go beyond the information found in their premises; (2) its conclusion generalize over its premise in a way that the conclusion can be applied to more instances other than instances mentioned in its premise. an example of inductive argument is that "if a white ball is found in a bag, then all balls in this bag are white." in this paper, we call the premises as "facts", and conclusions as "rules". prior computa-rule template @xcite rule template (natural language) ## dataset collection and new metrics in this section, we discuss the data collection process for our proposed dataset, and our proposed metrics for automatic and human evaluation. in general, we propose two datasets. the first one, named deer (inductive reasoning with natural language representation), contains 1.2k rulefact pairs, where rules are written by human annotators in english, and facts are existing english sentences on the web. the other one, named deer-let (classification of induced rules with natural language representation), including (fact, rule, label0, label1, label2, label3) tuples, where facts are the same as in deer, rules are generated output from plms, and label0/1/2/3 are classification labels describing different aspects of induced rules. specifically, rules in deerlet are collected from gpt-j @xcite using the in-context learning setting. we choose this setting because (1) gpt-j in this setting can generate reasonable rules, and (2) not all generated rules are correct so that the annotations on the generated rules can be used for fine-tuning. overall, deer is used as the main dataset for the task, and deerlet is used to measure the classification performance of specific capabilities described in §3.2. ## methodology in this section, we formally present the task definition and our proposed framework for natural language inductive reasoning. figure 1 illustrates the general architecture of our proposed approach. ## experiments in this section, we discuss the evaluation metrics and baselines, and then present the main results of our framework (all are averaged by 5 runs). ## analysis in this section, we investigate the question of "how well can pretrained language models perform inductive reasoning?". specifically, we provide analyses in terms of rule types, topics, variations of input fact, and scales of language models. except for ## overview and future perspectives of inductive reasoning the first version of this paper was finished in 2022. at that time, inductive reasoning-in the sense of deriving explicit natural language hypotheses (rules) from observations (input facts), where the hypotheses and observations adhere to specific relations defined by induction-was a new and unexplored research area. previously, the most closely related works came from the ilp (inductive logic programming) community, which focused on symbolic approaches to the task of inductive reasoning (inducing explicit formal language hypotheses). this paper aims to act as a bridge between the ilp and nlp communities by (1) demonstrating how natural language and related techniques (foundation models) can address challenges within the ilp community, and (2) introducing the definition and task of inductive reasoning to nlp. moreover, this paper can serve as a preliminary study, suggesting that language models have the potential to function as inductive reasoners. the transcription of requirements for inductive arguments from philosophical literature, as illustrated in section 4.2, could remain useful even in the era of powerful llms. the possible future challenges of research on inductive reasoning include (1) establishing and solving more challenging tasks for inductive reasoning, and (2) overcoming the fundamental challenges inherent in induction. ## conclusion to overcome the systematic problems of using formal language for inductive reasoning, we propose a new paradigm (task) of inducing natural language rules from natural language facts, and correspondingly propose a dataset deer and new evaluation metrics for this task. we provide the first and comprehensive analysis of plm's ability to induce natural language rules from natural language facts. we also propose a new framework, drawing insights from philosophical literature, which, as shown in the experimental section, surpasses baselines in both automatic and human evaluations.