FOLIO : Natural Language Reasoning with First-Order Logic (2024)

Abstract: Large language models (LLMs) have achieved remarkable performance on a variety of natural language understanding tasks. However, existing benchmarks are inadequate in measuring the complex logical reasoning capabilities of a model. We present FOLIO, a human-annotated, logically complex and diverse dataset for reasoning in natural language (NL), equipped with first-order logic (FOL) annotations. FOLIO consists of 1,430 examples (unique conclusions), each paired with one of 487 sets of premises used to deductively reason for the validity of each conclusion. The logical correctness of the premises and conclusions is ensured by their FOL annotations, which are automatically verified by an FOL inference engine. In addition to the main NL reasoning task, NL-FOL pairs in FOLIO constitute a new NL-FOL translation dataset. Our experiments on FOLIO systematically evaluate the FOL reasoning ability of supervised fine-tuning on medium-sized language models. For both NL reasoning and NL-FOL translation, we benchmark multiple state-of-the-art language models. Our results show that a subset of FOLIO remains a challenge for one of the most capable Large Language Model (LLM) publicly available, GPT-4.

## introduction large language models (llms) have achieved remarkable performance on a variety of natural language tasks @xcite @xcite . logical reasoning is a central component for intelligent systems and should be sufficiently and independently evaluated @xcite . however, existing natural language tasks are inadequate in measuring the complex logical reasoning capability of a model @xcite @xcite . several datasets related to logical reasoning have recently been proposed. however, existing benchmarks often exhibit limited complexity in reasoning or lack language naturalness. some of these common benchmarks do not specifically evaluate logical reasoning independently of other forms of reasoning @xcite . those specifically designed for measuring logical reasoning are insufficient in terms of logical reasoning complexity and natural language variety. as shown in we present a natural language reasoning dataset, folio, with first-order logic reasoning problems which require the models to decide the correctness of conclusions given a world defined by the premises. in folio, we aim to ensure high lanfolio is the first expert-written dataset for fol reasoning equipped with parallel fol formulas. the examples are mostly aligned with real-world knowledge and use highly natural wordings. it also has a greater variety than the previous datasets in terms of reasoning depths with a larger number of distinct logic patterns and a large vocabulary. a folio example based on the wild turkey wikipedia page: https://en.wikipedia.org/wiki/wild_turkey nl premises nl conclusions -> labels 1. there are six types of wild turkeys: eastern wild turkey, osceola wild turkey, gould's wild turkey, a. tom is an ocellated wild turkey. -> true merriam's wild turkey, rio grande wild turkey, and the ocellated wild turkey. b. tom is an eastern wild turkey. -> false 2. tom is not an eastern wild turkey. c. joey is a wild turkey. -> unknown 3. tom is not an osceola wild turkey. 4. tom is also not a gould's wild turkey. 5. tom is neither a merriam's wild turkey, nor a rio grande wild turkey. 6. tom is a wild turkey. c. wildturkey(joey) -> unknown 3. ¬osceolawildturkey(tom)) 4. ¬gouldswildturkey(tom) 5. ¬merriamswildturkey(tom) ∧ ¬riograndewildturkey(tom) 6. wildturkey(tom) under the few-shot setting, the most capable publicly available llm so far achieves only 53.1% on the stories written in a hybrid manner, which is slightly better than random. to sum up, the contributions of this paper are threefold. 1) we release a natural language reasoning dataset written by expert annotators, folio, with first-order logical reasoning problems. 2) we use formal logic, i.e., fol to ensure the logical validity of the examples written in nl and propose a new nl-fol translation task. 3) we benchmark the performance of lms by fine-tuning models and prompting llms with few-shot examples, on the folio reasoning task. we hope that folio, as a challenging logical reasoning dataset, will be used to facilitate measuring progress in the logical reasoning capabilities of language models. ## folio corpus construction we collected folio through a carefully designed manual annotation process to achieve high-quality examples that necessitate complex logical reasoning. writing natural language reasoning stories with fol requires sufficient knowledge in both semantic parsing and first-order logic, as well as strong analytical skills. given the complexities of such annotations, we selected annotators based on a few important criteria to ensures that our dataset is annotated with the highest level of precision and expertise, reflecting the complexity and nuance required for first-order logical reasoning. 1). our annotators are either college or graduate students who are native english speakers or possess nearnative proficiency in english.4 2). they possess formal education in first-order logic, having either completed relevant coursework or undertaken self-directed studies in first-order logic or seman-tic parsing. at the nl quality check stage, only annotators who are experts in natural language processing or computational linguistics are involved. for the fol quality check, only annotators who are experts in first-order logic are involved. we also give the annotators several training sessions on how to write a story, by providing them with detailed annotation guidelines. all stories and fol annotations in folio are written and reviewed by expert annotators, including cs undergraduate and graduate students, and senior researchers, who met the aforementioned criteria. we develop our dataset in six stages: wikilogic collection, hyblogic collection, nl quality control, fol quality control, nl-fol alignment and fol verification, spending 980 man-hours in total. ## task definition we define two new tasks based on folio, natural language reasoning with first-order logic and nl-fol translation. ## experiments in this section, we describe our experiments and main results. ## error analysis below we provide analysis of our results and key findings, providing additional insights into our dataset folio and the current capabilities of llms in logical reasoning. models have higher accuracy on examples with fewer reasoning depths than on those with higher number of reasoing depths we show the accuracy categorized by reasoning depths in figure 2. with few-shot prompting, gpt-3.5 and gpt-4 both perform much better on examples with a 0 ∼ 3 reasoning depth, indicating that examples with a 4 ∼ 7 reasoning depth pose a challenge to the sota lms. with fine-tuning, roberta has slightly higher performance on test examples with 0 ∼ 3 reasoning depth than on those with 4 ∼ 7 reasoning depth, but the difference is much smaller. this indicates that fine-tuning on longer and more difficult reasoning chains in the training set can improve model performance on equally-long test example chains. we note that the presence and prevalence of these difficult examples are unique to folio. folio's unique complexity reveals that current lms are limited in their ability to extrapolate to longer and more complex reasoning chains, and suggests an avenue for further study. ## conclusion we introduced folio, an expert-written dataset for logical reasoning equipped with fol formulas. the examples in folio are created based on real-world knowledge with natural language. it exhibits a large number of distinct logic patterns and a large vocabulary. experiments show that folio presents a challenge for one of the most capable large language model publicly available.