How much coffee was consumed during EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AI (2021)

Abstract: Many real-world problems require the combined application of multiple reasoning abilities—employing suitable abstractions, commonsense knowledge, and creative synthesis of problem-solving strategies. To help advance AI systems towards such capabilities, we propose a new reasoning challenge, namely Fermi Problems (FPs), which are questions whose answers can only be approximately estimated because their precise computation is either impractical or impossible. For example, “How much would the sea level rise if all ice in the world melted?” FPs are commonly used in quizzes and interviews to bring out and evaluate the creative reasoning abilities of humans. To do the same for AI systems, we present two datasets: 1) A collection of 1k real-world FPs sourced from quizzes and olympiads; and 2) a bank of 10k synthetic FPs of intermediate complexity to serve as a sandbox for the harder real-world challenge. In addition to question-answer pairs, the datasets contain detailed solutions in the form of an executable program and supporting facts, helping in supervision and evaluation of intermediate steps. We demonstrate that even extensively fine-tuned large-scale language models perform poorly on these datasets, on average making estimates that are off by two orders of magnitude. Our contribution is thus the crystallization of several unsolved AI problems into a single, new challenge that we hope will spur further advances in building systems that can reason.

## introduction how long is the drive from seattle to nyc? how big of an emergency fund do i need? we frequently encounter such questions in our daily lives. likewise, scientists are often faced with questions such as, how much would the ocean surface rise if the ice caps melted? known as fermi problems foot_0 (fps), these questions are problems whose answers can only be estimated within reasonable limits of error, as precisely measuring the required quantity is either impossible or impractical. solving a fp requires considerable life experience, ability to think through long chains of reasoning, and mathematical intuition -as illustrated by fig. 1 for a fp about rising sea levels. answering an fp correctly requires multiple different facts, and the correct estimate can be arrived via various reasoning paths -this open-ended nature further adds to their challenge. unsurprisingly, these questions are often used to test candidates in science olympiads and interviews. due to the complexity of reasoning that is required to answer these questions, we propose solving fps as a new task to drive progress in ai reasoning systems. a core skill required for solving fps is that of estimatione.g. "how much money do i need for a medical emergency?", "what is the thickness of ice sheets in antarctica", etc. this crucially requires abstracting out details to simplify a complex realworld problem, similar to the (in)famous metaphor of the spherical cow, 2 for instance, when estimating the volume of mt. everest foot_2 , abstracting that the mountain is a conical shaped object significantly simplifies the estimation problem at hand. to the best of our knowledge, our proposed fp challenge is the first of its kind that requires reasoning of this nature. through this challenge, we hope to spur research towards building ai reasoning systems that are capable of performing such abstractions, a key reasoning skill that is natural to humans. the complex reasoning involved in solving fps (see fig. 1 ) often means the question must be creatively decomposed into simpler ones. these simpler questions often themselves are open-ended fermi problems. we thus hope our challenge will encourage advances in recursive reasoning models. further, fps require the combined application of multiple reasoning strategies to solve the problem. unlike existing datasets and tasks geared towards specific reasoning skills (e.g. commonsense reasoning or question decomposition), we hope our work drives progress in not just the ability of ai to employ suitable abstractions and estimations, but also in models that can combine various reasoning abilities to produce a coherent solution. ## fermi problems the following properties of fps and their solutions make them an ideal candidate for evaluating and advancing ai reasoning -(1) recursive nature of sub-problems. as mentioned previously, problem decomposition is an important aspect of fps. an interesting property of fps is that decomposed sub-problems are also fps -e.g. "how many dump trucks to empty mt. everest?" requires answering the -"what is the volume of mt. everest?" and "what is the volume of a dump truck?", which are in-turn, fps. we employ this property of fps to create a richer synthetic dataset (see section 4 for more details). arriving at the correct answer requires one to make reasonable abstractions at each step. this requires a sufficiently accurate working model of world and is broadly categorized as life experience. for example -the fact that a mars bar can be eaten in a few bites can help determine its volume. similarly, understanding that pizza shops usually cater to homes within a few mile radius helps in estimating the number of pizza delivery persons in chicago. further, domain-specific reasoning might be required to solve some fps -for example fp illustrated in fig. 1 requires an understanding of physics to infer that only land ice leads to increase in sea levels. question decomposition fps require problem decomposition, in a way loosely similar to multihop inference. however, for fps, the appropriate decomposition is not explicit in the question itself, unlike early multihop datasets such as hotpotqa @xcite or webquestions @xcite . later multihop datasets, e.g., obqa @xcite , contained questions where the decomposition was not explicit in the question (e.g., "does a suit of armor conduct electricity?", implicitly requiring a subquestion about materials), but typically into just two (or at most three) steps. in contrast, fps typically require multiple levels of decomposition, significantly increasing complexity. this in turn requires identifying a solution strategy, namely how to factor an unknown quantity into a function of known (or recursively factorable) quantities. the strategyqa @xcite dataset illustrates this problem but for a different task, namely true/false questions about whether something is possible, and without recursive decomposition, a key feature of fps. commonsense in addition to mathematical reasoning, fps require significant commonsense knowledge, both for estimating quantities and for decomposing problems. for example, "how many pizza delivery trucks are in chicago?" requires significant commonsense about human behavior (how often do people order pizza? how many deliveries can a truck make per day?) to even begin to decompose the problem, let alone estimating basic quantities (population of chicago?). while new resources of commonsense knowledge are becoming available, e.g., @xcite , substantial development is still needed for the kind of world modeling that many fps require. numeric estimation large-scale language models trained on web-scale data have been shown to contain common numerical facts -e.g. number of days in a year, distance from earth to moon, number of hairs on a human head, etc. we leverage one such model (t5 @xcite ) for our baselines. more recently, researchers have shown that models can also perform estimation to some degree @xcite , and have proposed novel encoding strategies to improve number prediction and estimation @xcite . such techniques would be valuable for improved solutions to fps. ## datasets and tasks we present two datasets, realfp and synthfp, which are collections of real-world and synthetic fermi problems, respectively. we then define three fp challenge tasks, with varying difficulty levels. ## experiments we describe some baseline approaches to solve the fp challenge tasks, and report their performance on the test sets of the two datasets. ## conclusion in this work, we propose fermi problems (fps) as a reasoning challenge for ai systems. apart from introducing abstraction as a crucial reasoning skill, our work requires the combined application of various reasoning skills including creative decomposition of problems, commonsense reasoning, mathematical reasoning, etc. we collect two datasets -realfp with ∼1k real-world questions and syn-thfp with 10k templated questions. based on these datasets, we propose three concrete tasks of increasing difficulty that encompass the fp challenge. the baseline models we provide, despite being based on state-of-the-art language models and even with substantial fine-tuning, struggle on our challenge tasks. they are, on average, off by two orders of magnitude from the correct estimate and perform only slightly better than predicting a constant number. we thus hope to establish fermi problems as a hard reasoning challenge that motivates further advances in ai reasoning systems.