“John is 50 years old, can his son be 65?” Evaluating NLP Models’ Understanding of Feasibility (2023)

Abstract: In current NLP research, large-scale language models and their abilities are widely being discussed. Some recent works have also found notable failures of these models. Often these failure examples involve complex reasoning abilities. This work focuses on a simple commonsense ability, reasoning about when an action (or its effect) is feasible. To this end, we introduce FeasibilityQA, a question-answering dataset involving binary classification (BCQ) and multi-choice multi-correct questions (MCQ) that test understanding of feasibility. We show that even state-of-the-art models such as GPT-3, GPT-2, and T5 struggle to answer the feasibility questions correctly. Specifically, on (MCQ, BCQ) questions, GPT-3 achieves accuracy of just (19%, 62%) and (25%, 64%) in zero-shot and few-shot settings, respectively. We also evaluate models by providing relevant knowledge statements required to answer the question and find that the additional knowledge leads to a 7% gain in performance, but the overall performance still remains low. These results make one wonder how much commonsense knowledge about action feasibility is encoded in state-of-the-art models and how well they can reason about it.

## introduction commonsense reasoning has been a key aspect of ai since its advent in the 1950s. it is closely associated with reasoning about actions and their effects, which is considered a big challenge, especially for deep learning-based ai approaches and language models @xcite @xcite . while several datasets have been developed to evaluate large-scale language models, in this paper, we propose a dataset focused on reasoning about actions and their effects; specifically, the ability to reason if an action or its effect is feasible. figure 1 illustrates an example of our dataset; given the information "sam gave 50 dollars to the shopkeeper to buy a book and the shopkeeper returned some money", it is not possible to compute the exact price of the book; however, it can be established that the feasible price of the book is less than 50 since the shopkeeper returned some money. we often come across such questions in our daily life and find it trivial to reason about them. therefore, in order to develop nlp systems that can reliably reason about real-world situations, it is important to evaluate their understanding of feasibility. recently, many datasets have been created that test different reasoning skills such as pronoun resolution @xcite , commonsense reasoning @xcite @xcite , numerical reasoning @xcite @xcite @xcite @xcite , qualitative reasoning @xcite , discrete reasoning @xcite , and temporal reasoning @xcite . however, they do not have an ample number of examples that test understanding of feasibility. category knowledge context questions attribute comparison larger volume holds more amount of water. barrett's has two cylindrical shaped bottles. a bottle with a higher volume holds 32 units of water. amount of water in other bottle could be 28 units. amount of water in other bottle could be 33 units. what could be the amount of water in other bottle? change with time age increases with time today, while filling the application form edward filled the age field with 16. edward could have been 8 years old on his last birthday. edward could have been 19 years old on his last birthday. what could have been edward's age last year? change with action selling something reduces its quantity joshua organized a garage sale yesterday. joshua sold a total of 273 items at a minimum price of 1 usd each. joshua could have made 300 dollars in the garage sale. joshua could have made 260 dollars in the garage sale. how much money joshua could have made from the garage sale? implicit numerical knowledge 4 quarters make 1 dollar christopher is accepting quarters for a donation and fails to collect 12 dollars. he could have collected 35 quarters. he could have collected 52 quarters. how many quarters could christopher have collected? non numerical new movies can be watched after the release date. the latest superhero movie was releasing on 28th february 2022. ashton wanted to see the movie. he could have watched the movie on 2nd march 2022. he could have watched the movie on 3rd february 2022. when could ashton have watched the movie ? in this work, we address the above limitation and introduce feasibilityqa, a dataset consisting of questions that require an understanding of feasibility. this dataset comprises of two types of questions: binary classification (bcq) and multichoice multi-correct questions (mcq). in bcq, the task is to determine whether the question is feasible or not given a context; in mcq, the task is to select all feasible answers to the given question. the dataset consists of ∼5k instances covering diverse aspects of feasibility. we conduct comprehensive experiments with gpt-3, gpt-2, and t5 models @xcite @xcite in zeroshot and few-shot settings and show that all of these models struggle to correctly answer feasibility questions. specifically, on (mcq, bcq) questions, gpt-3 achieves an accuracy of just (19%, 62%) and (25%, 64%) in zero-shot and few-shot settings, respectively. prior work has found that explicitly providing relevant knowledge helps the model reason better and improves its performance @xcite @xcite . we explore this aspect of reasoning by explicitly providing relevant knowledge statements and find that it leads to ∼7% improvement in performance. however, the overall performance still remains low. we further investigate gpt-3's ability to reason about feasibility questions by prompting it to generate the reasoning chain. in many cases, we find that gpt-3 successfully generates the correct reasoning chain but still fails to output the correct answer. this analysis further leads to several interesting findings (section 3). ## conclusion in this work, we proposed feasibilityqa, a question-answering dataset that evaluates the understanding of feasibility. we conducted extensive experiments with several state-of-the-art models in zero-shot, one-shot, and few-shot settings and show that these models struggle to answer the feasibility questions correctly. we also experimented by providing additional knowledge (relevant to the question) and showed that it leads to a small gain in performance, but the overall performance still remains low. we further analyzed the performance of models that reveals several interesting findings. finally, we release our dataset and hope that our work will encourage further research in feasibility reasoning, an important yet underexplored aspect of commonsense reasoning.