A Comprehensive Evaluation of Inductive Reasoning Capabilities and Problem Solving in Large Language Models (2024)

Abstract: Inductive reasoning is fundamental to both human and artificial intelligence. The inductive reasoning abilities of current Large Language Models (LLMs) are evaluated in this research.We argue that only considering induction of rules is too narrow and unrealistic, since inductive reasoning is usually mixed with other abilities, like rules application, results/rules validation, and updated information integration.We probed the LLMs with a set of designed symbolic tasks and found that even state-of-the-art (SotA) LLMs fail significantly, showing the inability of LLMs to perform these intuitively simple tasks.Furthermore, we found that perfect accuracy in a small-size problem does not guarantee the same accuracy in a larger-size version of the same problem, provoking the question of how we can assess the LLMs’ actual problem-solving capabilities.We also argue that Chain-of-Thought prompts help the LLMs by decomposing the problem-solving process, but the LLMs still learn limitedly.Furthermore, we reveal that few-shot examples assist LLM generalization in out-of-domain (OOD) cases, albeit limited. The LLM starts to fail when the problem deviates from the provided few-shot examples.

## introduction recently, the development of llms has made great progress in various areas of artificial intelligence (ai), especially in natural language processing (nlp). the performance of llms like gpt-3.5 @xcite and @xcite can even outperform humans on some professional tests, proving their ability to understand and solve complex natural language questions. one of the intriguing abilities of llms is reasoning, which is also one of the core abilities of human intelligence. reasoning, following this definition @xcite , consists of deductive reasoning @xcite , inductive reasoning @xcite , and abductive reasoning @xcite . llms show surprisingly high performance on tasks requiring high-level reasoning ability, like programming @xcite and mathematical problem solving @xcite . however, as the llms memorize the statistical word co-occurrences from the pre-training corpora containing such examples, it is hard to know the real reasoning ability of llms as they always generate specious answers. therefore, evaluation at a fundamental level, e.g. symbolic level, is needed to accurately understand the reasoning abilities of llms. this research focuses on inductive reasoning, which is the ability to derive common principles from finite observations. recent inductive reasoning research in nlp @xcite focused mainly on rules induction from observations, but inductive reasoning in the real world is more complex than just rules induction. as inductive reasoning is based on finite observations, which may contain only partial information, we cannot always expect the induced rules or results to be fully correct. therefore, in the real world, under the surface of rules induction, the ability to validate induced rules/results and merge new rules with previous rules is equally important, and such ability to adapt to changing circumstances is important for building ai models suitable for real-world usage. to evaluate these abilities, we designed three symbolic tasks: 1) grouping polygons, 2) ordering named colors (color ordering), and 3) shifting characters in english text @xcite . we then define 3x5 experiments called rules application, rules induction, results validation, rules validation, and rules incorporation to evaluate the ability to apply rules, induce rules, validate induced results/rules, and merge new rules with previous rules, as depicted in figure 1 . we observe the llms failing on these tasks. subsequent experiments explored the role of few-shot examples for generalization, the scalability of llm perfor-mance with problem size, and the impact of the chain-of-thought prompts, namely: ## conclusion in this research, through designed symbolic probing tasks, we probed the llms' abilities centered around inductive reasoning, including rules induction, rules application, results/rules validation, and rules incorporation. we found that llms fail to correctly induce or apply rules in simple symbolic tasks and cannot or even fail to validate the correctness of results/rules or identify and merge new rules given new results. this suggests that not just improving prediction accuracy, but also making the llm identify what is correct and wrong, and being able to identify new information from new examples are important. our experiments show that near-perfect accuracy in small-sized tasks does not imply that llm performance scales well to a larger sized task. in this sense, it raises the question: how can we prove that the llm knows how to solve a problem/task? we also notice that few-shot examples help the model to generalize to unseen situations, but do not make the model able to solve the problem in all situations. through the cot-enhanced prompt, we see a significant performance improvement, stating that cot helps the model to understand the scalable solution of a task in which the cot prompt exposed more information about scalable solutions.