FR o G : Evaluating Fuzzy Reasoning of Generalized Quantifiers in LLM s (2024)

Abstract: Fuzzy reasoning is vital due to the frequent use of imprecise information in daily contexts. However, the ability of current large language models (LLMs) to handle such reasoning remains largely uncharted. In this paper, we introduce a new benchmark, FRoG, for fuzzy reasoning, featuring real-world mathematical word problems that incorporate generalized quantifiers. Our experimental findings reveal that fuzzy reasoning continues to pose significant challenges for LLMs. Moreover, we find that existing methods designed to enhance reasoning do not consistently improve performance in tasks involving fuzzy logic. Additionally, our results show an inverse scaling effect in the performance of LLMs on FRoG. Interestingly, we also demonstrate that strong mathematical reasoning skills are not necessarily indicative of success on our benchmark.

## introduction the capability to reason effectively is a critical dimension in evaluating the advancements of large language models (llms) @xcite @xcite @xcite . commonly, the progress of advancing reasoning abilities is mainly evaluated on mathematical reasoning benchmarks @xcite like gsm8k @xcite and math @xcite , which demands precise answers derived from clear, rule-based questions. however, much of human knowledge and daily decision-making processes are not based on precision but rather involve handling fuzzy, imprecise information @xcite . fuzzy reasoning, which deals with uncertainties and perceptual data, differs significantly from the deterministic processes evaluated in existing standard benchmarks. it often relies on natural language constructs that do not capture information with precise granularity @xcite . for instance, generalized quantifiers (gqs), such as "few" or "most", are frequently used in natural language to introduce vagueness @xcite ). an illustrative fuzzy problem employing gq might be: "there have been successive increases of 20% and then most in the price of gas from the previous month. by what percentage should a driver reduce gas consumption so that the expenditure does not change?" here, the term "most" introduces ambiguity concerning the extent of the price increase and necessitates an estimation of its semantics to solve the problem accurately. however, such gq-based fuzzy reasoning problems are under-explored. faghihi et al. (2020) introduces probabilistic fuzzy logic @xcite to enhance reasoning abilities. existing works like @xcite investigate employing fuzzy probabilistic theory in associationbased problems, like the relation between fuzzy features and labels (e.g. a lot of smoking and cancer). the mapping functions to encode fuzzy features are either simulated @xcite or collected from limited data with heuristics @xcite . those mapping functions fail to evaluate complex, real-world reasoning scenarios and realistically capture how llms utilize fuzzy reasoning in practical contexts. in this paper, we aim to explore reasoning challenges associated with fuzzy events (zadeh, 1968), which are mathematically ambiguous and articulated through fuzzy expressions of gqs, such as "most in the price of gas". specifically, we focus on mathematical reasoning problems that incorporate gqs. to this end, we have developed frog, a benchmark for fuzzy reasoning benchmark of generalized quantifiers, which transforms real-world mathematical problems from gsm8k @xcite and mathqa (amini step 1: find math question with percentage mentions question: here has been successive increases of 20% and then 10% in the price of gas from the previous month. by what percentage should a driver reduce gas consumption so that the expenditure does not change? step 2: mask the percentage mention answer: 24% step 3: search for nearest quantifiers step 4: construct the frog task a. none b. small amount c. few d. tiny amount a. some b. few c. tiny amount d. most hard choices: top k easy choices: randomly sampled frog c q frog @xmath0.4%), tiny amount (2.4%), small amount (18.3%), none (0.4%), some (22.5%), … figure 1: workflow of frog construction. et al., 2019) datasets into multiple-choice questions. these questions replace precise numerical data with gqs, requiring models to apply both precise reasoning like arithmetic computations and fuzzy reasoning to estimate the scope of gqs. our evaluation of several llms reveals that fuzzy reasoning remains a significant challenge in frog, with an inverse scaling effect observed in over half of the model families tested. furthermore, common methods designed to enhance reasoning capabilities like math-specialized tuning, code-specialized tuning, and general alignment, do not consistently improve performance in frog. interestingly, we find that strong mathematical reasoning skills are not necessarily indicative of success on frog. we also outline the primary strategies employed by llms to perform fuzzy reasoning. ## related work reasoning abilities, involving drawing conclusions from existing knowledge, are a cornerstone of human intelligence and are crucial for intricate tasks like decision-making and solving math word problems @xcite . recently, mathematical problem-solving has become a key dimension in assessing the reasoning capabilities of llms @xcite . various approaches have been developed to enhance the math reasoning ability of llms, like prompt-based designs @xcite @xcite . on the other hand, taylor et al. (2022); lewkowycz et al. (2022); paster et al. (2024); azerbayev et al. (2024) propose data construction for the pertaining stage. and further supervised fine-tuning, instruction tuning or alignment methods like direct preference optimization (dpo, rafailov et al. ( 2023 )) are used to enhance the reasoning abilities of llms @xcite @xcite @xcite . motivated by the fact that external tools are widely used in nlp tasks, tool integration is introduced to enhance math reasoning @xcite @xcite @xcite . @xcite employs programming methods to enhance the reasoning ability of llms. in frog, we build the fuzzy reasoning tasks based on mathematical reasoning problems and explore common approaches designed to improve reasoning capabilities including math-specialized tuning, codespecialized tuning, and general alignment. gqs are widely used to indicate proportions of predicate satisfaction in communication @xcite and benchmarks @xcite . they also contribute as a major source to the deficiencies of nlp systems like nli @xcite . given their real-world prevalence, we employ gqs as a natural approach to introduce fuzzy information in frog. existing methods for modeling fuzzy logic in natural language, as developed by @xcite and @xcite , depend on predefined mapping functions to process fuzziness. those mapping functions are mostly built from rule-based heuristics on limited data, or with simple distribution assumption (e.g. the gaussian distribution) and hard to be directly on real-world complex reasoning problems. in frog, we rely on llms that are pretrained from large-scale real world text corpora to process the fuzziness of gqs and conduct complex math reasoning problems. ## benchmark collection problems in frog are collected from two math word problem datasets from the real world: gsm8k and mathqa. gsm8k includes grade school math word problems that can be largely solved with basic arithmetic operations. mathqa consists of multiple-choice gre and gmat-level math problems. in frog, we include questions with percentage mentions, and figure 1 displays an overview of the collection workflow. specifically, eventually, frog includes 199 questions from gsm8k and 1,845 questions from mathqa. the average number of question tokens is 68.2, with each token separated by a space. and each question in frog contains an average of 1.6 percentage mentions. the total number of quantifiers involved in frog is 8, and the most common quantifiers used are few (25.3%), moderate amount (23.8%), and small amount (19.7%) (see top figure 2 for details). the bottom of figure 2 reveals the target percentage mentions mapped to each quantifier, e.g. the mean percentage value and 0.15 for few and around 0.4 for moderate amount. the mean strength difference between the golden quantifier and the incorrect quantifier that is closest to the target percentage mention is 0.105 in frog-hard. specifically, we would like to investigate the following three research questions: we employ the greedy decoding strategy with max tokens being 1,000, and temperature being 0.9 in experiments. the llms are instructed with task instructions (see appendix f) and 5 demonstrations @xcite with manually created chain-of-thought @xcite solutions to assist the reasoning procedure. the experiments are conducted on nvidia a100 80gb gpus, each experiment can be finished within 2 hours. moreover, we investigate the sensitivity of different masking strategies. specifically, we compare the performance between the mask and mislead or x% task by computing the pearson and spearman correlation of their accuracy. the results in ## conclusion the fuzzy reasoning ability is an under-explored direction of the reasoning ability of llms. to measure the fuzzy reasoning ability of llms, we collect a fuzzy reasoning benchmark frog that is based on generalized quantifiers. the experimental results show that fuzzy reasoning remains challenging for current llms, and an inverse scaling effect is observed on the performance of frog. besides, prevailing reasoning enhancement approaches including continuous pretraining, instruction tuning and general alignment may not stay effective on fuzzy reasoning of frog. lastly, llms can demonstrate diverse behaviors in fuzzy reasoning. chuanyang zheng, zhengying liu, enze xie, zhenguo li, and yu li. 2023. progressive-hint prompting improves reasoning in large language models. arxiv preprint arxiv:2304.09797. aojun zhou, ke wang, zimu lu, weikang shi, sichun luo, zipeng qin, shaoqing lu, anya jia, linqi song, mingjie zhan, and hongsheng li. 2024. solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. in the twelfth international conference on learning representations. yongwei zhou and tiejun zhao. 2024. dual instruction tuning with large language models for mathematical reasoning. arxiv preprint arxiv:2403.18295. kaijie zhu, jiaao chen, jindong wang, neil zhenqiang gong, diyi yang, and xing xie. 2024. dyval: dynamic evaluation of large language models for reasoning tasks. in the twelfth international conference on learning representations.