Generating Diverse Hypotheses for Inductive Reasoning (2025)

Abstract: Inductive reasoning — the process of inferring general rules from a small number of observations — is a fundamental aspect of human intelligence. Recent works suggest that large language models (LLMs) can engage in inductive reasoning by sampling multiple hypotheses about the rules and selecting the one that best explains the observations. However, due to the IID sampling, semantically redundant hypotheses are frequently generated, leading to significant wastage of compute. In this paper, we 1) demonstrate that increasing the temperature to enhance the diversity is limited due to text degeneration issue, and 2) propose a novel method to improve the diversity while maintaining text quality. We first analyze the effect of increasing the temperature parameter, which is regarded as the LLM’s diversity control, on IID hypotheses. Our analysis shows that as temperature rises, diversity and accuracy of hypotheses increase up to a certain point, but this trend saturates due to text degeneration. To generate hypotheses that are more semantically diverse and of higher quality, we propose a novel approach inspired by human inductive reasoning, which we call Mixture of Concepts (MoC). When applied to several inductive reasoning benchmarks, MoC demonstrated significant performance improvements compared to standard IID sampling and other approaches.

## introduction inductive reasoning, inferring general rules that explain a small number of observations (figure 1 ), is a key factor of human intelligence @xcite . recently, there has been exploration into automatic inductive reasoning using large language models (llms). trained with vast corpora and instruction tuning, llms can function as zero-shot rule proposers when given an inductive reasoning problem @xcite . by leveraging * corresponding authors. to address aforementioned problem, we begin by increasing the temperature, which is often considered a diversity parameter for lms, in iid sampling setup. our findings indicate that while both diversity and accuracy improve up to a certain point with increased temperature, they quickly reach saturation. this is because raising the temperature deteriorates the quality of hypotheses, and increases the rate of text degeneration @xcite . to generate diverse hypotheses while maintaining their quality, we propose a simple yet effective method called mixture of @xcite . inspired by human inductive reasoning @xcite , our approach consists of two stages: concept proposal and hypothesis generation. in the concept proposal, we instruct the llm to generate "a list of k concepts" that are likely to aid in the formation of hypotheses, and semantically non-redundant concepts are generated sequentially. in the hypothesis generation, these concepts are parsed and each concept is provided as a hint for generating hypotheses. our approach allows for the parallel generation of semantically diverse hypotheses without hurting the quality of hypotheses (figure 1 ). we conduct experiments on four inductive reasoning datasets from distinct domains. our methodology significantly improves performance across four llms by generating diverse hypotheses while generate a rule that maps the following inputs to their corresponding outputs. correct rule: mirror vertically "drop boxes" "drop objects" "push objects downward" "rotate clockwise" "transpose matrix" "flip vertically" "rotation" "transpose" "flip" "rotate clockwise" "drop objects" #%..@#!... increasing the temperature leads to frequent occurrences of text degeneration (middle). moc allows for the generation of diverse hypotheses without a decline in hypothesis quality (bottom). our contributions are as follows: ## hypothesis diversity and temperature in this section, we provide a description of the problem statement and the baseline method. subsequently, we analyze the impact of temperature on the diversity and accuracy of hypotheses generated by an llm. ## mixture of concepts to achieve aforementioned goal, we propose automatic inductive reasoning framework called mixture of concepts (moc). when humans engage in inductive reasoning, they begin by carefully observing examples and considering which concept or template might be effective in identifying the underlying rule @xcite . for instance, if presented with integer lists as depicted in figure 5 , one might initially contemplate concepts such as "arithmetic operations." using this concept, human would re-examine the given examples to discern what the hidden rule might be. however, in this instance, the correct rule is to output the length of the list, and the concept of "arithmetic operations" do not aid in discovering this rule. in such scenarios, the best strategy is to revisit the given examples from a different point of view and to contemplate novel concepts and rules. this process can be repeated until the correct rule is identified @xcite . inspired by this cognitive process, we propose a framework composed of two stages: concept proposal and hypothesis generation.