Idola Tribus of AI : Large Language Models tend to perceive order where none exists (2025)

Abstract: We present a tendency of large language models (LLMs) to generate absurd patterns despite their clear inappropriateness in a simple task of identifying regularities in number series. Several approaches have been proposed to apply LLMs to complex real-world tasks, such as providing knowledge through retrieval-augmented generation and executing multi-step tasks using AI agent frameworks. However, these approaches rely on the logical consistency and self-coherence of LLMs, making it crucial to evaluate these aspects and consider potential countermeasures. To identify cases where LLMs fail to maintain logical consistency, we conducted an experiment in which LLMs were asked to explain the patterns in various integer sequences, ranging from arithmetic sequences to randomly generated integer series. While the models successfully identified correct patterns in arithmetic and geometric sequences, they frequently over-recognized patterns that were inconsistent with the given numbers when analyzing randomly generated series. This issue was observed even in multi-step reasoning models, including OpenAI o3, o4-mini, and Google Gemini 2.5 Flash Preview Thinking. This tendency to perceive non-existent patterns can be interpreted as the AI model equivalent of Idola Tribus and highlights potential limitations in their capability for applied tasks requiring logical reasoning, even when employing chain-of-thought reasoning mechanisms.

## introduction recent achievements of large language models (llms) have raised expectations that they can perform well across a wide range of tasks in human activities, reducing labor and duties through automation enabled by artificial intelligence (ai) technologies @xcite . in particular, the ai agent framework is a promising approach for handling complex tasks by integrating llms with external systems, enabling self-situation understanding and action planning @xcite . to develop autonomous systems capable of executing complex real-world tasks, a deep understanding of llm behavior is essential. a key advantage of llms over other ai systems is that their input and output are in humanunderstandable natural language. users can provide task instructions through everyday conversation without strict formatting, much like assigning tasks to colleagues. there is ongoing discussion that llms exhibit behavior that can only be interpreted as genuine language understanding, making it increasingly difficult to dismiss the possibility that they truly comprehend natural language @xcite . additionally, results can be received through a conversational interface, allowing users to ask follow-up questions. in this context, it is crucial to ensure that llms accurately interpret human-provided prompts and generate reasonable and reliable outputs, rather than merely producing "language-like" lists of information. to assess this aspect, @xcite proposed a method for aligning llms with user objectives to enhance human-llm communication. similarly, the lcm team et al. @xcite introduced a new framework, the large concept model, which aims to address the abstract nature of natural language communication by incorporating higherlevel semantic representations called "concepts." one widely recognized issue in llm applications is hallucination, where llm outputs contain untruthful information @xcite . while one possible cause is the limited knowledge coverage in training data, studies have shown that hallucinations can occur even when an llm possesses sufficient knowledge to generate correct answers @xcite . to address knowledge gaps, various frameworks have been proposed to enhance llms with external information through input prompts, such as in-context learning @xcite and retrievalaugmented generation @xcite . another critical issue with llms is ensuring a reasonable thinking process and self-consistency. leaps in logic or unsupported conjectures can lead to false conclusions and may ultimately cause task execution failures. chain-of-thought (cot) prompting techniques have been proposed to address this issue by encouraging step-by-step reasoning @xcite . recently, major llm developers have introduced "thinking" models, such as openai o3, o4-mini (openai, 2025b), and google gemini 2.5 flash preview thinking @xcite , which incorporate built-in multi-step self-evaluation and modification mechanisms based on the cot concept. a reasonable thinking process involves not only deduction but also induction, both of which are crucial for executing real-world tasks. in practical societal settings, strict procedural definitions for tasks are often limited, making it necessary to formulate hypotheses and proceed accordingly. effective hypothesis formation requires not only consistency with the provided information but also the ability to abstract key patterns and principles. without proper abstraction, misunderstandings or biased assumptions can critically impact task execution. in this paper, we present an experiment to investigate the capabilities of llms, including thinking models, in forming hypotheses through precise information understanding and pattern abstraction. we use a simple task of identifying regularities in number series, allowing us to evaluate llms' pattern recognition ability independently of hallucinations related to knowledge accuracy. francis bacon, in his famous work novum organum @xcite , identified the tendency of biased belief and referred to it as idola, an inherent aspect of human nature. in particular, the tendency to over-recognize patterns in randomness is called idola tribus (idols of the tribe). the motivation of this research is to examine whether idola tribus, as described in the following citation, is a relevant concern for ai systems, particularly llms. ## method we conducted an experiment in which llms identified regularities in a prepared list of number series and then evaluated whether their descriptions accurately explained the series to investigate idola tribus, over-recognition biases. the details of the regularity identification configuration and evaluation method are provided in the following subsections. ## 81 quasiarithmetic almost arithmetic series with the same conditions as #1, but with one term deviating by 1 from the expected pattern. ## 81 quasigeometric almost geometric series with the same conditions as #2, but with one term deviating by 1 from the expected pattern. ## 100 quasidifference almost difference series with the same conditions as #3, but with one term deviating by 1 from the expected pattern. ## 100 randomincreasing randomly generated increasing integer sequences. not applicable to #1-#6. figure 1 shows an input prompt with a simple instruction for the regularity identification task. each prompt included five values from a given series. regularity becomes easier to explain when only four or fewer values are provided, such as when fitting a third-order polynomial function, while randomness becomes clearer when six or more values are provided in the random series. the same prompt was used across all models. the output is restricted to a single sentence via the prompt for ease of evaluation. we standardized the approach by providing instructions solely through user prompts, without using system prompts. prompt for identifying regularities role: user content: please explain the regularity in the following number series in one sentence, ensuring that all the numbers are not included together in your answer: 2, 33, 9, 25, 51, ... ## result we successfully obtained 3,620 regularity descriptions for 724 number series shown in all five models correctly identified the regularities in all arithmetic series, achieving a 100% success rate, and performance was also high for the geometric series. for the difference series, o3, o4-mini, gemini 2.5 flash preview thinking and llama 3.3 maintained strong performance, while the success rate declined for gpt-4.1. for the quasi-arithmetic, quasi-geometric, and quasidifference series, the success rate decreased across all llms, reflecting the fact that these series do not exhibit clear regularities. in particular, the nonthinking models gpt-4.1 and llama 3.3 performed worse than the self-iterative reasoning models o3, o4-mini, and gemini 2.5 flash preview thinking. in the random-increasing and random categories, the trend of thinking models outperforming nonthinking models remained consistent with the quasiordered series cases. overall, the success rate was high for the thinking models, with comparable performance among them. however, the success rate for the random series remained notably low across all llms. the fact that llms provide incorrect explanations for random series clearly demonstrates their tendency to overestimate regularities when interpreting information-specifically, the numerical values in the series used in the experimentindicating the presence of idola tribus in llms. as a result, our findings reveal a tendency to perceive ordered patterns that are inconsistent with the provided information-clear evidence that these ai models exhibit idola tribus. to investigate whether llms tend to force a pattern even when they fail to find a plausible one, we conducted an additional experiment by modifying the regularity identification prompt to explicitly allow the series to be random (fig. 3 ). the success rates with the random-allowing prompt configuration are shown in ## discussion as shown in the previous section, we presented a clear and comprehensive analysis-demonstrating for the first time-that llms occasionally generate false patterns that contradict the information they are given. when a clear regularity cannot be identified, the model should explicitly state this, rather than providing potentially inaccurate explanations. the lack of random-series explanations under the original prompt clearly illustrates this issue, where models incorrectly assert the presence of a pattern instead of acknowledging the absence of an identifiable structure. by comparing correct regularity explanations that align and do not align with the preset categories (evaluation result options 1 and 2), we found that in the arithmetic, geometric, and difference series, almost all correct explanations aligned with the preset categories. since the regularities in these categories are simple and well-defined, it is reasonable that llms did not attempt to explain them using alternative patterns. in contrast, for the quasiordered categories, which include one deviation from a clear regularity, there were comparable numbers of correct explanations both aligning and not aligning with the preset categories. this suggests that the evaluated models demonstrate strong performance in their ability to identify valid regularities, even when those regularities were not explicitly intended in the series generation process. for the o3 and o4-mini models, a greater variety of highly creative ideas were generated compared to other models, including interpretations based on atomic numbers, football players, piano, tarot, the holy bible, and telephone country codes-even though these were inconsistent with the given series. this level of creativity could be a significant advantage if the model's internal mechanism for logical self-validation functioned reliably. at present, however, both o3 and o4-mini tend to validate ideas they are not well-equipped to evaluate, leading to misinterpretations and false confirmations. it has been reported that llms often struggle with tasks such as recognizing characters in words, counting words, or identifying letter sequences @xcite -weaknesses that may contribute to the observed failures. in such cases, the models' wide-ranging ideas ultimately lead to lower-quality outputs. based on the behavioral changes observed with the random-allowing prompt, it is plausible that this bias stems from an implicit compulsion in llms to always provide an answer in response to a given instruction. this tendency can lead to false outputs resembling confabulation or improvisation-similar to hallucinations. in this context, frameworks designed to mitigate hallucinations may offer useful insights. several studies have explored such approaches, including prompting llms to state explicitly when they do not have certain knowledge @xcite , or even to explain why they cannot answer a given question @xcite . if llms were able to state that they do not know the regularities in a number series-or to explain why they cannot identify them-the impact of this phenomenon could be significantly reduced. further investigation is needed to determine whether these concepts can be applied to cases of logical reasoning, rather than purely knowledge-based tasks. the fact that the models with the best performance under the random-allowing prompt, o3 and o4-mini, achieve higher success rates for random series than for quasi-ordered series is notable. quasi-ordered series may appear to contain a regularity, as the deviation involves only a single value in the series. there are cases in which llms persist in trying to explain a pattern rather than recognizing the absence of a clear rule. this tendency provides evidence that the behavior aligns with the characteristics of idola tribus, suggesting that the phenomenon observed in llms shares the same nature. although the number of target llms is limited to five, we observed this tendency across all five models. this aspect also aligns with the concept of idola tribus, which refers to shared tendencies among humans, suggesting a comparable tendency within the "tribe" of llms. one possible cause of this phenomenon might be a tendency to process information efficiently during training or instruction tuning. providing additional prompts to help llms recognize this bias may reduce over-interpretation, although it could also limit their willingness to engage with more complex reasoning tasks. to overcome this bias through approaches other than fine prompt adjustments, fine-tuning with strategies proposed in previous studies for enhancing logical reasoning capabilities could also be effective in addressing the pattern recognition biases identified in this paper @xcite . ideally, developing a bias-free model through fine-tuning would be preferable, as relying on a biased model requires constant vigilance to mitigate bias through prompting. although the fine-tuning configurations discussed in these studies are primarily designed to improve deductive reasoning, the core principle of maintaining logical consistency with all available information is equally important for addressing inductive biases. it has also been noted that model performance depends not only on the quantity of training data but, more importantly, on its quality @xcite . in this context, exploring effective fine-tuning strategies is a key next step in tackling the issue. ## conclusion we conducted an experiment in which llms identified regularities in various types of number series, including randomly generated ones, and discovered a tendency for llms to over-recognize patterns that do not fully explain the provided information. this tendency can be regarded as the llm equivalent of idola tribus in humans. llms tend to force themselves to explain patterns even when they do not find a plausible one, unless explicitly instructed to acknowledge the absence of regularities. this tendency has been shown to share the same characteristic found in idola tribus in humans, particularly in its greater likelihood to appear when the series seems ordered. future research is expected to focus on mitigating the impact of this bias, and further model development will be necessary to address it. until this tendency is properly controlled, it is essential to remain aware of the issue when applying llms to practical tasks, just as humans must be cautious to avoid biases in general, in order to prevent unintended errors.