Understanding LLM s’ Fluid Intelligence Deficiency: An Analysis of the ARC Task (2025)

Abstract: While LLMs have exhibited strong performance on various NLP tasks, it is noteworthy that most of these tasks rely on utilizing the vast amount of knowledge encoded in LLMs’ parameters, rather than solving new problems without prior knowledge. In cognitive research, the latter ability is referred to as fluid intelligence, which is considered to be critical for assessing human intelligence. Recent research on fluid intelligence assessments has highlighted significant deficiencies in LLMs’ abilities. In this paper, we analyze the challenges LLMs face in demonstrating fluid intelligence through controlled experiments, using the most representative ARC task as an example. Our study revealed three major limitations in existing LLMs: limited ability for skill composition, unfamiliarity with abstract input formats, and the intrinsic deficiency of left-to-right decoding. Our data and code will be publicly released, and the data is also attached in the submission.

## introduction large language models (llms) have demonstrated impressive performance on a range of challenging nlp tasks @xcite @xcite @xcite , which naturally leads to the question: how close are llms to achieving human-level intelligence? to explore this question, it is useful to draw on established research in human intelligence @xcite , which categorizes intelligence into two major types: crystallized intelligence, the ability to apply prior knowledge to solve problems, and fluid intelligence, the ability to tackle new problems without relying on pre-existing knowledge. of the two, fluid intelligence is often viewed as more indicative of general cognitive ability (jaeggi * co-corresponding authors. @xcite @xcite , as it captures the capability to solve novel problems. moreover, evaluating fluid intelligence is essential for assessing the reasoning abilities of llms since llms have been exposed to and memorized vast amounts of knowledge during pre-training @xcite , potentially blurring the line between their reasoning ability and memorization capacity. drawing from both cognitive @xcite and ai research @xcite , an ideal approach to evaluate fluid intelligence in llms involves evaluating their ability to perform abstract inductive reasoning, i.e., induct a general pattern solely from given input-output examples and apply this pattern to deduce the correct outputs for new inputs. the abstraction and reasoning challenge (arc) @xcite , which requires models to induct transformation rules from input-output grid pairs (as shown in therefore, the arc task has become the de facto standard for measuring machine fluid intelligence, sparking a wave of recent studies aimed at improving llm performance on it @xcite @xcite . despite these efforts, llms continue to struggle with the arc task. for example, even the stateof-the-art gpt-4o with careful prompting can only correctly solve 19% of tasks (see ## breaking arc into atomic operations as mentioned in §1, the transformation rule of each arc task can be decomposed into several atomic operations (e.g., the rule in ## challenge on task composition in this section, we assess the deficiency of llm's fluid intelligence from a task composition perspective. first, we consider a simple composition experiment that controllably evaluates the composition for move and copy in araoc ( §4.1). moreover, we design a complex composition experiment utilizing arc tasks to evaluate llms' abilities to compose all atomic operations ( §4.2). ## challenge on input format since llms cannot process visual inputs, we follow @xcite to convert the 2d visual input-output grids in araoc tasks into matrixformat before feeding them to the llms ( §2.1). however, it remains uncertain that whether this conversion affects llms' performances on araoc, since llms are mostly trained on natural language data, and may not understand such matrix-format inputs well. in this section, we first try to answer this question ( §5.1), then investigate a strategy to remedy its potential challenges ( §5.2). ## modeling challenge in this section, we examine whether llms' modeling features affect their fluid intelligence from both the model architecture perspective and the information encoding perspective. ## conclusion this paper presents an in-depth study of llms' fluid intelligence deficiencies using the arc tasks, with a series of controlled experiments from multiple perspectives. through task decomposition, we introduce the atomic araoc benchmark, revealing that llms struggle with atomic operations despite their simplicity for humans. we further demonstrate that llms' task composition abilities are limited, as improvements on the decomposed araoc tasks via fine-tuning do not lead to better performance on arc tasks. additionally, our study shows that llms' difficulty in encoding abstract input formats is a major obstacle in addressing arc tasks. lastly, it shows an intrinsic limitation in the left-to-right paradigm of llms, which hinders their ability to achieve advanced fluid intelligence.