The Stochastic Parrot on LLM ’s Shoulder: A Summative Assessment of Physical Concept Understanding (2025)

Abstract: In a systematic way, we investigate a widely asked question: Do LLMs really understand what they say?, which relates to the more familiar term Stochastic Parrot. To this end, we propose a summative assessment over a carefully designed physical concept understanding task, P HYSI C O. Our task alleviates the memorization issue via the usage of grid-format inputs that abstractly describe physical phenomena. The grids represents varying levels of understanding, from the core phenomenon, application examples to analogies to other abstract patterns in the grid world. A comprehensive study on our task demonstrates: (1) state-of-the-art LLMs, including GPT-4o, o1 and Gemini 2.0 flash thinking, lag behind humans by ∼40%; (2) the stochastic parrot phenomenon is present in LLMs, as they fail on our grid task but can describe and recognize the same concepts well in natural language; (3) our task challenges the LLMs due to intrinsic difficulties rather than the unfamiliar grid format, as in-context learning and fine-tuning on same formatted data added little to their performance.

## introduction recent years have witnessed remarkable advancements in large language models (llms) @xcite @xcite . thanks to the substantial model capacity and massive training data, llms have achieved new stateof-the-arts on a variety of nlp tasks, even surpassing humans on some of them @xcite . nowadays the application of llms has become widespread, facilitating daily work and life, and profoundly influencing people's work and lifestyles @xcite @xcite . on the other hand, despite the great success of llms, many researchers argue that llms may not really understand what they claim they do (bender and koller, 2020; @xcite @xcite due to their strong memorization ability. in particular, @xcite questioned whether llms are just stochastic parrots that repeat words based on correlations without true understanding. this argument has been acknowledged by many research papers and dozens of them even include this term in their titles. 1 unfortunately, to our best knowledge, there are no quantitative experiments to verify the stochastic parrot phenomenon in llms. existing studies indicate that llms may fail on one particular challenging task @xcite @xcite ), but they do not demonstrate that llms claimed to understand those tasks by providing a controlled and paired evidence. this paper aims to provide quantitative evidence to validate the argument of stochastic parrot in llms. to this end, from the perspective of educational and cognitive psychology, we first employ the approach of summative assessment @xcite to measure understanding in llms. its key idea is to design various tasks that test different understanding levels regarding a specific concept. following the principle of bloom's taxonomy @xcite , we design tasks that reflect different levels of understanding. consequently, we develop physico, a task designed to assess understanding of basic physical concepts from high school such as gravity. our focus on physical concepts stems from both their fundamental relevance to important topics of world models and embodied systems @xcite @xcite , and their rich denotations and connotations that enable effective design of summative assessment tasks. specifically, physico includes two subtasks corresponding to two coarse levels of understanding in bloom's taxonomy, as shown in figure 1 . one is the low-level understanding subtask in the natural language format, aimed at measuring the remembering (or memorization) ability of llms. the other involves the same concepts but in an abstract representation format inspired by @xcite , which is designed to measure the high-level understanding beyond remembering of llms. we conduct comprehensive experiments on physico with representative open-source and commercial llms. 2 we obtain two key findings: (1) state-of-the-art llms perform perfectly on the low-level understanding subtask (>95% in accuracy) but lags behind humans by a large margin (∼40% in accuracy) on the high-level subtask, which verifies the stochastic parrot phenomenon in llms. @xcite further analysis shows that our highlevel subtask challenges llms due to the intrinsic difficulty of deep understanding rather than the unfamiliar format. this paper makes the following contributions: ## measuring concept understanding via summative assessment it is intrinsically challenging to measure the extent to which llms understand a sentence or concept. indeed, bender and koller (2020) provide a definition of "understanding" from a linguistic perspective, but this definition depends on another abstract and unmeasurable term, "meaning". therefore, even with this definition, accurately measuring "understanding" remains elusive. we approach the measurement of whether llms understand a concept from an educational and cognitive perspective, using summative assessment @xcite . summative assessment is widely used by educators as an appealing strategy to evaluate students' understanding and knowledge acquisition in educational and cognitive psychology. for example, when middle school physics teachers want to know whether a student truly understands the concept "gravity", they would design a series of questions specifically related to the concept of gravity to assess comprehension, e.g., the properties like inverse square law and examples like orbital motions. if a student struggles to answer many of these questions, the teacher may conclude that the student does not understand the concept well or has a poor grasp of it. we extend the idea of summative assessment to evaluating the concept understanding of machines. formally, assume s denotes an intelligent system and c is a specific concept. to evaluate the extent how s understands the concept c, our summative assessment includes the following two steps: • task design towards c: design several concept understanding tasks, each of which consists of several questions manually created towards understanding the concept c. ## validation on low-level subtasks to answer rq 1, we evaluate the llms' abilities to comprehend the definitions of these concepts and generate their descriptions and examples in natural language, as defined in section 3.2.1. ## experiments on high-level subtasks this section answers the research questions regarding our high-level understanding subtasks. ## conclusion we introduce physico, a novel task to assess machines' understanding of physical concepts at different levels. our experiments reveal that: 1) llms lag significantly behind humans on physico, indicating a lack of deep understanding of the covered concepts; 2) llms exhibit the stochastic parrot phenomenon, as they excel at low-level remembering tasks but struggle with high-level understanding tasks; 3) llms' poor performance stems from its intrinsic deficiencies, as neither in-context learning nor fine-tuning improves their results. an example of the masked description can be found in figure 6 .