Language Models Don’t Learn the Physical Manifestation of Language (2024)

Abstract: We argue that language-only models don’t learn the physical manifestation of language. We present an empirical investigation of visual-auditory properties of language through a series of tasks, termed H-Test.These tasks highlight a fundamental gap between human linguistic understanding and the sensory-deprived linguistic understanding of LLMs. In support of our hypothesis, 1. deliberate reasoning (Chain-of-Thought), 2. few-shot examples, or 3. stronger LLM from the same model family (LLaMA 2 13B -> LLaMA 2 70B) has no significant effect on H-Test performance. We bring in the philosophical case of Mary, who learns about the world in a sensory-deprived environment as a useful conceptual framework to understand how language-only models learn about the world (Jackson, 1986). Our experiments show that some of the strongest proprietary LLMs stay near random chance baseline accuracy of 50%, highlighting the limitations of linguistic knowledge acquired in the absence of sensory experience. Our code and data are available at <github.com/brucewlee/h-test>.

## introduction the advent of large language models (llms) like chatgpt has revolutionized the field of computational linguistics, offering unprecedented insights into the mechanics of language processing and generation. yet, a fundamental question persists: how does a language model "experience" and consequently learn language, and how does this differ from human language acquisition? our aim lies in the systematic exploration of these perceptual-level differences, focusing on the distinction between the acquisition of language through textual data alone versus the multimodal sensory experiences inherent to human learning. humans experience language through a rich sensory interplay involving visual, auditory, and tactile stimuli @xcite , integrating these sensory inputs to form a comprehensive understanding of language. in contrast, llms are trained on vast corpora of text data, quite devoid of sensory context. they process language as a series of tokens, learning patterns and associations between these tokens without any direct experience of the concepts they represent. this sensory deprivation raises critical questions about the completeness and depth of the "understanding" of language that llms can achieve through text-based learning alone. roughly speaking, llm "perceives without sensing". to probe these questions, we introduce a series of tasks, collectively referred to as the h-test, inspired by philosophical thought experiments. h-test is designed to assess the ability of llms to learn the physical manifestation of languagethat is, how our language looks in our eyes. these tasks challenge the models to demonstrate a form of language understanding that is closer to human cognitive processes. this exploration is motivated by the philosophical debate on the nature of knowledge and understanding, particularly the thought experiment involving mary, a scientist who knows everything about the color red but has never experienced it directly. by analogy, can an llm that has never "seen" or "heard" truly understand the full linguistic knowledge that derives its significance from visual and auditory experiences? figure 1 encapsulates the central philosophical conundrum that guides our inquiry. while there are a number of anecdotal examples of state-of-the-art llms struggling with understanding and utilizing these orthographic components of language (chat-gpt doesn't know if "r" is in the word blueberry in appendix h), we quantify and show that this deficiency is rather difficult to solve by commonly pursued llm research directions. adding more orthography data will not trivially solve this issue. in the realm of llms, a predominant belief is that scaling up data, model size, and computational power invariably leads to enhanced performance @xcite , sometimes exceeding expectations @xcite . the recent report on the emergence of world representations within llms further emphasizes this point @xcite , suggesting a diminishing return on the effort to identify tasks that cannot be addressed by scaling. an intriguing perspective on this topic, as proposed by @xcite , is the idea of comprehending the "experience" of a chatgpt-like model. this line of thought is crucial, as it highlights a common oversimplification: the tendency to describe llms as being trained in "text" or "language," as perceived by humans. this tendency to describe things as they seem to us is also discussed as anthropomorphic behavior @xcite . in defense of scientific accuracy, @xcite advances this discussion by distinguishing between two types of information contained within language: diegetic information, which encompasses the internal, semantic, and propositional aspects of language, and supradiegetic information, which refers to the physical form of language, such as the shape of letters and sounds of syllables. our interpretation aligns with this framework. diegetic information in training data roughly corresponds to the semantics of a language. the efficacy of language modeling in teaching this aspect is evident from the consistent improvement in tasks like knowledge-based question answering and other semantics-heavy applications @xcite . however, the supradiegetic aspect, which involves the sensory perception of language, remains a less explored territory for language modeling. this is evident from the fragmental evidence of language models' struggles in downstream tasks with heavy stylistic features, such as automated essay scoring @xcite and readability assessment @xcite . to anchor our discussion, we reference the thought experiment by @xcite , featuring "mary," a scientist confined to a black and white room, learning about the world through monochromatic means. despite her knowledge of the descriptive aspects of human color vision, her understanding is incomplete until she "sees" color firsthand. drawing parallels to our investigation, we posit that while llms like chatgpt are adept at processing the diegetic content of language, their understanding of supradiegetic elements remains rudimentary. the fundamental difference in how humans and machines experience language inevitably leads to some level of incompleteness in learned linguistic knowledge @xcite , and therefore, will not be able to solve certain language tasks, inherently. task criteria group a (example) group b (example) uppercase group a has one random letter in uppercase. owl calculates in the hall adventurously. iguana reads with interest expertly. starts vowel group a always starts with a vowel. engineer sings with interest eagerly. lion cooks delicious meals enthusiastically. end punctuation group a always ends with a punctuation. octopus in space butterfly... astronaut ... through the telescope elephant palindrome group a reads the same forward and backward. wow debby end ly group a always ends with "ly". tree reads in the studio adventurously. cat teaches with interest enough. spelled math group a contains no spelledout math notation. the cube of equals to 3. the square root + 4. spelled number group a contains spelled-out number. river talks four computers intently. butterfly ticks 3 apples intently. rhyme group a words always rhyme with each other. get fat ask go repeated word group a repeats one random repeated word. book wall fast fast jumps. intently drives meals car. hyphenated word groups a contains one hyphenated expression. a friend reads on the wall in a up-to-date manner. a car ticks with passion fast. ## h-test do language models truly comprehend what the letter "h" looks like (figure 1 )? this inquiry goes beyond the basic propositional knowledge and delves into the llms' capacity for visual representation and comprehension. as illustrated in figure 1 , advanced llms like chatgpt demonstrate a competent understanding of how the letter "h" is described. this capability, however, does not necessarily equate to a comprehensive understanding of how the letter "h" looks like, at least in the philosophical sense. this is reminiscent of mary's incomplete knowledge in jackson (1986)'s thought experiment. mary's knowledge of color, although extensive in a physical and descriptive sense, was fundamentally incomplete until mary saw color directly. similarly, visual elements, such as the appearance of letters, or auditory elements, such as the rhyming sounds of two words, may be inherently limited, or very difficult to understand or use for llms. to explore this hypothesis, we propose ten distinct classification tasks in each binary classification task has 200 balanced test instances (100 a, 100 b), along with 50 balwe test weaker models from the same family for models given in precondition. given each test case and a set of example cases, the goal is to classify whether the test case belongs more to group a or group b. such a task formulation is also often referred to as few-shot prompting, where an llm classifies the test cases from a number of pre-classified examples provided with the prompt. here, we use few-shot prompting instead of a direct inquisition like "does this sentence start with a vowel?" such prompts linguistically reveal the classification criterion, effectively making the task solvable by step-by-step sequencing of memorized facts like "elephant starts with e; e is a vowel; therefore, true." results. observation 1. insignificant intra-family improvements. if a stronger language-only system from the same model family, from the same lab, solves h-test better, it would strongly hint that developing a stronger llm can solve h-test better. in such a case, visual and auditory external features of language that we experience by sensing can be learned without actually "sensing" language, and our case will be negated. in figure 2 , we observe that a stronger model in the same model family often does not bring meaningful improvement on the h-test performance. some models, like jurassic 2 and luminous, showed almost neglectable average accuracy changes, tightly centered around the random baseline. though the details of most proprietary llms are not reported scientifically, the llama 2 model family has rather thorough reports and shows that simply increasing the model size is not an apparent solution. quite interestingly, gpt-4, the only model that was reported to have undergone multi-modal training (openai, 2023), made a particular improvement on h-test compared to its precedent. however, this initial performance of the gpt-4 api couldn't be replicated when we accessed the api again a few weeks later. instead, gpt-4o replicated a similar performance. due to this inconsistency, the initial gpt-4 performance was removed from in section 5, we make an attempt to explain this surprising improvement by comparing multi-modal and mix-of-experts open-source models. however, we fail to explain the improvement in gpt-4o level. though we are limited by the amount of knowledge on how gpt-4 was trained in a multi-modal way, such a result does hint at a good assumption that h-test is indeed solvable when given a welldevised multi-modal system, or possibly, a large enough system @xcite . however, without a thorough report, no presumption about gpt-4 or gpt-4o model architecture can be made. observation 2. number of examples has minimal impact. few-shot prompting that we have been using to report the results in figure 3 reports respective performances with a varying number of examples. we only test stronger language-only models in light of the recent reports that icl itself can be a learned ability @xcite . even though the range of the few-shot setup tested is rather wide (from @xmath1, the effect on the average accuracy is minimal. that is, the number of examples given neither increases nor decreases performance significantly, strongly hinting that the llm is simply not learning from h-test examples cases. such an observation strengthens our initial conjecture that the visual-auditory properties of language that we measure with h-test are analogous to the "blindspot" of current language modeling formulations and will be difficult to learn without sensory experience of language. now, given that the few-shot examples given in h-test are not "learnable," is there an experimental setup that we deem is generally helpful for llms but rather misleads them in h-test? in the given input sentence, the phrase "in the park" suggests a peaceful and natural setting, which might be more aligned with labels related to nature and tranquility. on the other hand, the word "umbrella" brings to mind protection from the elements, which could be associated with indoor activities or certain weather conditions. considering these factors, it is more likely that "kangaroo in the park umbrella" belongs to label b, which represents a description of animals in their natural habitats or surroundings. therefore, the final answer is: "kangaroo in the park umbrella." label: b. observation 3. deliberate reasoning often decreases performance. lastly, we prompt llms to generate a chain-of-thought (cot) before answering @xcite . cot is a widely explored prompting technique that is reported to consistently boost performance in a wide range of complex reasoning tasks, including mathematical, commonsense, symbolic, and logical reasoning @xcite . with the development of tree-of-thought, @xcite draws a particular connection to the "system 2" thinking of humans. if llms benefit from such logical, step-by-step semantic reasoning on h-test, this can also imply that h-test is fundamentally solvable by developing stronger language-only models. on the other hand, if cot decreased or did not impact performance, this can strongly cue that h-test is well-constructed to test our initial research target of simple, sensory-dependent experiential knowledge. in figure 4 , we observe that cot decreases performances in general, confirming our hypothesis. gpt-4, which previously solved h-test with over 80% accuracy, also experienced a drop in performance, more strongly confirming our case. to better understand why cot decreases performance, we analyze the cot responses generated by the llms. our findings reveal that while llms can produce coherent and logical chains of thought, these thought processes often do not align with the sensory aspects required to solve h-test as shown in ## reasoning challenge: letter geometry so far, we have presented h-test and showed that there are certain properties of language that are difficult to learn, as standard sensory-deprived language models do not go through the standard model of perception (@xmath2. a particular and only exception was gpt-4 among the tested models. as a further challenge that tests not only the visual-auditory understanding of letters but also visuospatial reasoning abilities, we devise a more challenging task: letter geometry. as shown in the evaluation results are depicted in figure 5 . most models remain close to a hypothetical baseline, struggling with the sensory-deprived nature of their training. this suggests that the capacity to process visuospatial information in the language is not simply a function of visual-auditory understanding of language but also of how models are trained to reason through sensory-like experiences. we also assessed the performance improvement across different models from the same research labs. in the case of the command models, the enhancement was marginal, at 1.11%, which essentially translates to one additional correct response. a standout observation was with openai's models: the shift from gpt-3.5 to gpt-4o marked a substantial increase of around 15%. ## gpt-4o and claude 3 started solving h-test but how? in figure 2 , we have reported the seemingly unexplainable (jumping) performance improvement on h-test from gpt-3.5 (language-only) to gpt-4 (multimodal). we observe a similar improvement from claude 2 (language-only) to claude 3 (multimodal) in we do not understand gpt-4o and claude 3 opus's performance on h-test. we had three hypotheses aimed at replicating this success: 1. data: training more orthography-specific data can improve h-test score, 2. modality: visionmodality can improve h-test score, 3. architecture: mixture-of-experts (moe) architecture @xcite can improve h-test score. however, we fail at pinpointing what really made solving h-test possible. analysis 1. training with more orthographyspecific language data does not improve h-test. this is our strongest observation so far that proves llms don't naturally learn to solve h-test tasks. we produced 1000 training instances model end ly end punctuation hyphenated word palindrome repeated word rhyme spelled math spelled number start vowel uppercase average a. scaling | did not improve we initially planned on running additional moe experiments using instruction-tuned mistral-7b-instruct-v2 @xcite and mixtral-8x7b-instruct-v1 @xcite but we skipped this evaluation as the models did not understand that task (at @xmath3. more about this difficulty of evaluating h-test on smaller models is discussed in appendix c. we could not specify a hypothesis that singlehandedly and meaningfully improved h-test performance. fortunately, these results strengthen our initial claim on the limitations of language-only models, but we fail to explain how gpt-4o and claude 3 opus are solving h-test. ## conclusion our research revisits the philosophical case of mary's room, illustrating that llms, much like mary, lack a comprehensive understanding of language due to their sensory-deprived training. we empirically show that scaling model size and data, while beneficial for some aspects of language understanding, does not address the deficiency in orthographic abilities. this finding suggests that there are some fundamental limitations in the current language-modeling paradigm that prevent a holistic understanding of language and that it is still an open research direction.