Do large language models resemble humans in language use? (2024)

Abstract: It is unclear whether large language models (LLMs) develop humanlike characteristics in language use. We subjected ChatGPT and Vicuna to 12 pre-registered psycholinguistic experiments ranging from sounds to dialogue. ChatGPT and Vicuna replicated the human pattern of language use in 10 and 7 out of the 12 experiments, respectively. The models associated unfamiliar words with different meanings depending on their forms, continued to access recently encountered meanings of ambiguous words, reused recent sentence structures, attributed causality as a function of verb semantics, and accessed different meanings and retrieved different words depending on an interlocutor’s identity. In addition, ChatGPT, but not Vicuna, nonliterally interpreted implausible sentences that were likely to have been corrupted by noise, drew reasonable inferences, and overlooked semantic fallacies in a sentence. Finally, unlike humans, neither model preferred using shorter words to convey less informative content, nor did they use context to resolve syntactic ambiguities. We discuss how these convergences and divergences may result from the transformer architecture. Overall, these experiments demonstrate that LLMs such as ChatGPT (and Vicuna to a lesser extent) are humanlike in many aspects of human language processing.

## introduction the formal linguistic competence apparent in llms has led to debates over whether they can serve as cognitive models of human language use (see @xcite . on the one hand, chomsky argued that humans are endowed with an innate universal grammar (e.g., @xcite , and he and colleagues maintain that this "genetically installed 'operating system'... is completely different from that of a machine learning program" @xcite ) such as chatgpt, which is simply "a lumbering statistical engine for pattern matching" (para. 5). more optimistic researchers, however, argue that deep neural networks suffice to learn syntactic structure @xcite , as evidenced by the fact that llms abide by complex grammatical rules (e.g., @xcite @xcite . this debate emphasizes grammar, but regularities in language range from phonology to pragmatics. for example, people associate different sounds with different referents (e.g., @xcite , automatically reinterpret implausible sentences (e.g., @xcite , and expect demographically appropriate content from speakers (e.g., @xcite . do llms share these regularities in language use? @xcite pointed out that llms integrate syntax and semantics (i.e., all aspects of usage are represented in a single vector space), so other humanlike regularities in language use might emerge along with grammaticality and coherence. we therefore subjected two llms-chatgpt, from openai (2022), and vicuna (with 13b parameters), from the large model systems organization @xcite -to a battery of psycholinguistic tests, in 12 preregistered experiments per llm (with default temperature). these experiments span a range of linguistic levels from sounds to discourse, with two experiments per level. in each experiment, each item was presented to each llm 1000 times. we used mixed effects modelling to analyse model responses as a function of the experimental manipulations. the preregistrations, data, and analytical codes are available at osf.io/vu2h3/ (chatgpt) and osf.io/sygku/ (vicuna). ## discussion our experiments showed that chatgpt replicated human patterns in language comprehension and production in 10 out of 12 psycholinguistic tasks and vicuna in 7 out of the same 12 tasks. we further note that the patterns of results mostly held when we removed example words/sentences presented in research papers (see appendix c), suggesting that these effects are unlikely to be a result of llms explicitly learning these effects in training. these findings suggest that both models largely approximate human language processing. both chatgpt and vicuna are built on transformer architectures @xcite , which allow them to vary how much weight they assign to different tokens within recent conversation history when predicting the subsequent token. this context sensitivity can explain llms' humanlike tendency to re-use previously-used meaning of ambiguous words, understand and produce words in light of the interlocutor's dialectic background, make inferences, and attribute causality according to verb semantics. in addition, the fact that llms change semantic representations of words to fit contexts @xcite may help to account for chatgpt's humanlike susceptibility to semantic illusions and adjust its interpretation of implausible sentences. the tokenization method might help to capture form-meaning associations available in languages. finally, the fact that llms are not trained on syntactic data but can be structurally primed suggests that they may have developed emergent syntax-like representations @xcite @xcite . in two of the experiments, neither chatgpt nor vicuna replicated the patterns of human participants. it is possible that the tokenization methods lead llms to fail to capture the effect of predictivity on word length. for example, gpt-4 segments roach into "ro" and "ach" and cockroach into "cock" and "roach"; thus, the model may fail to treat the two words as close in meaning as humans would do. in addition, both models failed to take context into account when resolving the vp/np syntactic ambiguity (e.g., the hunter killed the dangerous poacher with a rifle), which is reminiscent of a similar absence of contextual effects in pragmatic understanding observed in chatgpt @xcite . this finding is surprising given llm's superb ability using contextual information. it is also interesting that chatgpt replicated more humanlike patterns of language use than vicuna did (10 versus 7 out of the 12 experiments). given that increasing model size or training data improves performance (e.g., @xcite , we assume that this difference in mimicking the nuances of human language use should be attributed to vicuna being a smaller model than gpt-3.5. in conclusion, our results point to the interesting possibility that llms such as chatgpt (and vicuna to a lesser extent) can be used, by psycholinguists and cognitive psychologists, as models of language users (e.g., @xcite @xcite . perhaps researchers can experiment with llms to generate hypotheses, assess the replicability of existing psycholinguistic effects, estimate effect sizes, and model language development. ## limitations there are several limitations worth noting. first, the selection of the 12 psycholinguistic tasks might seem arbitrary and lack robust justification, raising concerns about the potential bias towards tasks where llms are inherently more successful. second, there were inherent discrepancies in the experimental designs used for llms compared to those for human studies, encompassing differences in materials, procedures, and contexts. third, many experiments do not include direct comparisons between llm and human behaviours due to the unavailability of data in corresponding human studies.