Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution: Humans or LLM s? (2025)

Abstract: This study explores how recent large language models (LLMs) navigate relative clause attachment ambiguity and use world knowledge biases for disambiguation in six typologically diverse languages: English, Chinese, Japanese, Korean, Russian, and Spanish. We describe the process of creating a novel dataset – MultiWho – for fine-grained evaluation of relative clause attachment preferences in ambiguous and unambiguous contexts. Our experiments with three LLMs indicate that, contrary to humans, LLMs consistently exhibit a preference for local attachment, displaying limited responsiveness to syntactic variations or language-specific attachment patterns.Although LLMs performed well in unambiguous cases, they rigidly prioritized world knowledge biases, lacking the flexibility of human language processing. These findings highlight the need for more diverse, pragmatically nuanced multilingual training to improve LLMs’ handling of complex structures and human-like comprehension.

## introduction natural language is inherently ambiguous, with single expressions often having multiple interpretations. this ambiguity poses significant challenges to both human cognition and computational models, especially in tasks requiring precise language understanding like machine translation, question answering, and dialogue systems. miscommunication can arise when different listeners or readers interpret the same expression differently, making ambiguity resolution a critical area of research @xcite @xcite @xcite @xcite . among various types of ambiguity, this study focuses on syntactic ambiguity, specifically relative clause (rc) attachment ambiguity. syntactic ambiguity occurs when a sentence's structure allows for multiple grammatical interpretations. rc attachment ambiguity arises when a relative clause can attach to more than one determiner phrase (dp), leading to different possible meanings. for instance, in (1), the rc who had a beard could refer to either the local dp (dp2) the man or the non-local dp (dp1) the son. the person who had a beard is the son. figure 1 illustrates this syntactic ambiguity with two distinct syntax trees. previous studies have demonstrated that attachment preferences are language-specific, influenced by multiple factors in the resolution of syntactic ambiguities @xcite @xcite @xcite . while world knowledge and biases often guide interpretation, the extent to which these factors override inherent syntactic parsing preferences remains underexplored -especially in large language models (llms) @xcite . moreover, much of the research on llms and am-figure 2 : multiwho dataset: the dataset creation started with a list of requirements and three different conditions. using a collaborative human-llm process, we started with developing english sentences and continued through translation and localization, resulting in a multilingual dataset across six languages. while not all sentences are pragmatically equivalent in all languages, they are structurally equivalent with regard to our requirements. these datasets were evaluated in two ways: the english dataset was evaluated by 65 human annotators for ambiguity/dpbias, and all 6 datasets were evaluated for ambiguity/dp-bias in three different answer order settings by llms. this study explores how humans and llms resolve rc attachment ambiguities across languages and whether world knowledge overrides syntactic preferences. 2 we investigate if llms show consistent attachment patterns across languages and how they align with human processing. we selected six languages-english (en), chinese (zh), japanese (jp), korean (ko), russian (ru), and spanish (es)-for their syntactic diversity, particularly regarding the position of rcs. for instance, en, es, and ru use postnominal rcs that follow the dp they modify, as shown in (2): in contrast, jp, ko, and zh use prenominal rcs that precede the dp they modify, as illustrated in the ko example (3): to resolve the ambiguities in rc attachment in different languages? will there be a certain pattern depending on the syntactic differences among languages? 2. how does the sensitivity of llms to world knowledge and biases compare to that of humans when resolving syntactic ambiguities? 3. does the presentation order (linear, reversed, random) of the possible response choices (dp1 or dp2) influence llm attachment preferences? to briefly summarize our results, in ambiguous cases in english, we found that humans exhibit a strong low attachment (la) preference, with an ha (high attachment) response rate of approximately 0.2, while llms show an even stronger la preference of around 0.1. extending this observation to other languages, llms default to la preference as well, aligning with human preferences in en and zh but contrasting with the ha preferences observed in humans for ko, jp, ru, and es. in unambiguous en cases, humans demonstrate a stronger and more consistent adherence to dp2-bias (0.88) compared to dp1-bias (0.64). this reflects a natural tendency toward la structures and showcases their flexibility in adjusting interpretations despite inconsistencies in world knowledge. in contrast, llms display near-perfect accuracy for both dp2-and dp1-biases, suggesting a greater sensitivity to world knowledge biases and a rigidity in interpretations that may reinforce existing social stereotypes. similarly, in unambiguous cases across other languages, llms show high accuracy ## multiwho dataset we introduce multiwho, a dataset designed to examine how humans and llms utilize world knowledge and biases to resolve syntactic ambiguities. multiwho comprises a total of 1728 sentences, with 96 sets, spanning three categories (ambiguous, dp1-biased, and dp2-biased), in six languages, en, es, jp, ko, ru, and zh. a sample english set is shown in recently, llms' fluency and instructionfollowing ability have prompted interest in human-llm collaboration for dataset creation @xcite . however, even advanced llms such as gpt-4o and claude 3 sonnet were unable to consistently generate sentences that met our criteria. this led us to develop a more flexible paradigm for specialized dataset creation, where human experts leverage llms for assistance and support rather than complete solutions. our approach involved (i) brainstorming with llms to generate options meeting individual constraints, (ii) manually combining these elements to create sentences satisfying all criteria, (iii) using llms for validation and discussion throughout the process, (iv) employing llms' fluency and cultural knowledge for initial drafts in multiple languages, and (v) validating and refining these drafts with help from native speakers. this collaborative method allowed us to work ef-ficiently while maintaining high standards of quality and consistency across diverse languages. ## experiment 1: humans to directly compare llm and human performance in rc ambiguity resolution, we conducted a forcedchoice experiment with human subjects using the en dataset. ## general discussion the findings in sections 3.3 and 4.3 can be summarized as follows: while llms like gpt-4o and claude 3 sonnet exhibit high accuracy in unambiguous conditions by effectively leveraging world knowledge to resolve syntactic ambiguities, this success may reflect a reliance on stereotypes and explicit biases embedded in their training data @xcite @xcite , leading to rigidity in interpretations and reinforcing existing biases. in contrast, humans these findings offer important insights into how llms and humans process syntactic ambiguities and incorporate world knowledge, carrying substantial implications for both the advancement of llm technology and the understanding of human language processing. while llms are becoming increasingly sophisticated, they do not yet fully capture the intricate interplay of syntactic, semantic, and pragmatic factors that characterize human language processing. human comprehension involves dynamic cognitive strategies that adapt to evolving social norms, contextual cues, and world knowledge. integrating insights from psycholinguistics and cognitive science is essential for developing llms that replicate not just outputs but also the underlying cognitive mechanisms of human language processing. future models would benefit from this integration, enabling them to handle ambiguities and culturally specific knowledge in a manner that reflects genuine human adaptability and flexibility. ## conclusion in summary, while llms have made considerable strides in handling syntactic ambiguities and incorporating world knowledge, they still exhibit limitations compared to human language processing. addressing these gaps will require a concerted effort to train llms with more diverse and nuanced data, enhancing their adaptability and integrating insights from human cognition to create models capable of truly human-like language comprehension. only by incorporating more context-sensitive, pragmatically rich, and culturally diverse training data can we develop llms that approach the depth and flexibility of human language understanding. by doing so, we will not only advance the capabilities of llms but also gain deeper insights into the complexities of human language processing, bridging the gap between artificial and human intelligence. limitations @xcite point out that language models' syntactic acceptability judgments improve in accuracy with longer context. our experiments contain only a single sentence, and it is possible that longer context would improve the models' alignment with human judgment. we sought to create an interdisciplinary team to ground our research well in both linguistics and natural language processing. along the way, we discovered several hurdles involved in this type of work. we, as linguists and computer scientists, speak different languages foot_3 . we look at data differently, use different terms, and consider different outcomes significant in different ways. each of these differences posed a barrier, and overcoming these barriers grew our capabilities as interdisciplinary researchers. we have learned much from each other, and would encourage others to take on similar challenges.