Scope Ambiguities in Large Language Models (2024)

Abstract: Sentences containing multiple semantic operators with overlapping scope often create ambiguities in interpretation, known as scope ambiguities. These ambiguities offer rich insights into the interaction between semantic structure and world knowledge in language processing. Despite this, there has been little research into how modern large language models treat them. In this paper, we investigate how different versions of certain autoregressive language models—GPT-2, GPT-3/3.5, Llama 2, and GPT-4—treat scope ambiguous sentences, and compare this with human judgments. We introduce novel datasets that contain a joint total of almost 1,000 unique scope-ambiguous sentences, containing interactions between a range of semantic operators, and annotated for human judgments. Using these datasets, we find evidence that several models (i) are sensitive to the meaning ambiguity in these sentences, in a way that patterns well with human judgments, and (ii) can successfully identify human-preferred readings at a high level of accuracy (over 90% in some cases).1

## introduction sentences like 'every farmer owns a donkey' are systematically ambiguous between two readings: one in which the embedded noun phrase (np) (e.g., 'a donkey') is interpreted within the scope of the quantifier that precedes it ('every'), and another in which the embedded np is interpreted outside its scope. as shown in figure 1 , 'every farmer owns a donkey', for example, could either mean (i) that each farmer simply owns their own (possibly unique) donkey, or (ii) that there is a specific donkey in question that all farmers jointly own. such constructions are examples of what are known as scope ambiguities. they are called so because the standard account of these ambiguities is that they arise when the respective scope of multiple semantic operators in the expression is ambiguous, yielding more than one possible semantic structure. consider the following example: importantly for this present work, english speakers (i) have access to both kinds of readings, and (ii) generally disambiguate between them to arrive at a preferred reading (see @xcite . for example, although (1a) has two possible interpretations, without further context, most people would prefer the surface reading, due to at least the surface positions of 'a' and 'every' in the sentence, as well as background world knowledge about farmers and donkeys (see @xcite @xcite , for insights into how such factors affect reading preferences). the focus of this paper is how large language models (llms) treat such ambiguities. assessing how they do so offers important insights into interactions between semantic structure and world knowledge, as well as the representation of scope in llms. ## background we focus on scope ambiguities involving quantifiers such as 'some', 'every', and 'most', as well as quantifier-like determiners, like indefinites and numbers. (1) is an example of scope ambiguity arising out of quantifier-quantifier interactions. but scope ambiguities can also arise out of quantifier-negation and quantifier-adverb interactions, as shown below: quantifier + negation: ## re-evaluation on expanded datasets the results from experiments 1a and 2a are promising, but rely on relatively small datasets that contain 153 and 29 unique datapoints, respectively-raising questions of how generalizable our results are. as a follow-up, we therefore considerably expand these two datasets, and rerun the same experiments described in sections 4 and 5 on the expanded datasets. ## discussion our results, which indicate that llms are both proficient at choosing the scope readings preferred by most humans, and sensitive to the meaning ambiguity in scope ambiguous constructions, offer further evidence of the capacity of large language models to induce semantic structure (see @xcite , and linguistic structure more generally (see @xcite . we rerun their experiment on this subset of our data; our results, shown in ## conclusion in this paper, we investigated how different autoregressive language models treat scope ambiguities. in doing so, we introduced novel datasets that contain a joint total of roughly 1,000 unique and diverse scope-ambiguous sentences, annotated for human judgments-the largest of this kind. our results indicate that llms are able to exhibit behavior in line with human preferences of interpretation-informed at least in part by background knowledge-as well as compatible with different types of semantic structures. finally, the contrast between our findings and those of other recent work emphasizes the need for diverse approaches in assessing the linguistic capacities of large language models.