Do Pre-Trained Language Models Detect and Understand Semantic Underspecification? Ask the DUST ! (2024)

Abstract: In everyday language use, speakers frequently utter and interpret sentences that are semantically underspecified, namely, whose content is insufficient to fully convey their message or interpret them univocally. For example, to interpret the underspecified sentence “Don’t spend too much”, which leaves implicit what (not) to spend, additional linguistic context or outside knowledge is needed. In this work, we propose a novel Dataset of semantically Underspecified Sentences grouped by Type (DUST) and use it to study whether pre-trained language models (LMs) correctly identify and interpret underspecified sentences. We find that newer LMs are reasonably able to identify underspecified sentences when explicitly prompted. However, interpreting them correctly is much harder for any LMs. Our experiments show that when interpreting underspecified sentences, LMs exhibit little uncertainty, contrary to what theoretical accounts of underspecification would predict. Overall, our study reveals limitations in current models’ processing of sentence semantics and highlights the importance of using naturalistic data and communicative scenarios when evaluating LMs’ language capabilities.

## introduction speakers can almost effortlessly deal with semantic underspecification, a widespread phenomenon that occurs when a linguistic signal does not fully convey all the information required for communication to succeed @xcite . this is possible because, in a normal state of affairs, humans have access to further linguistic or extra-linguistic information coming from the conversation, the surrounding context, or shared knowledge. if this is the case, speakers will have no trouble understanding the sentences "i'll meet you there" or "i saw you on the hill with the telescope", even though these sentences leave underspecified where "there" is, or which interlocutor had "a tele- ## dataset to study how lms deal with semantically underspecified language, and since there currently exists no comprehensive resource on underspecification, we construct dust, a dataset of underspecified sentences by type, consisting of 2,123 english underspecified sentences and equally many specified counterparts, based on egg's (2010) categorization; see type 1 egg (2010)'s first type consists of semantically and syntactically homogeneous expressions, i.e., sentences with multiple readings that all share the same syntactic structure, and word/token-level meaning. to cover this type of underspecification, we collect 53 sentences from the language and vision ambiguities (lava) dataset @xcite , a multimodal dataset containing ambiguous sentences and visual data that disambiguated them. for each sentence in this dataset, we created a more specified counterpart using its visual disambiguations. an example of such a sentence is andrei approached danny; yevgeni, too. which leaves underspecified whether yevgeni approached danny, or was approached by andrei. a potential specified version of this sentence is andrei approached danny; andrei approached yevgeni, too. type 2 the second type consists of semantically but not syntactically homogeneous expressions. the multiple readings of such expressions stem from the multiple ways of analyzing the expression's structure; different analyses can lead to different meanings. the lava dataset (berzak et al., 2015) provides 108 sentences containing vp and pp attachment ambiguity and conjunction ambiguity, for example andrei looked at the green bag with a telescope. type 3 the third type consists of syntactically but not semantically homogeneous expressions, which share a single syntactic structure but do not share the same semantic material in its constituent parts. it is unique in that, besides instances of referential ambiguity, all examples of type 3 in dust are underspecified but not ambiguous. it is thus particularly interesting for our work, as underspecified but not ambiguous expressions are important for lms to handle, but currently understudied. examples include deictic expressions and expressions that are underspecified due to missing information. as examples of this type of underspecification, we first collect 89 sentences from lava that contain referential ambiguity or missing information. we then extend our collection with sentences from the clarifying insertions from revision edits (claire) dataset @xcite , which consists of wikihow instructional texts, and revisions that clarify the original sentences by inserting additional information. we treat the pre-edit text as underspecified due to missing information, and the post-edit text as more specified. due to the original authors' pre-processing, some pre-edit sentences are ungrammatical; we thus score pre-edit sentences' grammaticality with gruen @xcite and exclude low-scoring sentences. we also include the original 273 sentences from the winograd schema challenge (wsc; @xcite in our dataset as examples of referential ambiguity. for each sentence from this dataset, we crafted a more specified counterpart by changing the gender or plurality of one of the antecedents in the sentence, removing the referential ambiguity. an example sentence of this type is don't spend too much. ## models we focus on pre-trained autoregressive models, including both older and newer (generally stronger) lms to consider the influence of general performance improvements on lms' underspecification processing. as our experiments require sentence probabilities, we use only openly available models, which provide these. specifically, we consider the following models, accessed using huggingface's transformers library @xcite ): foot_1 except for flan-t5, the models we use in our research were not instruction-tuned during pretraining. this allows us to test the abilities of base lms to process underspecification. ## detecting semantic underspecification we test whether lms recognize that a sentence is more or less specified by looking at the perplexity it assigns to a prompt comparing the degree of (under)specification of two embedded sentences. this task seeks to replicate the process of assessing human speakers' understanding of this phenomenon by asking them to provide a metalinguistic assessment. we use a perplexity-based approach, rather than evaluating the generative behavior of the models in response to a prompt, as there is growing evidence that prompt-based approaches are not suitable for this purpose @xcite . for comparison, we include a preliminary experiment exploring model-generated responses in appendix d. ## interpreting underspecified and specified sentences the results of our first experiment suggest that some lms may be able to recognize underspecification when asked about it. however, as discussed above, there is no guarantee that the results obtained using metalinguistic prompts are indicative of the actual capabilities of lms. in the second experiment, we use a more naturalistic setting where underspecification is not mentioned in the prompt. ## discussion underspecification, much like ambiguity @xcite , remains a challenging phenomenon for lms. older lms, such as gpt-2, perform near chance level at recognizing underspecification; newer models, such as llama 2 and mistral, perform much better, but still leave ample room for improvement. processing sentences containing underspecification is an even harder task for lms. models seem to fail to recognize when underspecified sentences license continuations that their more specified counterparts do not; moreover, their interpretations of more specified sentences are often incorrect. the striking difference between the results of our two experiments highlights the importance of carefully choosing a setup when evaluating model capabilities. in the first, metalinguistic prompts elicited good underspecification judgments from high-performing models. but, in surprising contrast to previous work @xcite , testing models' ability to process underspecification in a more naturalistic setting led to lower performance. this is important: lm use cases involving underspecification will most likely involve processing underspecification, rather than identifying it upon explicit request. our second experiment may thus be a better indicator of lms' practical abilities. however, future work may be needed to compare lm processing of underspecified sentences to results of human studies, which have shown that speakers do have default interpretations of such sentences @xcite . by introducing dust and studying underspecification in lms as distinct from ambiguity, we have taken the first step towards evaluating lms' performance on a commonplace but understudied phenomenon that can affect lm behavior. our findings show that current lms are limited in their ability to deal with underspecification, especially in genuine communicative scenarios. hence, a thorough evaluation of the abilities of lms should include (various facets of) underspecification, unlike current benchmarks, in which ambiguous and underspecified sentences are often systematically excluded. we hope that our research further showcases the relevance of underspecification as a direction of research in the study of language models.