When Your Language Model Cannot E ven Do Determiners Right: Probing for Anti-Presuppositions and the Maximize Presupposition! Principle (2023)

Abstract: The increasing interest in probing the linguistic capabilities of large language models (LLMs) has long reached the area of semantics and pragmatics, including the phenomenon of presuppositions. In this study, we investigate a phenomenon that, however, has not yet been investigated, i.e., the phenomenon of anti-presupposition and the principle that accounts for it, the Maximize Presupposition! principle (MP!). Through an experimental investigation using psycholinguistic data and four open-source BERT model variants, we explore how language models handle different anti-presuppositions and whether they apply the MP! principle in their predictions. Further, we examine whether fine-tuning with Natural Language Inference data impacts adherence to the MP! principle. Our findings reveal that LLMs tend to replicate context-based n-grams rather than follow the MP! principle, with fine-tuning not enhancing their adherence. Notably, our results further indicate a striking difficulty of LLMs to correctly predict determiners, in relatively simple linguistic contexts.

## introduction presuppositions have held a significant position in semantic and pragmatic studies over the past decades (e.g., @xcite ). also in the research on linguistic knowledge represented in large language models (llms) @xcite @xcite , focus has shifted more and more towards exploring llms' capabilities in semantic and pragmatic discourse processing @xcite @xcite , encompassing the exploration of presuppositions @xcite . in this context, the primary emphasis has largely been on classification tasks, examining whether language models capture inferences triggered by presuppositions, using datasets that were developed in particular for natural language inference (nli) tasks; and it became apparent that llms mostly acquire surface-level patterns rather than deeply engaging with underlying knowledge representations @xcite @xcite . our goal in this paper is to explore whether presuppositions and, in particular, pragmatic principles underlying the phenomenon of presuppositions, are captured in llms that are pretrained on large text corpora. rather than relying on nli sentence classification tasks, which typically involve various aspects of semantic and pragmatic linguistic knowledge, we aim to introduce a controlled linguistic diagnostic following the idea of minimal pair analysis, which has been well established in work on diagnosing syntactic knowledge in llms @xcite . more specifically, in this study, our focus lies on a special kind of presupposition that, despite its comparable prevalence in language, has not yet received equivalent attention in research, the so-called anti-presupposition @xcite and, furthermore, the pragmatic principle that accounts for it, @xcite 's maximize presupposition! principle @xcite . given that anti-presupposition triggers are simple instances found in minimal pairs that exhibit a clear preference for one trigger over the other (see section 2), they present an optimal opportunity to explore the degree to which language models incorporate linguistic competencies, particularly pragmatic principles like the mp! principle. still, to the best of our knowledge, it has remained unexplored how llms deal with antipresuppositions and, beyond, whether they consider the mp! principle as part of their predictive processes. in this paper, we use a simple minimal pair analysis that tests the llms' ability to predict determiners, when being presented with contexts featuring anti-presupposition triggers, utilizing data sourced from the field of psycholinguistics. given that determiners rank among the most frequent words in many languages, it may be expected that llms are able to not only pick up their surface-level occurrence patterns, but learn deeper generalizations about their discourse-level functions and principles of use. yet, as we show in this paper, llms show a striking difficulty to correctly predict determiners in our diagnostic setup, challenging the idea that determiners pose only minimal challenges for llms @xcite and supporting the hypothesis that they struggle with fundamental aspects of linguistic reasoning. in the following, section 2 presents background on anti-presuppositions and the maximize presupposition! principle as well as prior research in the domain, section 3 describes the setup of our experimental investigation and section 4 describes the results. please hand me a pen. what is more, the indefinite determiner raises an intriguing linguistic question, leading us to the central focus of this paper: its classification is uncertain in terms of whether it triggers a presupposition or is more fittingly categorized as a so-called anti-presupposition @xcite . that is, certain expressions containing presupposition triggers might appear inappropriate in situations where the truth of a presuppositionally stronger element is implied -in other words, where the presuppositionally stronger element is part of the common ground (cf., e.g., @xcite , @xcite , @xcite ). for a clearer illustration, see (2), where the presupposition that there is precisely one sun is fulfilled in the common ground, therefore, making the sentence appear odd. more precisely: the determiner "the" carries a stronger presupposition compared to the indefinite determiner "a" (i.e., it is presuppositionally stronger), as "the" implies both existence and uniqueness. therefore, in (2), using the indefinite determiner becomes inappropriate, as it is established that only one entity of the mentioned type exists. similarly, uttering (1) is appropriate in a context with three pens and a single pencil, yet it is unsuitable in a context featuring only one pen and three pencils. in the latter, due to uniqueness, a definite determiner would be anticipated. like presuppositions, anti-presuppositions are associated with specific words or constructions. apart from the (in-)definite determiners, other triggers include both and all, as well as the verbs know and believe, see ( 3 ) and ( 4 ) from @xcite . ## experimental setup we investigate whether language models follow the mp! principle by analyzing their predictions involving two different anti-presupposition triggers. to carry out this investigation, we conduct minimal pair analyses. this paradigm involves contrasting two linguistic items that are nearly identical except for a single aspect and is, therefore, particularly well-suited for investigating anti-presuppositions as these tend to appear in pairs. furthermore, the technique of minimal pair analysis is not only com-monly employed in linguistic experiments, but it has also been shown by several studies that it can furthermore be a productive approach for investigating the linguistic properties captured in language models (e.g., @xcite linzen (2018); warstadt et al. (2020); cong (2022); hu and levy (2023) ). 1 data. we ground our study on german data from @xcite , who investigated the mp! principle with regards to definite and indefinite determiners in the context of visualized stories. 2 the data from @xcite offer a valuable avenue to investigate anti-presuppositions, given that (anti-)presuppositions are sensitive to context, i.e., their interpretation very much depends on the shared knowledge or beliefs among the conversational participants (cf. section 2). concretely, in their study, @xcite utilized mouse-tracking to examine how definite and indefinite determiners are processed, both when used felicitously and infelicitously. for this, participants were asked to judge the appropriateness of sentences in the context of a visualized story. each experimental trial began with a context that featured a shopping basket with three pieces of fruit, accompanied with a context sentence, such as "jan's mother was shopping. she bought one banana and two pears". then, participants were presented with the next part of the story, where jan received one of the initially introduced fruit (e.g., "of these, jan received the banana."), and asked to judge this stimulus sentence against the provided context by selecting "true" or "false" response boxes. @xcite 's study had six conditions, resulting from combining two determiners (definite vs. indefinite) with three types of sentences (false vs. felicitous vs. infelicitous). while in infelicitous conditions the (anti-)uniqueness presupposition of the determiner used in the stimulus sentence was violated, in felicitous conditions, the context satisfied this presupposition: that is, for definite determiners, jan received the unique fruit, while for indefinite determiners, jan received one of the non-unique fruits. cf. (5) for sentences examples of the felicitous conditions. 3 (5) context: jan's mother was shopping. she bought one banana and two pears. a. felicitous definite: of these, jan received the banana. b. felicitous indefinite: of these, jan received a pear. @xcite find, among other things, that, for both felicitous and infelicitous conditions, the mouse cursor's deviation towards the final response location commenced later for sentences with indefinite determiners, and in general, that there was a delay in response for infelicitous sentences. their findings, thus, indicate that processing indefinite determiners is more challenging than processing definite determiners, providing evidence in support of the mp! principle. prompts. we utilize @xcite 's experimental items to construct prompts, in which we mask the position of the definite and indefinite determiners, respectively, allowing us to test the language models' predictions for these words (and with it to investigate whether determiners really are "easy" words for llms @xcite ). prompts always consist of a context sentence, followed by the sentence that contains the masked determiner, i.e. they follow the format in ( 6 ). just like in @xcite 's study, the second sentence of a prompt starts with the word "davon" ("of these") to underscore that jan received a fruit from the three fruits introduced in the context sentence. to evaluate if the choice of the proper name "jan" influenced the language models' performance, we experimented with alternative names, including potentially more internationally recognized ones like "tom" or "peter." as it turned out, the choice of names did not yield any discernible differences in the results. therefore, we opted to retain of any fruit starting with a vowel further offers the advantage of obviating the need to elicit the probability of english "an". conditions. in our study, we make use of the felicitous definite and felicitous indefinite conditions (cf. ( 5 )), which we refer to as unique fruit and non-unique fruit condition, respectively (cf. figure 1 ). furthermore, we go beyond @xcite 's original design and introduce supplementary conditions to enhance the robustness of our design and derive more insightful conclusions about the potential existence of the mp! principle in language models. that is, on the one hand, we extend the original german data with its corresponding english translations, following the pattern as seen in ( 6 ), for instance. firstly, this allows to explore possible crosslingual differences. secondly, as in english we introduce the unique fruit with the numeral "one" (e.g., "she bought one banana"), this allows us to investigate the inability to interpret the indefinite determiner as a numeral, a contrast to the german "eine" where such an interpretation is feasible (cf. schneider et al. (2019) ). on the other hand, we include another minimal pair in our investigation, namely the antipresupposition triggers "beide" (both) and "alle" (all), referred to with the pair of fruits condition (cf. figure 1 ). these triggers are not only well compatible with the experimental items, they further allow us to investigate potential differences between anti-presupposition triggers and to compare our investigation to other studies that included these triggers, e.g., @xcite (see section 2). to investigate both and all, we retain the sentence structure and keep the masked token at the same position as in (6), and simply change the num-ber of the received fruit to the plural form. here, as both is "presuppositionally stronger" than all (see section 2), language models should predict "both" rather than "all" if they follow the mp! principle in their predictions. all together, for both english and german, we investigate whether llms adhere to the mp! principle when making predictions with the conditions summarized and exemplified in figure 1 . 4 in accordance with mp!, the language models should predict the word highlighted in green with a higher score compared to the word highlighted in red. models. we use the hugging face framework for reproducibility, employing their fill-mask-pipeline and the models listed below: we focus our investigation on bert and bert model variants. as we are interested in evaluating whether language models adhere to the mp! principle, and fine-tuned models do not necessarily reflect the linguistic properties of language models in general (cf. ettinger (2020) or chang and bergen (2023)), we center our investigation on the predictions of these base models. however, due to less favorable outcomes regarding the models' compliance with the mp! principle, we also investigate whether there could be an effect from fine-tuning these models on exisiting nli datasets. furthermore, even though recent llm evaluations strongly rely on chatgpt or gpt-4 (e.g., cai et al. ( 2023), kocoń et al. (2023) ), we omit such models from our analysis. this is motivated by the absence of token probability access through the openai api. furthermore, in line with @xcite , we are of the view that that restricting our interactions with llms to high-level prompting might result in missing the opportunity to measure and understand their linguistic capabilities more comprehensively. for fine-tuning the models, we make use of @xcite 's superglue benchmark. 5 we preprocess the data by concatenating the pertinent input sequences ( similar to raffel et al. (2019) ) and then masking the specific minimal pairs of interest (i.e., the words "the," "a," "all," and "both"). we fine-tune all models on approximately 10.000 datapoints, using three epochs, a learning rate of 2e-5 and a weight decay of 0.01. as the super-glue benchmark is available in english only, we investigate fine-tuning only for english data. 6 evaluation. to assess whether the llms adhere to the mp! principle in their predictions, we prompt the models with our sentences and get their 150 predicted words and respective probability scores for the masked token. our assumption is that a language model adheres to the mp! principle in its predictions when the score for the felicitous token (e.g., "the"/"die" in the unique fruit condition) surpasses the score for the infelicitous token (e.g., "a"/"eine" in the unique fruit condition). we calculate ettinger (2020)' metrics of completion sensitivity and prediction accuracy. with completion sensitivity, we measure the percentage of items for which the models assign a higher score to the felicitous token than to the infelicitous token. with prediction accuracy, we calculate the percentage of items for which the felicitous token is among the model's top k predictions. further, in the tables in appendix a.2, we collect all mean probability scores. the efficacy of using such direct probability measurements for assessing language model generalization capabilities has been shown by, for example, @xcite and @xcite . ## results and discussion determiners. figure 2 depicts the probability scores for the definite and indefinite determiners, for german and english data, aggregated over the individual models (but see with this distribution it may seem that the models adhere to the mp! principle in their predictions at least for the non-unique fruit condition. however, we posit that this observed distribution, instead, is a result of the llms merely repeating the singular determiner or numeral present in the context of the prompt. this interpretation is supported by the crosslingual comparison: in german, the determiner "eine" is a homonym for the indefinite determiner "a" and the numeral "one" in english, and the german llms repeat this word in all conditions. plus, in the english prompts, where the unique fruit is introduced with the numeral "one", the llms clearly favour to predict this numeral in the continuation, across both conditions, as shown in figure 2 . turning our attention now to the results of the fine-tuned models which are presented in appendix a.2.1. first, in a clear inclination toward predicting the indefinite determiner more strongly than the definite determiner, again also within the unique fruit condition. this is also supported by the completion sensitivity scores in rather, the results indicate that masking the determiners during fine-tuning has caused the models to overly predict these words without effectively capturing underlying linguistic patterns. if we now briefly examine the scores of the finetuned models for this condition too, as shown in the tables in appendix a.2.1, it is evident that, even though there seems to be some enhancement in predictions concerning the adherence to the mp!, across all models, numbers (including those that do not make sense in the given context) are still predicted with the highest probability. these results, thus, again suggest that fine-tuning on nli data does not enhance the models' capacity to adhere to the mp! principle in their predictions. instead, they underscore that the models continue to replicate patterns from the context of the prompts, much like the outcomes observed in the other conditions. the rather limited difference in values between finetuned and base models for "all" and "both," unlike the distinct contrast observed for "the" and "a", could potentially stem from the less frequent occurrence of the words "all" and "both" in the datasets compared to the determiners. 7 see section a.2.1 for a more detailled discussion of the results of the fine-tuned models in the pair of fruits condition. ## conclusion this paper investigated whether llms follow the mp! principle by analyzing their predictions involving two different minimal pairs of antipresupposition triggers, i.e., expanding prior work on probing the semantic and pragmatic discourse knowledge captured in llms. our findings reveal that the language models we studied do not follow the mp! principle when predicting masked determiners but, instead, tend to repeat words from the given input prompt. furthermore, we find that fine-tuning language models on nli datasets does not enhance their capacity to adhere to the mp! principle; rather, it appears to lead to an excessive prediction of the triggers of interest, devoid of capturing underlying linguistic patterns. the observed deviation from the mp! principle contrasts with the assumption that determiners only pose minimal challenges for llms @xcite and lends further support for the hypothesis that these models cannot truly grasp and reflect fundamental aspects of language use that govern presupposition and pragmatics. however, it's important to acknowledge that our conclusions are based on our choice of models and a relatively limited range of examples, and further research is needed to confirm and qualify this observation. considering that (anti-)presuppositions are omnipresent in everyday language, it is imperative for language models to effectively capture them, for example, in tasks such as question answering @xcite . therefore, future research directions might involve exploring the potential utility of visual question answering tasks to gain deeper insights into the models' understanding of prompts and whether they are able to appropriately represent discourse entities within them.