When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it (2022)

Abstract: Understanding longer narratives or participating in conversations requires tracking of discourse entities that have been mentioned. Indefinite noun phrases (NPs), such as ‘a dog’, frequently introduce discourse entities but this behavior is modulated by sentential operators such as negation. For example, ‘a dog’ in ‘Arthur doesn’t own a dog’ does not introduce a discourse entity due to the presence of negation. In this work, we adapt the psycholinguistic assessment of language models paradigm to higher-level linguistic phenomena and introduce an English evaluation suite that targets the knowledge of the interactions between sentential operators and indefinite NPs. We use this evaluation suite for a fine-grained investigation of the entity tracking abilities of the Transformer-based models GPT-2 and GPT-3. We find that while the models are to a certain extent sensitive to the interactions we investigate, they are all challenged by the presence of multiple NPs and their behavior is not systematic, which suggests that even models at the scale of GPT-3 do not fully acquire basic entity tracking abilities.

## introduction in order to understand longer narratives or to participate in conversations, humans and natural language understanding systems have to keep track of the entities that have been mentioned in the discourse. for example, when someone tells you that arthur owns a dog, they have introduced the entity of a person named arthur and the entity of a dog owned by arthur into the discourse. once entities have been introduced to the discourse, it is natural to refer back to them with noun phrases or pronouns to elaborate further, e.g., by saying it has a red collar to elaborate on the dog. while no fully-specified account exists of how humans achieve this feat, many existing theories are based on the idea that humans maintain mental files (e.g., @xcite , i.e., explicit memory representations for each entity that encode all properties of an entity and its relation to other entities. when engaging in a conversation or reading a longer narrative, humans then update these representations as they encounter new entities or new information about existing entities. large pre-trained language models (lms) such as gpt-2 @xcite and @xcite , which in recent years have become the dominant foundation for natural language understanding and generation tasks, lack explicit representations of discourse entities. it remains largely an open question to what extent lms can match human behavior in tracking discourse entities. the most extensive investigation of this phenomenon has been through evaluations with the lambada dataset @xcite . lam-bada consists of a cloze task for which an lm has to predict the last word of naturalistic passages extracted from a corpus. solving this task requires keeping track of longer contexts, and making a correct guess frequently requires keeping track of the entities mentioned in the passage. while datasets such as lambada are an invaluable resource for monitoring high-level progress of lms in their ability to track discourse entities, such datasets lack the granularity to determine for which contexts lms can and cannot properly track discourse entities. in this work, we draw inspiration from recent targeted evaluation suites geared at lower linguistic levels (e.g., @xcite , and introduce a targeted evaluation suite for tracking of discourse entities in english. in particular, we focus on the interactions between different sentential operators and embedding verbs and indefinite noun phrases (see, e.g., @xcite and section 3); for example, we evaluate whether a language model correctly infers that because a sentence with a negation, such as arthur doesn't own a dog, does not introduce a dis-course entity for a dog, further elaborations about such a non-existent dog should be pragmatically odd, and, as such, their probability should be low compared to matched controls. to evaluate to what extent language models are sensitive to these interactions, we adapt the psycholinguistic assessment of language models paradigm @xcite for discourse entity tracking and discuss the methodological challenges that arise with using this paradigm for discourse phenomena. we introduce two expert-created evaluation suites and use them to evaluate gpt-2 and gpt-3 models. we find that while all the models we evaluated show some sensitivity to preceding context, they lack systematicity and are challenged when contexts contain multiple noun phrases. 1 ## related work the majority of systematic evaluations of autoregressive and masked language models so far have focused on the level of syntax, targeting abilities such as subject-verb agreement (e.g., @xcite @xcite , anaphora agreement and binding constraints (e.g., @xcite @xcite , or filler-gap dependencies (e.g., @xcite @xcite . at higher linguistic levels, ettinger (2020) compared bert's @xcite behavior on sentences with negation to data from neurolinguistic experiments with humans; @xcite investigated to what extent pre-trained language models can extract relevant information from the preceding context, both in the presence and in the absence of distractors; and @xcite investigated whether language models can predict connectives (e.g., but) for two given sentences. more closely related to our work, in the domain of discourse and reference, @xcite investigated whether gpt-2 and transformer-xl @xcite exhibit similar referential biases in their continuations as humans, i.e., they asked whether models provided with a sentence with two referents are biased similarly as humans when choosing the referent for the next sentence. @xcite used an acceptability judgment task to determine whether contextual lms correctly distinguish between definite and indefinite noun phrases. sorodoc et al. (2020) and tenney et al. (2019 ) used probing methods to investigate whether representations of lstm-and transfomer-based models contain information about coreference, which also provides some indication of entity tracking abilities. further, @xcite investigated to what extent attention weights of bert indicate coreference. these studies found that all evaluated representations contain some information about coreference but not consistently for all entities. ## background english indefinite noun phrases (nps) of the form a noun interact with the context in complex ways (see, e.g., @xcite @xcite , for more extensive discussions of this phenomenon). in affirmative statements, the indefinite np generally introduces a new entity to the discourse. however, several sentential operators and clause-embedding verbs modulate this behavior. for example, consider the following contrast between an affirmative and a negated sentence, where # indicates a pragmatically odd continuation: while in the affirmative sentence, the indefinite np introduces a novel discourse entity, the negation in (1b) prevents the np from introducing a new entity. in (1b), it is therefore pragmatically odd to refer back to a dog with the pronoun it. the implicative manage to and the negative implicative fail to in (2a-b) give rise to a similar contrast: the np under manage to introduces a discourse entity, the np under fail to does not. indefinite nps embedded under the factive know and the non-factive doubt introduce and fail to introduce a discourse entity, respectively: lastly, modals such as want also block the introduction of a discourse entity, as shown in (4): while these patterns generally hold, there are exceptions to these rules. for example, in the first sentence in (5), which could be paraphrased as (6), the indefinite scopes over the negation and therefore it is okay to refer back to the mistake. however, without additional context, listeners generally do not infer these so-called specific interpretations of sentences with an indefinite np, so like @xcite , we will largely ignore these cases throughout the remainder of this paper. ## experiments to what extent are gpt-2 and gpt-3 sensitive to the contrasts that we presented in section 3? to investigate this question, we adapted the methodology commonly used for syntactic evaluation of language models (e.g., @xcite and created minimal pairs of contexts that differ in whether they introduce a discourse entity or not. in experiment 1, we focus on contexts with a single indefinite np, and in experiment 2, we focus on sentences with multiple indefinite nps. ## likely continuations one drawback of the methodology of the previous two experiments is that we considered only one specific expected and one specific unexpected continuation for each item. thus, if both the expected and the unexpected continuations are very unlikely according to the lm, we may see poor performance on this task while at the same time, it would be very unlikely that either of the generations is ever sampled from the model. in that case, the evaluations in experiment 2 may underestimate the models' abilities @xcite and the results may not be very relevant for practical purposes for which one uses an lm to generate texts. for this reason, we also performed a manual analysis of randomly sampled generations @xcite ) from the two largest lms, gpt-2 xl and gpt-3. ## general discussion in his seminal work in 1976, karttunen introduced several challenges for natural language understanding systems that aim to track which entities are introduced in a larger discourse. in this work, we evaluated to what extent we made progress on these challenges in the past decades. in two sets of experiments, we found that transformer-based models are to some extent sensitive to different interactions between sentential operators and indefinite noun phrases. at the same time, however, we found in experiment 2 that models lack systematicity in their behavior, which suggests that models do not combine indefinite noun phrases and sentential operators as humans do. further, the analysis of likely continuations showed that this behavior can also be observed in high probability generations. learnability of meaning on the one hand, these results provide direct evidence for shortcomings of language models with respect to tracking entities. on the other hand, more broadly, these results also provide interesting data points with regard to the recent debate on whether language models could theoretically mimic human language understanding. bender and koller (2020) recently presented several thought experiments and argued that since lms are only trained on form and do not have access to meaning or intentions, they can never exhibit human-like language understanding (see also @xcite , for a more formal discussion of this claim). given that we evaluated the largest available gpt-3 model and still found that the model behavior is inconsistent despite its enormous amount of parameters and training data, our results suggest that at least current language model architectures indeed struggle with human-like understanding. interestingly though, while the thought experiments by @xcite focus on lack of world knowledge due to the lack of grounding of language models, our results suggest that additionally, language models fail at learning the meaning of more abstract words such as negation markers or embedding verbs. this is also in line with recent results, which showed that smaller models fail to learn the meaning of negation and discourse connectives. @xcite . lastly, the fact that gpt-2 and gpt-3 have been exposed to orders of magnitude more language data than human learners are and still do not fully succeed at tracking discourse entities underscores that there are differences between how humans and lms learn.