LIEDER : Linguistically-Informed Evaluation for Discourse Entity Recognition (2024)

Abstract: Discourse Entity (DE) recognition is the task of identifying novel and known entities introduced within a text. While previous work has found that large language models have basic, if imperfect, DE recognition abilities (Schuster and Linzen, 2022), it remains largely unassessed which of the fundamental semantic properties that govern the introduction and subsequent reference to DEs they have knowledge of. We propose the Linguistically-Informed Evaluation for Discourse Entity Recognition (LIEDER) dataset that allows for a detailed examination of language models’ knowledge of four crucial semantic properties: existence, uniqueness, plurality, and novelty. We find evidence that state-of-the-art large language models exhibit sensitivity to all of these properties except novelty, which demonstrates that they have yet to reach human-level language understanding abilities.

## introduction one central component of language understanding is the ability to recognize entities in text. a large body of research in natural language processing focuses on the task of named entity recognition, where a system must identify whether a noun phrase (np), typically a proper name, refers to a known individual of a certain semantic class @xcite . the recognition of discourse entities (des), in contrast, involves identifying not only the occurrence of known entities but also novel ones that are introduced within a text. the recognition of des takes place at two sites: introduction sites and reference sites. introduction refers to the first time where an entity appears in a discourse. reference sites are subsequent mentions of an entity that has been previously introduced. as humans, not only are we able to recognize des at both of these sites, but we also have knowledge of how to coordinate the introduction and subsequent reference to entities using appropriate linguistic means. for example, we know that the introduction of des is typically done using indefinite nps such as a man in 'a man walked into the room. @xcite we also know that subsequent mentions often involve definite nps like the man in 'the man sat down.' de recognition is an important component of more complex semantic understanding tasks such as coreference resolution. coreference relationships cannot be established between entities that have not been introduced into the discourse. semantics research has established properties of definite and indefinite nps from which their use in introducing and referring to entities follows, four of which are particularly relevant here: existence, uniqueness, plurality, and novelty. we will define and discuss these properties in detail in section 3. a good language model (lm) should reflect knowledge of all of these properties. in this paper, we provide a novel dataset, which builds on schuster and linzen's work, that provides a method of testing these properties directly. 1 our results, across a number of state-of-the-art (sota) large language models (llms), provide evidence for knowledge of existence, uniqueness, and plurality (all conditions on the use of definite nps to refer to des), but difficulty with novelty (a condition on the introduction of des by indefinite nps) unless information about distinctiveness is made explicit. in addition, we find that transformer lms, unlike humans, show strong sensitivity to linear distance in establishing de reference. taken together, these results suggest that sota llms do not reach human-level language understanding abilities. a. f: his dog follows him everywhere. b. i: his cat follows him everywhere. sl argue that if a language model is sensitive to the difference between contexts which do and do not introduce des (such as the two conjuncts in the first sentence in (2)), we would expect the following inequality to hold for the probabilities assigned to felicitous (f) and infelicitous (i) continuations: p(f|context) > p(i|context) they found that the models they examined (which included gpt-2 variants and gpt-3) showed above chance performance on distinguishing f and i continuations in the context of affirmative-negation, with gpt-3 showing the most human-like performance. sl further explored the systematicity of these contrasts, where a contrast is counted as systematic only when the model correctly predicts felicity on all possible orderings of the operators and nouns in the conjoined clauses that comprise the context: while intriguing, these results do not tell us what specifically causes difficulty for llms in this task, and how it relates to semantic properties of definite and indefinite nps. to evaluate language models' de recognition abilities, we believe that it is important to decompose models' performance in a more granular manner. we turn now to a paradigm that builds on this previous work to do precisely this. ## criteria for discourse entity recognition informed by theoretical research in natural language semantics, we propose that a thorough evaluation of de recognition and reference abilities should examine language models' knowledge of four fundamental semantic properties: existence, uniqueness, plurality, and novelty. we will define and discuss each of them in order. existence as sl argued, a language model with human-level understanding abilities should only use definite descriptions to refer to entities that have been introduced into the discourse. we define this requirement as existence @xcite . for example, given the context john doesn't own a dog, a language model should recognize that the continuation the dog barks at night is infelicitous because of the non-existence of a dog de. uniqueness a language model should use a singular definite description to refer to a previously introduced entity only when the referent is unique relative to the discourse. we will call this requirement uniqueness @xcite . for example, given the context john owns a dog and mark owns a dog too, the model should consider the continuation the dog barks at night as infelicitous. since more than one dog has been introduced into the discourse, the singular definite phrase is not licensed. on the other hand, if the context is john does not own a dog but mark owns a dog, uniqueness is satisfied, so the same continuation should be judged to be felicitous. plurality a language model should use a plural definite description only if the set of des contains more than one individual of the relevant sort, a requirement we call plurality @xcite . notice that uniqueness and plurality cannot be satisfied or violated at the same time -referent expressions that require uniqueness (i.e., singular definites) are infelicitous after contexts that satisfy plurality. in contrast, referent expressions that require plurality (i.e., plural definites) are infelicitous after contexts that satisfy uniqueness. for example, the context john owns a dog and mark owns a dog too satisfies plurality since there are two dog des, and therefore supports a plural but not singular continuation (i.e., the dogs bark/*dog barks). ## the lieder dataset the linguistically-informed evaluation of de recognition (lieder) evaluation dataset adapts the structure of sl's paradigm: a context example is provided that consists of two conjoined clauses, each containing an indefinite np with the same head noun. this is followed by a continuation, test sentence containing a definite description. as in sl, we vary the conjoined clauses as to whether they introduce des or not (affirmative or negative). however, in order to evaluate the four linguistic properties defined in the previous section, we make two innovations: (i) we allow zero, one, or both of the conjoined clauses to include negation and thereby fail to introduce a de; and (ii) we allow the definite description in the continuation to be either singular or plural. example items in our dataset are shown in the context type column indicates which sides of the conjunction introduce des. for example, pos_neg indicates that a de is introduced in the first conjunct (pos), and no des are introduced in the second conjunct (neg). since we consider all four context types with both singular and plural continuations, there are 8 different contextcontinuation combinations. these 8 combinations are then crossed with 16 distinct base sentence pairs for the context and continuation, resulting in 128 examples in total. different combinations of context type and continuation result in differences in felicity of our two-sentence discourse. the singular continuation is felicitous only when the context type is either pos_neg or neg_pos since these contexts introduce exactly one de. in contrast, the plural continuation is felicitous only when the type is pos_pos. this means that of the eight context-continuation pairs, three are felicitous and five are infelicitous. success in the lieder dataset requires that a model accurately distinguish felicitous from infelicitous context-continuation pairs. because metalinguistic judgments elicited from language models may not reflect the full extent of the model's knowledge @xcite , we instead compare felicity using the probabilities the model assigns to the continuation given the context. we assume that the probability a model assigns to a felicitous case should be greater than the probability it assigns to an infelicitous one. with 3 felicitous pairs and 5 infelicitous ones, this means we have 15 informative probability comparisons in total. these are depicted in importantly, success in each of these comparisons can be tied to the linguistic requirements described in section 3 that are involved in the introduction of and reference to des. for example, if the model assigns higher probability to a continuation with a singular definite in a neg_pos context as compared to a neg_neg context, this provides evidence for the model's awareness of the exis-tence requirement that singular definite np imposes; otherwise, the singular should be possible in this context. on the other hand, if the model assigns higher probability to a singular definite in a pos_neg context as compared to a pos_pos context, this indicates that the model is aware of the uniqueness requirement imposed on singular definiteness and the novelty condition imposed on indefinites, since otherwise the pos_pos context could be taken to introduce only a single de. in addition, if the model assigns higher probability to a singular definite in a neg_pos context than it does to a plural definite in the same context, this provides evidence of sensitivity to the plural-ity requirement on plural definites, as the context introduces only a single entity. because of the correspondences between the example types and the linguistic requirements, lieder can therefore be used to assess the details of the knowledge of des in a language model. human judgments in addition to evaluating large language models with our dataset, we also conducted an experiment to elicit human judgments. specifically, participants were asked to provide a rating on the acceptability of a continuation given a context using a continuous slider with possible values ranging from 1 to 7. each participant provided ratings for 16 experimental sentences (one for each context type-continuation pair), each with different lexical content. this means that comparisons of items were done across subjects. using the prolific platform, we recruited 80 participants who were native english speakers with perfect or corrected vision and without language disorders. acceptability ratings on the same sentences were averaged, and the resulting averages were compared within an item to compute accuracies that could be compared to the language model results. see appendix a for our experimental interface and more details on methodology. ## experiment 2: facilitating novelty as described in section 4, pos_pos contexts introduce two des using two instances of the same indefinite description. for example, in the con-text sentence john owns a dog and mark owns a dog, two dogs have been introduced using the same indefinite description a dog. however, language models might have difficulty understanding that these are two different dogs because not only do they need to recognize des through distinct occurrences of the indefinite description a dog, but they must also know the novelty condition to consider these distinct occurrences as distinct des. ## discussion in experiment 1, we saw that all llms displayed clear knowledge of existence and plu-rality. they inarguably consider singular continuations to be bad after contexts that introduce no des and plural continuations to be good only when more than one de has been introduced. however, they failed to show clear knowledge of uniqueness, as indicated by their at-chance performance for the comparison type p(sg|pos_neg)>p(sg|pos_pos). we believe that such underperformance is not due to a lack of knowledge of uniqueness because the results in figure 5 support knowledge of the uniqueness requirement. instead, we believe that the problem lies in novelty. specifically, the models cannot identify that pos_pos contexts introduce two distinct des. we suspect that the pos_pos context might be particularly challenging because the two des are introduced using identical indefinite nps (i.e., the two occurrences of a dog in john owns a dog and mark owns a dog too). one might imagine that it is often sufficient to associate unique nps with distinct des. for example, given the sentence john owns a cat and a dog, the information that a cat and a dog are two distinct des is lexically encoded. if a model adopts such a lexically dependent strategy for de introduction, then the pos_pos cases from experiment 1 are exactly the expected point of failure, as such cases require sensitivity to nov-elty in order to succeed. indeed, as shown in experiment 2, supplying explicit information about distinctiveness does improve model performance. taken together, these results all point to language models' mastery of existence, uniquness, and plurality and a lack of knowledge on novelty when minimal information is given. ## related work the current paper contributes to the body of literature that examines the semantic knowledge of neural language models. @xcite assessed llm comprehension of functional words, including a subtask of differentiating indefinite and definite determiners. their stimuli were constructed by swapping a/an with the in a sentence. such manipulation implicitly encodes the properties identified in lieder: using the in the place of a often violates uniqueness and existence, and novelty could also be violated vice versa. on the discourse level, aside from schuster and linzen (2022) from which the current paper draws inspiration, @xcite examined contexts with different biases concerning the de that will be mentioned next. they found that unlike humans, gpt-2 @xcite and transformer-xl @xcite are not sensitive to the manipulation of contexts when predicting the entity that will be mentioned next. @xcite built probing models to investigate whether pretrained representations encode information about an entity being newly introduced or having been mentioned before (which they call discourse-new/-old respectively). they found that while a high f1 score can be achieved for the classification of discourse-new vs. discourse-old, models struggle to locate the entities within a sequence, a finding that is in line with our result that models lack understanding of the novelty requirement. lastly, @xcite studied llm ability to accurately represent the states of discourse entities across long narratives and observed that only llms that have been pretrained on code exhibited non-trivial entity tracking abilities. the lieder dataset also adds to the body of work that uses linguistic insights in the development of semantic benchmarks for llms. this work includes cogs @xcite , recogs @xcite , and slog @xcite on compositional generalization and imppres @xcite , nope @xcite , @xcite , and (qa) 2 @xcite that assess models' ability in handling presuppositions. ## conclusion in this paper, we proposed the linguistically-informed evaluation for discourse entity recognition (lieder) dataset. our paradigm allows for a detailed examination of language models' knowledge of four properties of definite and indefinite nps concerning their ability to introduce and refer to des. these properties are existence, uniqueness, plurality, and novelty. we demonstrated that despite mastering existence, uniqueness, and plurality, the llms we examined lack understanding of the novelty requirement. in spite of this deficiency, we showed that language models of the llama 2, llama 3, and gpt series reflected the human preference of referring to des that are introduced closer to their reference point, which we label an effect of distance. we recognize that given the fast-paced development of llm research, it is highly likely that the performance presented in the current paper will be surpassed by future generations of lms. however, the success of lieder in helping to identify language models' deficiency in de recognition highlights the importance of linguistic considerations in evaluating the strengths and weaknesses of future language models.