We Understand Elliptical Sentences, and Language Models should Too: A New Dataset for Studying Ellipsis and its Interaction with Thematic Fit (2023)

Abstract: Ellipsis is a linguistic phenomenon characterized by the omission of one or more sentence elements. Solving such a linguistic construction is not a trivial issue in natural language processing since it involves the retrieval of non-overtly expressed verbal material, which might in turn require the model to integrate human-like syntactic and semantic knowledge. In this paper, we explored the issue of how the prototypicality of event participants affects the ability of Language Models (LMs) to handle elliptical sentences and to identify the omitted arguments at different degrees of thematic fit, ranging from highly typical participants to semantically anomalous ones. With this purpose in mind, we built ELLie, the first dataset composed entirely of utterances containing different types of elliptical constructions, and structurally suited for evaluating the effect of argument thematic fit in solving ellipsis and reconstructing the missing element. Our tests demonstrated that the probability scores assigned by the models are higher for typical events than for atypical and impossible ones in different elliptical contexts, confirming the influence of prototypicality of the event participants in interpreting such linguistic structures. Finally, we conducted a retrieval task of the elided verb in the sentence in which the low performance of LMs highlighted a considerable difficulty in reconstructing the correct event.

## introduction a key phenomenon of natural languages is ellipsis, the omission of a word or phrase that is expected to occupy a place in the syntactic structure of a sentence (mcshane, 2005). @xcite elliptical sentences are usually composed of a standard sentence (aka antecedent clause) and an elliptical clause, which is not fully propositional and apparently not wellformed from a syntactic point of view @xcite literature tends to distinguish between syntactic and semantic ellipsis. here we focus on the former type. and jackendoff, 2005). consider the following example, where the antecedent is underlined and the elliptical one is characterized by the verb omission: the engineer completed the project, but the student didn't. since ellipsis represents a deviation from the simple compositional mapping between form and meaning, elliptical sentences have been the focus of many studies that seek to investigate how ellipsis is mentally represented, how the interpretation of the elided material is recovered, and consequently, how meaning can arise in the absence of form (ginzburg and sag, 2000; schwabe and winkler, 2003; culicover and jackendoff, 2006; jacobson, 2012; merchant, 2013, 2018; van craenenbroeck and temmerman, 2018). over the years, such theoretical discussions have proven the presence of a structural parallelism between the two sentence components through which ellipsis resolution mechanisms can be activated. currently, the most popular one is the indirect licensing mechanism (culicover and jackendoff, 2005) which rejects any kind of hidden (syntactic) level in the ellipsis site and involves a semantic identity procedure that consists of the recovery of linguistic material in the syntactic structure of the antecedent which, therefore, becomes relevant not only to the interpretation of the elliptical clause but also to its syntactic well-formedness. @xcite elliptical items (aka orphans) are licensed by this inter-clause parallelism or by a single lexical licensor in the antecedent. in many cases, however, the establishment of such a co-reference relation with some contextual elements does not guarantee the perfect resolution of this syntactic gap and the speaker must search for a link to a real-world referent, relying on external event knowledge. for such reasons, ellipsis resolution is not a trivial task in human and machine language processing. the goal of this work is to explore the ability of lms to cope with elliptical sentences and to recover the missing elements. in particular, we investigate the role of event knowledge in ellipsis resolution. we focus our attention on verbal ellipsis, and ask the question whether different degrees of thematic fit (mcrae and matsuki, 2009), that is the compatibility between the omitted verb in the ellipsis site and its arguments, affect the capacity of a language model to interpret such linguistic structures. for example, in (1) there is a high thematic fit in the antecedent clause between the predicate completed and the two arguments engineer (as an agent) and project (as the patient/theme). the thematic fit relation defines a typicality gradient, ranging from highly typical, preferred arguments to violations of the selectional restrictions of the verb, at the lower side of the spectrum. are thematic fit relations transferred to elliptical clauses? are typical verbargument combinations somehow facilitating the job in reconstructing a full semantic representation when the verb is being omitted? with those questions in mind, we explore the issue of how the prototypicality of event participants affects lms in handling elliptical sentences, and whether these models are able to identify the omitted elements at different degrees of thematic fit. our contribution to these issues is the creation of ellie, foot_1 the first dataset of elliptical utterances which is perfectly suited for a dynamic evaluation of thematic fit since it is composed of sentences that differ for their filler-argument typicality, ranging from highly typical to semantic anomalous ones. the paper is organized as follows. section 2 discusses previous works in this specific research area. section 3 presents the design and structure of ellie. in section 4, we discuss the experiments conducted with the lms on ellie. section 5 reports and discusses the results, while section 6 shows how these can lead to further research. ## the ellie dataset to the best of our knowledge, ellie is the first dataset created to explore the complexity of the ellipsis phenomenon and its relation with thematic fit. its structure was conceived to include multiple types of elliptical constructions, covering different thematic roles, and with the omitted elements (i.e., the verb or the whole verb phrase) having different degrees of thematic fit with the arguments in the context. the dataset is useful to investigate to what extent computational models encode the structured semantic information necessary for ellipsis resolution, and use it to make an accurate representation of the event context. ## experiments we used ellie as an evaluation dataset to test two transformer-based lms and analyze their behavior with elliptical constructions. ## results and analysis we report here the results of the experiments carried out on ellie. figures 1 and 2 show the probability distribution of sentences in the five candidate filler typicalityconditions extracted both from gpt-2 (figure 1 ) and bert (figure 2 ). as can be seen from the two sets of boxplots, the models' behavior is quite @xcite such an evaluation method might look strict, but we think it is consistent with the linguistic properties of the ellipsis phenomenon: the elliptic gap corresponds to an exact copy of some material in the antecedent clause. similar: they can assign significantly higher scores to the t-t condition compared to the conditions containing an atypical filler (i.e., t-at, at-t and at-at) or to the conditions including a selectional preference violation (t-sp_v). by contrast, both the models are unable to make a meaningful distinction between atypical conditions and a selectional preference violation (t-sp_v). statistical significance was assessed with the kruskal-wallis test, followed by a pairwise wilcoxon test to examine among which pairs of conditions differences were statistically significant. this shows that gpt-2 and bert apparently cannot distinguish a plausible (even if atypical) event from an impossible one, when such events occur in elliptical constructions. furthermore, we observe that the patient role is the most affected one by argument atypicality or by semantic preference violation among all thematic roles, as it records the lowest probability scores (see ## conclusion in this paper, we proposed a new framework to evaluate ellipsis and its relationship with thematic fit and selectional preferences. we did this by creating ellie, the first dataset composed of elliptical utterances and structurally suited for estimating the effect of argument thematic fit in solving ellipsis. we tested two lms with a transformer-based architecture in three different tasks to understand whether their ability to process elliptical constructions is affected by argument typicality and event knowledge. experimental results suggest a limited mastery of elliptical sentences and a significant influence of prototypicality of event's participants. moreover, the tested models greatly struggle to recover the missing elements of elliptical clauses and, thus, to reconstruct the whole event context. their performance (especially in task 3) may also depend on the low occurrence of such constructions in the training corpora, since the ellipsis phenomenon tends to be more frequent in speech than in writing. finally, the influence of event typicality suggests that lms tend to rely on frequent lexical co-occurrences, without being able to reconstruct the implicit syntactic and semantic structure necessary to interpret elliptical sentences.