The Typology of Ellipsis: A Corpus for Linguistic Analysis and Machine Learning Applications (2024)

Abstract: State-of-the-art (SotA) Natural Language Processing (NLP) technology faces significant challenges with constructions that contain ellipses. Although theoretically well-documented and understood, there needs to be more sufficient cross-linguistic language resources to document, study, and ultimately engineer NLP solutions that can adequately provide analyses for ellipsis constructions. This article describes the typological data set on ellipsis that we created for currently seventeen languages. We demonstrate how SotA parsers based on a variety of syntactic frameworks fail to parse sentences with ellipsis, and in fact, probabilistic, neural, and Large Language Models (LLM) do so, too. We demonstrate experiments that focus on detecting sentences with ellipsis, predicting the position of elided elements, and predicting elided surface forms in the appropriate positions. We show that cross-linguistic variation of ellipsis-related phenomena has different consequences for the architecture of NLP systems.

## introduction ellipsis is a linguistic phenomenon that results in the omission of words in sentences that are usually obligatory in a given syntactic context and that the speaker and hearer can understand and reconstruct without effort. simple noun phrase (np) or forward conjunct reduction (fcr), as in example (1), is common cross-linguistically. the possibility to elide phrases or words in coordinated constructions has universal and languagespecific aspects to it. common fcr is possible in all languages we are aware of. it is not only possible but the preferred form of presentation in text or spoken language whenever coordination occurs. if ellipsis can be applied in unmarked cases, it is applied. the form in (1b) without ellipsis might be perceived as emphatic or, in a pragmatic or semantic sense, as specific, in contrast to the unmarked default in example (1a). other variants of ellipsis include so-called gapping, as in (2a) where the verb complex is reading is elided. in example (2b), a case of vp-ellipsis, the entire predicate or verb phrase (vp) is elided. context-dependent forms of ellipsis can be found in responses to questions, as in example (3). in the response (3b), the words each candidate will talk are elided. while english exhibits limited examples with lexical mismatches of elided word forms, as in example (4a), highly inflecting languages like hindi or @xcite show that the elided words do not have to be homophonous. the words in round brackets in (4) are preferably elided in unmarked contexts. as discussed in @xcite and @xcite , ellipsis constructions are very common and often accompanied by specific semantic effects. while various quantifier scope effects 1 can be observed in ellipsis constructions, common semantic issues involve so-called zeugma @xcite effects as in example ( 6 ). the second conjunct in example (6a) includes an idiomatic predicate causing semantic deviation without significantly impacting grammaticality judgments. we observed that nlp pipelines fail to provide appropriate syntactic structures for such sentences in downstream tasks and for common information extraction from business reports or medical documents. using ellipsis constructions from our corpus, we tested the most recent versions of stanza @xcite , spacy @xcite , benepar @xcite , and the xerox linguistic environment (xle) @xcite . none of the parse trees based on the different grammar formalisms were adequate in our evaluation. our team of syntacticians judged the adequacy of parse trees. this is true for sota dependency parsers, neural constituency parsers, as well as for rule-based systems like the xle-based lexical-functional grammar (lfg) parser using the english, german, or polish grammar. llms are as challenged with such constructions as these rule-based, statistical, or neural syntactic parsers. figure 1 shows an example in which the overt subject of the first conjunct is labeled as the subject. the same element is the syntactic subject and semantic object of the second conjunct. these functional relations are missing in the dependency 1 a discussion of semantic changes caused by quantifier scope effects in ellipsis constructions would go beyond the scope of this article. the following data and corpus creation and experiments were motivated by the fact that document types like business reports, medical or technical documentation, as well as social media content, chat, or spoken language discourse, contain a large number of sentences with ellipses. given that common sota nlp pipelines fail to provide adequate syntactic representations as tree structures, higher-level processing of discourse and semantic properties is not possible using their output. our motivation to create larger data sets for a larger group of languages was not only driven by this fact but also by a lack of a comparative typological description of ellipses in different languages and across different language groups. as the example in (4) shows, morphologically rich languages allow lexically matching words to be elided, although the morpho-phonological surface form does not match. this does not seem to be a challenge for native speakers of these languages. however, it is a significant computational challenge to identify the correct morpho-phonological forms that were subject to ellipsis. scattered ellipsis, as in example (5), does not appear to be cognitively challenging, either; however, from a machine learning (ml) and nlp perspective, we expect to see significant errors and issues in identifying the ellipsis slots and guessing the elided words. other typologically interesting aspects of ellipsis and cross-linguistic comparisons are related to the unmarked underlying word order. while vpellipsis might manifest itself in different ways in svo languages, it might result in very different surface phenomena in sov languages like hindi or german. in the german example (7) the vp containing the direct object and main verb (mutter helfen) can be elided in the first conjunct. while we have a good understanding of vpellipsis in english, it must be made clear whether an elided verb in a transitive predicate construction in hindi is similar to gapping or, rather, the result of partial vp-ellipsis. there are numerous research questions that we try to address. on the one hand, syntax internal constraints license sentence-internal ellipsis. in gapping constructions, the gapped verb is not necessarily licensed by discourse conditions and previously mentioned context. on the other hand, the ellipsis of discourse-introduced and -linked words and phrases cannot be assumed to be restricted by purely syntactic constraints. at the same time, complex gapping constructions seem to indicate that ellipsis is not restricted by syntactic phrase boundaries, but rather licensed by phonological correspondence of word sequences. as example 8 shows, the repeated word sequence can be elided, ignoring syntactic phrase structure boundaries. one interpretation of 8, the default one, implies that mary was always dreaming about going to tokyo. the elided sequence of words, in this case, does not match with clear-cut syntactic phrase boundaries. the corpus and research presented in this article are part of the hoosier ellipsis corpus (hec) project at the nlp-lab. the goal of the hec project is to provide a resource for qualitative and quantitative typological studies of ellipsis over different language types and groups, as well as to provide corpora for the evaluation and development of nlp pipelines that can generate semantically more adequate syntactic structures for ellipsis constructions. ## the hoosier ellipsis corpus the hoosier ellipsis corpus v 0.1 consists of data from seventeen languages. among those languages are low-resourced languages like navajo, a lan-guage of the athabaskan branch of the na-dené language family, and kumaoni, an indo-aryan language spoken in northern india and parts of western nepal, as well as common slavic languages (russian, ukrainian, polish), germanic languages (english, german, swedish), as well as hindi, arabic, japanese, and korean. the corpus includes the following ellipsis types: vp-ellipsis, sluicing, gapping, stripping, forward (fcr), and backward coordinate reduction (bcr). the collected data set consists of sentence pairs and possible contexts that precede or follow the target sentence in a text or discourse. the examples in the corpus are collected from linguistic and typological literature. example sentences from lowresourced languages were collected and validated by native speakers. we selected a simple unicode text-based format to encode the data using separator lines and line prefixes to indicate the data entry type. in the encoded data files, the target sentence with an ellipsis is followed by a line of 4 dashes. within the ellipsis target structure, three underscores indicate each canonical position of the elided word sequence. complex ellipsis constructions can contain numerous elided slots. the pair of sentences with ellipses and the full form are optionally accompanied by meta information indicated by lines that start with the hash symbol. in the meta-information lines we provide the opportunity to translate the sentence, to provide the original source of the example from publications, and to specify who contributed this example to the data collection. each example sentence in the data file is followed by at least one empty line. figure 6 shows a sample entry with the core elements. wird sie kommen oder ___ er gehen? ----wird sie kommen oder wird er gehen? # tr eng: will she come or will he go? # added by: john smith # source: wolfgang klein (1981) # some rules of regular ... this annotation format allows us to indicate and study the distribution of elided elements in the clause. it also provides the 'understood' or 'implied' sequence of words as understood by human native speakers. from a computational perspective, this format allows us to train models that detect the positions of elided elements in sentences. we can also train models that generate the elided word forms. we can use the data set to evaluate existing models and, in particular, llms, as discussed in the following section. the format allows us to convert most of the ellipsis and full-form pairs into the ud 2 format for encoding ellipsis. foot_0 at the same time, tree structures based on the different grammar formalisms can be encoded as bracketed-notation strings, triple sets for dependencies, or c-and f-structure strings in the meta-information section of each example. the ellipsis corpus is continuously expanded. many languages in the corpus are expanded using examples from peer-reviewed publications and theoretical or documentary linguistics publications. for low-resourced languages, we rely on contributions from native speakers and their speaker communities. while some of the languages are as of writing this article under-represented, we describe the following experiments and results for a couple of languages that we collected sufficient data on for training models and evaluating their performance, or testing the performance of pre-trained llms. ## nlp experiments: methods & results we designed three main experimental settings to test the capabilities of current sota nlp technologies. the tasks are described as follows: these tasks we compare across three different nlp-approaches: ## results in the zero-shot gpt-4 setting, we used the context "you are a linguistic expert." the prompt "classify the following sentence as containing ellipsis or not and return a 1 for a sentence with ellipsis and a 0 for a sentence without ellipsis" was preceding each sentence. we tested various llms, including gpt-4, gpt-3.5, falcon, llama, zephyr. we decided to focus on gpt-4 only, since none of the other llms turned out to be useful in any of the three tasks, given accuracies below 0.5. for task 1 for english the results are given in in task 1, the zero-shot gpt-4 experiment achieved using the arabic data resulted in a surprising accuracy of 0.87. in the default setting, the output from gpt-4 is not discrete for a given example sentence. with the temperature set to the default 0.7 when requesting a label for a sentence in task 1, 2 of 10 responses were the opposite label. when we reduce the temperature to 0, there are no mismatches; the judgments were deterministic. however, with the temperature set to 0, the accuracy was 70%. with the temperature set to 0.7, the accuracy was 75%. in task 2, we tested an initial bert-based ellipsis position guesser and achieved first test accuracy of 0.7. the gpt-4-based experiments on task 2 were challenging. the prompt engineering for the zero-shot experiment resulted in an accuracy of only 0.15 for the english data. for task 3, we exclusively focused on the evaluation of gpt-4. in this task, using the zero-shot strategy, we achieved an accuracy of 0.25 with gpt-4. ## conclusion ellipsis constructions are obviously still challenging for all the common sota nlp pipelines, including rule-based systems like the lfg-based xle. use of dependency or constituency parse trees, or even lfg c-and f-structures for syntactic and semantic processing of real-world data from different genres or registers is limited due to the fact that ellipsis is a common and widespread phenomenon in all languages. the problem can be partially linked to grammar frameworks like dependency grammar or lfg, which do not necessarily foresee opaque linguistic elements (e.g., elided words or phrases) to be active rule elements modeled in grammar rules or descriptive formal annotation frameworks. while ud provides the instruments for annotating or handling ellipses, those instruments need to be more extensive to describe the different intra-and crosslinguistic ellipses types. we also suspect that parsing algorithms and the training of parsers need to include such opaque elements and potentially new learning strategies. the fact that specific models trained on the prediction of ellipses in sentences outperform llms seems to indicate that the lack of explicit data and pure self-supervised machine learning is not sufficient to handle opaque elements in language, either. training llms on purely overt data ignores significant properties of language. ellipsis phenomena are grammatical and systematic, and it seems problematic for current llms to guess covert continuations. given that there is too little data on ellipsis in general, and none at all for most languages, it seems necessary to continue our ellipsis corpus project and provide not only sufficient data for the different languages, but also a good typological overview of the different manifestations of ellipsis phenomena in different languages and language groups.