Is neural semantic parsing good at ellipsis resolution, or isn’t it? (2025)

Abstract: Neural semantic parsers have shown good overall performance for a variety of linguistic phenomena, reaching semantic matching scores of more than 90%. But how do such parsers perform on strongly context-sensitive phenomena, where large pieces of semantic information need to be duplicated to form a meaningful semantic representation? A case in point is English verb phrase ellipsis, a construct where entire verb phrases can be abbreviated by a single auxiliary verb. Are the otherwise known as powerful semantic parsers able to deal with ellipsis or aren’t they? We constructed a corpus of 120 cases of ellipsis with their fully resolved meaning representation and used this as a challenge set for a large battery of neural semantic parsers. Although these parsers performed very well on the standard test set, they failed in the instances with ellipsis. Data augmentation helped improve the parsing results. The reason for the difficulty of parsing elided phrases is not that copying semantic material is hard, but that usually occur in linguistically complicated contexts causing most of the parsing errors.

## introduction semantic parsing is the task of providing a formal meaning representation for an input sentence of a natural language such as english, dutch, or italian. semantic parsing is crucial for applications that require the precise translation of unstructured data (i.e., text and images) into structured data (e.g., databases and robot commands). currently, the most promising approaches to semantic parsing are based on neural models @xcite @xcite trained or fine-tuned on large semantically annotated corpora @xcite , reaching high performance with f scores greater than 90%. little is known about the ability of neural semantic parsers to cope with ellipsis, a lin-guistic construction in which elements are omitted and are supplied by the discourse context. in this paper, we will study how neural semantic parsers deal with verb phrase ellipsis (vpe) in english. an example of a vpe is shown in (1) together with its fully expressed surface interpretation in (2). as far as we know, this is the first in-depth study of vpe interpretation in neural semantic parsing. related, but taking a different perspective, is work by @xcite , who found that large language models have difficulty processing ellipsis. in section 2 we give an overview of earlier computational approaches to vpe. in section 3 we introduce the parallel meaning bank (pmb) and a vpe challenge test set distilled from the pmb. in section 4 we outline our approach to enhance the semantic parsing for vpe, while in section 5 the parsing results are presented, showing that neural approaches face a difficult time in interpreting elliptical constructions, even with substantial finetuning, but not for the reasons we initially thought would cause the difficulty. data-driven approaches based on annotated corpora @xcite @xcite demonstrated the large gap between theoretical ideas and practical implementations @xcite @xcite , and were considered to be specific tasks rather than an integral part of widecoverage semantic parsing. in this paper, we take a different computational perspective and depart with an overall well-performing general-purpose semantic parsing and investigate how well it succeeds on ellipsis data. ## data the parallel meaning bank the pmb abzianidze et al., 2017 is a multilingual corpus enriched with semantic annotations, covering a wide range of linguistic phenomena. it contains a substantial set of parallel texts, each paired with a formal meaning representation known as a discourse representation structure (drs) based on discourse representation theory (drt, @xcite . while drss are typically presented in a human-readable box format, a clause-based linear representation was introduced by @xcite to enable their use in sequence-based models. more recently, @xcite proposed sequence box notation (sbn), a simplified, variable-free version of drs aimed at further facilitating sequence processing. in this paper we use sbn as meaning representation format (see figure 1 ). sentence with vpe: life never ends but earthly life does. drs: life.n.11 % life never [0-10] @xmath0.v.01 theme -1 time +1 % ends [11-15] time.n.08 equ now % @xmath1.a.01 attributeof +1 % but earthly [16-27] life.n.11 % life [28-32] end.v.01 theme -1 time +1 % does. [33-38] time.n.08 equ now % drg: life.n.11 negation contrast end.v.01 time.n.08 theme time now equ earthly.a.01 life.n.11 end.v.01 time.n.08 attributeof theme time now equ figure 1: an example sentence with verb phrase ellipsis and meaning representation in sequence notation and drawn as a directed acyclic graph. annotated vpe instances as occurrences of vpe are relatively rare @xcite , it is rather challenging to yield a reasonably sized corpus. a total of 120 cases were identified in the pmb and their corresponding meaning representations manually corrected. slightly more than half of the cases (71) contained some kind of negation in the elided construction (e.g., "and neither am i", "greenland is not", "but she didn't"). half of the instances are accompanied by the auxiliary verb to do, a third by to be, and the remaining cases are formed by other auxiliary verbs, the infinitival particle to or instances of gapping. an annotated example taken from the corpus is shown in figure 1 , where the elliptical phrase "life does" is semantically interpreted as "life does end". ## experimental setup training sets for training our neural semantic parsers, we consider two settings: (1) the standard training set, the default training data provided by pmb version 5.1.0, with all texts that are included in the vpe test set removed; and (2) the augmented training set, an augmented dataset to enhance the model's ability to handle verb phrase ellipsis. we construct the augmented dataset apply-ing the following data augmentation strategies: ## results and analysis the performance of models on both test sets is presented in performances on vpe120 all models exhibit substantially degraded performance on vpe120 relative to the standard test, confirming the inherent difficulty of parsing elliptical constructions semantically. when trained solely on the standard dataset, models achieve vpe120 smatch scores between 66.22 and 83.11, accompanied by markedly increased ifr (e.g., 33.17% for mbart-large and 29.83% for mt5-large), indicating frequent generation of malformed outputs. vpe-specific data augmentation yields substantial improvements across all architectures. ministral-8b achieves the highest score of 90.61 with aug600, closely followed by llama3.1-8b (89.21) and gemma2-9b (89.20). these topperforming models also demonstrate the most significant ifr reductions (e.g., ministral-8b: 13.17% → 6.50%). encoder-decoder models also benefit from augmentation: mbart-large improves from 70.90 (standard) to 78.10 (aug600), while mt5-large advances from 70.38 to 75.50. notably, byt5-large shows improvement from 66.22 to 73.00 with aug300. these findings demonstrate that vpe-specific data augmentation effectively narrows the performance gap between the standard and vpe test sets, particularly for larger decoder-only models. the convergence of performance scores beyond aug300 (see figure 2 ) suggests diminishing gains from additional augmentation data, indicating that current models may be approaching their capacity limits for ellipsis resolution. this shows the need for more advanced architectures or specialized training strategies to further improve performance on complex elliptical phenomena. ## conclusion although open-domain semantic parsing achieves good overall performance on the standard test sets, its shortcomings arise at the surface when looking at more complex linguistic phenomena. we demonstrated this by looking specifically at how neural parsing models deal with cases of english vp ellipsis. although we observed a drop in performance, the reason for the drop was not the context-sensitive nature of ellipsis, but rather the fact that elliptical phenomena are often surrounded by complex phenomena such as tense, negation, and discourse structure, causing parsing errors. so, is neural semantic parsing good at ellipsis resolution? yes, it is! out that the developed datasets will be valuable for the community, and indeed the vpe dataset will be made public via the parallel meaning bank data releases. reviewer 2 wondered how many different vp targets there are for each case of ellipsis, as the complexity for ellipsis resolution "depends greatly on the number of available vp targets for each vpe". we added a discussion in this topic in section 5. reviewer 3 noted that the question in the paper's title is not actually answered. this was a correct observation. but now we did explicitly in section 6. special thanks go to juri opitz, who pointed out that using a hill-climber for evaluation (as we did in the submitted version of this paper) is not optimal. so we recalculated our scores using ilp instead.