Holistic Exploration on Universal Decompositional Semantic Parsing: Architecture, Data Augmentation, and LLM Paradigm (2024)

Abstract: In this paper, we conduct a holistic exploration of Universal Decompositional Semantic (UDS) parsing, aiming to provide a more efficient and effective solution for semantic parsing and to envision the development prospects after the emergence of large language models (LLMs). To achieve this, we first introduce a cascade model for UDS parsing that decomposes the complex task into semantically appropriate subtasks. Our approach outperforms prior models while significantly reducing inference time. Furthermore, to further exploit the hierarchical and automated annotation process of UDS, we explore the use of syntactic information and pseudo-labels, both of which enhance UDS parsing. Lastly, we investigate ChatGPT’s efficacy in handling the UDS task, highlighting its proficiency in attribute parsing but struggles in relation parsing, revealing that small parsing models still hold research significance. Our code is available at https://github.com/hexuandeng/HExp4UDS.

## introduction a long-standing objective in natural language understanding is to create a structured graph of linguistic meaning. various efforts have been made to encode semantic relations and attributes into a semantic graph, such as abstract meaning representation (amr; @xcite , universal conceptual cognitive annotation (ucca; @xcite , and semantic dependency parsing formalisms (sdp; @xcite . recently, universal decompositional semantics (uds; @xcite introduced a more advanced hierarchical approach, as shown in figure 1 . it can automatically construct semantic relations from syntactic annotations @xcite and annotates semantic attributes following decompositional semantics @xcite ), * corresponding author: meishan zhang. sounds like we may actually get these documents executed early next week . root something assets subspace attribute value factuality factual -1.529 genericity dynamic 1.145 genericity hypothetical 1.145 time instant -0.894 ... ... ... assets subspace attribute value protoroles exists-before 1.504 protoroles exists-during 1.507 protoroles exists-after 1.504 protoroles change-of-state 0.059 ... ... ... words semantic node (predicate) semantic node (argument) syntactic edge instance edge non-head edge semantic edge using only simple questions about words or phrases, significantly lowering the annotation cost. however, existing solutions on uds parsing remain suboptimal. previous parsing approaches mainly rely on the seq2seq transduction framework @xcite , suffering from poor parallelism and long inference times that increase with sentence length. in this paper, we propose a cascade architecture that decomposes the complex parsing task into multiple semantically appropriate subtasks. within each subtask, our model predicts all corresponding sentence elements simultaneously, enhancing parallelism and substantially reducing inference time. experimental results show that our approach outperforms previous models while maintaining high efficacy during inference. furthermore, the structured data and highly automated annotation procedure of the uds dataset enable low-cost data augmentation schemes. to take full advantage of this, we design two approaches. firstly, we incorporate syntactic information, proven beneficial to various tasks @xcite @xcite . we use multi-task training @xcite to incorporate syntactic information, and propose several approaches for further improvement. secondly, to utilize the automated tool in uds that derives semantic relation from syntax, predpatt @xcite , @xcite attempted to introduce this tool during inference but do not achieve improvements. in contrast, we propose a data augmentation method that effectively exploits predpatt's capabilities, leading to significant performance gains in relation parsing. detailed analysis is also provided to guide the design of parsing systems. lastly, we reveal the strengths and weaknesses of large language models (llms), such as chat-gpt @xcite , in semantic parsing tasks, providing guidance for future developments in this field. llms have shown considerable performance in various tasks @xcite @xcite . for semantic parsing tasks, we conduct preliminary experiments on chatgpt for the uds task, directly applying it for parsing, as well as using it for data augmentation. due to poor performance when directly answering by chatgpt, we carefully design prompts, breaking down questions into small steps. results show that chatgpt excels in attribute parsing but struggles in relation parsing, revealing that small, specialized models still hold research significance. uds datasets silveira et al. (2014) create a standard set of stanford dependency annotations for the english web treebank (ewt, silveira et al., 2014) corpus. subsequently, white et al. (2016) proposed a framework aimed at constructing and deploying cross-linguistically robust semantic annotation protocols and proposed annotations on top of the ewt corpus using predpatt @xcite . several works have then been proposed to provide semantic annotations within this framework, including annotations for semantic roles @xcite , entity types @xcite , event factuality @xcite , linguistic expressions of generalizations about entities and events @xcite , and temporal properties of relations between events @xcite . all of these efforts culminated in @xcite , which presents the first unified decompositional semantics-aligned dataset, namely, universal decompositional semantics (uds). uds parser uds parsing has been conducted using transition-based parser @xcite , deep biaffine attention parser @xcite , and sequence-to-graph transductive parser @xcite . the latter significantly outperforms the others by employing an efficient seq2seq transduction framework @xcite . this approach is initially used in amr parsing @xcite and later extended to cover other semantic frameworks, such as ucca and sdp, by @xcite in a unified transduction framework, which predicts nodes and corresponding edges simultaneously in a seq2seq manner. for uds, an attribute module is added by @xcite . syntactic information is incorporated by @xcite , yielding further improvements. despite these attempts, cascade models with better parallelism and shorter inference time have not yet been explored. incorporating syntactic information syntactic information has been shown to improve the performance of downstream tasks. multi-task learning is widely used to incorporate syntactic information. @xcite improve the performance of semantic parsing by using multi-task learning, with syntactic and other semantic parsing tasks serving as auxiliary tasks. @xcite use syntactic and semantic information to improve the efficacy of two low-resource translation tasks. @xcite employ a single model to parse syntactic and semantic information simultaneously to improve semantic parsing. graph convolutional networks (gcn, @xcite are also widely used. @xcite use gcns to incorporate syntactic information in neural models and construct a syntax-aware semantic role labeling model. @xcite propose an extension of gcns to help relation extraction models capture longrange relations between words. @xcite presents a syntax-aware approach based on dependency gcns to improve opinion role labeling tasks. ## preliminaries the uds dataset comprises three layers of annotations: syntactic annotations, semantic relation annotations, and decompositional semantic attribute annotations at the edge and node levels. syntactic annotation is derived from the ewt dataset, which provides consistent annotation of grammar, including part-of-speech (pos) tag, morphological features, and syntactic dependencies, for human languages. these annotations are used to construct the syntactic tree, where each word is tied to a node. as shown in figure 1 , the headword of "we" is "get", and the headword of the root "sounds" is defined as itself. semantic relations consist of predicates, arguments, and edges between them, which forms the semantic relations. it is generated by predpatt tool @xcite automatically, using the pos tag and the syntactic tree as input. each semantic node explicitly corresponds to one word in the sentence called the center word, demonstrated by the instance edge. additionally, each semantic node is also tied with several non-repetitive words with the non-head edge, which forms a multi-word span. as shown in figure 1 , the leftmost predict node has a span "sounds like" with the center word "sounds". note that two semantic nodes may correspond to the same word in the case of clausal embedding. then, an extra argument node "something" is introduced as the root of the clause, e.g., "executed" corresponds to an extra argument node. semantic attributes consist of crowdsourced decompositional annotations tied to the semantic relations, detailed in §2. these annotations can be further categorized into node-level and edge-level attributes, corresponding to the table on the left and right in figure 1 , respectively. for each node or edge, all attributes have a value in range @xcite . besides, each attribute also has a confidence in range [0, 1], which shows how likely it is to have the property. following stengel-eskin et al. (2020), we discretized it into {0, 1} by setting every nonzero confidence to one. ## methodology in this section, we introduce our cascade model and methods to improve its performance. ## conclusion in this paper, we conduct a holistic exploration of semantic parsing, focusing on universal decompositional semantic (uds) parsing. first, we develop an efficient cascade model that offers improved performance and reduced training and inference costs. additionally, we examine data augmentation methods that incorporate syntactic information and employ the predpatt tool to strengthen the model's syntactic and semantic comprehension. lastly, we find that chatgpt performs poorly in relation parsing and data augmentation but excels in attribute parsing. this reveals that small, specialized models still hold research significance in semantic parsing.