Towards Collaborative Neural-Symbolic Graph Semantic Parsing via Uncertainty (2022)

Abstract: Recent work in task-independent graph semantic parsing has shifted from grammar-based symbolic approaches to neural models, showing strong performance on different types of meaning representations. However, it is still unclear that what are the limitations of these neural parsers, and whether these limitations can be compensated by incorporating symbolic knowledge into model inference. In this paper, we address these questions by taking English Resource Grammar (ERG) parsing as a case study. Specifically, we first develop a state-of-the-art, T5-based neural ERG parser, and conduct detail analyses of parser performance within fine-grained linguistic categories. The neural parser attains superior performance on in-distribution test set, but degrades significantly on long-tail situations, while the symbolic parser performs more robustly. To address this, we further propose a simple yet principled collaborative framework for neural-symbolic semantic parsing, by designing a decision criterion for beam search that incorporates the prior knowledge from a symbolic parser and accounts for model uncertainty. Experimental results show that the proposed framework yields comprehensive improvement over neural baseline across long-tail categories, yielding the best known Smatch score (97.01) on the well-studied DeepBank benchmark.

## introduction semantic parsing is the task of mapping natural language to machine interpretable meaning representations, and graph-structured semantic representations, which encode rich semantic information in the form of semantic graphs, have played an important role in natural language processing @xcite . parsing natural language sentences into the semantic-graph representation (e.g., figure 1 ) has * part of the work was done while zi was an ai resident at google. been extensively studied in the recent decade. work in this area has shifted from the symbolic (grammar-based) approach to the neural approach. thanks to the flourishing of deep learning technologies, sequence-to-sequence (seq2seq) models have shown great performance on data sampled from the training distribution. these neural semantic parsers reduce the need for domain-specific grammar and feature engineering, but comes at a cost of lacking interpretability, as the model directly outputs a (linearized) graph without revealing the underlying meaning-composition process. moreover, these neural models often generalize poorly to tail and out-of-distribution (ood) examples, and previous work has shown that combining high-precision symbolic approaches with neural models can address this issue for task-oriented semantic parsing @xcite @xcite . however, this type of approach requires complex architecture engineering to incorporate the grammar formalism. the grammar formalism being utilized is usually primitive, and was not tested beyond simple datasets such as scan @xcite or @xcite . therefore they are likely not sufficient for handling complex graph-based meaning representations derived from realistic corpora. in this work, we aim to develop a simple yet principled neural-symbolic approach for graph semantic parsing to address long-tail generalization, which leverages the information from an a priori grammar parser while maintaining the convenience of neural seq2seq training built on top of massively pre-trained embeddings @xcite . we take graph semantic parsing for english resource grammar (erg) as our case study @xcite . erg is a compositional semantic representation explicitly coupled with the syntactic structure. compared to other graph-based meaning representations, erg has high coverage of english text and strong transferability across do-mains @xcite @xcite , rendering itself has an attractive target formalism for automated semantic parsing. the classic erg literature has focused on developing grammar-based erg parser. however, they can suffer from issues such as incomplete categorization of lexical items and multi-word expression, and yields low coverage for realistic corpus such as wikipedia @xcite . on the other hand, multiple neural erg parsers have also been proposed @xcite @xcite . however, they are commonly structured as a pipelined system and often rely on external tools (e.g, aligners, part-of-speech taggers, and named entity recognizers), with the performance of the upstream component significantly impacting the final performance. this motivates us to build a pure end-to-end neural parser for erg parsing that directly maps the input sentences to target graphs. first, we present an end-to-end seq2seq model based on t5 @xcite ) that achieves the state-of-the-art results for erg parsing. this model goes beyond the conventional multi-step predictions for node and edge in previous work, and does not require specialized architecture that explicitly incorporate the erg rules or the synaptic structure as part of inductive bias. despite the complicated syntax and semantic structures encoded in semantic graphs, we have shown that by devising proper linearization and tokenization, we can successfully transfer erg parsing problem to translation problem (section 3.1). second, we conduct a comprehensive study of the generalization behavior of the neural parser, interrogating its performance within fine-grained linguistic categories. comparing with a state-ofthe-art symbolic parser ace, the neural parser exhibits complementary strengths. particularly, the neural model yields much higher coverage than the symbolic parser, generating valid parses for a wider range of examples. however, the quality of the top-1 parse degrades severely in the long-tail situation. perhaps remarkably, we also observed that the neural model's top-k parses in fact often contain candidate that generalizes well on long-tail, but the vanilla mle-based inference fell short in selecting them (section 4 and 5). the above observation motivates our third contribution: to develop a practical framework for collaborative neural-symbolic parsing. the key lies in designing a principled decision making strategy for this neural-symbolic collaboration that performs optimally during inference time. to this end, we design a new decision criterion for neural model inference (e.g., beam search) that incorporates both model uncertainty and the prior knowledge from a symbolic parser, leveraging the theoretical framework of optimal decision-making under the incomplete knowledge of the world @xcite @xcite . the basic idea is to utilize uncertainty estimates of the neural parser as a switch between the optimistic, mlebased inference and the conservative, prior-based inference, such that the neural parser seeks the guidance from a symbolic parser during its decoding stage when encountering low-confident examples. this proposed approach achieves comprehensive improvement compared to the original neural parser, across almost all linguistic categories. our result suggests that sometimes the limitation of the neural approach lies not necessarily in the model architecture or the training method, but in a suboptimal inference procedure that naively maximize the a posteriori likelihood (e.g., the beam search) without questioning the reliability of the prediction (section 3.2). in summary, our contribution are three-fold: the resulting framework not only comprehensively improved the model performance in tail linguistic categories, but further boosted the performance of the neural model on the standard in-domain test set (an extra 9.5% error rate reduction), establishing a new state-of-the-art smatch 97.01. 1 _the_q<0> _drug_n_1<1> parg_d _introduce_v_to<3> loc_nonsp _in_p<4> @xmath0 @xmath1. <9> figure 1 : an example of semantic graph for english resource grammar (erg). some nodes are surface concepts, meaning that they are related to a single lexical unit, e.g. _introduce_v_to (the number in the angle brackets indicates their token alignments in the sentence), while others are abstract concepts representing grammatical meanings, e.g. compound (multiword expression), parg_d (passive) and loc_nonsp (temporal). color red indicates the root of this semantic graph. it also supports light-weight named entity recognition (e.g., "west germany" is labeled as two named in the graph). ## experiments dataset we conduct model training on deep-bank v1.1 that correspond to erg version 1214, and adopt the standard data split. the pydelphin foot_3 library is leveraged to extract eds graphs and transfer them into penman format. implementation details t5 @xcite is a pre-trained sequence-to-sequence transformer model that has been widely used in many nlp applications. we use the open-sourced t5x foot_4 , which is a new and improved implementation of t5 codebase in jax and flax. specifically, we use the official pretrained t5-large (770 million parameters) and finetuned it on deepbank in-domain training set. despite the general fact that larger model size will lead to better performance on finetuning for some tasks, our empirical results show that adopting model sizes larger than t5-large will not lead to further gain for erg parsing. for the collaborative neural-symbolic parsing, we set the beam size to 5, i.e., our combined predictions will be selected from the top 5 predictions produced by the model. for the monotonic transformation α(x) in (2), we set we set λ = 0.1 and @xmath2.1. evaluation metrics for evaluation, following previous work, we adopt the smatch metric @xcite , which was originally proposed for evaluating amr graphs. it measures graph overlap, but does not rely on sentence alignments to determine the correspondences between graph nodes. specifically, smatch is computed by performing inference over graph alignments to estimate the maximum f1-score obtainable from a one-to-one matching between the predicted and gold graph nodes. this is also ideal for measuring the divergence between predicted and prior graphs in our collaborative framework. ## fine-grained linguistic evaluation though performs better than symbolic parser, we find that actually neural and symbolic parsers yield different distributions on the test set (see appendix c for details). this has motivated us to dive deeply into more fine-grained evaluation for our models. erg provides different levels of linguistic information that is beneficial to many nlp tasks, e.g., named entity recognition, semantic role labeling, and coreference. this rich linguistic annotation can help us quantify different types of errors the model makes. we reported the detailed evaluation results in argument structure in erg, there are different types of core predicates in argument structures, specifically, verbs, nouns and adjectives. we also categorize verb in to basic verb (e.g., _look_v_1) and verb particle constructions (e.g., _look_v_up). the verb particle construction is handled semantically by having the verb contribute a relation particular to the combination. coreference erg resolves sentence-level coreference, i.e., if the sentence referring to the same entity, the entity will be an argument for all the nodes that it is an argument of, e.g., in the sentence, "what we want to do is take a more aggressive stance", the predicates "want" (_want_v_1) and "take" (_take_v_1) share the same agent "we" (pron). as discussed before, this can be presented as reentrancies in the erg graph, we notice that one important type of reentrancies is the passive construction (e.g., parg_d in figure 1 ), so we also report evaluation on passive construction in as shown, the t5 parser performs much better than ace, especially for compound recognition. this indicates that local semantic information such as compound constructions or named entities can be easily captured by those pretrained embeddingbased models. for argument structure, though performs better than ace in most cases, the t5 parser still has relatively low accuracy for arg3 and noun structure recognition. this is mainly due to their relatively low frequency in the training set (1.94% for arg3 and 5.54% for noun argument structures). our analysis in this section is consistent with previous work: the t5 parser, similar to many other neural parsers, is fragile to tail instances that do not have sufficient representation in the training data. we also further report the evaluation results for our collaborative neural-semantic parsing framework (collab.), where we can see that it brings improvement for the issues above, which validates the effectiveness of the collaborative framework. ## conclusions and future work in this paper, we present a simple, uncertaintybased approach to collaborative neural-symbolic parsing for graph-based meaning representations. in contrary to the prior neural-symbolic approaches, we maintain the simplicity of the seq2seq training, and design a decision-theoretic inference criteria for beam candidate selection, incorporating model uncertainty and prior knowledge from an existing symbolic parser. remarkably, despite the simplicity of the method, our approach strongly outperform all the previously-known approach on the deepbank benchmark ( in the future, we plan to apply this approach to a broader range of graph meaning representations, e.g., amr @xcite and ucca @xcite , and build a more advanced uncertainty estimation approach to quantify model uncertainty about sub-components of the graph, thereby allowing more fine-grained integration between neural prediction and symbolic derivations. the term -of is used for reversing the edge direction for graph traversing. nodes in the graph get identifiers (e.g., x0, e0), which can be referred to later to establish a reentrancy, e.g., the node _drug_n_1 serves as arg2 of _introduced_v_to and arg2 of parg_d at the same time, so the identifier x_1 appears twice in the notation. however, in our settings, these identifiers can be randomly set to any unique symbols, which will confuse the model to learn the real meaningful mappings. to tackle this issue and create a variable-free version of the penman notation, we replace these identifiers with star markers to indicate reentrancy, e.g., replacing x1 with _drug_n_1 * . to illustrate more about reentrancies, we consider two different types of cases: in other words, the max number of star markers * indicates the total number of different reentrancies in the sentences. this will not confuse the model to do the reentrancy prediction as it can always refer to how many reentrancies have been predicted in the previous sequences.