Benchmarking Meaning Representations in Neural Semantic Parsing (2020)

Abstract: Meaning representation is an important component of semantic parsing. Although researchers have designed a lot of meaning representations, recent work focuses on only a few of them. Thus, the impact of meaning representation on semantic parsing is less understood. Furthermore, existing work’s performance is often not comprehensively evaluated due to the lack of readily-available execution engines. Upon identifying these gaps, we propose , a new unified benchmark on meaning representations, by integrating existing semantic parsing datasets, completing the missing logical forms, and implementing the missing execution engines. The resulting unified benchmark contains the complete enumeration of logical forms and execution engines over three datasets \times four meaning representations. A thorough experimental study on Unimer reveals that neural semantic parsing approaches exhibit notably different performance when they are trained to generate different meaning representations. Also, program alias and grammar rules heavily impact the performance of different meaning representations. Our benchmark, execution engines and implementation can be found on: https://github.com/JasperGuo/Unimer .

## introduction a remarkable vision of artificial intelligence is to enable human interactions with machines through natural language. semantic parsing has emerged as a key technology for achieving this goal. in general, semantic parsing aims to transform a natural language utterance into a logic form, i.e., a formal, machine-interpretable meaning representation (mr) @xcite . 1 thanks to the recent development mr geo atis job prolog --91.4 lambda 90.4 91.3 85.0 funql 92.5 --sql 78.0 69.0 -prolog 89.6 -92.1 lambda ---funql ---sql 82.5 79.2 - upon identifying the three biases, in this paper, we propose unimer, a new unified benchmark, by unifying four publicly available mrs in three of the most popular semantic parsing datasets: geo, atis and jobs. first, for each natural language utterance in the three datasets, unimer provides annotated logical forms in four different mrs, including prolog, lambda calculus, funql, and sql. we identify that annotated logical forms in some mr × dataset combinations are missing. as a result, we complete the benchmark by semi-automatically translating logical forms from one mr to another. second, we implement six missing execution engines for mrs so that the execution-match accuracy can be readily computed for all the dataset × mr combinations. both the logical forms and their execution results are manually checked to ensure the correctness of annotations and execution engines. after constructing unimer, to obtain a preliminary understanding on the impact of mrs on semantic parsing, we empirically study the performance of mrs on unimer by using two widely-used neural semantic parsing approaches (a seq2seq model @xcite and a grammar-based neural model @xcite ), under the supervised learning setting. in addition to the empirical study above, we further analyze the impact of two operations, i.e., program alias and grammar rules, to understand how they affect different mrs differently. first, program alias. a semantically equivalent program may have many syntactically different forms. as a result, if the training and testing data have a difference in their syntactic distributions of logic forms, a naive maximum likelihood estimation can suffer from this difference because it fails to capture the semantic equivalence @xcite . as different mrs have different degrees of syntactic difference, they suffer from this problem differently. second, grammar rules. grammar-based neural models can guarantee that the generated program is syntactically correct @xcite @xcite . for a given set of logical forms in an mr, there exist multiple sets of grammar rules to model them. we observe that when the grammar-based neural model is trained with different sets of grammar rules, it exhibits a notable performance discrepancy. this finding alias with the one made in traditional semantic parsers @xcite that properly transforming grammar rules can lead to better performance of a traditional semantic parser. in summary, this paper makes the following main contributions: ## preliminaries in this section, we provide a brief description of the mrs and neural semantic parsing approaches that we study in the paper. ## benchmark to provide an infrastructure for exploring mrs, we construct unimer, a unified benchmark on mrs, based on existing semantic parsing datasets. currently, unimer covers three domains, namely geo, atis, and job, each of which has been extensively studied in previous work and has annotated logical forms for at least two mrs. all natural language utterances in unimer are written in english. geo focuses on querying a database of u.s. geography with natural language. to solve the problem, @xcite designed a prologstyle mr and annotated 880 (utterance, logical form) pairs. @xcite and @xcite proposed to use sql and funql to represent the meanings, respectively. almost the same time, @xcite proposed to use lambda calculus and manually converted the prolog logical forms to equivalent expressions in lambda calculus. following their work, we adopt the standard 600/280 training/test split. atis is a dataset of flight booking questions. it consists of 5,418 questions and their corresponding sql queries. @xcite proposed to use lambda calculus to represent the meanings of natural language and automatically map these sql queries to its equivalent logical forms in lambda calculus. following the work of @xcite , we use the standard 4480/480/450 training/dev/test split. job is a dataset about job announcements posted in the newsgroup austin.jobs (califf and @xcite . it consists of 640 utterances and their corresponding prolog logical forms that query computerrelated job postings. similarly, in geo, @xcite proposed using lambda calculus and manually converted them to equivalent expressions in lambda calculus. we use the same training-test split with them, containing 500 training and 140 test instances. since not all the four mrs that we introduce in section 2.1 are used in these three domains, we semi-automatically translate logical forms in one mr into another. this effort enables researchers to explore mrs in more domains and make a fair comparison among them. take the translation of lambda calculus to funql in atis as an example. we first design predicates for funql based on those defined in lambda calculus and implement an execution engine for funql. then, we translate logical forms in lambda calculus to funql and compare the execution results to verify the correctness of the translation. in this process, we find that there is no ready-to-use lambda calculus execution engine for the three domains. hence, we implement one for each domain. these engines, on the one hand, enable evaluations of semantic parsing approaches with both exact-match accuracy and execution-match accuracy. on the other hand, they enable exploration of weakly supervised semantic parsing with lambda calculus. in addition, we find some annotation mistakes in logical forms and several bugs in existing execution engines of prolog and funql. by correcting the mistakes and fixing the bugs in the engines, we create a refined version of these datasets. section a.1 in the supplementary material provides more details about the construction process. we plan to cover more domains and more mrs in unimer. we have made unimer along with the execution engines publicly available. 3 we believe that unimer can provide fertile soil for exploring mrs and addressing challenges in semantic parsing. ## experimental setup based on unimer, we take the first attempt to study the characteristics of different mrs and their impact on neural semantic parsing. ## conclusion in this work, we propose unimer, a unified benchmark on meaning representations, based on established semantic parsing datasets; unimer covers three domains and four different meaning representations along with their execution engines. unimer allows researchers to comprehensively and fairly evaluate the performance of their approaches. based on unimer, we conduct an empirical study to understand the characteristics of different meaning representations and their impact on neural semantic parsing. by open-sourcing our source code and benchmark, we believe that our work can facilitate the community to inform the design and development of next-generation mrs. implications. our findings have clear implications for future work. first, according to our experimental results, funql tends to outperform lambda calculus and prolog in neural semantic parsing. additionally, funql is relatively robust against program alias. hence, when developers need to design an mr for a new domain, funql is recommended to be the first choice. second, to reduce program alias' negative effect on neural semantic parsing, developers should define a concrete protocol for annotating logical forms to ensure their consistency. specifically, given an mr, developers should identify as many as possible sources where program alias can occur. take sql as an example. to express the argmax semantics, one can either use subquery or the orderby clause. 8 having identified these sources, developers need to determine using which expression in what context, e.g., argmax is always expressed with subquery, and the unordered expressions in conjunctions are always sorted by characters. kai zhao and liang huang. 2015. type-driven incremental semantic parsing with polymorphism. in proceedings of the 2015 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1416-1421, denver, colorado. association for computational linguistics. zexuan zhong, jiaqi guo, wei yang, jian peng, tao xie, jian-guang lou, ting liu, and dongmei zhang. 2018. semregex: a semantics-based approach for generating regular expressions from natural language specifications. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 1608-1618, brussels, belgium. association for computational linguistics. a supplemental material algorithm 1: translation of lambda calculus logical forms to funql input: a lambda calculus logical form p output: a funql logical form f 1 @xmath0; 2 @xmath1; 3 expressions, @xmath2, []; 4 @xmath3; 5 for @xmath4,i); 8 else if @xmath5; geo. since the four mrs we study have annotated logical forms and have execution engines except for lambda calculus in geo, we directly use them (sql taken from @xcite , prolog taken from @xcite , funql taken from @xcite , and lambda calculus taken from @xcite ). we implement an execution engine with haskell for lambda calculus. with these resources, we crossvalidate the correctness of annotations and execution engines by comparing the execution results of logical forms. as a result, we found nearly 30 prolog logical forms with annotation mistakes and two bugs in the execution engines of prolog and funql. atis. there are only annotated logical forms for lambda calculus and sql in atis. we directly use lambda calculus logical forms provided by @xcite and sql queries provided by @xcite . to provide annotations for funql and prolog, we semi-automatically translate lambda calculus logical forms to equivalent logical forms in funql and prolog. algorithm 1 presents the pseudo code for translating a lambda calculus logical form to funql. basically, we first approach mr exact exec geo kwiatkowksi et al. (2010) funql 84.3 -shaw et al. (2019) * funql 89.3 -jia and liang (2016) * prolog -89.3 -data augmentation prolog -85.0 chen et al. (2020) * prolog -89.6 kwiatkowksi et al. (2010) λ 87.9 -wang et al. (2014) λ 90.4 -zhao and huang (2015) λ 88.9 -dong and lapata (2016) * † λ 87.1 -rabinovich et al. (2017) * † λ 87.1 -dong and lapata (2018) * † λ 88.2 -sun et al. (2020) * † λ 89.1 -iyer et al. (2017) * † sql -82.5 atis wang et al. (2014) λ 91.3 -zhao and huang (2015) λ 84.2 -jia and liang (2016) * λ 83.3 --data augmentation λ 76.3 -dong and lapata (2016) * † λ 84.6 -rabinovich et al. (2017) * † λ 85.9 -yin and neubig (2018) * † λ 88.2 -dong and lapata (2018) * † λ 87.7 -cao et al. (2019) * † λ 89.1 -sun et al. (2020) * † λ 89.6 -shaw et al. (2019) * λ 87.1 -iyer et al. (2017) * † sql -79.2 jobs zettlemoyer and collins (2005) λ 79.3 -zhao and huang (2015) λ 85.0 -dong and lapata (2016) * † prolog 90.0 -rabinovich et al. (2017) * † prolog 91.4 -chen et al. (2020) * prolog -92.1 job. there are only annotated logical forms for lambda calculus and prolog in job. we directly use prolog logical forms provided on the website. 9 to provide annotations for funql, we semi-9 http://www.cs.utexas.edu/ ml/nldata/jobquery.html automatically translate prolog logical forms to equivalent logical forms in funql with an algorithm similar to algorithm 1. in terms of sql, the logical forms in job are relatively simple and can be expressed with the select, from and where clauses of sql. we simply translate each subexpression in prolog to an expression in where clause and use a conjunction to join the resulting expressions. since we cannot find the annotated lambda calculus logical forms provided by @xcite , we also translate the prolog logical forms to lambda calculus. we implement execution engines for lambda calculus and funql.