Exploring the Secrets Behind the Learning Difficulty of Meaning Representations for Semantic Parsing (2022)

Abstract: Previous research has shown that the design of Meaning Representation (MR) greatly influences the final model performance of a neural semantic parser. Therefore, designing a good MR is a long-term goal for semantic parsing. However, it is still an art as there is no quantitative indicator that can tell us which MR among a set of candidates may have the best final model performance. In practice, in order toselect an MR design, researchers often have to go through the whole training-testing process for all design candidates, and the process often costs a lot. In this paper, we propose a data-aware metric called ISS (denoting incremental structural stability) of MRs, and demonstrate that ISS is highly correlated with the final performance. The finding shows that ISS can be used as an indicator for MR design to avoid the costly training-testing process.

## introduction semantic parsing is the task of transforming a natural language utterance into a meaning representation (mr), whose accuracy has been significantly improved recently @xcite @xcite @xcite . for example, in a semantic parsing dataset of text-to-sql, sql is the meaning representation. previous work @xcite @xcite has shown that the design of mr significantly influences model performance of a neural semantic parser. as shown in in order to reduce learning difficulty of mrs, some researchers have put efforts to design customized mrs for their tasks. for example, @xcite propose a new mr named semql to bridge the gap between utterances and sql. compared with sql, the structure of semql is simpler, decreasing learning difficulty of semql. in addition, @xcite propose an mr named lir by discarding sql components that cannot correspond well to natural language utterances. their experimental results show that the models can reach better model performance on lir compared with sql. despite the key role of mrs, a quantitative indicator is lacking to guide the design of easy-tolearn mrs. in order to evaluate whether a design choice of mr is good or not, researchers have to go through the whole model training-testing process iteratively @xcite @xcite , incurring huge costs. hence, it is important to find a quantitative indicator so that researchers can design mrs under a clear guideline before the model training-testing process. @xcite conduct an investigation on potential factors that influence learning difficulty of mrs. their work studies three factors including the number of grammar rules, the number of production rules in an mr, and the proportion of program alias. however, none of these factors can be directly used as an indicator because each of them only partially reflects the underlying learning difficulty. in this paper, we propose the notion of incremental structural stability and argue that it is a good quantitative indicator for learning difficulty of mrs. incremental structural stability refers to the degree of the structural changes in an mr instance (denoted as mri for simplicity) when its corresponding natural language utterance is changed by adding a constraint condition. intuitively, larger changes in an mri structure bring more challenges to neural models, leading to higher learning difficulty. as shown in figure 1 (a), the structure of sql changes a lot when a constraint condition ("in usa") is added into its utterance. this simple change drastically influences the contents of the from clause and where clause of sql. in the where clause, a condition for cross-table joining is added, and this condition addition is hard to predict because it depends on the database structure. in figure 1(b) , compared with sql, funql @xcite (the best performing mr among the four mrs in in order to measure the incremental structural stability of mrs on datasets, we propose iss (denoting incremental structural stability), a dataaware metric. iss quantifies the structural difference between a pair of mris whose corresponding utterances' semantics differ by only a constraint condition. in particular, given an utterance dataset d (for a specific database schema) annotated in an mr m, the value of iss(m, d) is computed in three steps: (1) anonymize the utterances @xcite ; (2) extract all pairs of mris (in d) whose corresponding anonymized utterances differ by only a constraint condition; (3) measure the degree of the structural changes between each pair of mris. to evaluate the effectiveness of iss on measuring learning difficulty of mrs, we conduct experiments on the unimer benchmark @xcite . unimer contains three widely used semantic parsing datasets: atis @xcite , geo-root from select where table job_id cond job job.req_exp is not null root from select where tablelist job_id condlist job job.req_exp is not null country country. @xmath0.@xmath1.job_id nl: show me the job that needs experience ? sql: select job.job_id from job where job.req_exp is not null nl: list jobs in usa requiring experience. sql: select job_id from job, country where job.req_exp is not null and country.@xmath2.@xmath3.job_id (a) an example of sql root predicate answer predicate req_exp job all nl: show me the job that needs experience ? funql: answer(req_exp(job(all))) nl: list jobs in usa requiring experience. funql: answer(req_exp(job(loc_1(const(usa))))) root predicate answer predicate req_exp job predicate loc_1 const usa (b) an example of funql query @xcite , and job (tang and mooney, 2001), each of which is annotated in four mrs: sql, lambda calculus, prolog, and funql. the experiments consist of three steps. first, we compute the iss value for each combination of four mrs × three datasets (12 iss values in total). second, we measure model performance (shown in in summary, this paper makes the following main contributions: it can be used to predict model performance, decrease the experimental cost, and help researchers design easy-to-learn mrs. we propose a data-aware metric named iss to measure the incremental structural stability of mrs. in this section, we introduce the definition of iss and computation of the iss. ## experimental setup in this section, we introduce the details of our experiments. we present the used datasets and the metric for evaluating iss. we also present the implementation details of iss and two baselines used to compare with iss. ## results and discussion in this section, we present the experimental results to show the effectiveness, robustness, and low cost of iss, respectively. ## related work mrs play an important role in semantic parsing. researchers have made a lot of efforts on designing new mrs in order to improve model performance. variants of typed lambda calculus are designed to outperform lambda calculus @xcite . liang (2013) designs dcs for querying knowledge bases. @xcite propose the funql mr, which performs better than sql on domain-specific datasets. there are also research efforts on decreasing learning difficulty of sql. @xcite propose semql to bridge the gap between natural language and sql, and achieve better performance on the spider benchmark @xcite . @xcite propose rir and lir based on sql, improving model performance on domain-specific datasets. rir and lir also improve the compositional generalization of sql. sql u f @xcite and an extension of relational algebra @xcite are also designed to address the compositional generalization challenge of sql. however, the preceding mrs are usually designed based on a hunch or trial-and-error. as the final goal, designing easy-to-learn mrs is so abstract that there is no standard guideline for researchers during the design process. the motivation of designing semql is to align the natural language utterance with the mr, while rir and lir are designed to reduce the mismatch between mris and natural language, but the designers do not explain their motivation. in this paper, we argue that incremental structural stability could be one of the important guidelines for designing mrs. our experimental results also support our hypothesis. previous work has proposed some hypotheses on which characteristics of mrs are related to learning difficulty. @xcite show that lambda calculus outperforms funql on the geo-query dataset, and use the depth of asts to explain this phenomenon. @xcite propose the unimer benchmark and compare learning difficulty of the four mrs under a fair condition. they guess that the number of production rules and the size of asts are factors leading to the difference in model performance. however, their experimental results do not support their hypothesis. as a result, to the best of our knowledge, we are the first to propose a quantitative indicator that can reflect learning difficulty of mrs and can be used to guide researchers on designing mrs. ## conclusion in this paper, we have studied the difference of mrs in order to find a quantitative value to indicate learning difficulty of mrs. we have proposed the notion of incremental structural stability and argued that it is a reasonable indicator for learning difficulty of mrs. we have proposed a data-aware metric named iss to measure the incremental structural stability of mrs on the given datasets. we have conducted experiments to show the high correlation between iss and model performance of the neural semantic parser. our findings have important implications for future work. for example, our experimental results have demonstrated that iss can be used to guide the design of new mrs. researchers can design an mr to achieve better parsing accuracy by improving the incremental structural stability of the mr. more importantly, the iss-based process of designing mrs helps researchers largely avoid the high cost brought by the iterative training and testing process.