On the Effects of Structural Modeling for Neural Semantic Parsing (2023)

Abstract: Semantic parsing aims to map natural language sentences to predefined formal languages, such as logic forms and programming languages, as the semantic annotation. From the theoretic views of linguistic and programming language, structures play an important role in both languages, which had motivated semantic parsers since the task was proposed in the beginning. But in the neural era, semantic parsers treating both natural and formal language as sequences, such as Seq2Seq and LLMs, have got more attentions. On the other side, lots of neural progress have been made for grammar induction, which only focuses on natural languages. Although closely related in the sense of structural modeling, these techniques hadn’t been jointly analyzed on the semantic parsing testbeds. To gain the better understanding on structures for semantic parsing, we design a taxonomy of structural modeling methods, and evaluate some representative techniques on semantic parsing, including both compositional and i.i.d. generalizations. In addition to the previous opinion that structures will help in general, we find that (1) structures must be designed for the specific dataset and generalization level, and (2) what really matters is not the structure choice of either source or target side, but the choice combination of both sides. Based on the finding, we further propose a metric that can evaluate the structure choice, which we believe can boost the automation of grammar designs for specific datasets and domains.

## introduction semantic parsing is the task to transduce source sentences in natural languages (nl), into the target representations, which are usually artificial formal languages (fl), such as lisp, λ-calculus, and sql. theoretically natural languages are processed in structures @xcite , and the formal languages are also defined to have a context-free syntax @xcite . therefore inevitably semantic parsers such as the ccg-based are aware of source structures, and adopt the compositional semantics foot_0 of the targets. but they usually parse to λ-calculus @xcite and do not support programming languages. in the neural era, seq2seq based parsers add supports to any sequential languages, but they can make grammar errors despite the effectiveness. grammar-based parsers are proposed to ensure the grammatical correctness by decoding the rule sequences of the target ast. recently, the development of the text-to-sql has motivated specialized parsers to support the sql language. but the nl structures on the source side are seldom handled and left to pretrained large models. on the contrary, nl structures are the key issues of treebanks like ptb and supervised parsers. the grammar induction field has also invented many methods to induce grammars with restricted forms from unsupervised training data. these parsers can infer trees for new sentences, but don't process the semantic annotations obviously. unfortunately, no investigations had been conducted on the combination of the success of the two fields. our research question (rq) is thus as follows: is structural modeling of the natural language or the formal language useful for neural semantic parsing? to answer the question, we use the encoder-decoder architecture with the attention mechanism to connect structures of two sides, due to its success of modeling token-level correlations. our investigations are kept diverse in several important factors, such as the dataset variety, categories of structures, and generalization levels (i.i.d., compositional, or zero-shot). under every possible combination of these factors, results are believed more faithful than single datasets @xcite . our evaluations add new knowledge to prior insights @xcite . we find it's not safe to claim the effectiveness for specific structural models for either nl or fl. the structures of nl and fl must be evaluated as a whole, and their effects even vary across datasets and generalization levels. therefore, we make the conclusion that the combination of structural choices are more important than the structural choice on either the source or target side. the result is consistent with the one of the findings from @xcite in that different grammars, leading to different tree structures, have significantly different performance when keeping the same semantic representations and datasets. these arguments in total suggest we can expect improvements from searching for better structural combinations on specific application domains. however, grammar search is not trivial but can be highly expensive. inspired by the recent works in large language models (llms) which can handle the code inputs well, we propose the metric, dis-struct, for evaluating the structural combination of the source and target sides based on the representations given by the llms and the optimal transport. the metric can be interpreted as the discrepancy between the specific training and testing splits under the structural choices. the metric is shown negatively correlated with the parser performance. it thus will help the automation of the grammar search theoretically. in summary, we make three contributions as: ## metric for structural evaluation taking sequences as the right-branching trees, the models we've discussed can all parse an example (x, y) to its structures (s(x), t(y)). but the gener-alization performance is not only determined by some smart structural choices. it also depends on the dataset and the generalization level. however, it's expensive to manually design good structures, or to optimize a parameterized structural policy. because on one hand we have to train and then evaluate a parser every time we need to confirm the effectiveness of that policy. on the other hand, even a parser jointly learning mappings and latent structures may work poorly according to above findings. inspired by the recent success on large language models (llms) @xcite such as the codex @xcite which can read and write programming source codes well, we propose a learning-free metric for the structures based on the representations generated by llms, such that it's correlated with the performance. specifically, to evaluate a pair of structural models (s, t) for a dataset @xmath0, y) i , we first define the distance between a parallel sequence (x, y), e x , e @xmath1, llm (y) since each fitted linear model has a low r 2 value (i.e., plots far from the fitted line), we examine the results by datasets. as long as the metric can indicate performance for datasets, it'll be possible to probe or search structural choices for a specific dataset we're interested in. for each dataset under a generalization level, we only have 2 points. we computed the slope of the line determined by the two points, and plot the histogram of the slopes in figure 9 . hopefully, the slopes are negative at more than 50% times, and are also relatively small even it's positive. we also find the metrics based on chatglm-6b and falcon-7b are more ideal than baichuan-7b. ## related works many representations have been used for semantic parsing. popular representations include semantic roles, fol or λ-calculus @xcite @xcite , λ-dcs @xcite @xcite , application-specialized query graphs @xcite @xcite , and programming languages like sql @xcite , java @xcite , and python @xcite . linguists also design meaning representations such as amr @xcite , ers @xcite , and umr @xcite . @xcite had reviewed many semantic representations in a linguistic-centric perspective, and @xcite had proposed a metric to evaluate different representations. our discussions are not at representation level (only the lispress, λ-calculus, and sql are used), but on structure effects under maybe a fixed representation. classic semantic parsers used to assign categories to linguistic or semantic fragments, and com-pose them in a bottom-up fashion. some typical implementations are based on ccg @xcite , scfg @xcite , hyperedge replacement grammar @xcite , and am algebra @xcite @xcite . other parsers do not define linguistic categories, but use feature engineering or types to guide composing algorithms @xcite . neural parsers like seq2seq @xcite ) adopt end-to-end mappings but can make grammar errors. seq2tree @xcite is then proposed to generate grammatically valid trees for untyped λ-calculus. grammar-based decoding @xcite turns to generate rule sequences converted from the target ast. some parsers design intermediate patterns for an easier abstraction over the targets @xcite @xcite @xcite @xcite . the abstraction layer can be seen as handcrafted structures for the targets. we only consider cfg-based structures due to their generality. similarly, graph-based targets and parsers are also beyond our discussing. llms as semantic parsers @xcite are found not performing well on the cogs dataset before structural discussions. we leave some results and discussions in appendix c. recently the compositional generalization has attracted much focus @xcite @xcite . but they either devise special parsers other than the encoder-decoder architecture, or handle representations like funql, therefore not direct applicable to other general parsers. @xcite reports the entanglement problem where seq2seq models entangle irrelevant semantic factors during generation. @xcite induces token and span level alignments. our structural discussions are orthogonal to their model improving works. ## conclusion by evaluations on a variety of settings, we find the structural modeling is not guaranteed to give better performance. we conclude that structural biases for sources and targets must be chosen as a whole, and that choices also depend on the specific dataset and generalization level. we propose the disstruct metric to facilitate structure finding, which is negatively correlated with the performance.