A Globally Normalized Neural Model for Semantic Parsing (2021)

Abstract: In this paper, we propose a globally normalized model for context-free grammar (CFG)-based semantic parsing. Instead of predicting a probability, our model predicts a real-valued score at each step and does not suffer from the label bias problem. Experiments show that our approach outperforms locally normalized models on small datasets, but it does not yield improvement on a large dataset.

## introduction semantic parsing has received much interest in the nlp community @xcite @xcite . the task is to map a natural language utterance to executable code, such as λ-expressions, sql queries, and python programs. recent work integrates the context-free grammar (cfg) of the target code into the generation process. instead of generating tokens of the code @xcite , cfg-based semantic parsing predicts the grammar rules in the abstract syntax tree (ast). this guarantees the generated code complies with the cfg, and thus it has been widely adopted @xcite @xcite @xcite . typically, the neural semantic parsing models are trained by maximum likelihood estimation (mle). the models predict the probability of the next rules in an autoregressive fashion, known as a locally normalized model. however, local normalization is often criticized for the label bias problem @xcite @xcite . in semantic parsing, for example, grammar rules that generate identifiers (e.g., variable names) have much lower probability than other grammar rules. thus, the model will be biased towards such rules that can avoid predicting identifiers. more generally, the locally normalized model will prefer such early-step predictions that can lead to low entropy in future steps. in this work, we propose to apply global normalization to neural semantic parsing. our model scores every grammar rule with an unbounded real value, instead of a probability, so that the model does not have to avoid high-entropy predictions and does not suffer from label bias. specifically, we use max-margin loss for training, where the ground truth is treated as the positive sample and beam search results are negative samples. in addition, we accelerate training by initializing the globally normalized model with the parameters from a pretrained locally normalized model. we conduct experiments on three datasets: atis @xcite , conala @xcite , and spider @xcite . compared with local normalization, our globally normalized model is able to achieve higher performance on the small atis and conala datasets with the long short-term memory (lstm) architecture, but does not yield improvement on the massive spider dataset when using a bert-based pretrained language model. ## related work early approaches to semantic parsing mainly rely on predefined templates, and are domain-specific @xcite @xcite . later, researchers apply sequence-to-sequence models to semantic parsing. @xcite propose to generate tokens along the syntax tree of a program. @xcite generate a program by predicting the grammar rules; our work uses the tranx tool @xcite with this framework. globally normalized models, such as the conditional random field (crf, @xcite , are able to mitigate the label bias problem. how-ever, their training is generally difficult due to the global normalization process. to tackle this challenge, @xcite propose learning as search optimization (laso), and wiseman and rush (2016) extend it to the neural network regime as beam search optimization (bso). specifically, they obtain negative partial samples whenever the ground truth falls out of the beam during the search, and "restart" the beam search with the ground truth partial sequence teacher-forced. our work is similar to bso. however, we search for an entire output, and do not train with partial negative samples. this is because our decoder is tree-structured, and different partial trees cannot be implemented in batch efficiently. we instead perform locally normalized pretraining to ease the training of our globally normalized model. ## methodology in this section, we first introduce the neural semantic parser tranx, which servers as the locally normalized base model in our work. we then elaborate how to construct its globally normalized version. ## experiments datasets. we conduct experiments on three benchmark parsing datasets: atis @xcite , conala @xcite , and spider @xcite , which contain 4473, 2379, and 8695 training samples, respectively. it should be pointed out that much work adopts data anonymization techniques to replace entities with placeholders @xcite @xcite . this unfortunately causes a large number of duplicate samples between training and test. this is recently realized in @xcite , and thus, in our work, we only compare the models using the original, correct atis dataset. settings. our globally normalized semantic parser is developed upon the open-sourced tranx foot_1 . we adopt the cfg grammars provided by tranx to convert lambda calculus and python programs into asts and sequence of grammar rules (actions). for atis and conala datasets, we use long lstm models as both the encoder and the decoder. their dimensions are set to 256. for the spider dataset, we use a pretrained bert model foot_2 @xcite 2019) and the relation-aware transformer @xcite as the encoder and an lstm as the decoder. the architecture generally follows the work by @xcite . the beam size is set to 20 to search for negative samples, and is set to 5 for inference. the margin ∆ in (5) is set to 0.1. we use the adam optimizer (kingma and ba, 2015) with a learning rate of 5e-4 for training. for both atis and conala datasets, we report the best results on the development sets and the corresponding results on the test set. for the spider dataset, we only report the results on the development set as the ground truth of the test set is not publicly available. we observe that the copy mechanism largely affects the accuracy on the test set, although it has little effect on the development set. this is because the training and validation distributions closely resemble each other, whereas the test distribution differs largely. therefore, the copy mechanism is important for handling unseen entities in the test set, and our proposed copy variant in section 3.3 is also essential to globally normalized models. we then train our model with the max-margin loss. our globally normalized model consistently improves the accuracy on both development and in addition, we notice that a large number of entities in atis have a form like "ap:denvor" (denver airport). we thus use the combination of characterlevel elmo embeddings @xcite and word-level glove embeddings @xcite . this further improves the accuracy, which outperforms the previous methods by ∼1.9% in the setting without data augmentation. conala dataset. for conala, bleu is treated as the main metric in previous work @xcite , because accuracy is generally very low (<3%) on this dataset. from we further compare our model with @xcite , which reranks beam search results by heuristics. our method is outperformed by the reranking approach. note that reranking can be considered as alleviating label bias with postprocessing, as the locally normalized model fails to assign the correct sequence with the highest joint probability. however, the reranking method requires training several reranking scorers, combined with an ad hoc feature (namely, length). by contrast, our global normalization does not rely on ad hoc human engineering. spider dataset. ## conclusion in this work, we propose to apply global normalization for neural semantic parsing. our approach predicts the score of different grammar rules at an autoregressive step, and thus it does not suffer from the label bias problem. we observe that our proposed method is able to improve performance on small datasets with lstm-based encoders. however, global normalization becomes less effective on the large dataset with a bert architecture.