Transferring Knowledge from Structure-aware Self-attention Language Model to Sequence-to-Sequence Semantic Parsing (2022)

Abstract: Semantic parsing considers the task of mapping a natural language sentence into a target formal representation, where various sophisticated sequence-to-sequence (seq2seq) models have been applied with promising results. Generally, these target representations follow a syntax formalism that limits permitted forms. However, it is neither easy nor flexible to explicitly integrate this syntax formalism into a neural seq2seq model. In this paper, we present a structure-aware self-attention language model to capture structural information of target representations and propose a knowledge distillation based approach to incorporating the target language model into a seq2seq model, where grammar rules or sketches are not required in the training process. An ablation study shows that the proposed language model can notably improve the performance of the baseline model. The experiments show that our method achieves new state-of-the-art performance among neural approaches on four semantic parsing (ATIS, GEO) and Python code generation (Django, CoNaLa) tasks.

## introduction semantic parsing aims to map a natural language sentence into a machine executable formal representation, which has been considered as one of the prime challenges nowadays in natural language processing (nlp). these target formal representations can generally be divided into three categories (kamath and das, 2018), i.e., logical forms, like first order sentences or λ-calculus expressions @xcite , programming language statements, like python code or sql programs, and graph-based forms, like labeled graphs in abstract meaning representation (amr) @xcite . in this paper, we focus on semantic parsing that yields logical forms. target logical forms often follow a syntax formalism that limits permitted formulas, which can be used to filter the output and improve the performance of semantic parsing. for example, in the preneural era, ccg based approaches @xcite achieved significant performance gains by introducing a linguistically motivated grammar induction scheme. some neural semantic parsers @xcite first transduce the natural language utterance into an abstract syntax tree (ast), then serve it as an intermediate meaning representation to incorporate with grammar rules for the target logical form. semantic parsing can also be considered as a seq2seq transduction problem, where the decoder can leverage structural features of target representations. in particular, hierarchical tree decoders are applied in @xcite @xcite to take into account the tree structure of the logical expression. decoders constrained by a grammar model are applied in @xcite @xcite . the uncertainty-driving adaptive decoding is used to guide the decoder in @xcite . relatively sizeable monolingual corpus of the target programming language is used in @xcite to improve performance. note that, manually specified grammar rules and sketches for target logical forms are required in most of these approaches, which limits their adaptabilities and scalabilities to a new semantic parsing task with updated target logical forms. in this paper, we consider using a structure-aware language model to capture formal patterns for target representations and incorporating the language model into seq2seq models for semantic parsing. we first train the structure-aware language model on target logical forms to capture structural information. then, we incorporate the language model to a seq2seq model for semantic parsing. integrating a language model into a seq2seq model has been considered in automatic speech recognition (asr) and neural machine translation (nmt). in particular, shallow fusion and deep fusion @xcite are two such approaches in nmt. cold fusion @xcite ) is tested on asr tasks. @xcite proposes a knowledge distillation based training approach to transferring knowledge from a language model to a seq2seq model for asr. here, we follow the knowledge distillation structure to integrate the language model to the baseline seq2seq model for semantic parsing. we evaluate our approach on two semantic parsing datasets, atis @xcite and geo @xcite datasets, where target logical forms are λ-calculus expressions and two code generation tasks, django @xcite and conala @xcite , where target logical forms are python code. we train the target language model based on target logical forms. the experimental results show that our approach achieves state-of-the-art performance among neural network based approaches on atis, geo, django and conala datasets. in this paper, we show that the proposed language model can be used to capture structural features of target logical forms and the knowledge distillation structure can be used to transfer knowledge to a seq2seq model for semantic parsing, where manually specified grammar rules or sketches are no longer required. notice that, this approach can be applied to various sophisticated seq2seq models, which results a more flexible and scalable method for neural semantic parsers to leverage structural features of target representations. the main contributions of the paper are summarized as follows: ## method in this section, we specify details of our method, i.e., using a knowledge distillation based structure to transfer knowledge from a structure-aware target language model to a seq2seq model. we first introduce the architecture of the new model. then, we describe the proposed target language model. at last, we provide details of the method in the training process. ## experiments in order to evaluate the performance of our proposed model, we conduct the experiments detailed below. ## conclusion in this paper, we present a structure-aware selfattention language model to capture structural information of target representations and propose a knowledge distillation based approach to incorporating the target language model into a seq2seq model. we show that using knowledge distillation from a target language model provides a flexible and scalable way for neural semantic parsers to leverage structural features of target representations. our method achieves strong results and doesn't need any manually designed rules or sketches. for future direction, we are interested in exploring other datasets to verify the model's ability for structural data. we will also attempt to integrate grammar rules to this model to have a better performance on semantic parsing tasks.