Transformer Semantic Parsing (2020)

Abstract: In neural semantic parsing, sentences are mapped to meaning representations using encoder-decoder frameworks. In this paper, we propose to apply the Transformer architecture, instead of recurrent neural networks, to this task. Experiments in two data sets from different domains and with different levels of difficulty show that our model achieved better results than strong baselines in certain settings and competitive results across all our experiments.

## introduction semantic parsing maps natural language sentences to meaning representations including, but not limited to, logical formulas, structured query language (sql) queries, or executable codes. in recent years, end-to-end neural semantic parsing has achieved good results @xcite @xcite @xcite . the main advantage of these models is that they do not require intermediate representations, lexicons, manually designed templates, or handcrafted features. current neural semantic parsing models use encoder-decoder architectures with recurrent neural networks (rnns). one drawback of rnns is their inability to capture long distance relationships between input tokens or between input and output tokens @xcite . hence, to model dependencies disregarding their distance, the best performing models also include some kind of an attention mechanism @xcite , which allows to model dependencies more accurately; specifically, the transformer architecture introduced in this last reference has become the new state-of-the-art for sequence-tosequence problems. the transformer architecture consists of a selfattention mechanism and does not include recurrent layers. unlike rnns, in which sequences are processed sequentially -word by word -transformer models process the entire sentence as a whole. this characteristic is particularly beneficial in capturing long distance dependencies as the self-attention mechanism sees all the words at once. despite the success of transformer, to the best of our knowledge, prior to @xcite that founded our paper, this framework has never been applied to semantic parsing before. thus, in this paper, we propose the transformer architecture for semantic parsing. a well-known limitation of sequence-tosequence models is their inability to learn competitive parameter values for words that are rare in a given data set. to alleviate this problem, a common method is to anonymise entities with their respective types. for example, city names such as denver are anonymised as ci0 and later, as part of post-processing, put back in the output utterance. neural semantic parsing models are usually trained and tested using data sets where variables are identified and anonymised beforehand -as in the example above -which considerably reduces the difficulty of the semantic parsing task @xcite . as a result, many input sentences of the test set are seen prematurely while training. consequently, we have a twofold approach to evaluate our model more extensively in this work and demonstrate its contributions to semantic parsing: first, we use non-anonymised versions of two data sets for semantic parsing from different domains, as well as different data splits. second, we test the ability of our model to compose new output meaning representations. these experiments give evidence of the transformer model outperforming strong baselines in certain settings. its results are competitive on other settings across the data sets. the rest of the short paper is organised as follows: section 2 includes the related work in neu-ral semantic parsing. section 3 describes our model architecture, data sets, and experimental setup. section 4 introduces our experimental results. section 5 concludes our study. ## related work encoder-decoder architectures based on neural networks have been applied in the past five years (i.e., up to, and including, 2019) to semantic parsing, but they have typically learnt from sentences paired with meaning representations without using explicit syntactic knowledge. @xcite have proposed two models with an attention mechanism as follows: the first model generates sequences and the second one generates trees as logical formulae that are hierarchically structured. both models include an attention mechanism over a rnn that has the ability of focusing on a subset of input tokens or features. they also provide soft alignments between the input sentences and the logical formulae. later, @xcite have proposed to use a twostep (coarse-to-fine) decoder to better model the compositionality of the logical formulae. finally, to overcome the limitations of semantic parsing data sets being small and domain-dependent, several methods, such as multi-task learning @xcite @xcite , transfer learning @xcite , and data augmentation @xcite , have been applied. however, the aforementioned models cannot address the fact that the distance between tokens is not always an indication of a weak relationship. this problem becomes significantly worse for long sentences paired with long logical formulae. thus, we propose to use a transformer-based model in which token relations are not affected by the (long) distance. as mentioned, data sets for semantic parsing are small and neural models do not tend to be good at learning the appropriate parameters for the long tail of rare words. to mitigate this problem, a common method is to apply variable or entity anonymisation as a pre-processing step. later, the entities are put in the output sequence in a postprocessing step. another strategy is to use pointer networks @xcite where input tokens are copied to the output sequence at each decoding step. moreover, attention-based copying @xcite refers to a mechanism in which the decoder can either choose to copy over a word to the output sequence or to pick from a softmax over the entire vocabulary. in our study, we use two data sets from different domains with and without variable anonymisation, and different splits to reflect diffing complexity levels of the semantic parsing task. ## methods as our model architecture, we have implemented a self-attention neural semantic parsing model with the mechanism of transformer (see @xcite 1 ) for further information). as in other state-of-the-art sequence-to-sequence architectures, transformer is essentially an encoderdecoder structure with blocks for encoding and decoding. similar to other neural generation models with attention, the output of the last layer of the encoder is used as part of the input of each layer of the decoder. the most significant difference between transformer and other sequenceto-sequence models is that transformer uses neither convolutional neural networks (cnns) nor rnns. instead, it uses self-attention, which reduces path lengths within the network, thus minimising the loss of information due to computations. identical encoder blocks consist of multi-head self-attention and fully-connected feed-forward layers. after each layer there is a residual module, followed by a normalisation step. this produces a 512-dimensional output. each of identical decoder blocks has a multihead self-attention layer, fully-connected feedforward layer, and multi-head attention layer over the output of the encoder stack. again, a residual module and normalisation step are applied to each output layer. multi-head self-attention layers in encoding and decoding are similar, except in the latter one adds a mask operation between scaling and the softmax activation function. the reason for adding this look-ahead mask is to avoid that at the timestamp t, the tokens after t are used for predicting token at t . in most natural language processing tasks, the model should capture the order and position information from the sequential inputs. this is one of the advantages of rnns and cnns. thus, our model includes a position embedding in the source and target inputs as in vaswani et al. @xcite . furthermore, word embeddings are randomly initialised for source and target inputs to treat them equally. we use the adam optimiser. the learning rate is set to 3×10 -4 . the dimension of the self-attention model is 1, 024. from 6 to 8 encoder and decoder blocks are used. the dropout rate is 0.4 and maximum number of epochs 720 with early stopping. we have used two semantic parsing data sets -namely atis with queries from a flight booking system @xcite @xcite and geo with queries about us geographical information (zelle and @xcite 84.6 86.07 tisp @xcite 84.2 88.9 neural baselines seq2seq + attention @xcite 84.15 84.6 there are two types of data set splits: question-split and query-split. in the former, training and testing examples are divided based on questions, thus based on the input sequence. in the latter, training and test examples are divided according to the similarity of their meaning representations, thus based on output sequences. in other words, training and testing examples in query-split are strictly controlled to have a more diverse set of logical formulae. therefore, its use is more appropriate when evaluating the model's capability to compose output sequences (i.e., lambda calculus expressions here). for each split, data sets might contain variables with or without anonymisation (tables 1 and 2 , resulting in two versions of the first data set (i.e., atis question-split and atis question-split non-anonymised) and three versions of the second data set (i.e., geo question-split, geo question-split non-anonymised, and geo query-spit non-anonymised). versions of geo without anonymisation are from @xcite and splits originate from @xcite . as output logical formulae cannot be partially correct, we report the exact match by computing @xmath0. its accuracy of 87.95% ( ## conclusion and future work we evaluated the transformer architecture for semantic parsing. the model was extensively evaluated with two data sets from different domains -with and without anonymisation -across a range of complexity levels. experiments shows transformer is competitive with other state-of-theart models and outperformed strong baselines in some settings. for future work, it would be interesting to design a tree-structure self-attention model. as logical forms are tree-structures, adding some constraints in the decoder to enforce tree-based decoding would be of particular interest.