How to S EQ 2S EQ for SQL Anonymous EMNLP submission
Abstract
Translating natural language to SQL queries for table-based question answering has recently attracted more research attention.
Previous approaches develop models whose architecture is speciﬁcally tuned to the structure of the task, such as separately predicting the arguments of the SELECT clause.
In this work, we show that a more general attention-based sequence-to-sequence model outperforms more specialized state-of-the-art approaches by only adapting the input and output layers.
In particular, we extend it with onthe-ﬂy embedding and output vectors as well as an input copying mechanism, which are used throughout the whole decoding process.
We also investigate the potential order-matters problem that could arise due to having multiple correct decoding paths and investigate the use of a dynamic oracle in this context.
Introduction
Semantic parsing, the task of converting natural language utterances to their representation in a formal language, is a fundamental problem in
NLP and has many applications, including Question Answering (QA) over structured data.
In this work, we focus on QA over tabular data (NL2SQL), which recently attracted research efforts (Zhong et al., 2017; Xu et al., 2017; Neelakantan et al., 2017).
As in many other semantic parsing tasks, in this task, we are confronted with tree-structured decoding targets and challenging output vocabularies.
In NL2SQL, we must handle table columns, which may be different for every example and unseen during testing.
We also need to handle other rare words that refer to values in the tables and thus are essential for building the correct query.
We develop a S EQ 2S EQ -based approach that outperforms the state-of-the-art on the recently introduced W IKI SQL (Zhong et al., 2017) dataset.
With 80654 examples annotated with their logical forms, W IKI SQL is, to the best of our knowledge, the largest fully annotated and manually verbalized dataset for semantic parsing and question answering with complex questions 1 .
In contrast to previous works on W IKI SQL (Zhong et al., 2017;
Xu et al., 2017) that adapt the architecture of their models to the structure of the data, our approach is more general as it only adapts the input and output layers of a standard S EQ 2S EQ model to better suit the task of generating SQL queries.
In particular, we add on-the-ﬂy embeddings (Bahdanau et al., 2017; Hill et al., 2016) and on-the-ﬂy output vectors and incorporate a pointer-based (Vinyals et al., 2015) copying mechanism as part of the output layer.
In contrast to previous works on W IKI SQL, which used separate losses for SQL generation subtasks, our approach can be trained using normal sequence supervision.
The order of decoding of unordered sets using
S EQ 2S EQ models can have a signiﬁcant impact on performance (Vinyals et al., 2016), as mentioned by previous works on W IKI SQL (Zhong et al., 2017; Xu et al., 2017).
We investigate the use of a dynamic oracle (Goldberg and Nivre, 2012) in an effort to improve the generation of logical forms with unordered children (in our case conditions in the WHERE clause).
To summarize, our contributions are as follows:
We outperform the state-of-the-art on the W IKI SQL task using a S EQ 2S EQ architecture with specialized input and output layers.
The devised model is more extensible to tasks with a different structure than previous work.
We investigate the potential order-matters problem (Vinyals et al., 2016) when having multiple correct decoding paths.
In this context, we also investigate two types of dynamic oracle in training semantic parsers instead of teacher forcing.
To the best of our knowledge, dynamic oracles have not been investigated for neural network-based semantic parsing before.
The paper is structured as follows: First, we illustrate the task with an example (Section 2).
Then, we describe our model (Section 3), followed by a description of the training methods (Section 4).
We then present our experimental setup and results in Section 5 and provide an extensive discussion in Section 6 that goes deeper into the most related work.
We conclude with an overview of other related work (Section 7) and a conclusion (Section 8).
Queries, Trees and Linearizations
As an illustration, consider the question “ Which L1 Cache can we get with an FSB speed of 800MHz and a clock speed of 1.2GHz? ”, which should be mapped to the following query:
SELECT
L1 Cache
WHERE
FSB Speed = 800 (Mhz)
AND
Clock Speed = 1.0 (GHz) This query will be executed over a table listing processors, the sizes of their caches, their clocks speeds, etc.
The query can be represented as a tree (see Figure 1), where the root node has two children:
SELECT and WHERE .
Note that the order of the two conditions appearing in the WHERE clause is arbitrary and does not have any impact on the meaning of the query or the execution results. [Figure] WHERE
[Figure]
Trees containing such unordered nodes can be linearized into a sequence in different, equally valid, ways (”FSB Speed” ﬁrst or ”Clock Speed” ﬁrst in the example).
We refer to the linearization where the original order given in the data set is preserved as the canonical linearization .
The two valid linearizations for the example are shown in Figure 2.
See Supplement A for speciﬁcs on our implementation of tree linearization.
[Figure]
Model
We start from a sequence-to-sequence model with attention and extend the embedding and output layers to better suit the task of QA over tabular data.
In particular, we compute on-the-ﬂy embeddings (Bahdanau et al., 2017) and output vectors for column tokens and implement a pointerbased (Vinyals et al., 2015) mechanism for copying tokens from the question.
The S EQ 2S EQ model, the embedding and output layers are described in Sections 3.1, 3.2 and 3.3, respectively.
3. The S EQ 2S EQ Model
The general architecture of our model follows the standard sequence-to-sequence (S EQ 2S EQ ) attention-based architecture.
The S EQ 2S EQ model consists of an encoder, a decoder and an attention mechanism.
3.1. Encoding
We are given a question Q =
[ q 0 , q 1 , . .
. , q N ] of natural language (NL) tokens q i from a set V E (i.e., the encoder vocabulary).
We ﬁrst pass the tokens through an embedding layer that maps every token q i to its vector representation q i ∈ R d emb , that is
[Equation]
where W E ∈ R |V E |× d emb is a learnable weight matrix and one hot ( · ) maps a token to its one-hot vector.
Given the token embeddings, a bidirectional two-layered
LSTM (Hochreiter and
Schmidhuber, 1997) (see
Supplement B) encoder outputs encoding vectors [ h 0 , h 1 , . . .
, h N ] = BiLSTM
([ q 0 , q 1 , . .
. , q N ]) , where h i ∈ R d enc .
3.1. Decoding
The decoder produces a sequence of output tokens s t from an output vocabulary V D conditioned on the input sequence Q .
It is realized by a unidirectional multi-layered LSTM.
First, the previous output s t − 1 is mapped to its vector representation:
[Equation]
The applied embedding function EMB ( · ) is described in Section 3.2 in detail.
Taking the embedding as input, the multi-layered LSTM updates the hidden states in every layer (see Supplement B for the update equations).
Finally, the probabilities of the elements of V D are calculated based on the output y t of the ﬁnal LSTM layer and a context vector ˆh t , which is deﬁned in Section 3.1.3:
[Equation]
ˆh
[Equation]
[Equation]
[Equation]
p (
[Equation]
t |
[Equation]
t
[Equation]
[Equation]
= [Equation] [Equation] where the function OUT ( · ) is the output layer that computes unnormalized scores over V D as explained in Section 3.3.
3.1. Attention
We use attention (Bahdanau et al., 2014) to compute the context vector ˆh t , that is a ( t ) i
= h
i · y t , (5) α ( t ) i = [Equation] t [Equation]
[Equation] [Equation] i
[Equation] (6) ˆh
[Equation] N (cid:88) [Equation]
(7) where softmax ( ˙ ) i denotes the i -ith element of the output of the softmax function and h 1 , . . .
, h N are the hidden vectors of the encoder described in Section 3.1.1.
3. Embedding Function of the Decoder
The whole output vocabulary V D can be grouped in three parts: (1) structure tokens from V STR , (2) column ids from V COL , and (3) input words from the encoder vocabulary V E , i.e., V D = V STR ∪
V COL ∪ V E .
Whereas structure tokens are shared among examples, column ids refer to columns that can have different names in every example depending on the corresponding table.
In the following paragraphs, we describe how each of the three types of tokens is embedded in the decoder.
Structure tokens:
These are tokens used to represent the structure of the query, such as SQL speciﬁc tokens like SELECT and WHERE .
Since the available tokens V STR are the same for every example, and their meaning stays the same across examples, we use a vanilla embedding matrix W STR and reuse it across all examples.
See Supplement A for details on the used structure tokens.
Column tokens:
These are column ids that are used to refer to a certain column in the table the question is being asked against.
Since these ids themselves are largely meaningless, we map them to the corresponding column names to get a more meaningful representation.
We encode the column names using a uni-directional single-layer LSTM to get the embedding vector representing the ids in V COL .
Column names may consist of several words, which are ﬁrst embedded using an embedding matrix W CT containing an embedding vector for every word in V CT that occurs in any column name in any table.
The word embedding are then fed into the LSTM and the ﬁnal state of the LSTM is taken as the embedding vector for the column id token.
This approach for computing column representations is similar in spirit to other works that encode external information to get better representations for rare words (Bahdanau et al., 2017; Ling et al., 2015; Hill et al., 2016).
Input
words:
To represent input words in the decoder we reuse the vectors from the embedding matrix W E that is also used for encoding the question.
3. Output Layer of the Decoder
The output layer of the decoder takes a the context ˆh t and the hidden state y t of the decoder’s
LSTM and produces scores over the output vocabulary V D (see OUT ( · ) in Eq. 3).
For each of the three subsets of V D ( V STR , V COL and V E , as deﬁned in Section 3.2), the scores are computed as described in the following paragraphs.
The whole output layer is visualized in Figure 3.
Structure tokens:
For the structure tokens we apply a trainable matrix U STR ∈ R |V STR |× d out that [Figure] is shared across examples.
The output scores for tokens in V STR are computed by the linear transformation: U STR · [ y t , ˆh t ] .
Column tokens:
Similarly as in the embedding function, the output vectors for every columnfrom V COL are computed by encoding the corresponding column names using an LSTM.
Let the matrix U COL k contain the computed vectors for every column present in the table of the current example. The output scores for all column id token are then computed by U COL k · [ y t , ˆh t ] .
Input words:
To enable our model to copy tokens from the input question, we follow a pointerbased approach to compute scores over these words.
That is, we compute scores b
i between y t and the hidden state of the encoder LSTM at every input position i : b
[Equation]
y t , (8) and then obtain the score for every token q that occurs in the question sequence Q by taking the maximum over all positions in Q =
[ q 0 , . . .
, q i , . .
. , q N ] where q occurs, i.e. the score for q is given by:
[Equation]
The scores for all input tokens q ∈ V E that do not occur in the question Q are set to −∞ .
We also ran experiments without the pointer mechanism just described to test its inﬂuence (see “without pointer” experiment in Section 5.3).
There, we replace the pointer mechanism by a simple linear transformation U E D · [ ˆh t , y t ] with a matrix U E D to yield scores over the set of tokens in the question. U E D is populated using pretrained word vectors which are kept ﬁxed during training. The alternative, training randomly initialized U E D resulted in worse performance, and is not reported.
3. Pretrained Embeddings and Rare Words
We initialize all NL embedding matrices 3 using Glove embeddings for words covered by Glove (Pennington et al., 2014) and use randomly initialized vectors for the remaining words.
Whereas randomly initialized word embeddings are trained together with the remaining model parameters, we keep Glove embeddings ﬁxed, since ﬁnetuning them led to worse results in our experiments.
We also replace rare words that do not occur in Glove with a rare word representation in all embedding matrices.
Training
We train our models by minimizing the crossentropy loss over the output probabilities:
L = − (cid:88) (
[Equation] [Equation]
log( p ( G ( k ) | Q ( k ) )) , (10) p ( G
[Equation]
k ) , Q ( k ) ) = |
[Equation]
where G ( k ) is a sequence of tokens g ( k ) t ∈ V D representing a valid linearization of the correct tree R ( k ) corresponding to question Q ( k ) from the training data D .
The probabilities of the tokens g ( k ) t are computed as described in Section 3.3.
For constructing the target sequence G ( k ) from the tree R ( k ) during training we use teacher forcing (TF) and also experiment with a dynamic oracle (Goldberg and Nivre, 2012).
The dynamic oracle was proposed for dependency parsing, where different sequences of actions can lead to the same parse tree. We notice that in semantic parsing, when the target trees contain at least some unordered children, we are confronted with a similar setting where different sequences of tokens can describe the same query. To the best of our knowledge, dynamic oracles have not been applied before in training neural networks based semantic parsers.
4. Teacher Forcing
Teacher forcing takes the canonical linearizations of the query trees (as provided in the dataset) and uses it both for supervision and as input to the decoder.
Thus, teacher forcing does not explore any deviations from the canonical path during training.
In the presence of different correct sequences (as resulting from different correct linearizations in this scenario), teacher forcing can suffer from suboptimal supervision order (Vinyals et al., 2016).
This might also be the case for semantic parsing, as pointed out by previous works on W IKI SQL (Zhong et al., 2017; Xu et al., 2017).
4. Dynamic Oracle
Instead of forcing the model to follow the canonical decoding sequence, our dynamic oracle enables the exploration of alternative correct sequences (i.e. different valid linearizations of the same tree) and is an adaptation of Goldberg and
Nivre’s (2012) dynamic oracle with spurious ambiguity .
It is formally described in Algorithm 1, which is invoked at every decoding step t to get a token g t (used for supervision) and a token x t + (that will be used as input to the decoder in the next time step) from the set VNT t of possible next tokens that would lead to a valid linearization of the given tree.
Essentially, the algorithm always picks the best-scored correct token for supervision and uniformly samples one of the correct tokens to be used as decoder input in the next time step, if the overall best-scored token (over the whole output vocabulary) does not belong to the correct ones.
Thus, the oracle explores alternative paths if the decoder would make a mistake in free-running mode.
Note that, for decoding steps t where VNT t is of size one, the oracle is equivalent to teacher forcing.
In the algorithm, p t is the decoder’s output distribution over V D at time step t .
The set of valid next tokens VNT t ⊂ V D , from which the correct tree can be reached, is returned by the function get valid next ( · ) .
In order to compute VNT t , we keep track of where in the query tree we are Algorithm then 6: : return and of what is left to decode, given the previously decoded tokens.
The query tree can have nodes with either ordered or unordered children (for example, children of the WHERE clause are unordered).
If we are currently decoding the children of a node with unordered children, all the children that have not been decoded yet are returned as VNT t .
In other cases, VNT t contains the next token according to the canonical sequence order.
See Supplement C for more details on the construction of VNT t .
Zhong et al.
(2017) propose policy gradient (PG)
based training for query generation. The presented oracle is similar to PG training in that it explores alternative paths to generate the same query.
In contrast to the oracle, a PG-based method would sample the next token ( x t +1 ) according to the predictive distribution ( p t ) and then use the sampled sequence to compute gradients for policy parameters:
[Equation]
Replacing argmax V D p t in line 3 of Algorithmwith sample VNT t p t (sample a token from VNT t according to p t ) yields an algorithm equivalent to a speciﬁc case of REINFORCE (Williams, 1992): where A t is the reward for the episode, and A t is set to +1 if the sampled sequence produces the full correct query and 0 otherwise.
With such rewards, the resulting vanilla REINFORCE does not update policy parameters when an incorrect query is sampled, ignoring the sampled episode. In contrast,
Zhong et al.
(2017) use a different reward scheme, assigning rewards of − 2 for invalid queries and − 1 if the query is valid but the execution result is incorrect and +1 otherwise.
Experiments
To evaluate our approach, we obtain the W IKI SQL dataset by following the instructions on the
W IKI SQL website 4 .
5. Dataset
Each example in the dataset contains a NL question, its SQL equivalent and the table against which the SQL query should be executed.
The original training/dev/test splits of W IKI SQL use disjoint sets of tables with different schemas.
This allows to obtain a performance estimate for the developed approaches when confronted with questions over previously unseen tables.
Whereas many different tables occur in W IKI SQL, a single query is limited to only a single table.
Nevertheless, the data set poses a signiﬁcant challenge due to the variation of table schemata across examples and the variable structure of expected SQL queries.
In addition, the task is complicated by a high occurrence of rare words, which can not be omitted since they can be essential for constructing the correct SQL query.
However, W IKI SQL is constructed such that the values in the conditions of the query occur as substrings in the NL question.
For details on the construction of the dataset and how it compares to existing datasets, we refer the reader to the W IKI SQL paper (Zhong et al., 2017).
5. Experimental Setup Evaluation:
We report results with the same evaluation metrics as previous works (Zhong et al., 2017;
Xu et al., 2017) on both the development and test sets:
• logical form accuracy Acc LF = K LF /K , • query match accuracy Acc QM =
K QM /K , • query execution accuracy Acc EX = K EX /K
where K is the total number of examples, K LF the number of examples for which the predicted sequence matched exactly the one provided by the dataset, K QM the number of examples for which the predicted query matched exactly (i.e. the predicted sequence is one of the correct linearizations) and K EX the number of examples for which the execution results matched exactly.
Note that while Acc LF accepts only the canonical ordering of conditions in the WHERE clause, Acc QM and
Acc EX accept alternative orderings.
Training details:
After a hyperparameter search, we obtained the best results by using two layers both in the encoder and decoder LSTMs, using d dec = 600 , d emb = 300 and applying time-shared dropouts on the inputs of the recurrent layers (dropout rate 0.2) and recurrent connections (dropout rate 0.1).
We trained using
Adam (Kingma and Ba, 2014), with a learning rate of 0.001 and a batch size of 100, a maximum of 50 epochs, and early stopping and clipped gradient norms at 5.
Experimental settings:
While training with an oracle does not depend on the chosen linearization of the query tree, training with TF does.
Therefore, for experiments with TF we explored variations in the linearizations of the query tree by changing the order of conditions in the WHERE clause, that is the order of conditions in the canonical query linearization is (1) reversed or is (2) reassigned randomly.
We also run experiments with the oracle in two variants: (1) as described in Algorithm 1 and (2) replacing the argmax in line 3 of Algorithmwith sampling according to p t , which is equivalent to REINFORCE with zero reward for incorrect queries (see Section 4.2).
For more details on the experimental setup, we refer to Supplement D. Our preprocessed data, code, and sample outputs will be made freely available after the double-blind review process is ﬁnished.
5. Results
We compare our obtained results under different settings with the performance reported for previous methods in Table 1.
The proposed S EQ 2S EQ model outperforms the current state-of-the-art by at least 4% Acc QM .
Using a single representation for rare words (indicated by “ + rare ” in the table) provides a small improvement.
The results indicate that the order of conditions in the linearization matters for the performance of
TF based training to a small degree.
Training with randomly reassigned order of conditions in the
WHERE clause results in a 1.5% drop in accuracy.
However, reversing the order of conditions does not seem to affect the results.
The REINFORCElike dynamic oracle (indicated by “ oracle - sample ” in the table) also does not seem to affect the results.
However, the dynamic oracle as described in Algorithm 1 seems to provide a small improvement.
We can also see that Acc LF for the oracle is signiﬁcantly lower compared to TF while Acc QM is on par with TF.
Given that Acc LF is sensitive to the order of arbitrarily ordered clauses and Acc QM is not, this means that the oracle-trained models effectively learned to use alternative paths.
Comparing the oracle to TF with arbitrarily reordered conditions in the WHERE clause shows that when the supervision sequences are not consistently ordered, training with TF can suffer.
When training with the oracle, the order of unordered nodes as provided in supervision sequences does not matter.
We also ran one setting (indicated by “ without pointer ” in the table) without the copying mechanism (described in Section 3.3), where we obtained signiﬁcantly worse results, indicating the beneﬁt brought by the pointer mechanism.
See Supplement E for separate accuracies for the WHERE and SELECT clauses.
Discussion
First, we discuss our model by comparison to
Seq2SQL (Zhong et al., 2017) and
SQLNet (Xu et al., 2017), followed by a discussion of the training methodology used by previous approaches and our approach.
6. Model Discussion
Seq2SQL:
The baseline in Seq2SQL (Zhong et al., 2017) was based on the earlier S EQ 2S EQ work by Dong and Lapata (2016) for semantic parsing with lambda expressions.
Our model uses the same architecture (S EQ 2S EQ ) but in contrast to Seq2SQL’s adaptation of Dong and
Lapata (2016) to W IKI SQL, we develop taskadapted embedding and output layers in the decoder.
Given the poor performance of an unadapted S EQ 2S EQ model, Zhong et al.
(2017) propose an augmented pointer network that is better suited for copying words from the input.
Seq2SQL authors obtain further improvement by introducing separate predictors with dedicated parameters in order to predict the two arguments of the SELECT clause: (1) the SELECT column and (2) the aggregator.
This also requires the addition of two losses in the ﬁnal training loss.
For the WHERE clause, Seq2SQL uses the augmented pointer network trained using policy gradient RL.
The SELECT column prediction uses column names encoded using an LSTM, similar to our approach.
However, our approach uses the same column name encodings throughout the whole decoding process .
SQLNet:
Similarly to Seq2SQL, SQLNet uses dedicated networks for predicting the two parts of the SELECT clause, using column-conditioned question summaries and explicitly encoding task-speciﬁc prediction dependencies into the architecture.
For the WHERE clause, SQLNet proposes a sequence-to-set approach that essentially makes a set inclusion prediction over the set of available columns in order to avoid the order-matters (Vinyals et al., 2016) problem.
However, in addition to producing scores for the available columns, SQLNet needs a dedicated network to predict the number of columns that will ultimately be taken.
Compared to Seq2SQL and SQLNet, we believe our model is simpler and more general, while achieving better performance.
Thus, we believe that our model can be applied to other tasks, such as machine translation, if they can beneﬁt from the rare word handling extensions.
6. Training Discussion
The oracle we implement is similar to reinforcement learning (RL) in that it explores alternative paths to generate the same query and similar to the work of Vinyals et al.
(2016) in that it’s aimed at ﬁnding and using more effective linearizations during training.
The dynamic oracle with non-optimal transitions from Goldberg and
Nivre (2012) also explores incorrect paths but is able to provide more useful supervision at every time step on how to build the best possible tree after a mistake.
Our oracle implements only the dynamic oracle with spurious ambiguity (Goldberg and Nivre, 2012) and thus explores only correct paths.
We leave the oracle with non-optimal transitions for future work.
Other Related Work Some
examples of semantic parsers that do not rely on neural networks include
Zettlemoyer and Collins (2007),
Berant et al. (2013) and Reddy et al. Recently, many neural network-based approaches emerged Dev Accuracies (%)
Test Accuracies (%)
Acc LF
Acc QM
Acc EX
Acc LF
Acc QM
Acc EX
Seq2Seq baseline (Zhong et al., 2017) 23.
Aug. Ptr.
Net (Zhong et al., 2017) 44.
Seq2SQL (no RL) (Zhong et al., 2017) 48.
Seq2SQL (RL) (Zhong et al., 2017) 49.
*Seq2SQL (SQLNet) (Xu et al., 2017) 52.
[Table]
SQLNet (Xu et al., 2017) – [Table]
[Table] [Table] [Table] Seq2Seq+DP+C (TF) 65 .67 .74 [-.64 .
-].
[Table]
Seq2Seq+DP+C (TF - reversed)
Seq2Seq+DP+C (TF - arbitrary) – 65 .73 .– 64 .72 .
Seq2Seq+DP+C (TF) without pointer 60 [-.
-].62 .
[Table]
Seq2Seq+DP+C (TF) + rare 66.
[-Seq2Seq+DP+C (oracle) 
-][Table] [Table] [Table] [Table]
[Table] Seq2Seq+DP+C [-(oracle - sample) 
-](oracle) [Table] [Table]
[Table] [Table] [Table] Seq2Seq+DP+C (oracle - sample)
[Table] [Table]
[Table] [Table] [Table] Seq2Seq+DP+C (oracle) + rare
[Table] [Table] [Table]
[Table]
focusing on semantic parsing and question answering.
Works focusing on answering simple questions (Bordes et al., 2015; Yin et al., 2016;
Lukovnikov et al., 2017; Dai et al., 2016) typically employ a ranking approach to rank logical forms (which always consist of 2 elements).
Regarding more complex questions, some approaches (Yih et al., 2015; Yu et al., 2017) also rely on the ranking of (parts of) candidate queries and use rules to complete the query.
Some other works on complex questions use reinforcement learning (Liang et al., 2016; Zhong et al., 2017).
Some interesting approaches for answering questions over tables include the Neural Programmer (Neelakantan et al., 2016) and Neural Enquirer (Yin et al., 2015), both of which use only the execution results to supervise query composition, relying on end-to-end decoding and execution instead of reinforcement learning.
Neelakantan et al.
(Neelakantan et al., 2017) further evaluate the Neural Programmer on the W IKI T ABLEQ UESTIONS (Pasupat and Liang, 2015) dataset.
The W IKI T ABLE Q UESTIONS dataset is similar to the W IKI SQL dataset in that it focuses on learning to query tables, however, W IKI T ABLE Q UESTIONS does not provide logical forms.
Conclusion
In this work we present a S EQ 2S EQ model adapted to the semantic parsing task of NL2SQL.
It outperforms the state-of-the-art on the W IKI SQL dataset, while being more general than previous works in the sense that we only adapt the input and output layers of a S EQ 2S EQ model.
We also investigated the order-matters problem, concluding that order of conditions in the WHERE clause matters to small degree.
We also evaluated two dynamic oracles for training the neural network-based semantic parser, which in our experiments revealed that the RL-like oracle does not improve results and the oracle most similar to the work of Goldberg and Nivre (2012) provides a small improvement.