How to S EQ 2S EQ for SQL Anonymous submission
Abstract
Translating natural language to SQL queries for table-based question answering has recently attracted more research attention.
Previous approaches develop models whose architecture is speciﬁcally tuned to the structure of the task, such as separately predicting the arguments of the SELECT clause.
In this work we show that a more general attentionbased sequence-to-sequence model outperforms more specialized state-of-the-art approaches by only adapting the input and output layers.
In particular, we extend it with on-the-ﬂy embedding and output vectors as well as an input copying mechanism, which are used throughout the whole decoding process.
We also investigate the potential order-matters problem that could arise due to having multiple correct decoding paths and investigate the use of a dynamic oracle in this context.
Introduction
Semantic parsing, the task of converting natural language utterances to their representation in a formal language, is a fundamental problem in
NLP and has many applications, including Question Answering (QA) over structured data.
In this work, we focus on QA over tabular data (NL2SQL), which recently attracted research efforts (Zhong et al., 2017; Xu et al., 2017; Neelakantan et al., 2017).
As in many other semantic parsing tasks, in this task, we are confronted with tree-structured decoding targets and challenging output vocabularies.
In NL2SQL, we must handle table columns, which may be different for every example and unseen during testing.
We also need to handle other rare words that refer to values in the tables and thus are essential for building the correct query.
We develop a S EQ 2S EQ -based approach that outperforms the state-of-the-art on the recently introduced W IKI SQL (Zhong et al., 2017) dataset.
With 80654 examples annotated with their logical forms, W IKI SQL is, to the best of our knowledge, the largest fully annotated and manually verbalized dataset for semantic parsing and question answering with complex questions 1 .
In contrast to previous works on W IKI SQL (Zhong et al., 2017;
Xu et al., 2017) that adapt the architecture of their models to the structure of the data, our approach is more general as it only adapts the input and output layers of a standard S EQ 2S EQ model to better suit the task of generating SQL queries.
In particular, we add on-the-ﬂy embeddings (Bahdanau et al., 2017; Hill et al., 2016) and on-the-ﬂy output vectors and incorporate a pointer-based (Vinyals et al., 2015) copying mechanism as part of the output layer.
In contrast to previous works on W IKI SQL, which used separate losses for SQL generation subtasks, our approach can be trained using normal sequence supervision.
The order of decoding of unordered sets using
S EQ 2S EQ models can have a signiﬁcant impact on performance (Vinyals et al., 2016), as mentioned by previous works on W IKI SQL (Zhong et al., 2017; Xu et al., 2017).
We investigate the use of a dynamic oracle (Goldberg and Nivre, 2012) in an effort to improve the generation of logical forms with unordered children (in our case conditions in the WHERE clause).
To summarize, our contributions are as follows:
We outperform the state-of-the-art on the W IKI SQL task using a S EQ 2S EQ architecture with specialized input and output layers.
The devised model is more extensible to tasks with a different structure than previous work.
We investigate the potential order-matters problem (Vinyals et al., 2016) when having multiple correct decoding paths.
In this context, we also investigate two types of dynamic oracle in training semantic parsers instead of teacher forcing.
To the best of our knowledge, dynamic oracles have not been investigated for neural network-based semantic parsing before.
The paper is structured as follows: First, we illustrate the task with an example (Section 2).
Then, we describe our model (Section 3), followed by a description of the training methods (Section 4).
We then present our experimental setup and results in Section 5 and provide an extensive discussion in Section 6 that goes deeper into the most related work.
We conclude with an overview of other related work (Section 7) and a conclusion (Section 8).
Queries, Trees and Linearizations
[todo: lets include also a NL querey and a table as example]
As an illustration, consider the following query:
SELECT
L1 Cache
WHERE
[Equation]
AND
[Equation]
The query can be represented as a tree (see Figure 1), where the root node has two children:
SELECT and WHERE .
Note that the order of the two conditions appearing in the WHERE clause is arbitrary and does not have any impact on the meaning of the query or the execution results.
[Figure]
Trees containing such unordered nodes can be linearized into a sequence in different (”FSB
Speed” ﬁrst or ”Clock Speed” ﬁrst in the example), equally valid, ways.
We refer to the linearization where the original order is preserved as the canonical linearization .
The two valid linearizations for the example are shown in Figure 2.
See
Supplement A for speciﬁcs on our implementation of tree linearization.
[Figure]
Model
We start from a sequence-to-sequence model with attention and extend the embedding and output layers to better suit the task of QA over tabular data.
In particular, we compute on-the-ﬂy embeddings (Bahdanau et al., 2017) and output vectors for column tokens and implement a pointerbased (Vinyals et al., 2015) mechanism for copying tokens from the question.
The S EQ 2S EQ model, the embedding and output layers are described in Sections 3.1, 3.2 and 3.3, respectively.
3. The S EQ 2S EQ Model
The general architecture of our model follows the standard sequence-to-sequence (S EQ 2S EQ ) attention-based architecture.
The S EQ 2S EQ model consists of an encoder, a decoder and an attention mechanism.
Encoding
We are given a question Q =
[ q 0 , q 1 , . .
. , q N ] of natural language (NL) tokens q i from a set V E (i.e., the encoder vocabulary).
We ﬁrst pass the tokens through an embedding layer that maps every token q i to its vector representation q i ∈ R d emb , that is
[Equation]
where W E ∈ R |V E |× d emb is a learnable weight matrix and one hot ( · ) maps a token to its one-hot vector.
Given the token embeddings, a bidirectional two-layered LSTM (Hochreiter and
Schmidhuber, 1997) (see
Supplement B) encoder outputs encoding vectors [ h 0 , h 1 , . . .
, h N ] = BiLSTM
([ q 0 , q 1 , . .
. , q N ]) , where h i ∈ R d enc .
Note that in our BiLSTM ( · ) , instead of concatenating the output vectors h fwd i and h rev i of the forward and reverse LSTM, respectively, we interleave their elements, that is h i =
[ h fwd i, 0 , h rev i, 0 , h fwd
i, 1 , h rev i, 1 , . . . ] .
3.1. Decoding
The decoder has to produce a sequence of output tokens s t (from an output vocabulary V D ) conditioned on the input sequence Q .
It is presented by an uni-directional multi-layered LSTM.
First the previous output s t − 1 is mapped to its vector representation [Equation]
[Equation]
The applied embedding function EMB ( · ) is in detail described in Section 3.2.
Taking the embedding as input, the multi-layered LSTM updates the hidden states in every layer (see Supplement B for the update equations). Finally, the probabilities for the elements of V D are calculated based on the output y t of the ﬁnal LSTM layer and a context vector ˆh t , which is deﬁned in Section 3.1.3:
[Equation] where the function OUT ( · ) is the output layer that computes scores over V D as explained in Section 3.3.
Attention
We use attention (Bahdanau et al., 2014) to compute the context vector ˆh t , that is a ( t ) i
= h
i · y t , (5) α ( t ) i = softmax ( a ( t ) 0 , . .
. , a ( t ) i , . .
. , a ( t ) N )
i , (6) ˆh t =
N (cid:88) i = α ( t ) i h i , (7) where softmax ( ˙ )
i denotes the i -ith element of the output of the softmax function and h 1 , . . .
, h N are the hidden vectors of the encoder described in Section 3.1.1.
Embedding Function of the Decoder
The whole output vocabulary V D can be grouped in three parts: (1) structure tokens from V STR , (2) column ids from V COL , and (3) input words from the encoder vocabulary V E , that is V D = V STR ∪ V COL ∪ V E .
Whereas structure tokens are shared among examples, column ids refer to columns that can have different names in every example (depending on the corresponding table).
In the following paragraphs, we describe how each of the three types of tokens is embedded in the decoder.
Structure tokens:
These are tokens used to represent the structure of the query, such as SQL speciﬁc tokens like SELECT and WHERE .
Since the available tokens V STR are the same for every example, and their meaning stays the same across examples, we use a vanilla embedding matrix W STR and reuse it across all examples.
See Supplement A for details on the used structure tokens.
Column tokens:
These are column ids that are used to refer to a certain column in the table the question is being asked against.
These ids themselves are largely meaningless, however, we can map them to the corresponding column names to get a more meaningful representation.
We encode the column names using a uni-directional singlelayer LSTM to get the embedding vector representing the ids in V COL .
The column names may consist of several words, which are ﬁrst embedded using an embedding matrix W CT containing an embedding vector for every word ∈ V CT that occurs in any column name in any table. The word embedding are then fed into the LSTM and the ﬁnal state of the LSTM is taken as the embedding vector for the column id token.
This approach for computing column representations is similar in spirit to other works that encode external information to get better representations for rare words (Bahdanau et al., 2017; Ling et al., 2015; Hill et al., 2016).
Input words:
To represent input words in the decoder we reuse the vectors from the embedding matrix W E that is also used for encoding the question.
3. Output Layer of the Decoder
Since in our task, the column names can be different for every example, we use on-the-ﬂy output
vectors for column tokens, similarly to the embedding case (Section 3.2).
In addition, the task requires copying input tokens into the value part of the conditions in the WHERE clause in the query, which we accomplish using a pointer mechanism.
The output layer of the decoder takes a the context ˆh t and the decoder’s LSTM’s output y t and produces scores over the output vocabulary V D (see OUT ( · ) in Eq. 3).
For each of the three subsets of V D ( V STR , V COL and V E , as deﬁned in Section 3.2), the scores are computed as described in the following paragraphs.
The whole output layer is visualized in Figure 3.
[Figure]
Structure tokens:
Similarly to the embedding case (Section 3.2), we construct a trainable matrix U STR ∈ R |V STR |× d out that is shared across examples.
The output scores for tokens in V STR are computed by a linear transformation: U STR · [ y t , ˆh t ] .
The dimension of the output layer is d out = d dec ∗ 2 .
Column tokens:
Similarly to the embedding case (Section 3.2), the vectors for column id tokens V COL are computed by encoding them using an LSTM to produce output vectors
u i ∈ R d out for every token in V COL .
The output scores for each column id token in V COL are then computed by U COL k · [ y t , ˆh t ] , where U COL k contains the output vectors u i for every available column id token for example k .
Input words:
The set of input tokens present in the output vocabulary for a certain example is limited to the set of unique tokens in the question of the example.
Constructing a trainable matrix
U E D ∈ R |V E |×
d out and using it for a linear transformation of [ ˆh t , y t ] will yield scores over the set of tokens V E from the question of the k -th example.
However, in order to improve the model’s copying abilities, we instead follow a pointer-based approach to compute scores over the words present in the input (question).
First, we compute scores b
i between y t and every input position i : b i
[Equation]
i
[Equation]
(8) and then obtain the score for every token q ∗ j ∈ V E that occurs in the question sequence Q ( k ) by taking the maximum over all positions i in Q ( k ) =
[ q 0 , . . .
, q i , . .
. , q N ] where q ∗ j occurs: output
[Equation]
The scores for tokens q ∗ j ∈ V E that do not occur in the question Q ( k ) are set to −∞ .
Output probabilities:
The three sets of scores for the tokens in V D are taken together and normalized using a softmax ( · ) to produce probabilities over V D .
3. Pretrained Embeddings and Rare Words
We initialize all natural language embedding matrices using Glove embeddings for words covered by Glove (Pennington et al., 2014): W E for question word embedding in the encoder, which is shared with the decoder’s embeddings for input words, and the two W CT embedding matrices for words occurring in column names in the column name encoders in both the encoder and the decoder.
Whereas vanilla word embeddings are trained together with the model, we keep Glove embeddings ﬁxed, since ﬁnetuning them led to worse results in our experiments.
Glove embeddings are also used (and ﬁxed) in the output layer for the - copy ablation experiment (Section 5), which resulted in a large improvement compared to a fully vanilla, trainable output layer.
We also replace embeddings of rare words with a single rare word, in the question encoding embedding layer, in the embedding and output layers of the decoder (for input words only), and in column name encoders’ embedding layers.
During testing, the embeddings of all rare or unseen words that do not occur in Glove are replaced with the rare word vector.
Training
For training, we use teacher forcing (TF) and also experiment with training using a dynamic oracle (Goldberg and Nivre, 2012).
The dynamic oracle was proposed by Goldberg and Nivre for dependency parsing, where different sequences of actions can lead to the same parse tree.
We notice that for semantic parsing, when the target trees contain at least some unordered children, we are confronted with a similar setting where different sequences of tokens can lead to the same query.
To the best of our knowledge, dynamic oracles have not been applied before in training semantic parsers with neural networks.
With both methods, we minimize the crossentropy loss over the output probabilities:
L [Equation] − (cid:88) (
[Equation] [Equation]
(10) p ( G (
[Equation]
) , Q ( k ) ) = | G ( k ) | (cid:89)
[Equation]
where G ( k ) is the (gold) sequence of tokens g ( k ) t ∈
V D leading to the correct (gold) query for example k , and occurs together with its question Q ( k ) in training data D .
The probabilities of the gold tokens g ( k ) t ∈ V D are computed as described in
Section 3.3 and Eq 3.
Teacher forcing and our adaptation of Goldberg and Nivre’s (2012) dynamic oracle differ in how the correct sequence of tokens used in Eq.
11 is constructed and which token is fed to the decoder in the next time step ( x t +1 ).
4. Teacher Forcing
Teacher forcing takes the canonical linearization of the query trees (as provided in the dataset) and uses it both for supervision and as input to the decoder.
Thus, teacher forcing does not explore any deviations from the canonical path.
In the presence of different correct gold sequences, teacher forcing can suffer from suboptimal supervision order (Vinyals et al., 2016).
This might also be the case for semantic parsing, as pointed out by previous works on W IKI SQL (Zhong et al., 2017; Xu et al., 2017).
4. Dynamic Oracle
Instead of forcing the canonical decoding path, our dynamic oracle enables the exploration of alternative correct decoding paths and is an adaptation of
Goldberg and Nivre’s (2012) dynamic oracle with spurious ambiguity .
Our dynamic oracle is formally described in Algorithm 1, which is invoked at every decoding step t to get the supervision (gold) token g t and the token x t +1 that will be used as input to the decoder in the next decoding step t + 1 .
Note that the oracle is only used during training.
During testing, the decoder is always free-running.
Algorithm[-
-]
1: function G ET N [Equation]
A [Equation] G OLD ( p t , t , x ≤ t ) 2:
VNT t ← get valid next ( t, x ≤ t ) 3: s t ← argmax V D p t 4:
if s t ∈ VNT t then 5: g t ← s t 6: x t +1 ← s t 7: else 8: g t ← argmax VNT t p t 9: x t +1 ← random ( VNT t ) 10: return g t , x t +
In the algorithm, p t is the decoder’s output distribution over V D at time step t .
The function get valid next ( · ) returns a set of tokens
VNT t ⊂ V D from which the gold tree can be reached.
In order to compute VNT t , we keep track of where in the query tree we are, based on the previously decoded tokens, and produce VNT t based on what is left to decode from the current node in the tree.
In summary, the whole set of the so far undecoded children of the last decoded parent node is returned as VNT t if they are unordered.
In other cases, it follows the canonical sequence order.
See Supplement C for more details on VNT t .
Essentially, the algorithm always picks the bestscored correct token as supervision and uniformly samples one of the correct tokens as the next token, if the overall best-scored token (over the whole output vocabulary) is not correct.
Thus, it explores alternative paths only when the decoder would make a mistake in free-running mode.
Note that, for decoding steps t where VNT t is of size one, the oracle is equivalent to teacher forcing.
The presented oracle is similar to reinforcement learning (RL) in that it explores alternative paths to generate the same query.
In contrast to the oracle, RL samples the next token ( x t +1 ) according to the predictive distribution ( p t ).
Policy gradient (PG) methods then use the sampled sequence to
compute gradients for policy parameters:
[Equation]
Replacing argmax V D p t in line 3 of Algorithm 1 with sample VNT t p t (sample a token from
VNT t according to p t ) yields an algorithm equivalent to a speciﬁc case of REINFORCE: where A t is the reward for the episode, and A t is set to + if the sampled sequence produces the full correct query and 0 otherwise.
With such rewards, the resulting vanilla REINFORCE does not update policy parameters when an incorrect query is sampled, ignoring the sampled episode.
Experiments
To evaluate our approach, we obtain the W IKI SQL dataset by following the instructions on the
W IKI SQL website 2 .
We use the original training splits and report the same metrics as previous works.
5. Dataset
Each example in the dataset contains a question, its SQL equivalent and the table against which the
SQL query should be executed.
The original training/dev/test splits of W IKI SQL use disjoint sets of tables with different schemas.
This allows to obtain a performance estimate for the developed approaches when confronted with questions over previously unseen tables.
Whereas many different tables occur in W IKI SQL, a single query is limited to only a single table.
Nevertheless, this poses a signiﬁcant challenge due to the variation of table schemata across examples and the variable structure of expected
SQL queries.
In addition, the task is complicated by a high occurrence of rare words, which can not be omitted since they can be essential for constructing the correct query.
However, W IKI SQL is constructed such that the values in the conditions of the query occur as substrings in the question.
For details on the construction of the dataset and how it compares to existing datasets, we refer the reader to the W IKI SQL paper (Zhong et al., 2017).
5. Experimental Setup Evaluation
We report results with the same evaluation metrics as previous works (Zhong et al., 2017;
Xu et al., 2017) on both the development and test sets:
• logical form accuracy Acc LF = K LF /K , • query match accuracy Acc QM =
K QM /K , • query execution accuracy Acc EX = K EX /K
where K is the total number of examples, K LF the number of examples for which the predicted sequence matched exactly,
K QM the number of examples for which the predicted query matched exactly and K EX the number of examples for which the execution results matched exactly.
Note that while Acc LF accepts only the canonical ordering of conditions in the WHERE clause, Acc QM and
Acc EX accept alternative orderings.
Training:
After a hyperparameter search, we obtained the best results by using two layers both in the encoder and decoder LSTMs, using d dec = 600 , d emb = 300 and applying time-shared dropouts on the inputs of the recurrent layers (dropout rate 0.2) and recurrent connections (dropout rate 0.1).
We trained using
Adam (Kingma and Ba, 2014), with a learning rate of 0.001 and a batch size of 100, a maximum of 50 epochs and early stopping and clipped gradient norms at 5.
Order experiments:
To investigate the inﬂuence of the linearization of the query, we ran experiments with teacher forcing under different linearizations, as well as the oracle in two modes.
For teacher forcing experiments, we explored a variation in the linearizations of the query by changing the order of conditions in the WHERE clause: the order of conditions in the canonical query linearization is (1) reversed or is (2) reassigned randomly.
We also run experiments with the oracle in two variants: (1) as described in Algorithmand (2) replacing the argmax in line 3 of Algorithm 1 with sampling according to p t , which is equivalent to REINFORCE with zero reward for incorrect queries (see Section 4.2).
For more details on the experimental setup, we refer to Supplement D. Our preprocessed data, code, and sample outputs will be made freely available after the double-blind review process is ﬁnished.
5. Results
We compare our obtained results under different settings with previously reported numbers in Table 1.
The proposed S EQ 2S EQ model outperforms the current state-of-the-art by at least 4% Acc QM .
Using a single embedding vector for rare words (Section 3.4) provides a small improvement.
The results indicate that the order of conditions in the linearization matters to a small degree.
Randomly reassigning the order of conditions in the
WHERE clause results in a 1.5% drop in accuracy.
However, reversing the order of conditions does not seem to affect the results.
The REINFORCElike dynamic oracle also does not seem to affect the results.
However, the dynamic oracle as described in Algorithm 1 seems to provide a small improvement.
We can also see that Acc LF for the oracle is signiﬁcantly lower compared to TF while Acc QM is on par with TF.
Given that Acc LF is sensitive to the order of arbitrarily ordered clauses and Acc QM is not, this means that the oracle-trained models effectively learned to use alternative paths.
Comparing the oracle to TF arbitrarily reordered conditions in the WHERE clause shows that when the supervision sequences are not consistently ordered, training with TF can suffer.
When training with the oracle, the order of unordered nodes in supervision sequences does not matter.
We also ran one experiment (“ - copy ”) without the copying mechanism (Section 3.3), obtaining signiﬁcantly worse results.
See Supplement E for separate accuracies for the WHERE and SELECT clauses.
Discussion
First, we discuss our model by comparison to
Seq2SQL (Zhong et al., 2017) and
SQLNet (Xu et al., 2017), followed by a discussion of the training methodology used by previous approaches and our approach.
6. Model Discussion
Seq2SQL:
The baseline in Seq2SQL (Zhong et al., 2017) was based on the earlier S EQ 2S EQ work by Dong and Lapata (2016) for semantic parsing with lambda expressions.
Our model uses the same architecture (S EQ 2S EQ ) but in contrast to Seq2SQL’s adaptation of Dong and
Lapata (2016) to W IKI SQL, we develop taskadapted embedding and output layers in the decoder.
Given the poor performance of an unadapted S EQ 2S EQ model, Zhong et al.
(2017) propose an augmented pointer network that is better suited for copying words from the input.
Seq2SQL authors obtain further improvement by introducing separate predictors with dedicated parameters in order to predict the two arguments of the SELECT clause: (1) the SELECT column and (2) the aggregator.
This also requires the addition of two losses in the ﬁnal training loss.
For the WHERE clause, Seq2SQL uses the augmented pointer network trained using policy gradient RL.
The SELECT column prediction uses column names encoded using an LSTM, similar to our approach.
However, our approach uses the same column name encodings throughout the whole decoding process .
SQLNet:
Similarly to Seq2SQL, SQLNet uses dedicated networks for predicting the two parts of the SELECT clause, using column-conditioned question summaries and explicitly encoding task-speciﬁc prediction dependencies into the architecture.
For the WHERE clause, SQLNet proposes a sequence-to-set approach that essentially makes a set inclusion prediction over the set of available columns in order to avoid the order-matters (Vinyals et al., 2016) problem.
However, in addition to producing scores for the available columns, SQLNet needs a dedicated network to predict the number of columns that will ultimately be taken.
Compared to Seq2SQL and SQLNet, we believe our model is simpler and more general, while achieving better performance.
Thus, we believe that our model can be applied to other tasks, such as machine translation, if they can beneﬁt from the rare word handling extensions.
6. Training Discussion
The oracle we implement is similar to reinforcement learning (RL) in that it explores alternative paths to generate the same query and similar to the work of Vinyals et al.
(2016) in that it’s aimed at ﬁnding and using more effective linearizations during training.
The dynamic oracle with non-optimal transitions from Goldberg and
Nivre (2012) also explores incorrect paths but is able to provide more useful supervision at every time step on how to build the best possible tree after a mistake.
Our oracle implements only the dynamic oracle with spurious ambiguity (Goldberg and Nivre, 2012) and thus explores only correct paths.
We leave the oracle with non-optimal transitions for future work.
Dev Accuracies (%)
Test Accuracies (%)
Acc LF
Acc QM
Acc EX
Acc LF
Acc QM
Acc EX
Seq2Seq baseline (Zhong et al., 2017) 23.
Aug. Ptr.
Net (Zhong et al., 2017) 44.
Seq2SQL (no RL) (Zhong et al., 2017) 48.
Seq2SQL (RL) (Zhong et al., 2017) 49.
*Seq2SQL (SQLNet) (Xu et al., 2017) 52.
SQLNet (Xu et al., 2017) – 63.
Seq2Seq+DP+C (TF) 65 .67 .74 .64 .
Seq2Seq+DP+C (TF - reversed)
Seq2Seq+DP+C (TF - arbitrary) – 65 .73 .– 64 .72 .
Seq2Seq+DP+C (TF) - copy 60 .
Seq2Seq+DP+C (TF) + rare 66.
Seq2Seq+DP+C (oracle) 
 Seq2Seq+DP+C (oracle - sample)
Seq2Seq+DP+C (oracle) + rare 53 .68.
Other Related Work
Some examples of semantic parsers that do not rely on neural networks include Zettlemoyer and
Collins (2007); Berant et al.
; Reddy et al.
Recently, many neural network-based approaches emerged focusing on semantic parsing and question answering.
Works focusing on answering simple questions (Bordes et al., 2015;
Yin et al., 2016; Lukovnikov et al., 2017; Dai et al., 2016) typically employ a ranking approach to rank logical forms (which always consist of 2 elements).
Regarding more complex questions, some approaches (Yih et al., 2015; Yu et al., 2017) also rely on the ranking of (parts of) candidate queries and use rules to complete the query.
Some other works on complex questions use reinforcement learning (Liang et al., 2016; Zhong et al., 2017).
Some interesting approaches for answering questions over tables include the Neural Programmer (Neelakantan et al., 2016) and Neural Enquirer (Yin et al., 2015), both of which use only the execution results to supervise query composition, relying on end-to-end decoding and execution instead of reinforcement learning.
Neelakantan et al.
(Neelakantan et al., 2017) further evaluate the Neural Programmer on the W IKI T ABLEQ UESTIONS (Pasupat and Liang, 2015) dataset.
The W IKI T ABLE Q UESTIONS dataset is similar to the W IKI SQL dataset in that it focuses on learning to query tables, however, W IKI T ABLE Q UESTIONS does not provide gold logical forms.
Conclusion
In this work we present a S EQ 2S EQ model adapted to the semantic parsing task of NL2SQL.
It outperforms the state-of-the-art on the W IKI SQL dataset, while being more general than previous works in the sense that we only adapt the input and output layers of a S EQ 2S EQ model.
We also investigated the order-matters problem, concluding that order of conditions in the WHERE clause matters to small degree.
We also evaluated two dynamic oracles for training the neural network-based semantic parser, which in our experiments revealed that the RL-like oracle does not improve results and the oracle most similar to the work of Goldberg and Nivre (2012) provides a small improvement