On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries (2020)

Abstract: Large-scale semantic parsing datasets annotated with logical forms have enabled major advances in supervised approaches. But can richer supervision help even more? To explore the utility of fine-grained, lexical-level supervision, we introduce SQUALL, a dataset that enriches 11,276 WIKITABLEQUESTIONS English-language questions with manually created SQL equivalents plus alignments between SQL and question fragments. Our annotation enables new training possibilities for encoderdecoder models, including approaches from machine translation previously precluded by the absence of alignments. We propose and test two methods: (1) supervised attention; (2) adopting an auxiliary objective of disambiguating references in the input queries to table columns. In 5-fold cross validation, these strategies improve over strong baselines by 4.4% execution accuracy. Oracle experiments suggest that annotated alignments can support further accuracy gains of up to 23.9%.

## introduction the availability of large-scale datasets pairing natural utterances with logical forms @xcite @xcite has enabled significant progress on supervised approaches to semantic parsing @xcite @xcite . however, the provision of logical forms alone does not indicate important finegrained relationships between individual words or phrases and logical form tokens. this is unfortunate because researchers have in fact hypothesized that the lack of such alignment information hampers progress in semantic parsing @xcite . by providing expert annotations, consisting not only of target logical forms in sql, but also labeled alignments between the input question tokens (e.g., "how many") and their corresponding sql fragments (e.g., count(. . .)). figure 1 shows two squall instances. these new data enable training of encoderdecoder neural models that incorporates manual alignments. consider the bottom example in figure 1 : a decoder can benefit from knowing that order by . . . limit 1 comes from "the highest" (where rank 1 is best); and an encoder should match "who" with the "athlete" column even though the two strings have no overlapping tokens. we implement these ideas with two training strategies: using bert features, our models reach 54.1% execution accuracy on the wtq test set, surpassing the previous weakly-supervised state-of-the-art 48.8% (where weak supervision means access to only the answer, not the logical form of the question). more germane to the issue of alignment utility, in 5-fold cross validation, our additional fine-grained supervision improves execution accuracy by 4.4% over models supervised with only logical forms; ablation studies indicate that mappings between question tokens and columns help the most. additionally, we construct oracle models that have access to the full alignments during test time to show the unrealized potential for our data, seeing improvements of up to 23.9% absolute logical form accuracy. through annotation-cost and learning-curve analysis, we conclude that lexical alignments are cost-effective for training parsers: lexical alignments take less than half the time to annotate as a logical form does, and we can improve execution accuracy by 2.5 percentage points by aligning merely 5% of the logical forms in the training set. our contributions are threefold: 1) we release a high-quality semantic parsing dataset with manually-annotated logical forms; 2) we label the alignments between the english questions and the corresponding logical forms to provide additional supervision; 3) we propose two training strategies that use our alignments to improve strong base models. our dataset and code are publicly available at https://www.github.com/tzshi/squall . ## task: table-based semantic parsing our task is to answer questions about structured tables through semantic parsing to logical forms (lfs). formally, the input @xmath0, t ) consists of a question q about a table t , and the goal of a semantic parser is to reproduce the target lf y for q (and thus have high lf accuracy) or, in a less strict setting, to generate any query lf y that, when executed against t , yields the correct output z (and thus have high execution accuracy). in a weakly supervised setting, training examples consist only of input-answer pairs (x, z ). recent datasets @xcite , inter alia) provide enough logical forms, i.e., (x, y ) training pairs, to learn from mappings from x to y in a supervised setting. unsurprisingly, supervised models are more accurate than weakly supervised ones. however, training supervised models is still challenging: both x and y are structured, so models typically generate y in multiple steps, but the training data cannot reveal which parts of x generate which parts of y and how they are combined. just as adding supervised training improves accuracy over weak supervision, we explore whether even finer-grained supervision further helps. since no large-scale datasets furnishing fine-grained supervision exist (to the best of our knowledge), we introduce squall. ## squall: our new dataset squall is based on wikitablequestions (wtq; @xcite . wtq is a largescale question-answering dataset that contains diverse and challenging crowd-sourced questionanswer pairs over 2,108 semi-structured wikipedia tables. most of the questions are more than simple table-cell look-ups and are highly compositional, a fact that motivated us to study lexical mappings between questions and logical forms. we hand-generate sql equivalents of the wtq queries and align question tokens with corresponding sql query fragments. 2 we leave lexical alignments of other text-to-sql datasets and cross-dataset model generalization @xcite ) to future work. ## (state-of-the-art) base model: seq2seq with attention and copying recent state-of-the-art text-to-sql models extend the sequence-to-sequence (seq2seq) framework with attention and copying mechanisms @xcite @xcite . we adopt this strong neural paradigm as our base model. the seq2seq model generates one output token at a time via a probability distribution conditioned on both the input sequence representations and the partially-generated output sequence: the training objective is the negative log likelihood of the gold y , defined for each timestep as ## using alignments in model training the model design in §4 includes many latent interactions within and across the encoder and the decoder. we now describe how our manual alignments can enable direct supervision on such previously latent interactions. our alignments can be used as supervision for the necessary attention weights ( §5.1). in an oracle experiment where we replace induced attention with manual alignments, the jump in logical form accuracy shows alignments are valuable, if only the models could reproduce them ( §5.2). moreover, alignments enable a column-prediction auxiliary task ( §5.3). the loss function l of our full model is a linear combination of the loss terms of the seq2seq model, supervised attention, and column prediction: attention type acclf (dev) ∆ induced attention 37.8 ± 0.6 oracle attention encoder only 51.5 ± 1.4 +13.7 decoder only 49.4 ± 0.9 +11.6 encoder + decoder 61.7 ± 0.4 +23.9 ## conclusion we introduce squall, the first large-scale semantic parsing dataset with both hand-produced target logical forms and manually-derived lexical alignments between questions and sql queries. our dataset enables finer-grained supervision than existing datasets have previously supported. we incorporate the alignments into encoder-decoder-based neural models through supervised attention and an auxiliary task of column prediction. experiments confirm our intuition that finer-grained supervision is helpful to model training. our oracle studies also show that there is large unrealized further potential for our annotations. thus, it remains an exciting challenge for future research to use our lexical alignment annotations more effectively. our annotation cost analysis shows that collecting additional lexical alignments is more costeffective for improving model accuracy than having only logical forms. we hope that our findings will help future dataset design decisions and extensions of other existing datasets. one potential future direction is to further investigate the utility of lexical alignments in a cross-dataset/domain evaluation setting @xcite .