Expanding, Retrieving and Infilling: Diversifying Cross-Domain Question Generation with Flexible Templates (2021)

Abstract: Sequence-to-sequence based models have recently shown promising results in generating high-quality questions. However, these models are also known to have main drawbacks such as lack of diversity and bad sentence structures. In this paper, we focus on question generation over SQL database and propose a novel framework by expanding, retrieving, and infilling that first incorporates flexible templates with a neural-based model to generate diverse expressions of questions with sentence structure guidance. Furthermore, a new activation/deactivation mechanism is proposed for template-based sequence-to-sequence generation, which learns to discriminate template patterns and content patterns, thus further improves generation quality. We conduct experiments on two large-scale cross-domain datasets. The experiments show that the superiority of our question generation method in producing more diverse questions while maintaining high quality and consistency under both automatic evaluation and human evaluation.

## introduction with a growing demand for natural language interfaces for databases, automatic question generation from structured query language(sql) query has been of special interest @xcite . recently, diversity-aware question generation has shown its effectiveness in improving down-stream applications such as semantic parsing and question answering tasks @xcite . although neural sequence-to-sequence based generation has been a dominant approach and is able to produce a meaningful description for sql queries, existing methods still suffer from the lack of diversity as well as bad sentence structures @xcite . in the neural-based approaches, conventional ways of generating diverse sentences focus on approximate decoding techniques such as beam search @xcite and temperature sweep @xcite . those decoding strategies generate diverse samples while sacrificing the quality of sentences. variational auto-encoders (vaes) have been used to generate various sentences by applying additional information as latent variables @xcite @xcite @xcite . however, implicit latent representation provides limited controllability over sentence structure and can be difficult to adapt to a new domain. paraphrase @xcite and syntactic-based methods @xcite have also been studied. however, learning a paraphrasing model relies on a large number of domain-specific paraphrase pairs, which is difficult to obtain for target databases. besides, syntacticbased approaches apply syntactic parsers or semantic rules to the natural language utterance, thus are not applicable to sql-to-question generation. in the rule-based generation systems, the templates work as essential prior knowledge that contains the structural information of sentences @xcite @xcite . this ensures the generation contains fewer grammatical errors and performs better with extractive metrics @xcite . however, their template formats are mostly strict and the valid content for each chunk should be pre-defined, which makes large set of templates difficult to obtain. in this paper, we propose a novel method that incorporates template-based generation with a neural sequence-to-sequence model for diversity-aware question generation. instead of applying strict templates, we use flexible templates that can be collected efficiently with less expense. these flexible templates provide high-level guidance of sentence structure while also enable sufficient flexibility for a neural-based model to fill chunks with content de-tails. we present our method as a three-stage framework including expanding, retrieving, and infilling. in the expanding stage, we take advantage of existing large-scale cross-domain text-to-sql datasets to extract and collect the flexible template set automatically. in the retrieving stage, given a sql query, the best templates are retrieved from the collected template set by measuring the semantic distance of sql and templates in a joint template-sql semantic space. in the filling stage, we treat each template as a masked sequence and explicitly force the generator to learn question generation with the constraint of the template. in order to help the generator to discriminate template patterns and content patterns, a unique activate/deactivation mechanism is designed for the generator to learn when to switch between template-copying state and content-filling state. we conduct experiments on two large-scale cross-domain text-to-sql datasets. compared to existing approaches, our method achieves the best diversity result for both datasets with both automatic evaluation and human evaluation, while maintaining competitive quality and high consistency with sql queries. we further demonstrate that the designed modules each contribute to a performance gain through an ablation study. ## problem formulation given a sql query as the input sequence, question generation over database aims to generate a natural language question as an output sequence that accurately reflects the same meaning as the given sql query. in this work, we generate the question by introducing an intermediate template in the generation process. therefore, by applying proper templates sql query question query 1 query 2 query 3 query 4 ... question 1 question 2 question 3 question 4 ... q1 q2 q3 q4 t1 t2 t3 t4 ... t1 t2 t3 t4 ... t1 t2 t3 t4 ... t1 t2 t3 t4 ... sql query i gru encoder gru encoder joint semantic space query embedding template embedding soft classifier save sql query query embedding template embedding template 1 template 2 ... ... ... ... min(d(e sql , e t )) e t template i fc μenc h t h x gru gru gru gru soft attention generation vector copy vector fc @xmath0.11 (59) <@xmath1: question: e sql template dataset expansion template retrieval text infilling text-to-sql training corpus training phase stage 1: dataset expansion training corpus template candidates lcs t len sql query question template query 1 query 2 query 3 query 4 ... question 1 question 2 question 3 question 4 ... template 1 template 2 template 3 template 4 ... our training corpus stage 2: template retrieval weight sharing dictionary inference phase gru encoder template set hard filter retrieve question example: template set sql query diverse questions sql query hidden states henc h i σ enc z stage 3: text infilling template t gru gru gru sql query x framework overview template t figure 1: architecture of our framework. different templates, we can generate diverse expressions of questions. let @xmath2, x 2 , ..., x |x| ] denote the given sql query, @xmath3, y 2 , ..., y |y| ] denote the gold-standard question, and @xmath4, t 2 , ..., t |t| ] denote the corresponding template. given a neuralbased system with a set of learnable parameters θ * t and θ * q , the two-stage objective in this work can be formulated as follows: θ * @xmath5, t) ## methodology in this section, we introduce our framework, which learns question generation from sql queries with the guidance of various templates, so as to increase the diversity of generation. ## experiment dataset to validate our framework's capability in generating diverse and controllable questions, we conduct experiments on two large-scale crossdomain text-to-sql datasets: wikisql @xcite and spider @xcite . wikisql contains 80654 query-question pairs derived from 24241 different schemas, with both validation set and test set released. spider contains 10181 queryquestion pairs in 138 domains, with the validation set published. we follow the provided split settings for training and testing. setup we first construct the template set from the training set using our expansion method. for each sql query in the test set, we retrieve the k -1 most relevant templates from the template set to generate k -1 different questions. we also include one question with @xmath6. thus k questions for each sql query are provided for evaluation. for our experiments, we set @xmath7. baselines we compare our model(eri)'s performance to models based on the baseline ap-proach qg @xcite with different diversesentence generation strategies. (1) latent variable(qglv): guo et al. ( 2018 ) applies a sequenceto-sequance network with a copy mechanism for question generation from sql. a latent variable is introduced to generate diverse questions. (2) temperature sweep(temps): we apply temperature sweep (ts) @xcite for decoding in qg. (3) beam search(beams): we further combine qg with beam search @xcite to generate diverse questions. in practice, we set the beam width to 5 and obtain sentences with top-5 highest probabilities for comparison. a boltzmann temperature parameter α is applied to modulate the entropy of the generator. in practice, we set α = 0.7 as suggested and obtain 5 generated sentences for each query. evaluation metrics we adopt the following automatic metrics to measure the quality of generated questions. (1) maxbleu: the max bleu-4 score among 5 generated questions. (2) coverage: @xcite this metrics measures the average proportion of input query covered by the generated questions. (3) parseacc: we use neural semantic parsers sqlova @xcite and global-gnn @xcite to parse wikisql and spider respectively. the semantic parsers translate the generated question into sql query and calculate the exact-match accuracy with the input sql query as the ground-truth. a higher accuracy score means the generated questions are more natural and consistent with the given sql queries. we adopt the following automatic metrics to measure the diversity of generated questions. (1) self-bleu: @xcite the average bleu-4 of all pairs among the k questions). (2) self-wer: @xcite the average word error rate of all pairs among the k questions. a lower self-bleu score and a higher self-wer score indicate more diversity in the result. (3) distinct-4: @xcite it measures the ratio of distinct n-grams in generated questions. ## conclusion in this paper, we present a novel framework for question generation over sql database to produce more diversified questions by manipulating the templates. we expand the template set from crossdomain sql-to-text datasets, and retrieve proper templates from a template set by measuring the distance between the templates and the sql query in a joint semantic space. we propose an activation/deactivation mechanism to make full use of templates to guide the question generation process. experimental results have shown that the presented model can generate various questions while maintains their high quality. the model has also improved the matching between the templates and the content information of sql queries.