Syn- QG : Syntactic and Shallow Semantic Rules for Question Generation (2020)

Abstract: Question Generation (QG) is fundamentally a simple syntactic transformation; however, many aspects of semantics influence what questions are good to form. We implement this observation by developing Syn-QG, a set of transparent syntactic rules leveraging universal dependencies, shallow semantic parsing, lexical resources, and custom rules which transform declarative sentences into question-answer pairs. We utilize PropBank argument descriptions and VerbNet state predicates to incorporate shallow semantic content, which helps generate questions of a descriptive nature and produce inferential and semantically richer questions than existing systems. In order to improve syntactic fluency and eliminate grammatically incorrect questions, we employ back-translation over the output of these syntactic rules. A set of crowd-sourced evaluations shows that our system can generate a larger number of highly grammatical and relevant questions than previous QG systems and that back-translation drastically improves grammaticality at a slight cost of generating irrelevant questions.

## introduction automatic question generation (qg) is the task of generating question-answer pairs from a declarative sentence. it has direct use in education and generating engagement, where a system automatically generates questions about passages that someone has read. a more recent secondary use is for automatic generation of questions as a data augmentation approach for training question answering (qa) systems. qg was initially approached by syntactic rules for question-generation, followed by some form of statistical ranking of goodness, e.g., @xcite . in recent years, as in most areas of nlp, the dominant approach has been neural network generation @xcite , figure 1 : the srl structure is leveraged to invoke a template, and a simple rearrangement of the modifying arguments is performed. in particular using a sequence-to-sequence architecture, which exploits the data in the rapidly growing number of large qa data sets. previous rule-based approaches suffer from a significant lack of variety in the questions they generate, sticking to a few simple and reliable syntactic transformation patterns. neural architectures provide a pathway to solving this limitation since they can exploit qa datasets to learn the broad array of human question types, providing the usual neural network advantages of a data-exploiting, end-toend trainable architecture. nevertheless, we observe that the quality of current neural qg systems is still lacking: the generated questions lack syntactic fluency, and the models lack transparency and an easy way to improve them. we argue that in essence qg can be governed by simple syntactic "question transformations"while the implementation details vary, this is in accord with all major linguistic viewpoints, such as construction grammar and chomskyan generative grammar, which emphasize grammatical rules and the existence of finite ways to create novel utterances. however, successful, fluent question generation requires more than just understanding syntactic question transformations, since felicitous questions must also observe various semantic and ## related work with the advent of large-scale qa datasets @xcite , recent work in qg @xcite has primarily focused on training sequence-tosequence and attention-based architectures. @xcite fine-tuned the question generation task by taking advantage of a large pre-trained language model. success in reinforcement learning has inspired teacher-student frameworks @xcite treating qa and qg as complementary tasks and performing joint training by using results from qa as rewards for the qg task. @xcite ; @xcite ; @xcite used evaluation metrics like bleu, sentence perplexity, and qa probability as rewards for dealing with exposure bias. traditionally, question generation has been tackled by numerous rule-based approaches @xcite @xcite @xcite . @xcite introduced an overgenerate-and-rank approach that generated multiple questions via rule-based tree transformations of the constituency parse of a declarative sentence and then ranked them using a logistic-regression ranker with manually designed features. @xcite described transformations of minimal recursion semantics representations guaranteeing grammaticality. other transformations have been in the past defined in terms of templates @xcite @xcite , or explicitly performed @xcite by searching tree patterns via tregex, followed by their manipulation using tsurgeon @xcite . @xcite provide a comprehensive summary of qg, analysing and comparing approaches before and after 2014. vis-à-vis current neural question generators, rule-based architectures are highly transparent, easily extensible, and generate well-formed questions since they perform clearly defined syntactic transformations like subject-auxiliary inversion and whmovement over parse structures whilst leveraging fundamental nlp annotations like named entities, co-reference, temporal entities, etc. however, most of the existing rule-based systems have lacked diversity, being mostly focused on generating what-type and boolean questions and have mainly exploited parse structures which are not semantically informed. @xcite ; @xcite use dependency, srl, and ner templates but do not handle modalities and negation in a robust manner. moreover, there is plenty of availability of core linguistic resources like verbnet and propbank, which provide ## evaluation and results most of the prior qg studies have evaluated the performance of the generated questions using automatic evaluation metrics used in the machine translation literature. we use the traditional bleu scores @xcite and compare the performance of syn-qg on the squad @xcite test split created by @xcite . bleu measures the average n-gram precision on a set of reference sentences. a question lexically and syntactically similar to a human question would score high on such n-gram metrics. despite not utilizing any training data, syn-qg performs better than the previous sota on two evaluation metrics bleu-3 and bleu-4 and close to sota on bleu-1 and bleu-2 (