Complex question generation using discourse-based data augmentation (2024)

Abstract: Question Generation (QG), the process of generating meaningful questions from a given context, has proven to be useful for several tasks such as question answering or FAQ generation. While most existing QG techniques generate simple, fact-based questions, this research aims to generate questions that can have complex answers (e.g. “why” questions). We propose a data augmentation method that uses discourse relations to create such questions, and experiment on existing English data. Our approach generates questions based solely on the context without answer supervision, in order to enhance question diversity and complexity. We use an encoder-decoder trained on the augmented dataset to generate either one question or multiple questions at a time, and show that the latter improves over the baseline model when doing a human quality evaluation, without degrading performance according to standard automated metrics.

## introduction question generation is the task of automatically producing varied questions about a document or a set of documents. it is used to facilitate matching real users' questions looking for information contained in those documents, for instance in the context of customer relationship management or producing faqs @xcite , in dialogue systems to improve interaction with users @xcite , to develop interactive learning for educational purposes @xcite @xcite or as auxiliary tasks for e.g. summarization @xcite . more generically, it can help question-answering (qa) systems by augmenting the amount of instances available for training, as in @xcite where automatically generated questions are integrated within a text-based qa system, or in @xcite where they are used as adversarial data to improve robustness. as pointed out in e.g. @xcite , question diversity is crucial, meaning that a qg system should be able to produce different types of questions, with varied lexical content and associated explicit and implicit answers. however, the majority of the current research techniques in qg have primarily focused on factoid and multiple-choice questions, where the systems are designed to retrieve factual information or require short-span answers. since they rely more on reasoning, complex questions might help the user to gain deeper and multiple perspectives on a topic. this makes them especially useful in learning environments, complex dialogue systems, and applications that call for a better understanding of text. on the other hand, generating complex questions is a challenging task, as the system must have a grasp of underlying semantic relationships between different parts of the text. this is where discourse relations can play an important role: discourse, or rhetorical, relations are the semantic-pragmatic links between sentences or clauses within a text, describing e.g. causal, temporal or manner connections. we assume that including discourse relations into the generation process could help the system to produce complicated questions that accurately represent the depth and complexity of the text while also being contextually relevant. for instance, recognizing a "cause-effect" discourse relation can inspire "why" questions that aim to go deeper into the reasons behind a certain occurrence or circumstance addressed in the text. in this paper, we present an answer-agnostic qg system, based on a transformer-driven model finetuned specifically for question generation. the emphasis of our qg system is on generating complex questions using discourse relations, with a particular focus on causality related questions to enhance contextual understanding. our approach relies on data augmentation: the system is fine-tuned on reference datasets for qa that are reversed to perform the qg task, and augmented with "why" questions that are automatically built from discourse annotated data using simple heuristics. by using gold annotated data for discourse, we ensure the quality of our synthetic data. we use several datasets, the stanford question answering dataset or squad @xcite and explain like i'm five, or eli5 @xcite for training a generator, and the penn discourse treebank 2.0 @xcite for data augmentation. we evaluate the results using both automatic evaluation metrics comparing generated questions to existing reference questions about the same paragraphs, and a human assessment of the quality of the generated questions, since automated metrics do not account well for the variety of outputs from answer-agnostic models. ## related work question generation aims at producing relevant questions from documents, that could be a single text or a collection, or other types of inputs such as knowledge bases or images. in this paper, we focus on generating questions from a single document, using datasets in which each source text (i.e. context) is associated to question-answer pairs. in this context, many annotated datasets, primilarly built for qa, have been used for qg with two different settings: answer-aware systems provide the context and the targeted answer to generate the question, while answer-agnostic ones only rely on contexts. first systems for qg were rule-based: @xcite proposed to apply syntactic modifications to generate question from declarative sentences, while dhole and manning (2020) refined generating patterns using semantic resources. interestingly, @xcite demonstrated the importance of discourse connections for qg by designing patterns also relying on discourse connectives, i.e. specific expressions that can trigger discourse relations (e.g. because, but, as a result...), and that also constrain the type of the question to be generated. we also rely on syntactic templates and discourse information, but we significantly extend this line of work by using gold discourse annotations and by also including implicit relations. current approaches rely on neural architectures, either rnns @xcite or transformers @xcite @xcite . as in our work, @xcite ; lopez et al. ( 2020 ) proposed an answer-agnostic qg system based on a transformer architecture but only evaluated on squad, where complex questions are almost nonexistent. within the same setting, @xcite demonstrated the ability of a t5-model to generate relevant and natural questions, but the authors highlighted the challenge of evaluating generated questions using squad: while the answer-agnostic setting encourages diversity, the generated questions could be far from the reference ones, an issue we address through human evaluation (see section 8). while these studies successfully applied transformer models such as t5 to qg, they primarily focuses on generic, simple questions, leaving complex questions less explored. beside @xcite , discourse information was also leveraged in @xcite where rules are used to extract cause-effect relations in squad: a language model then generates questions on both the cause and effect aspects, and the synthetic questions are evaluated via a qa task. contrary to this work, we use causal relations that are manually annotated to create synthetic data to augment a generic qg model. in addition, relevant to our work is the approach introduced in @xcite : the authors propose simple transformations based on syntactic templates to create a corpus of "why" questions. our heuristics to generate questions are inspired by this work, but our evaluation is not done directly on these synthetic, possibly noisy questions, but on a natural, classic benchmark (e.g. squad). also using data augmentation, ashok kumar et al. (2023) rely on prompting an llm using context-answer-question triplets to generate a set of new questions, using varied decoding strategies with the aim of increasing diversity. these questions are then ranked, based on perplexity or on a separate model, and the best ranked is added to the training set of a flan-t5 model fine-tuned on fairytaleqa @xcite to generate questions given context-answer pairs. the evaluation demonstrates that the approach allows to generate questions for which the answer is implicit, i.e. no directly present as text span but need to be inferred. our approach is much simpler, relying on heuristics to generate questions, with a focus on difficult, complex questions while their approach aims at producing generic diversity, with no insight on the types of questions generated. ## methodology the pipeline for our question generation task is illustrated in figure 1 and consists of the following elements: pcmq makes for a more complex decoding, but is supposed to encourage question diversity and avoid redundant generations. this setup is made possible because the reference answer for each question is ignored, and so a given paragraph is associated to several different questions in squad. given the scarcity of complex questions in existing datasets, we aim to expand our training examples by integrating more "why" based questions. we thus use the pdtb2 dataset which contains documents annotated with discourse relations, including causality relations. these can be signaled by discourse markers, such as "because", "as", and "since", or be implicit, and the annotation consists of a typical marker that could be inserted. we take the sentences from the pdtb2 dataset for both implicit and explicit relations that represent causal relations and produce questions based on some predefined rule-based templates. the rules operate on the syntactic structure of a sentence to identify the main verb and auxiliary, and transform it to produce a grammatically correct interrogative sentence, in a manner similar to how data was produced in the dataset of @xcite . appendix a. ## datasets there are numerous datasets available for question generation tasks, including but not limited to newsqa @xcite , ms marco @xcite , natural questions @xcite , fairytaleqa @xcite , squad @xcite , and eli5 @xcite . initially, these datasets were designed for question-answering tasks, yet they are now also broadly used in question generation research. for the present work we rely on two datasets, namely squad and eli5, to perform question generation from a given text. squad is chosen for its diverse range of source paragraphs and questions from wikipedia, it is commonly used as a reliable benchmark for both qa and qg. the dataset was produced by stanford university academics and contains a sizable number of paragraphs that were taken from wikipedia articles @xcite . for our experiment, we use the training and development datasets from squad v2.0, which were created by rajpurkar et al. @xcite in 2018. 1 however, squad focuses mostly on simple factoid questions, so the eli5 dataset, consisting of more complex questions, is incorporated. eli5 which stands for "explain like i'm five", is another popular benchmark dataset used for tasks like qa, qg, and other nlp tasks. it is sourced from the subreddit r/explainlikeimfive. it provides long-form answers and is available from the hug-ging face website. 2 in the eli5 dataset, each instance consists of a question and user-provided answers on reddit. in our context, we consider the answer as the source paragraph and the questions as our system's input. pdtb2.0: additionaly, we use the penn discourse treebank version 2.0 (pdtb2) @xcite that provides discourse annotated texts. the pdtb2 is used here to leverage discourse marker-based annotations and produce additional data to augment the training set. other corpora exist for discourse annotations, but the pdtb is the largest annotated dataset for english including annotations for discourse relations (e.g. cause, result, manner), both explicit -that is triggered by a discourse connective (e.g. because, as a result, then...) -, and implicit -no lexical marker. of particular interest, the pdtb2 has annotations of causal relations that we use to create "why" questions. we use the version provided from the conll 2016 shared task @xcite , with level-2 annotations (15 different relation types). ## experiments our experiments aim at evaluating the influence of the training data composition, the model size, and the generating procedure as outlined in section 3. pcsq vs pcmq: we build the training set differently for the pcsq and pcmq setups: for pcsq -i.e. one question per paragraph -, we select paragraphs and all questions about them to generate one instance per paragraph-question pair ; for pcmq, input: many locals and tourists frequent the southern california coast for its popular beaches, and the desert city of palm springs is popular for its resort feel and nearby open spaces. reference question: other than the desert city why do many locals and tourists frequent southern california? baseline: how many locals and tourists frequent the southern california coast? squad+eli5: what city has a beach? +eli5+pdtb (exp): why do many locals and tourists frequent the southern california coast? +eli5+pdtb (exp+imp): why do many locals and tourists frequent the southern california coast? training data composition: for the training data, we use squad data as a baseline, and vary the training set by adding either (i) eli5 data only, or (ii) eli5 and the generated questions from the explicit examples of the pdtb, or (iii) eli5 and the generated questions from both the implicit and explicit examples of the pdtb. for the baseline dataset (squad) we keep approximately 50k instances for the pcsq setup, and compare to similarly-sized datasets, by having 20k instances from squad and 30k instances from eli5. the additional augmentation from the pdtb is much smaller, with about 1, 600 instances generated from explicit relations, and 1, 550 from implicit relations. for the pcmq setup we cannot hold the number of instances constant without restraining squad too much (there are only 19k paragraphs in total), so we chose to start from a baseline including all of squad + 30k eli5 instances (note that there are much less questions per paragraph in eli5). we kept the squad-only setup for comprehensiveness, but the pcmq setup is not entirely fair to this dataset compared to the others. while the training and development sets of squad are publicly available, the test set is not accessible to the public. so we divided the development set evenly, allocating 50% for validation and the remaining 50% for testing. ## automated evaluation we generated questions in both approaches on the squad left-out paragraphs and evaluated against the corresponding reference questions using automated evaluation metrics: bleu (bilingual evaluation understudy) @xcite , me-teor (metric for evaluation of translation with explicit ordering) @xcite , and rouge-l (recall-oriented understudy for gisting evaluation-longest common subsequences or lcs) @xcite . these metrics are widely adopted in the literature for evaluating question generation. the evaluation tasks involved the use of the following library packages: the natural language toolkit (nltk), rouge, and meteor in calculating bleu (1 to 4), rouge-l, and me-teor scores. note that those measures, relying on common ngrams or subsquences between reference and system outputs, are moderately appropriate to our setup, where we try to generate more diverse questions than are present in the reference, without a target answer. we address this problem with a human evaluation in section 8. a total of 500 paragraphs were chosen from the squad test dataset to assess the questiongeneration capability of our model. these paragraphs consist of multiple reference questions, and correspondingly, our model generates multiple questions for each paragraph. to accommodate the presence of multiple references and generated questions per paragraph in the squad dataset, we implemented a mapping approach to find out which reference and generated question pairs are more relevant to each other for the evaluation, especially focusing on one-to-one match between reference and generated questions. for both pcsq and pcmq approaches, we combined questions generated for each context's four outputs from the beam. using automatic evaluation metrics such as bleu, rouge-l, or meteor, we then calculated scores for each pair of matched generated and reference questions. this filtering resulted in a oneto-one matching between generated and reference questions, ensuring meaningful evaluation of our model's question-generation accuracy. given the decoding procedure, the average number of nonduplicate generated questions was about 3.9 for pcsq, and 9.5 for pcqm (with small variations depending on the training data). ## results the results presented in when train with pdtb derived instances, the number of "why" question is higher (+38% when using explicit and implicit with pcmq wrt the baseline, +24% with only explicit). in pcsq, the increase in "why" questions is limited (going from 0 for the baseline to 10 for the full training data), reflecting its lower effectiveness in generating this question type. questions in "how" do not seem positively affected (each system generates almost the same amount), but we did not distinguish simple "how" questions (asking for quantities, i.e "how much/many") and more complex ones. we just observed that some generated "how" questions were causal in nature, but more manual analysis is needed to evaluate this precisely. thus, aligned with our objective, our augmentation techniques effectively increased the number of generated "why" questions, particularly within the pcmq models, without detrimentally affecting the quality of the questions generated as a whole, at least according to the automated metrics. this is notable since our models are not trained on example answers, meaning they can generate questions about any aspect of the chosen paragraph, for which it is likely the reference does not include any question-answer pair. this is why it is important to have a separate, more fine-grained evaluation of the quality of the generated answers, and this is the subject of the following section. we conducted a human evaluation to assess the quality of questions generated in pcmq approach by three models: baseline model, +eli5+pdtb (exp), and +eli5+pdtb (exp+imp), all finetuned from the t5-base model. two of the authors annotated a subset of randomly selected questions and their context from the squad test dataset using a set of 7 predetermined categories that included subcategories for incorrect questions (more details are provided in appendix 12); the selection was done by a third author, who kept hidden the system that produced each question. there were 137 annotated questions, some generated by more than one system. adjudications of annotations were done by the two annotators. it turned out some of the error subcategories were quite similar, and the final categories were restricted to three cases: (1) the generated question is good: fluent, and can be answered from the source paragraph, (2) the generated question is almost good: minor disfluency and the answer is in the paragraph, (3) the question is either impossible to understand or too vague, or the paragraph does not contain an answer to the question. cohen's kappa (κ) was 0.48 on the 7 original categories, indicating a moderate level of interannotator agreement, but was 0.74 when only distinguishing between good questions and all the rest. moreover, from the annotated questions, we determined the distribution of good, almost okay, and bad questions for each question type (e.g., what, why, etc.), see ## conclusion we presented an approach based on discourse relation annotations to augment a question generation training set, in the case of a general answeragnostic question generation system, and with a focus on causal questions. our experiments show that with a small set of additional instances we can make the system generate more causal questions with a good quality, as evaluated by human annotators, and with almost no difference with respect to classic automated metrics for question generation. this is only preliminary, as the results would need to be tested on different base question-answer corpora, and more human evaluation would be precious to better separate the roles of the different factors at play here. it would also be interesting to investigate the impact of including other discourse relation types to generate different kinds of questions (e.g. "how" questions with relations of the type "goal" or "manner").