Automatically Generating Cause-and-Effect Questions from Passages (2021)

Abstract: Automated question generation has the potential to greatly aid in education applications, such as online study aids to check understanding of readings. The state-of-the-art in neural question generation has advanced greatly, due in part to the availability of large datasets of question-answer pairs. However, the questions generated are often surface-level and not challenging for a human to answer. To develop more challenging questions, we propose the novel task of cause-and-effect question generation. We build a pipeline that extracts causal relations from passages of input text, and feeds these as input to a state-of-the-art neural question generator. The extractor is based on prior work that classifies causal relations by linguistic category (Cao et al., 2016; Altenberg, 1984). This work results in a new, publicly available collection of cause-and-effect questions. We evaluate via both automatic and manual metrics and find performance improves for both question generation and question answering when we utilize a small auxiliary data source of cause-and-effect questions for fine-tuning. Our approach can be easily applied to generate cause-and-effect questions from other text collections and educational material, allowing for adaptable large-scale generation of cause-and-effect questions.

## introduction automated question generation (qg) can have a large educational impact since questions can be generated to check learners' comprehension and understanding of textbooks or other reading materials @xcite . a high-quality qg system could reduce the costly human effort required to generate questions as well as free up teachers' time to focus on other instructional activities @xcite . furthermore, a robust qg system could also expand the variety of ## related work in this section, we discuss past work in causal extraction, question answering, question generation, and applications of generated questions. ## cause-and-effect question generation pipeline we propose a novel pipeline which combines a causal extractor with a neural question generation system to produce cause-and-effect questions. the entire pipeline can be seen in figure 1 . ## evaluating extracted causal relations in order to ensure the extracted causes and effects are suitable to generate questions from, we evaluated the original @xcite and our improved causal extractor for accuracy via a crowdworking task. workers evaluated if an extracted cause and effect were causal or not, judged from a passage with the extracted cause and effect highlighted (see appendix c). we utilized amazon mechanical turk to recruit crowdworkers for data labeling. we required crowdworkers to have a 98% hit approval rating, have completed at least 5,000 past hits, and be located in the united states. we estimated the task would require no more than 10 minutes to complete; therefore, we paid @xmath010 we sampled 100 passages from tqa and 100 from squad for both the original causal extraction system and our improved system. we used proportionate stratified sampling where the size of sample typology group is proportional to the size of population typology group, while also ensuring that all typology groups are represented in the sample. after tiebreaking using majority vote, for the original @xcite causal extraction system we find an overall 70% of tqa and 68% of squad are rated as causal. in comparison, for our improved system, we find an overall 83% of tqa and 79% of squad are rated as causal. results segmented by typology for our improved system can be seen in fleiss's kappa @xcite however, is only slight to fair for this evaluation (0.21 for base extractor and 0.10 for our improved extractor). this could be due to a high prior probability of an extracted cause and effect being causal (62% of questions have 4 or 5 annotators in agreement) @xcite . however, future work should additionally investigate alternatives to this task to achieve higher agreement. overall, the high accuracy of our improved extractor provides evidence that we can reliably extract a cause and effect to pass into a question generation system. ## qg and qa models this section describes the design of both the question generation (qg) and question answering (qa) models, which occur in the question generation and evaluation framework components of figure 1 , respectively. we describe them together because they share an auxiliary data source. the qg models are evaluated in sections 6 (automated) and 7 (manual). to generate cause-and-effect questions where the intended answers are the causes and effects from our improved extractor, we train and evaluate two question generation models based on prophet-net. we also make use of question answering models to assess the quality of the generated questions. we choose a transformer-based qa model from the huggingface transformers library @xcite which is bert-large fine-tuned on squad 2.0 with whole-word masking. foot_3 f1 performance for this model on the original squad 2.0 test set is 0.93. for both qa and qg, we utilize an additional data source for fine-tuning, to examine the effect of augmenting the models with additional cause-andeffect question data. we use a syntactic question generation system combined with backtranslation, syn-qg foot_4 @xcite , to generate questions and answers for fine-tuning. we limit syn-qg to two patterns, purpose (pnc and prp) and cause (cau), which generate cause-and-effect questions. we input three-sentence passages which our system has identified as including a cause and effect to syn-qg, resulting in 2,082 questions from tqa and 1,753 from squad. while the wording and syntactic structure of syn-qg questions lack diversity, we hypothesize that prophetnet will benefit from fine-tuning on a set of cause-and-effect questions as this question type was not well-represented in the squad training set. additionally, while syn-qg uses its own syntactic patterns to generate cause-and-effect questions directly from text, we observe their system is not able to cover all causal relationships included in @xcite . from randomly-sampled passages which our improved extractor has identified as including a cause and effect, syn-qg is able to generate cause-and-effect questions for 309 out of 500 passages for squad and 233 out of 500 passages for tqa. we compare two models for qg and qa experiments in sections 6 and 7: base (with no finetuning) and syn-qg fine-tuned to explore the effect of fine-tuning qg and qa models on a small auxiliary dataset of cause-and-effect questions. for all experiments, we split the syn-qg auxiliary dataset into 80% training and 20% test data. the qg models' resulting questions are fed as input into the qa models (see figure 1 ). additional model specifications are in appendix d. ## automated evaluation of generated questions as part of our evaluation framework, we develop two automated metrics to evaluate the generated cause-and-effect questions: (i) cause/effect presence, which measures whether the cause or effect is present in the question, and (ii) qa system performance, which measures whether a question answering system can answer the generated question. ## human evaluation of generated questions while automated metrics are scalable, they may not be as reliable as human evaluation. thus, we conducted a crowdworker task to evaluate generated questions. to ensure we provide the model with high-quality causal relations, we evaluated questions generated from the extracted relations that were labeled as causal in section 4 (83 from tqa and 79 for squad). we performed human evaluation for base prophetnet, to establish a baseline human-rating of generated questions. crowdworkers were presented with a passage (with the intended answer highlighted) and a generated question (see appendix c). workers were asked to label (1) if the question is a causal question and ( 2 ) whether the answer highlighted in the passage correctly answers the question. if a question was too ill-formed or ungrammatical to provide these classifications, crowdworkers were asked to select a checkbox and provide a text justification. the conditions for this task were the same as in section 4. results can be seen in for the task of determining whether an intended answer is correct, when the effect is the intended answer, the proportion that are rated as correct is higher than for cause. this is consistent with the higher performance for effect questions on the cause/effect presence metric. results stratified by typology can be seen in overall, human ratings indicate that 80%-90% of the time, the generated question is both causal and correct for the intended answer. ## future work our proposed pipeline is generalizable to a wide range of corpora, allowing for the generation of cause-and-effect questions from other educational texts at-scale. human evaluation ratings indicate our model can reliably generate questions which are both causal and correct for the intended answer. however, more work needs to be done to generate questions which sound more natural and are less reliant on the wording of the input passage. the types of grammar errors made should also be examined before assessing the educational benefits of the generated questions. additionally, our goal in generating cause-andeffect questions was to generate questions which fall higher on bloom's taxonomy. while our results indicate cause-and-effect questions are more challenging than straightforward factual questions for a qa model, further work should assess the difficulty of these questions in an educational setting. the complexity of causal relationships can be further explored. we observe instances of overlapping causal relationships extracted from our system, such as the subordination example in future work can also examine diversifying question wording. by generating multiple questions testing the same causal relationship, students can have multiple opportunities to solidify their knowledge. our pipeline allows for straightforward experimentation with the question generation model. ## conclusion we propose a new task, cause-and-effect question generation, along with a novel pipeline to utilize extracted causes and effects as intended answers for a question generation system. we provide automatic and manual evaluation metrics and show that fine-tuning qg and qa models on an auxiliary dataset of cause-and-effect questions improves performance. our publicly-released pipeline can automatically generate cause-and-effect questions for educational resources at scale.