Temporal Question Generation from History Text (2021)

Abstract: Temporal analysis of history text has always held special significance to students, historians and the Social Sciences community in general. We observe from experimental data that existing deep learning (DL) models of ProphetNet and UniLM for question generation (QG) task do not perform satisfactorily when used directly for temporal QG from history text. We propose linguistically motivated templates for generating temporal questions that probe different aspects of history text and show that finetuning the DL models using the temporal questions significantly improves their performance on temporal QG task. Using automated metrics as well as human expert evaluation, we show that performance of the DL models finetuned with the template-based questions is better than finetuning done with temporal questions from SQuAD.

## introduction major events in history have always held significance for the social sciences community. understanding the history of a nation, a society, an era or historic personalities involves analysing the timelines of major events that happened, their locations, the actors, and the consequences that followed. given a set of history documents (wikipedia pages, books, papers), it is a challenging problem to automatically extract timelines from them and to use these timelines for downstream applications such as q&a @xcite @xcite @xcite . another important application is to generate temporal questions from historical narrative text, which can be used for testing and improving the students' understanding of the temporal aspects of history. while much research has focused on generation of general questions from text, generation of temporal questions has received less attention @xcite @xcite . we experimented with two deep learning (dl) based language models, unilm @xcite and prophetnet @xcite finetuned on squad @xcite for qg from history documents. the percentage of temporal questions generated and the acceptability of the questions was quite low (details in section 4). to improve the quality and quantity of temporal questions generated, we propose linguistic knowledge based methods (templates). each template analyzes the given sentence to generate a temporal question having a specific structure, by looking at the relationships among nominal and verbal events, time expressions (timex), verbs and its arguments in the dependency parse tree. our manual evaluations show that the templates generate temporal questions with high acceptability. then we leveraged the generated temporal questions to finetune prophetnet and unilm for the temporal question generation task. we evaluated the performance of the finetuned models using both domain-expert evaluation and automated metrics of bleu-4, me-teor, and rouge-l. the results show that temporal questions created through our templates significantly improve the performance of dl models on the task of temporal qg. ## related work compared to general purpose question generation (qg) and question answering (qa), the temporal aspect of qg has been relatively less explored in the literature. tequila @xcite is a system for temporal qa over knowledge bases (kb). it identifies temporal questions, converts them into non-temporal sub-questions and temporal constraints and then uses the underlying kb-qa engine to extract the answers to the sub-questions. @xcite generate and rank answers to complex questions by creating event graphs from text using dependency parser mainly focusing on temporal and causal relations. both of these methods focus on temporal qa whereas our work is focused on temporal qg. recent trends in deep learning (dl) based methods for qg are mainly driven by neural sequenceto-sequence modeling @xcite . however, acceptability of temporal questions generated using these methods is low. compared to other literature, the work by @xcite is closer to the scope of this paper. they have used @xmath0, predicate, @xmath1, a structured knowledge-base and a rulebased method to generate temporal questions. however, our approach uses raw input text and does not need any external kb for generating temporal questions. further, we use the template-based questions to finetune and improve the dl methods for temporal qg. extraction of timex and events: as shown in figure 1 , the processing pipeline of stage 1 starts with identification of tokens that indicate temporal expressions (i.e., timex). we use hei-deltime @xcite to identify and normalize timex entities in the input sentence. to capture the verbal events in the input text, we make use of the past-tense propagation technique proposed by @xcite . to identify the algorithm 1: algorithm for template: input: @xmath2; @xmath3; t @xmath4; n @xmath5; output: temporal question of the form: ''when did sub v n?'' let w be a verb in past tense in s which is not modal or auxiliary verb; if there is no such w then return; let v be the present tense form of w; let sub be the complete text connected to w using dr "nsubj" in pt; if there is no such sub or sub contains a pronoun then return; let p1 be a preposition in s such that p1 is connected to w using dr "prep" in pt and t is a timex in t es connected to p1 using dr "pobj" in pt; if there is no such t then return; if there is n in n es such that n is connected to a preposition p2 using dr "pobj" in pt and p2 is connected to w using dr "prep" in pt then print ''when did sub v n?'' else if there is n in n es such that n is connected to w using dr "dobj" in pt then print ''when did sub v n?'' else if there is verb u in present tense in s connected to w using dr "xcomp" in pt and there is n in n es such that n is connected to u using dr "dobj" in pt then print ''when did sub u n?'' temporal question generation templates: the templates are designed to probe different aspects of history text such as spatio-temporal details of an event, key players involved in it, relative temporal order among events, consequences of an event etc. we note that the set of templates is open to further extension based on the interest of historians and analysts. the proposed approach is flexible such that the questions generated by the dl based methods can be adapted to the additional templates. sr template sentence question 1 when @xmath7, romania switched sides on august 23, 1944. when did the jassy-kishinev offensive happen? 2 when @xmath8 in june 1941, hitler ordered an invasion of the soviet union. when did hitler order an invasion of the soviet union? 3 what happened @xmath9. we generate the dependency parse tree of the sentence using spacy @xcite and apply the template patterns to generate temporal questions. we traverse through the partof-speech (pos) tags and dependency relations (dr) in the parse tree of the sentence to extract phrases and tokens required to fill the relevant parameters of the templates. for instance, for template #2, we need the subject (sub), verb (v), nominal event (n) with the constraint that a temporal expression (t) is appropriately associated. algorithm 1 gives the details of how we verify the constraints and extract the parameters of the template #2. as an example, consider the sentence: in june 1941, hitler ordered an invasion of the soviet union. figure 2 shows its dependency parse tree and the postags of tokens. from this sentence, the algorithm 1 extracts @xmath10, @xmath11, @xmath12, @xmath13, @xmath14. finally, after verifying the appropriate constraints, algorithm 1 generates the question when did hitler order an invasion of the soviet union?. fine-tuning dl models for temporal qg: the second stage of our approach overcomes the limitations of existing deep learning (dl) models for temporal qg. we use two different dl models to emphasize the flexibility and robustness of this approach. prophetnet uses future n-gram prediction and n-stream self attention mechanism to achieve state-of-the-art performance on many nlp tasks. unified pre-trained language model (unilm) employs a shared transformer network and specific self-attention masks. both prophetnet and unilm have shown superior performance on general purpose qg task for squad dataset @xcite . we observe that off-the-shelf qg models of prophetnet and unilm do not perform satisfactorily when used directly for temporal qg from history text ( ## experimental evaluation datasets: we evaluate the proposed approach for temporal qg using history text intended for diverse audience and focusing on different topics such as historical accounts of famous personalities (e.g. napoleon), important phenomenon (e.g., fascism), battles, wars, and global conflicts. we use 3 chapters from history text books as well as 186 wikipedia articles on historical topics. the pointers to the book chapters and wikipedia articles are provided as a part of supplementary data. we use two different datasets of temporal questions for the second stage of our approach i.e. finetuning experiments with prophetnet and unilm. first dataset consists of 868 question generated by our template based approach for training and 217 for validation of the dl model finetuning. the second dataset consists of temporal questions (i.e., questions with explicit date-time expressions as well as implicit expressions such as before, after, during etc.) extracted from squad dataset. the resulting subset of squad dataset consists of 35794 questions as training set and 2492 questions as the validation set. the dataset used for experimentation is available for research purposes upon email request. finetuning details: to evaluate the effectiveness of finetuning the dl models for temporal qg task, we use two different models, prophetnet and unilm. prophetnet: the input provided for finetuning prophetnet is in the form of answer [sep] narrative and output is the generated question. we use the hyperparameter values as suggested by the authors on their github page foot_0 for finetuning task. we use adam optimizer @xcite with a learning rate of 0.00001. we set the dropout value as 0.1 and train for 10 epochs. unilm: the input provided for finetuning unilm is in the form of narrative [sep] answer and output is the generated question. here also, we use the hyperparameter values as suggested by the authors on their github page foot_1 for finetuning task. we use bert adam optimizer (bert version of the adam algorithm with weight decay fix) with a learning rate of 0.00002. we train unilm for 10 epochs. evaluation methodology: we employ both automated and human expert evaluation. for human evaluation, we asked experts (people well versed with english language and global history) to mark the generated question as acceptable or non-acceptable following the human evaluation in @xcite . a question is marked as acceptable if it is grammatically correct, readable, sensible and not too vague. more details can be found in the guidelines proposed by @xcite . for human evaluation, we use a random sample of 100 generated questions for each experimental setting. q acc metric is used for the percentage of generated temporal questions which are found acceptable by a human expert. q temporal measures the fraction of temporal questions generated with respect to total number of generated questions. for automated evaluation, we use bleu-4 @xcite , meteor @xcite and rouge-l @xcite which are standard metrics for natural language generation tasks. the questions generated by the template based approach are used as reference questions for calculating these automated metrics. results and discussion: the evaluation details of our template based approach (i.e., the stage 1 of our approach) are given in from ## conclusion we proposed a two-staged method for temporal qg from history text. first, we use templates motivated by linguistics and domain knowledge to carry out syntactic and semantic transformations to generate temporal questions. then, the generated temporal questions are used to finetune dl models for qg. we experimentally validated the approach with two different dl models to demonstrate improvement due to finetuning as well as flexibility and robustness of this approach for temporal qg.