CONSISTENT : Open-Ended Question Generation From News Articles (2022)

Abstract: Recent work on question generation has largely focused on factoid questions such as who, what,where, when about basic facts. Generating open-ended why, how, what, etc. questions thatrequire long-form answers have proven more difficult. To facilitate the generation of openended questions, we propose CONSISTENT, a new end-to-end system for generating openended questions that are answerable from and faithful to the input text. Using news articles asa trustworthy foundation for experimentation, we demonstrate our model’s strength over several baselines using both automatic and human based evaluations. We contribute an evaluationdataset of expert-generated open-ended questions. We discuss potential downstream applications for news media organizations.

## introduction factoid questions are relatively straightforward questions that can be answered with single words or short @xcite . however to obtain the central idea of a long piece of text, one can ask an open-ended question (e.g. why, how, what) @xcite , which can essentially be viewed as an extreme summary of the text @xcite in the form of a question. the ability to generate such questions is particularly difficult because the generated questions must be answerable from and faithful to the given input text (see ## data 6955 it's springtime of the pandemic. after the trauma of the last year, the quarantined are emerging into sunlight, and beginning to navigate travel, classrooms and restaurants. and they are discovering that when it comes to returning to the old ways, many feel out of sorts. do they shake hands? hug? with or without a mask? how are people adapting to life after the pandemic? training data most prior work has successfully trained models for question generation using squad @xcite , triviaqa @xcite , or nq @xcite datasets, the answers to which are typically short. to account for the open-ended nature of our desired questions, we rely on the eli5 @xcite , explain like i'm five) dataset. the dataset comprises 270k english-language threads in simple language from the reddit forum of the same name foot_0 , i.e easily comprehensible to someone with minimal background knowledge. compared to existing datasets, eli5 comprises diverse questions requiring long-form answers. it contains a significant number of open-ended how/why questions. interestingly, even what questions tend to require paragraph-length explanations (what is the difference...). as seen in evaluation data since our goal is to generate open-ended questions from news articles, we specifically design our evaluation data to reflect the same. to achieve this goal we obtain englishlanguage articles from the new york times website from january 2020 to june 2020. we obtained written consent to use this content for research purposes by the copyright holder. one of the additional advantages of crawling data from the the new york times website is that we can divide news articles by domain, as each news article appears in a specific section of the website. from the given url foot_1 , we can tell that the article belongs to the science domain. additionally, as most pre-trained language models were trained prior to the covid-19 pandemic, we also test how well they generalize to covid-19 related news topics. each news article from a particular domain is segmented into several paragraphs. we randomly sample 529 paragraphs spanning six domains. this includes 55 paragraphs from science, 66 from climate, 98 from technology, 110 from health, 100 from nyregion, and 100 from business. while we understand that selecting standalone paragraphs might sometimes ignore the greater context, or suffer from co-reference issues, we carefully replace any such paragraphs from our bigger pool. as we do not have gold questions associated with each paragraph, we crowd-source humanwritten questions for each paragraph on amazon mechanical turk. each paragraph is shown to a distinct crowdworker who is then instructed to read the paragraph carefully and write an open-ended question that is answered by the entire passage. we recruit 96 distinct crowd workers for this task. after the questions are collected from first round of crowd-sourcing, two expert news media employees approve or reject them based on quality. the paragraphs with rejected questions are put up again and through this iterative process and careful quality control we obtain one high quality open-ended question associated with each paragraph. ## consistent model the backbone of our approach is a fine-tuned bart-large @xcite model on the eli5 dataset of question-answer pairs. however, there are two major factors to consider in our end-to-end question generation pipeline. the generated questions i) must be relevant and factually consistent to the input paragraph, and ii) must have the answer self-contained in the input paragraph. our con-sistent model (figures 2 and 3 ) addresses these issues as described below. factual consistency to ensure faithfulness to the input paragraph, we need to design our model in such a way that the generated question is about a topic or concept mentioned in the paragraph. in traditional fine-tuning of a seq2seq model where x denotes input paragraphs in the training set and y denotes the corresponding question our goal is to learn p θ (y|x) where recently @xcite proposed ctrl, a conditional language model that is conditioned on a control code c and learns the distribution p θ (y|x, c) to provide explicit control over text generation. the distribution can still be decomposed using the chain rule of probability and trained with a loss that takes the control code into account. owing to this modification, language models can generate text conditioned on control codes that specify domain, style, topics, dates, entities, relationships between entities, plot points, and taskrelated behavior. we rely on the same underlying principle for training question generation models. during training, we extract keywords from questions and feed the input paragraph along with the extracted keyword to the encoder of bart. the extracted keyword here acts as the control code. since we do not have any supervision for these keywords we use yake @xcite , an unsupervised keyword extraction tool. for example, as shown in figure 2 , given the question: where do presidential campaign donations actually get spent?, we extract the top-most salient trigram "presidential campaign donations" using yake. we then feed the (control code, answer) to the encoder, the original question to the decoder, and fine-tune the model as shown in figure 2 . @xcite propose a bert @xcite based answer extraction model on natural questions (nq) by predicting p(a|c) = p([a start , a end ]|c) where "a" is an answer and "c" is a passage containing "a". this model first feeds the passage "c" through bert, before concatenating the start and end token representations of all possible spans of up to length 30, and then feeds them into an mlp to compute p(a|c). at generation time, the answer extraction component extracts a constant number of spans from each passage, ranked by their extraction probabilities. these extracted spans, while originally designed for a different purpose, can act here as control codes for our question generation model. to encourage the question to refer to a concept mentioned in the passage, we extract salient key phrases as control codes from the input paragraph using a combination of yake and the answer extraction model p(a|c) (figure 3 ). it should be noted that during training the keywords are taken from the question, while during inference the keywords are produced from the article. this is because at training time we want to minimize the generation loss with respect to the training question so encouraging the model to obey the training keyword is beneficial. at inference time we do not have access to any question so using keywords from article is the only option. answerability prior work @xcite @xcite has relied on filtering methods to ensure answerability of generated questions. a filtering qa model p f (a|q, c) generates an answer for a given question. if an answer generated by p f does not match the answer a question was generated from, the question is discarded. such filtering methods are not applicable for our task because i) our question generation model treats the entire input paragraph as an answer instead of short answer spans typically common in odqa tasks, and ii) the length of the answers are typically long-ranging across several sentences which is beyond the capabilities of most generative models @xcite and additionally would be hard for string matching purposes. we propose two filtering methods to ensure answerability: model confidence and instruction prompting. primary filtering: model confidence qa models trained on squad 2.0 @xcite are capable of asserting when a question is unanswerable by signaling lower confidence scores. taking advantage of this fact we first rely on an albert-based qa model finetuned on squad 2.0 foot_2 . the intuition behind this is such a model would typically have lower confidence scores for most poorly formed / unanswerable questions and can be used as a primary filtering step. while it may appear that a model trained on squad to determine the answerability of questions conflicts with open-ended nature of questions requiring long-form answers, it is often not the case. as we have seen in the case of natural questions @xcite at least 35% question requiring a long form answer often has a short answer associated with it. this means the specific span that is returned by the squad based qa model when prompted with the generated question and the input paragraph can often be an approximate short answer. for instance for the generated question q3 in figure 3 , the squad model gives an answer via art, sports, entertainment and media. which isn't inaccurate but requires further elucidation. this motivates us to use a model trained on squad v2.0 as our initial primary step. however different questions can have different model confidence. to decide on an appropriate threshold for model confidence we observe the distribution of confidence scores. we observe a median model confidence of 0.42. we then experiment with 3 different thresholds κϵ{0.35, 0.4, 0.45} for selecting generated questions. the quality on a held out set of 50 generated questions by is evaluated by human judges and finally decide on a model confidence threshold κ = 0.4 such that any generated question having a confidence score below κ is discarded. it should also be noted that we tried higher values of κ between (0.6,0.9) but having such a strict high confidence score sometimes leaves us with no generated question for an input paragraph. as can be seen in figure 3 , a generated question "what are bitcoins and how have the made a lot of people very rich?" while being open-ended, grammatically correct, and relevant to the input does not meet the answerability threshold and hence is discarded. ## results and analysis in an effort to better understand why or how the consistent model is better than the baselines, we carefully analyze outputs from all systems. without an explicit supervision on what to ask, bart often asks generic questions or deviates from the input source and hallucinates content as can be seen in to test the effect of the primary filtering we choose the candidate question with the highest confidence given by the qa model. to test the effect of secondary filtering we choose the candidate question with the highest confidence of generating yes. the obtained bertscore for primary, secondary and consistent(both) are 64.0, 64.5 and 66.4 showing that the best filtering mechanism is the combined one. ## downstream applications we believe open-ended question generation might enhance the news experience through new q&a tools, enhanced search, improved recommendations, and more. media organizations have used faq pages to help readers better understand complex news topics, from covid-19 vaccines foot_5 to personal finance foot_6 . the ability to automatically generate open-ended question about a given topic could make it easier for news organizations to launch an faq page for a new topic. we envision an admin tool (figure 4 ) that presents the users with a list of generated questions and allows them to approve/reject, edit, and publish the results quickly. this human-in-the-loop approach is essential for maintaining reader trust when the generated questions may be presented directly to readers @xcite . another potential application of this human-inthe-loop version of the system is improved news article recommendations (figure 5 ). while reading a news article, a recommended article interface may be presented showing a series of questions related to the topics in the article. this level of functionality could in some ways anticipate questions a reader may ask and point them in the direction of other news articles that may provide them with answers. for other use cases that might allow automatically generated open-ended question to be used more broadly in production systems, we envision a human-validated database of such question-answer pairs where the reliability of the results could be controlled. this can help improve the search experience on news media websites. for instance, if a user were to search what are some of the issues with nfts?, the search experience could fuzzily match questions generated by consisted to prioritize the article containing the answer relevant to the user's query which can help them discover things pertaining to what they are curious about. additionally, question clustering algorithms could be deployed to better match searched questions with the generated questions. finally automatic question generation can also used to improve interactive news podcast @xcite . ## limitations we note that our training dataset is automatically collected from the r/eli5 subreddit and as such we don't account for any sensitive text. we focus on open ended question generation from news articles where our inputs are paragraph level and longer than sentence level inputs in factoid qg. however our model is not capable of handling longer sequences like an entire news article or opinion piece. we believe models like longt5 @xcite might be useful for such inputs however we leave this for future task. even though we control for hallucination by incorporating control codes from input text, it does not ensure 100% hallucination free output. in regards to answerability judgements our methods are useful and bridge the gap in distinguishing unanswerable questions however it in itself is a difficult task and our approaches based on squad v2.0 and t0pp can still make errors.this means our models are still capable of generating unacceptable questions and should be deployed based on due deliberation. finally temporal misalignment is an issue and owing to the fact that our training data is from a few years back it sometimes fails on newly coined scientific terms or expressions related to covid-19 pandemic. continually fine-tuning our models on newer data with experience replay can mitigate these issues. we leave this for future work. figure 6 : screenshot of the evaluation tool where employees of a news media organization are asked to select the best option given two generated questions, an input paragraph, and headline from a news article input at the two new horizons homes in gainesville, the medical director, dr. swati gaur has held six staff town halls, in person or online, including one at 2 a.m. for the night shift, and offered rewards like free meals. about half of the workers have been vaccinated, dr. gaur said.