Graph Guided Question Answer Generation for Procedural Question-Answering (2024)

Abstract: In this paper, we focus on task-specific question answering (QA). To this end, we introduce a method for generating exhaustive and high-quality training data, which allows us to train compact (e.g., run on a mobile device), task-specific QA models that are competitive against GPT variants. The key technological enabler is a novel mechanism for automatic question-answer generation from procedural text which can ingest large amounts of textual instructions and produce exhaustive in-domain QA training data. While current QA data generation methods can produce well-formed and varied data, their non-exhaustive nature is sub-optimal for training a QA model. In contrast, we leverage the highly structured aspect of procedural text and represent each step and the overall flow of the procedure as graphs. We then condition on graph nodes to automatically generate QA pairs in an exhaustive and controllable manner. Comprehensive evaluations of our method show that: 1) small models trained with our data achieve excellent performance on the target QA task, even exceeding that of GPT3 and ChatGPT despite being several orders of magnitude smaller. 2) semantic coverage is the key indicator for downstream QA performance. Crucially, while large language models excel at syntactic diversity, this does not necessarily result in improvements on the end QA model. In contrast, the higher semantic coverage provided by our method is critical for QA performance.

## introduction asking questions is a natural way for humans to understand how to perform a task. questions that pertain to a given procedure (i.e., a structured task such as cooking a recipe) encompass both factual questions about a given step (e.g., what tools are used in a given step), as well as questions that span across multiple steps (e.g., the order of steps). a smart ai agent should be able to handle both types of questions to assist humans. while gpt models and competing alternatives have shown impressive results on multiple applications, including qa, they require large amounts of cloud computing resources due to their extreme sizes, thus being inviable as the qa models behind a smart assistant. we show that it is possible to train task-specific small models (e.g. suitable for running on a mobile phone), that at the same time are as accurate and complete as gpt variants on the target task. high-quality in-domain training data is however required but, unfortunately, most qa datasets focus on general text comprehension where the answers can be spans from the text @xcite @xcite , free-style answers about a specific context @xcite or obtained from a conversation history in conversational qa @xcite . similarly, collecting highquality qa data at scale requires expensive labeling efforts. this motivates our paper, where we propose a method that ingests large quantities of procedural instructions (e.g. cooking recipes) and automatically generates extensive procedural qa (pqa) training pairs that can be used to fine-tune a well-performing small language model. in particular, the goal is to automatically generate pqa pairs that elicit information both from single sentences (or steps) in a procedure as well as information that requires reasoning over multiple steps to understand the temporal aspect of a procedure. while there have been efforts to create multi-modal qa datasets from recipes that require alignment between vision and text @xcite , to the best of our knowledge, our work is the first that specifically concentrates on extracting a rich set of qa pairs from procedural text. we focus on cooking recipes as a type of procedural text. in a cooking sce-nario, single sentence-based questions span local concepts (e.g., quantities of ingredients, cooking times, and tools), while temporal questions cover multiple steps (e.g., order of actions, the contents of mixtures at certain steps). to tackle this problem, we propose to leverage the highly-structured nature of procedural text and represent the semantics of the procedure as graphs from which we can automatically generate pqa pairs. specifically, to cover all question types pertaining to individual steps in a recipe, we rely on abstract meaning representation (amr) graphs @xcite . we perform a controlled set of transformations on the amr graph of a step to generate a number of question amrs and then generate questions from those amrs using a pretrained amr-to-text model. for temporal questions that span across multiple steps, we start by converting the recipe into an action flow graph @xcite @xcite using a neural graph parser @xcite . we then extract all potential temporal answers by traversing the graph, and generating temporal question templates in the amr space. we then, once again, rely on amr-to-text models to generate corresponding questions. 1 optionally, our approach can take advantage of llms (e.g., gpt3 @xcite ) to increase the syntactic diversity and semantic coverage of the generated questions, either by improving the wording and paraphrasing the generated questions or via directly replacing the graph-to-text generation model with a gpt-based solution that relies on content selected with our graph-guided approach. extrinsic evaluation shows the usefulness of our generated data in training question-answering models, which outperforms all considered baselines. our results highlight the importance of devising an approach dedicated to generating qa pairs from procedural text, as we show small models (e.g., t5-base with around 220m parameters) can compete with gpt3 and chatgpt (175b params) when finetuned on specialized high-quality data. in addition, intrinsic evaluation of our generated data demonstrates its superiority in terms of diversity, coverage, and overall quality, compared to data generated using several baselines including gpt3based methods. 1 we use action flow graphs rather than more recent work on multi-sentence docamr @xcite as docamr does not consider the temporal nature of procedural text. contributions. in summary the contributions of our paper are threefold: ## related work question generation is an important topic within the natural language generation community @xcite , where given a source text (i.e., context) and a target answer, the task is to generate the corresponding question. the answer is either provided @xcite @xcite @xcite or automatically extracted from the context @xcite @xcite . our work follows the latter approach, where we automatically extract answers and generate corresponding questions. moreover, compared to acs-qg @xcite , our question generation method does not require additional clue and style information extracted from the input text, as they are represented in the semantic graph of the text. existing methods either rely on hand-crafted rules and templates @xcite @xcite , or use annotated data (in the form of text spans as answers, along with corresponding ground-truth questions) to learn to automatically generate qa pairs @xcite @xcite @xcite . rule-based methods offer more control over the generated data, but are not easily scalable. learning-based methods offer better scalability, but require costly annotations. our graph-based approach combines the benefits of the two while addressing their short-comings. specifically, our graph-guided content selection offers the desired control over the content extracted from procedural text (to form the answer), and draws on generic models for graph-to-text generation to generate questions @xcite . recent work has shown that large language models (llms) such as gpt3 can be used for generating qa pairs based solely on the input context texts @xcite . however, these models are sensitive to the prompt used to generate the data, offer less control over the generated qa pairs, and are not cost-effective. in contrast, we provide evidence that our graph-controlled qa generation approach yields high-quality and diverse data that can be used for training question answering models that compete with llms while being orders of magnitude smaller. ## methodology in this section, we present our approach to qa generation from procedural text. we introduce the amr and flow graphs that our method relies on ( §3.1), and detail our method for generating questions from single instructions ( §3.2) and temporal questions spanning across multiple instructions ( §3.3). lastly, we propose to use llms to improve the language quality of generated questions ( §3.4). ## evaluation we describe our experimental setup and questionanswer generation baselines in §4.1, where we also introduce a new human-annotated pqa set (reference set) used for evaluation. §4.2 shows extrinsic evaluations, where question-answering models are trained with the generated data and evaluated on the aforementioned reference set. §4.3 describes intrinsic evaluations, assessing the distributions and quality of our generated qa pairs, which further amplifies the significance of our qa generation approach. ## conclusion in this paper we tackled task-specific qa from procedural text. to this end, we proposed a novel method for automatic generation of questionanswer pairs from procedural texts in a comprehensive manner, both in their semantic content as well as syntactic diversity. we do so by exploiting the structured nature of the procedural data by using graph-based representations, and devise a systematic way of generating semantically-comprehensive question-answer pairs. we further enrich the syntactic correctness and diversity through the use of llms. we show that 1) using automaticallygenerated in-domain data to train a simple t5 model results in question-answering performance competitive with very large language models such as chatgpt and gpt3. 2) our method results in excellent coverage of human-generated questions.