Competence-based Question Generation (2022)

Abstract: Models of natural language understanding often rely on question answering and logical inference benchmark challenges to evaluate the performance of a system. While informative, such task-oriented evaluations do not assess the broader semantic abilities that humans have as part of their linguistic competence when speaking and interpreting language. We define competence-based (CB) question generation, and focus on queries over lexical semantic knowledge involving implicit argument and subevent structure of verbs. We present a method to generate such questions and a dataset of English cooking recipes we use for implementing the generation method. Our primary experiment shows that even large pretrained language models perform poorly on CB questions until they are provided with additional contextualized semantic information. The data and the source code is available at: https: //github.com/brandeis-llc/CompQG.

## introduction developing an artificial intelligence (ai) system with inferencing and reasoning capabilities that enables itself to communicate with intellectual human users has been a holy grail of the research community. for example, there has been considerable effort put on large challenges in question answering formats to measure such inference and communication capabilities of neural end-to-end systems @xcite @xcite . however, we argue that a broader effort needs to be put on measurements more focused on linguistic competencies, and not just on extractive comprehension skills or "challenge checklisting". out argument is in line with some moves in this direction @xcite , but there is still no generally accepted distinction in current natural language processing (nlp) between challenge-based tasks and competence-based (cb) performance @xcite . analogous to human cognitive compe-tencies, there is both a methodological and modeling advantage to focusing a system's performance on competence-based learning rather than a narrowly defined task or challenge. the term competence-based (cb) has been applied to a number of different concepts, from linguistics @xcite to both the science of learning and educational communities @xcite @xcite @xcite . the common core to both is a concept capturing a coherent set of abilities that an individual has in a specific domain @xcite . more recently, we proposed a new multimodal question answering task r2vq that focuses on querying competence-based knowledge from cooking recipe texts and videos @xcite . for the text part, we created a dataset with rich annotations of hidden arguments and coreference that focuses on lexical competence as deployed in both single and multiple sentence composition @xcite @xcite . as a natural complement and extend to the r2vq task @xcite , here we define the task competence-based question generation that aims to generate cb questions that require lexical competence. the lexical competence involves understanding the hidden arguments (due to syntactic ellipsis or semantic defaulting or shadowing) given the event context; and understanding the dynamics of events and objects change in the text. this competence requires non-extractive question answering (qa) capabilities of some sort. thus we define a competence-based question as one that queries any aspects of lexical competence mentioned above. in the rest of the paper, we first review recent related work (sec. 2), and give more detailed definitions of the cb question generation task (sec. 3). we then present a dataset of cb knowledge and how we collected and annotated it (sec. 4). section 5 provides details of generation of cb questions on the dataset we created. the generated questions are evaluated and the results are discussed in section 6 and 7. then we conclude our work in the final section (8). ## related work question generation (qg) is an essential nlp task that can be used for supplementing educational materials @xcite , data augmentation for qa systems @xcite and understanding semantic relations within the text @xcite . with the advent of large pretrained language models and large qa datasets, recent qg approaches are primarily based on transformer-based neural architectures. @xcite finetuned a unified language model for both qg and language understanding tasks. @xcite enhanced the language model for qg with additional embeddings of linguistic features. to make generated questions more diverse, @xcite took advantage of the fully text-to-text generation model to to generate questions with mixed types of answers. @xcite applied graph networks to generate complex questions requiring broader contexts to answer. comparing with the neural methods, traditional rule-based qg systems @xcite did not receive much attention due to the lack of diversity of the generated questions. to solve this, @xcite proposed qa-srl that use predicate-argument structure to generate more semantic-informed qa pairs. more recent work also has shown the success in incorporating existing explicit semantic annotations and linguistic resources into the rule-or templatebased qg systems @xcite . in this work, we adopt a template-based approach to generate competencebased questions that solicit implicit information and the event dynamics across a broad context. more broadly related to our general purpose to probe the above-defined competence-based knowledge with qg, @xcite proposed a new qa task that queries competence-based knowledge structures; @xcite proposed a new commonsense inference task that involves the reasoning of intents and reactions to the events; @xcite used context information for commonsense qa. there also exist semantic tasks that explore the implicit or underspecified components of a linguistic expression @xcite @xcite . ## task definition we define our task as generating a competencebased question set. unlike traditional qg tasks whose answers are mostly extractive and contexts are largely dependent on surface lexical locality conditions, the goal of our task is to generate questions that represent commonsense semantic inferences from a large context in an abstractive way. ## data description due to the lack of existing datasets suitable to our task, we curate a collection of english cooking recipes as the data for our task. recipes along with other procedural text like instructions tend to be task-oriented and stepwise. the content is composed of step sentences that describe small goals to accomplish the final task. comparing to text of news or narratives, we think recipe texts are a good fit for our task as it involves the understanding of how to reach the goal locally for each step, as well as how each step contributes to the final task globally. further, the stepwise progression inherent in the goal-oriented narrative contributes both an interpretative dynamics as well as contextualized elision of arguments. our proposed dataset has also been used as part of the data resource for the semeval-2022 task 9: r2vq @xcite . here we describe the annotation process and the annotation alignment step for our qg task. data preprocessing we build our qg dataset with a collection of public domain recipes. 1 we first run the stanza pipeline @xcite on the raw text of each recipe to get tokenization and other linguistic features including lemmatization, part-of-speech tagging and dependencies. we also run the state-of-the-art semantic role labeling (srl) parser from @xcite to label each recipe sentence with its semantic roles. subsequently, we ask 2 students annotators to validate and correct both frames and argument labels. recipe title: appelkoek passage: peel and cut apples into eighths (wedges). sift together flour, baking powder and salt with 4 tablespoons of the sugar. cut in butter. combine egg and milk and add to flour mixture. turn batter into greased 8 inch square cake pan. press apple wedges partly into batter. combine remaining 2 tbsp sugar and cinnamon. sprinkle over apple. bake at 425 degf for 25 to 30 minutes. train val # of recipes 800 100 avg. # of sentences per recipe 8 7.9 max./min. # of sentences 26/4 16/4 avg. sentence length per recipe 12.5 13.4 max./min. sentence length 32/6 25/6 ## experiment method now we describe a template-based method to generate competence-based questions. we explore the data with crl annotation in two phases: first we adopt the concept of question family in §5.1 by proposing question templates and heuristics to generate qa pairs that involve hidden arguments for each family. for example, given a sentence sauté the onions until browned., we generate the question where should you sauté the onions?. the correct answer in the pan is not in the current sentence, thus needing to be inferred from its context. in §5.2 we propose the concept dense paraphrasing that can record previous states of the entities as being processed though events, e.g. sauté onions will be rewritten as sauté chopped onions in generated questions based on the given context. then we apply it to enhance question families by generating new questions and answers that cover a bigger context and dynamics of events. ## evaluation and discussion in this section we describe the steps to evaluate our system. first we adopt crowd-sourced evaluation to measure the quality of generated questions in- intrinsic evaluation similar to other text generation task @xcite , human evaluation plays a critical role in question generation, as it can provide a more reliable evaluation and capture the nuance between generated text. we measure the generated qa pairs on a 5-point likert scale from grammatical correctness, relevance and answer completeness. the first two metrics are proposed in @xcite and widely used in previous work @xcite . they measure the grammaticality and the relevence of the questions to the context, respectively. in addition, we propose answer completeness for our qg task. it measures whether the answer is complete, e.g., some ingredients are missing from the answer or redundant. this metric is helpful to our task as we generate the answers that involve hidden objects which also traverse the whole passage. we conduct the intrinsic evaluation on the hid-den and the enhanced question subsets of the qg validation set. we assign the generated questions to 4 trained annotators who had the experience with crl annotation, and ask them to score each qa pair. to understand whether crl annotation can help answer cb questions, we fine-tune the t5 model on the enhanced set again, but we modify the input string to include the crl by inserting hidden entities as a human-readable format to the in comparison to other qg systems as a qualitative study of our system, we compare it to two qg systems: syn-qg (dhole and manning, 2021) and roleqg @xcite . due to the fact that all three systems are essentially different regarding their domain and model i/o, we focus on comparing how the systems differ in the final outcome and how our system fits into the larger scope of qg. syn-qg leverages syntactic and semantic rules to generate template-based questions. roleqg proposes a hybrid method of templates and seq2seq models to generate questions for any semantic role that may exist in the context. we adopt syn-qg and roleqg on a sample data and summarize our 5 e.g., input sauté until browned is changed to {using a spatula} {on the cutting board} sauté {chopped onions} until browned {resulting in sautéed onions} observations from comparing results. 6 procedural vs. descriptive text both synqg and roleq are developed on datasets with narratives, such as ontonotes 5.0 @xcite and squad @xcite . however we choose to apply our method on procedural text like cooking recipes. this difference in data format and scope can impact the operations for qg tasks. procedural text tends to be imperative and instructional. this may lead to certain semantic roles being naturally omitted from sentences. for example in competence vs. ubiquitous qg traditional qg methods are designed to generate as many questions as possible to have a wide coverage of the context. to achieve that, synqg includes named entities and hypernyms into srl arguments to increase the variety of srl templates (synqg [2]); roleq generates questions for all possible semantic roles of the predicate even though the answers can not be identified in the text (roleq [1]). as a com-parison, we focus on competence-based qg with special attention to answer completeness. answers to srl questions are just semantic roles that can be retrieved locally with ease (srl row in ## error analysis and postprocessing based on the intrinsic evaluation results from to solve or alleviate aforementioned errors, we postprocess the data based on some heuristics to fil-ter out the low-quality qa pairs. before the events are populated into the templates, we fill in possible missing annotation for hidden tools or habitats. we assign the most likely tool or habitat to the given event based on the frequency distribution of the entity from the existing annotation. for example, if the event head cut has no hidden tool annotated, knife as the most frequent tool to this event head will be assigned. to make the questions less ambiguous, the templates that are used to generate questions involve common verbs such as add, cook, put, place need to contain as least one adjunct slot. this can generate more specific questions. we also remove qa pairs that contain pronouns and the pairs whose answer is only one ingredient and overlapped with the entities that are asked about. although the errors from the annotation and automation process are hard to avoid, the annotation overall can be improved through more careful post validation and adjudication of the data. answer ambiguity can also be partially solved by setting rules in the specification to favor more specific entities over the general ones (e.g. rolling pin vs. hands, large spoon vs. spoon), or allowing multiple valid answers. we leave these to the future discussion. ## conclusion we have proposed a method for generating competence-based questions that leverages lexical semantic knowledge involving, implicit arguments, subevent structure of verbs, and document-level event dynamics. to that end, we have constructed a new dataset that encodes manually annotated lexical semantic knowledge in a corpus of cooking recipes. our proposed method includes a set of rich targeted question families and a suite of dense paraphrasing operations that facilitate systems to generate answers both locally and globally. we have conducted intrinsic evaluation of the system to show the quality and usefulness of cb questions. we have also performed qa tasks by letting systems answer generated questions; this in turn has provided potential insights into further improvement of existing large language models for understanding such questions. by comparing to existing qg systems, our work has focused on generating questions from the subdomain of procedural text, and the questions are designed to query competence-based knowledge. in future research, we intend to further improve our method by minimizing the annotation errors and ambiguous answers as indicated in the error analysis, and then apply the method to a broader range of text genres.