PROTEGE : Prompt-based Diverse Question Generation from Web Articles (2023)

Abstract: Rich and diverse knowledge bases (KB) are foundational building blocks for online knowledge sharing communities such as StackOverflow and Quora, and applications such as conversational assistants (aka chatbots). A popular format for knowledge bases is question-answer pairs (or FAQs), where questions are designed to accurately match a multitude of queries. In this paper, we address the problem of automatic creation of such Q&A-based knowledge bases from domain-specific, long-form textual content (e.g., web articles). Specifically, we consider the problem of question generation, which is the task of generating questions given a paragraph of text as input, with a goal to achieve both diversity and fidelity of the generated questions. Towards this goal we propose PROTEGE, a diverse question generation framework which consists of (1) a novel encoder-decoder based Large Language Model (LLM) architecture which can take a variety of prompts and generate a diverse set of candidate questions, and (2) a hill-climbing algorithm that maximizes a sub-modular objective function to balance diversity with fidelity. Through our experiments on three popular public Q&A datasets, we demonstrate that PROTEGE improves diversity by +16% and fidelity by +8% over diverse beam search and prompt-based baselines.

## introduction in a data rich era, identifying, extracting and generating responses to user's questions has become the next challenge. while search engines provide a simple interface for users to get responses to their queries, getting answers to complex queries still remains a challenge @xcite . as a result, specialized knowledge bases that extract and store question-answer pairs have become prevalent. many applications rely on a knowledge base of generated question-answer pairs, to ensure reliable, accurate and as close to human-generated information to their users. for instance, a key frustration for online shopping is the difficulty in identifying the right product that suits the requirement. for high-consideration products such as laptops and smartphones, at times customers lack the human touch that they would otherwise experience in an offline store where trained sales agents can explain features of each product and provide high-level guidance to select the right one. the sales agent can proactively query the customer to understand her requirement, help refine her needs and finally recommend the right products. in order to bridge the gap between online and offline shopping experience, multi-turn goal-oriented dialog systems, also known as chatbots offer a promising direction. chatbots help users to familiarize technical concepts, acquire domain knowledge, get recommended products that they are likely to buy, closely mimicking the offline shopping experience. towards curating a large scale knowledge bank, large language models (llms) @xcite @xcite have shown remarkable success in learning in-depth knowledge from data. they do so without access to any external memory as the knowledge is imbibed in the model parameters. while this is fascinating, on the downside, the model may hallucinate @xcite and generate answers that are factually incorrect. as a result, modern chatbot systems employ a retrieval-augmented generation (rag) architecture @xcite ) that has two components: a) an encoder-decoder network that does the natural language understanding of user queries and language generation and b) a back-end knowledgebase (kb) that indexes relevant bits of information for the task at hand. the encoder maps user inputs to a dense representation which is used to query the kb and retrieve evidences. the evidences, as well as the input, are fed to the decoder to generate the final response. scalable generation of knowledge-bases is fundamental to the success of the rag and the underly-ing chatbot application. in this work, we investigate automatic generation of a knowledge-base. unlike traditional rag system @xcite that indexes webpages and large documents, we look for a knowledge-base in the form of question-answer pairs. not only this helps us improve the accuracy of evidence retrieval, but also allows rich applications to be easily built on top of it e.g., suggesting related questions, navigation of the kb etc, for improved customer experience. our focus is on educational questions, i.e., questions that help user familiarize with product concepts ("what is the difference between ssd and hdd?") and use-case guidance ("what is the recommended configuration for a gaming laptop?"). question generation is the task of generating questions given a paragraph of text as input. question generation quality can be attributed to two characteristics: a) fidelity that measures the semantic coherence of generated questions and our ability to answer them from the input paragraph, and b) diversity which measures lexical and semantic dissimilarity between generated questions. many previous works @xcite @xcite have addressed the task of generating questions from text. while it is essential to generate questions that are of high fidelity, for knowledge-base completion, it is imperative to have a diverse question set. to promote diversity, current question generation models rely on beam search. the resulting set, however contains many structurally similar questions with minor lexical changes that warrant the same answer. there has been prior work @xcite @xcite in nlp on diversity. in particular, song et al. @xcite addresses diversity via determinantal point processes (dpp) for neural conversation models, it can be adapted for question generation task. while these approaches are helpful in maximizing diversity, they fall short in terms of generating high fidelity output. naively borrowing these techniques may allow the model to hallucinate and generate questions that are not answerable from the input paragraph. addressing the task of diverse question generation through the lens of monotone sub-modular function @xcite alleviates this problem and provide additional benefits. on one hand, this formulation provides flexibility in controlling diversity and fidelity of the output. on the other hand, we can leverage a well-known greedy algorithm @xcite to generate a near optimal set of questions, therefore, increasing yield and quality simultaneously. we propose protege (prompt-based diverse question generation), a diverse question generation framework which consists of two stages (1) a novel encoder-decoder based llm architecture which can take a variety of prompts and generate a diverse set of candidate questions, and (2) a greedy hill-climbing algorithm that maximizes a sub-modular objective function to balance diversity with fidelity. we demonstrate that protege improves diversity by +16% and fidelity by +8% while also improving text generation metrics, over strong baselines. our experiments on three popular public q&a datasets indicate that protege consistently outperforms both diverse beam search-based and prompt-based baselines. ## related work rule-based @xcite and dnn based @xcite models are used for question generation from text corpora. answer extraction @xcite or machine comprehension @xcite is a branch of nlp where the goal is to extract answer snippet from text documents given a question as input. in both cases, either the question or the answer is given as input. qa extraction models @xcite @xcite are generally pipeline-based which generates the question and the answer in a sequence. boros et al. @xcite uses a question-answering system to detect specific events in textual content (e.g., tweets, blogs). in this context, the entity information is used to frame template-based question (e.g., where did the [attack] happen?) where attack is an event of interest). zhang et al. @xcite propose combining entity linkage with a qa system. however, our objective is differ-ent as we enrich the qa extraction technique by augmenting it with entity level metadata. ## conclusion and future work in this paper we present protege, a transformer based two-stage question generation framework based on prompts that balances diversity of the generated questions with their fidelity. through extensive experiments on multiple datasets we show that protege significantly improves diversity (by +16%) and fidelity (by +8%) compared to strong baselines. as a future work, we will extend our models to simultaneously generate both questions and answers. in preliminary experiments on the task of extracting answers for questions from a given context, we have observed that providing the "entities" in the question as additional prompt signals to a bert-based model improves the answer extraction quality by up to +4.2% in f1 score. similar applications to other nlg tasks such as document summarization and faq creation are possible using the framework proposed in our paper. extension of our work to non english languages is part of future work.