Prompting Few-shot Multi-hop Question Generation via Comprehending Type-aware Semantics (2024)

Abstract: Given several documents, multi-hop question generation (MQG) is a task aims to generate complicated questions that require reasoning over multiple pieces of these documents to find the answer. To perform this task, existing studies focus on designing advanced architectures to locate essential keywords or sentences in multiple documents and then generate questions accordingly, where they normally do not note that question types could provide crucial hints for extracting key information from the documents for MQG. In general, supervised approaches are used that rely on large annotated data, which is not available in many low-resource scenarios and thus makes MQG hard in these domains. Consider the recent success of large language models (LLMs) on natural language processing tasks using limited labeled data under few-shot settings, in this paper, we propose an approach named type-aware semantics extraction-based chain-of-thought method (TASE-CoT) for few-shot MQG. Specifically, our approach firstly extracts question types and essential semantic phrases from the given documents and the answer. Then, we design a three-step CoT template to leverage the extracted question type and semantic phrases to predict multi-hop questions. Extensive experiments and the results demonstrate the effectiveness of our approach and the proposed modules.

## introduction question generation (qg) aims to generate questions that are relevant to the given document. it is a vital task in the field of question answering (qa) owing to its wide applications, e.g., helping chatbots start conversations with intriguing questions @xcite . most existing approaches for qg (du et al., 2017; † corresponding author. 1 the source code and relevant resources of the paper are available at https://github.com/synlp/tase-cot . figure 1 : the figure presents examples where the question type affects the process of generating questions. in this example, the first model selects the inappropriate question type "when", and thus extracts irrelevant semantics. on the contrary, the second model selects the appropriate question type "who", and thus extracts the appropriate key semantics to generate questions. important semantic phrases are highlighted in red color. @xcite @xcite focus on generating simple one-hop questions based on a single document, which cannot cover the cases that need complicated multi-hop questions requiring a deep understanding of multiple documents to answer. under this circumstance, multi-hop question generation (mqg), which aims to generate multi-hop questions where answering the questions requires reasoning over multiple documents, has attracted increasing interest from both the academia and industry community @xcite @xcite @xcite @xcite . existing studies @xcite @xcite on mqg usually regard semantic phrases as nodes and build graphs over them. they utilize graph neural networks (gnn) or node classification to extract reasoning chains or keywords from multiple documents. then, they use the reasoning chains and keywords to help mqg models generate relevant multi-hop questions. these approaches generally require a large amount of labeled training data to learn a well-performing mqg model and are hard to apply to low-resource situations. owing to the recent success of large language models (llms) on few-shot learning @xcite , it is intuitive to perform mqg with llms under the few-shot setting when the training resources are limited. however, prompting llms with straightforward instructions to generate multi-hop questions is not trivial to obtain satisfying performance. llms are inefficient at extracting and utilizing key semantic phrases that are essential to produce a multi-hop question. consider prompting llms with relevant information is demonstrated to be effective for many nlp tasks @xcite @xcite , a carefully designed prompting strategy with semantic information is expected to be helpful for better mqg. meanwhile, we notice that the question type provides hints in extracting key semantic phrases and producing multi-hop questions that satisfy human preferences, as illustrated in the examples in figure 1 . in with the given set of n context documents @xmath0, ..., d n } and an answer a related to c, where a is the answer to the generated question q using information from at least two documents in c. our approach consists of two parts, the first tase module f 1 (see the left of figure 2 ) extracts the question type t and semantic phrases s that provide essential information for mqg. the second part f 2 leverages the extracted t and s to construct the type-aware cot and use it to instruct llms to generate multihop questions. thus, the overall objective of our proposed framework is defined as follows: ## related work this paper is relevant to multi-hop question generation and cot. the following text presents the details of the related work in the two fields. ## conclusion in this paper, we propose tase-cot for the fewshot mqg. tase-cot extracts the question type and type-aware semantic phrases from the given documents and the answer, then utilizes them to conduct the question generation with type-aware cot. we run experiments on benchmark datasets, and the results on benchmark datasets show that our approach achieves state-of-the-art performance.