Type-dependent Prompt C ycle QAG : Cycle Consistency for Multi-hop Question Generation (2022)

Abstract: Multi-hop question generation (QG) is the process of generating answer related questions, which requires aggregating multiple pieces of information and reasoning from different parts of the texts. This is opposed to single-hop QG which generates questions from sentences containing an answer in a given paragraph. Single-hop QG requires no reasoning or complexity, while multi-hop QG often requires logical reasoning to derive an answer related question, making it a dual task. Not enough research has been made on the multi-hop QG due to its complexity. Also, a question should be created using the question type and words related to the correct answer as a prompt so that multi-hop questions can get more information. In this view, we propose a new type-dependent prompt cycleQAG (cyclic question-answer-generation), with a cycle consistency loss in which QG and Question Answering (QA) are learnt in a cyclic manner. The novelty is that the cycle consistency loss uses the negative cross entropy to generate syntactically diverse questions that enable selecting different word representations. Empirical evaluation on the multi-hop dataset with automatic and human evaluation metrics outperforms the baseline model by about 10.38% based on ROUGE score.

## introduction question generation (qg) problem that automatically generates a question from a given document with a correct answer is a challenging and an interesting task in the field of natural language processing @xcite @xcite . with the advent of deep learning, the pre-trained language models @xcite @xcite @xcite were proposed, after which the study of natural language processing began to develop rapidly. these works not only use single-hop qa dataset such as squad @xcite , which is a representative of research on question answering (qa), but also the multi-hop qa dataset such as hotpotqa @xcite . the qa dataset consists of (context, question, answer) pairs along with a lot of qa data, that enables research on automatic question generation @xcite . most of the question generation methods evaluated questions using the single-hop qa datasets @xcite @xcite . however, in real-world situations, the questions can be very complex and sometimes require a complicated reasoning process @xcite @xcite . multi-hop qg requires combining several pieces of information and reasoning over them to derive an answer related a question, making it a dual task. multi-hop questions that can be encountered in the real world are largely divided into two types, bridges and comparisons. as shown in fig. 1 , the middle side is an example of a bridge-type question. when the question is "who played selby wall in the film that charlize theron won an academy award for?", the first thing we need to know is what film theron won the academy award. second, we should be able to obtain information about the actors who played selby wall among the actors in the movie. here, monster, the movie that connects the two, serves as a bridge. on the other hand, the comparison type shown on the right side of fig. 1 is to create a question that can be answered by comparing two objects. some of the methods for multi-hop qg transform the input text into an intermediate representation such as a parsing tree @xcite , and then convert the resulting form into a question by some well-designed templates or general rules. in @xcite , they use multi-task learning with an auxiliary loss for sentence-level supporting fact prediction. graph-based methods @xcite , 2020; kumar et al., 2019) used graph convolution networks(gcns) to capture dependencies among different pieces of information for reasoning. however, those approaches should predefine graphs only from the question and candidate answers, lacking much key information for multi-hop reasoning. to alleviate the above problems, we introduce the automated question generation to handle the lack of information for multi-hop reasoning and solve the predefined graph issue in an end-to-end manner. in this view, we propose a type-dependent prompt cycleqag, which provides additional information and loss of cycle consistency for multihop qg. at first, an intermediate task of cyclically learning qg and qa is performed before the finetuning stage. in this process, we use cycle consistency loss, and in particular, we introduce the negative cross entropy (nce), which is used to increase the lexical diversity of multi-hop questions. and in the final step, we use a prompt-based fine-tuning method that maximizes the information obtained from the intermediate task by giving information that can be provided according to the types of questions (eg, type and answer related words). using the proposed model, we can generate complex questions by an end-to-end manner with semantically similar but diverse vocabulary. we use the hotpotqa distractor setting and perform experiments with a multi-hop qa dataset. the proposed model outperforms the baseline models in automatic evaluation results such as rouge @xcite for quantitative evaluation. however, qualitative part of the multi-hop question is evaluated using fluency, relevance, answerability, complexity, and diversity for human evaluation. we evaluate the diversity of vocabulary through qualitative evaluation, and show examples to prove this. question generation. the ultimate goal of the qg task is to automatically generate questions from texts or knowledge data. with the advent of machine reading comprehension datasets such as squad and pre-trained language models, qg research is conducting the multi-hop reasoning research that deals with more complex and inference-demanding thorny questions, to mimic humans @xcite @xcite . @xcite proposed a whole generator evaluator network for generating questions by creating an entity graph to integrate various entities scattered in the texts. @xcite proposed a multi-hop qg method that used predefined basic operators to search, generate, and aggregate information of each input according to the types of inputs. they also defined and used six inference types of reasoning graphs. in particular, an off-the-shelf template was used for generating a comparison type question that compares two subjects. although such pre-defined templates or structured models can generate accurate questions for given data, they can be fatal in both quantitative and qualitative aspects when new complex data are given. to overcome some of these issues, we propose a new end-to-end approach to generate multi-hop questions. dual task of qa and qg. qa and qg are separate but closely related tasks. in @xcite , they jointly train the two tasks by exploiting the probabilistic correlation between qa and qg. in particular, the parameterized model was jointly trained to minimize the loss function according to the constraints. @xcite used question generation as an auxiliary task to improve the text-based qa task. they calculated the relevance score between the input question and the answer candidates, and chose the highest relevance score as a correct answer. @xcite generated additional training instances to further improve the qa model in @xcite , each consists of a question, an answer, and a label for a category. in addition, the question was created by clamping the answer part and providing the answer to the qg model. many efforts have been made to improve each module by using qa and qg together. in this view, we not only propose a method of using cycle consistency to increase the robustness of qa and qg but also introduce the nce that increases the diversity of questions. ## proposed model the proposed model for question generation includes an intermediate task execution phase before the fine-tuning step. in the intermediate task, qa and qg are trained to have cycle consistency, where question paraphrasing and similarity are additionally used to increase the robustness of the question generation. we focus on using the multihop qg and qa together as a supplement to increase the performance of qg. we define qa as the top-down approach to find a right answer, and qg as the bottom-up approach to use abundant information from entities or sentences. the overall framework of our proposed model is explained in fig. 2 . ## conclusion in this work, we propose type-dependent prompt cycleqag with cycle consistency. since multihop qg needs to know more diverse information because it needs to gather more scattered pieces of information for generating a question, we introduce the nce for the first time in the qg task. also, we demonstrate that the intermediate task is effective in the qg task. furthermore, we show a significant performance improvement by using prompt-style fine-tuning to make the most of the information obtained from the intermediate task. the experiments show that the proposed model outperforms in all automatic evaluations comparing with the existing text-based multi-hop model and several qg models. although we use only multi-hop, single-hop-based datasets, experiments can be performed without additional datasets later using the type-dependent prompt cycleqag method. in other words, it is possible to learn qa and qg models using unsupervised learning. in the future, we would like to investigate a model that generates questions and answers by itself enough to imitate humans from knowledge through self-cyclic learning that is less influenced by data.