Reasoning Circuits: Few-shot Multi-hop Question Generation with Structured Rationales (2023)

Abstract: Multi-hop Question Generation is the task of generating questions which require the reader to reason over and combine information spread across multiple passages employing several reasoning steps. Chain-of-thought rationale generation has been shown to improve performance on multi-step reasoning tasks and make model predictions more interpretable. However, few-shot performance gains from including rationales have been largely observed only in +100B language models, and otherwise require large-scale manual rationale annotation. In this paper, we introduce a new framework for applying chain-of-thought inspired structured rationale generation to multi-hop question generation under a very low supervision regime (8- to 128-shot). We propose to annotate a small number of examples following our proposed multi-step rationale schema, treating each reasoning step as a separate task to be performed by a generative language model. We show that our framework leads to improved control over the difficulty of the generated questions and better performance compared to baselines trained without rationales, both on automatic evaluation metrics and in human evaluation. Importantly, we show that this is achievable with a modest model size.

## introduction recently, there has been a surge of interest in the nlp community in the idea of providing supervision to language models(lms) in the form of human-written rationales @xcite @xcite @xcite which explain why and how the target label is arrived at. using human-written explanations as an intermediate step has been shown to improve performance on a variety of predictive tasks, compared to the cases where no rationales are provided @xcite . however, rationales are expensive to collect through manual annotation at a large scale. explanations can take several forms, such as textual highlights, free-text explanations and structured explanations. in this work, we focus on the latter two. by rationales we refer to several structured sentences enumerating intermediate steps of reasoning required to solve a multi-step reasoning problem before producing the target text. in chain-of-thought rationale generation paradigm @xcite , lms learn to generate rationales -a step toward explainable nlp models. prompting lms with few-shot rationale examples has been shown to improve performance for multi-step reasoning tasks compared to standard prompting without rationales @xcite . however, this effect is only observed in extremely large language models (xllms) with +100b parameters @xcite . at the same time access to xllms is limited in the research community due to costs and infrastructure required to fine-tune and inference them. in many cases these models are never released publicly. in essence, supervision from rationales is a richer signal compared to supervision from only target labels, especially for multi-step reasoning tasks. however, only xllms have been shown to capture this signal in few-shot regimes. in this paper we assume large-scale rationale annotation to remain unavailable since it is tedious and arguably more expensive to generate than standard target label annotation. this leads to a currently unaddressed challenge of solving multi-step problems by smaller lms than xllms in a few-shot regime. this work deals with the complex task of multihop question generation(mqg) where, given multiple passages and a pre-defined answer, the objective is to generate challenging questions that cannot be answered only from reading a single passage, this task requires many steps of reasoning to ac-complish and we further constrain ourselves to the case where supervision available is restricted to a few number of labelled examples. our contributions can be summarized as follows. we propose a new framework called reasoning circuits applicable specifically for the often encountered constraints faced in real-world where: 1. large-scale annotation is not possible or available, only a limited number of examples of a multistep reasoning problem are available. 2. access limited to modest neural compute infrastructure that can support training models up to a maximum of 3 billion parameters. 3. budget for rationale annotation is limited. in this work we apply this framework to mqg task in a few-shot setting. this entails identifying reasoning steps human annotators employ to generate multi-hop questions and codifying them into a structured rationale annotation scheme, and manually producing rationale annotations for the few examples, capped at a maximum of around 200 examples. a generative model is then fine-tuned with a mixture of tasks where each "task" refers to a single smaller step of reasoning derived from the structured rationales designed for the mqg task. we report improvements over baselines where no rationale was employed on automatic evaluation metrics as well as human evaluation. we also show reduced gap in performance between our system only trained with approimately 150 examples(training and validation combined) and prior art that has been trained with 9,000 to 90,000 examples without rationales on automatic evaluation metrics. ## structured rationales for multi-hop question generation hotpotqa dataset @xcite is one of the most widely used benchmarks of multi-hop question answering and consists of broadly two types of questions: bridge-entity questions and comparison questions. in bridge-entity questions, which constitute 75% of the dataset, annotators get pairs of passages where at least one entity (called the bridge entity) is present in both passages. in comparison questions, annotators are provided a pair of passages about entities drawn from a similar theme such as musicians, authors, films, plants among several categories. a comparison question typically compares some quality of the central entities in the two passages. question types found in this dataset cover 5 out of 6 total sub-types of multi-hop questions as identified in recent survey on multihop question answering and generation @xcite originally identified in @xcite with the only exception of commonsense reasoning. below, we identified the reasoning steps that annotators needed to follow in order to create the questions of each type. ## rationale steps as a mixture of tasks the step-by-step nature and clearly defined structure of the rationales identified above motivated us to formulate this multi-step problem into a mixture of tasks. a step of reasoning is treated as a task, only the information required to perform the reasoning step is treated as the input and the result of the reasoning step is the expected output. we formalise bridge multi-hop question generation into nine separate smaller steps of reasoning and we identify 3 steps of reasoning for generating comparison multi-hop questions. additionally, a reasoning step should identify whether a given pair of passages and answer are more suitable to generate a bridge question from or a comparison question instead. for this purpose we create the first task of predicting the question type. these reasoning steps or tasks can be categorized into: 1. control tasks where the outcome is a control variable whose value decides what reasoning path to follow and 2. generative tasks which essentially generate free-form text to be treated as input for later reasoning steps or as the final output. we describe each of these tasks below. bridge type 2 example passage 1: ... the kennedy compound consists of three houses on six acres ... on cape cod ... passage 2: hyannisport club: ... the course is located adjacent to the kennedy compound and the kennedy family have long been members of the club. ... answer: kennedy compound bridge entity: kennedy compound answer entity same as bridge entity: yes statement extracted from passage 1: the kennedy compound consists of three houses on six acres on cape cod. statement extracted from passage 2: hyannisport club is located adjacent to the kennedy compound. combined statement: hyannisport club is located adjacent to the kennedy compound that consists of three houses on six acres and on cape cod. contract answer from combined statement: hyannisport club is located adjacent to a certain property that consists of three houses on six acres and on cape cod. multi-hop question: hyannisport club is located adjacent to which property that consists of three houses on six acres and on cape cod? bridge type 1 example passage 1: romy ruyssen is a french mixed martial artist. she ... and headlined the first invicta fighting championships event against marloes coenen ... passage 2: marloes coenen ... is a retired dutch mixed martial artist. ... answer: invicta fighting championships bridge entity: marloes coenen answer entity same as bridge entity: no statement extracted from passage 1: romy ruyssen headlined the invicta fighting championships event against marloes coenen. statement extracted from passage 2: marloes coenen is a dutch mixed martial artist. combined statement: romy ruyssen headlined the invicta fighting championships event against marloes coenen who is a dutch mixed martial artist. contract bridge entity from combined statement: romy ruyssen headlined the invicta fighting championships event against a dutch mixed martial artist. contract answer entity from previous statement: romy ruyssen headlined a certain event against a dutch mixed martial artist. multi-hop question: romy ruyssen headlined which event against a dutch mixed martial artist? figure 1 : two examples of bridge rationales that lead to the creation of multi-hop question. example 1 shows rationale annotation when answer span is not found in both passages. example 2 shows another of example of rationale annotation when the answer is present in both passages, so that the step 4 is skipped. the highlights in green and yellow show the answer and bridge entities, in the second example they are the same. in the penultimate steps, the lighter highlights indicate the substitution of bridge and answer entities with a common noun preceded by "certain". in the last step typically a wh-word substitutes "certain" or "some" words, however more radical transformations also take place in our annotations. ## experiments we conduct all our experiments on t5-3b v1.0 model with 3 billion parameters foot_0 . as a baseline, we consider tuning the same t5-3b model to directly generate multi-hop questions, without any rationale input and provided with only the two passages and answer as input. for the baseline we use a simple prompt in appendix a. for the reasoning circuit experiment we utilise rationale prompts described in appendix a to generate a mixture of task examples from our few-shot annotation. we use a learning rate of 2e-5 with no warm-up and keep constant learning rate throughout. we train for 35 epochs or 5000 steps whichever is higher. for fair comparison, we follow the data splits similar to sggdq-dp @xcite , qa4qg @xcite and sggdq-dp @xcite to get 90,440 training examples and 6,072 test examples respectively, note however that instead of using the entire test set as validation set as done in prior work we only validate with same or lower number of examples as that used for training, see section 4.4 for details. we use two settings as input to the encoder: 1. the original training data in hotpotqa, in which each question is paired with two long documents, and 2. a pre-processed version of the data where only supporting sentences required to answer the gold question are kept. we conduct experiments with 8, 16, 32, 64, 128 training examples where 75% of the examples are drawn from the bridge-type while the remaining 25% examples are of the comparison type, this is done to mimic the distribution of question types in the original hotpotqa dataset. the number of validation examples is equal to the number of training examples until the 32-shot experiment. after this the number of validation examples gets capped at 32 bridge type questions and 24 comparison type questions. we also conduct an experiment with 148 training examples that constitute all collected annotations. we tune on the average of bleu1, bleu2, bleu3 and bleu4 @xcite , me-teor (lavie and agarwal, 2007), and rouge-l @xcite scores on our few-shot validation sets. ## discussion and future work through automatic and human evaluations we show that larger language models generate similar questions to reference questions with orders of magnitude less labelled data. the proposed approach also is found to generate a much higher percentages of multi-hop questions than the baseline. one avenue of future work is in the area of selftraining. self-training, involves generating predictions from a weaker model on unlabelled data and using these predictions as additional training data, where the training set now includes the silver predictions on unlabelled data. self-training may be detrimental for or not improve overall model performance strongly especially when the task is hard for the weaker model @xcite . since prediction errors of the weaker model in the silver annotations further reinforce wrong predictions during self-training. in reasoning circuits, silver predictions on unlabelled data for many steps of reasoning can be approximately validated with simple heuristics. for instance, predictions from task 9 can be verified by checking whether they still contains the answer and bridge spans or not, if they do then these predictions can be deemed unfit for self-training. filtering prediction errors from the initial trained weaker model on unlabeled data should lead to stronger improvements from self-training compared to vanilla self-training. ## conclusion in this paper, we propose reasoning circuits, a new framework suited to real-world scenarios where the nlp task at hand requires multiple steps of structured reasoning, with only a limited number of available labelled examples, and a small annotation budget, also only a modest deep learning computational infrastructure/budget is accessible. in this work, we apply this framework to the task of fewshot multi-hop question generation which fits all these criteria. we identify structured multi-step rationales that break down this problem into many discrete reasoning steps. each step in these rationales is treated as a single "task" within a mixture of similar "tasks". the individual tasks can be categorized into control tasks, which control the flow of information between tasks, and generative tasks, that generate free-form text for successive tasks in the reasoning circuit. the framework is relatively easy to implement, since only a single generative model is fine-tuned with a mixture of all reasoning steps; at inference time, the same model can generate all reasoning steps sequentially. we show that fine-tuning with only around 64 to 128 labelled rationale examples with our approach is enough to improve automatic evaluation metrics compared to a baseline trained without rationales on the hot-potqa dataset. more importantly, with human evaluation, we find that this framework can strongly improve the central objective of multi-hop qg, to generate challenging questions which cannot be answered from reading only a single passage.