Successive Prompting for Decomposing Complex Questions (2022)

Abstract: Answering complex questions that require making latent decisions is a challenging task, especially when limited supervision is available. Recent works leverage the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting by demonstrating how to output intermediate rationalizations while solving the complex question in a single pass. We introduce “Successive Prompting” where, we iteratively break down a complex task into a simple task, solve it, and then repeat the process until we get the final solution. Successive prompting decouples the supervision for decomposing complex questions from the supervision for answering simple questions, allowing us to (1) have multiple opportunities to query in-context examples at each reasoning step (2) learn question decomposition separately from question answering, including using synthetic data, and (3) use bespoke (fine-tuned) components for reasoning steps where a large LM does not perform well. The intermediate supervision is typically manually written, which can be expensive to collect. We introduce a way to generate synthetic dataset which can be used to bootstrap model’s ability to decompose and answer intermediate questions. Our best model (with successive prompting) achieves an improvement in F1 of ~5% when compared with a state-of-the-art model with synthetic augmentations and few-shot version of the DROP dataset.

## introduction compositional reading comprehension datasets like hotpotqa @xcite and drop @xcite have inspired a range of model architectures that learn to answer complex questions with weak supervision from the final answer. one recent direction is to leverage large language models (lms) to solve compositional tasks with very few examples by generating latent reasoning steps before answering the question @xcite who kicked the longest field goal in the first half? q: what are all the field goals in first half? a: 42-yard q: who kicked the 42-yard field goal? a: matt bryant there are no more questions left to ask. the final answer is matt bryant figure 1 : example decomposition used by successive prompting's question decomposition and question answering stage on a drop example. the model iterates between predicting a simple question to ask and answering the simple question. @xcite . given a complex question, this approach first finds nearestneighbor training examples from a dataset of (question, reasoning, answer) triples and then concatenates them to create an input for the lm. a large lm is then prompted with this input to generate the intermediate reasoning steps needed, while answering the complex question in a single pass. while promising, this approach discards many of the benefits of prior approaches to this task @xcite by coupling the supervision for question decomposition to the supervision for performing the intermediate steps. moreover, its non-modular nature does not allow using alternate symbolic reasoning engines in cases where they perform better than lms. additionally, the model gets exposed to only a single set of incontext examples, selected based on their proximity to the complex question, which may not contain optimal supervision for the intermediate steps that need to be taken. we propose "successive prompting", where we iteratively decompose the complex question into the next simple question to answer, answer it, and then repeat until the complex question is answered (figure 1 ). each of these steps is performed with separate a query to the lm. since the decomposition and answering steps are performed separately, we can decouple the supervision of each step, providing two primary benefits. first, when performing in-context learning, we get multiple opportunities to select different in-context examples, which can be tailored to the particular decomposition or answering step being performed, instead of selecting a single set of examples based only on the complex question. second, when fine-tuning (with or without in-context examples @xcite ), we can provide training examples for each step independently, so the model only has to learn to perform one step at a time. this decoupling additionally allows us to judiciously inject synthetic data into the learning process, e.g., to help the model answer a particular kind of simple question that it could not previously answer, or a new reasoning composition it did not know how to decompose. because the steps are separate, we can isolate model failures and develop synthetic approaches to fill in the gaps. it also allows us to replace the lm with other, purpose-built components to perform symbolic reasoning when appropriate @xcite @xcite . we demonstrate the utility of successive prompting using a few-shot variant of the drop dataset @xcite , selecting 300 examples for training (either fine-tuning or in-context example selection). these 300 examples are manually annotated with simple qa pairs as decompositions. we find that performance of all models is quite low in this few-shot setting, so we develop a synthetic data generator that produces complex questions with their decompositions from semi-structured wikipedia tables @xcite . this synthetic data provides not just complex question supervision, but also supervision for the intermediate steps. we augment this data with the 300 (complex) training examples and their decompositions from drop. in this few-shot setting, our best performing successive prompting model shows a ∼5% improvement in f1 when compared to state-of-the-art model on drop. the code and data are available at https://github.com/ddua/succesive_prompting ## decomposing complex questions the goal of compositional question answering is to answer a complex question q in the context of a passage p (together denoted as x) by reasoning through latent sequential decisions @xmath0, z 2 , ..., z s to reach the final answer, y. many models have been proposed to accomplish this with varying amounts of supervision and interpretability. in prompting methods like chain-of-thought (cot, @xcite the latent steps are supervised, interpretable sentences; in other models these latent steps might be a program @xcite or even just the (unsupervised) hidden states in the model @xcite we focus on models that take in-context examples and produce a discrete, language-encoded z, with cot being the primary exemplar. we write the general form for cot, given an input x, a language model encoder l and n in-context examples obtained from querying an index i-each containing a triplet of passage with complex question (x n ), latent steps (z n ) and final answer (y n )-as follows: ## synthetic dataset any method that prompts lms to produce intermediate reasoning steps to answer complex questions needs some amount of supervision for those reasoning steps. this kind of annotation can be expensive to collect and often requires expert knowledge. 3 ). prior work has typically relied on a small handful of manually-written example decompositions. we find that such small collections lead to very poor performance on a dataset as varied as drop, even for large models. to mitigate these data issues, we propose a way to synthetically generate complex questions and their decompositions using semi-structured data which is easy to parse. we show that we can bootstrap model learning with this out-of-domain, synthetically generated data so it can adapt better when fine-tuned with limited in-domain supervision. ## experiments and results the drop dataset contains a variety of reasoning compositions which are not uniformly distributed. in order to get a fair representation of drop examples, we first embed the examples using a sentence embedding method trained on the qqp dataset @xcite . we then use cosine similarity to get the top-50 nearest neighbor questions for each training example. the connection graph between each training question to its neighbors is then used to obtain 300 questions that cover the majority of the training data, via the vertex cover algorithm. we manually annotate these 300 examples with decomposed qa pairs in the same format as our synthetic data (figure 3 ). for synthetic examples, since we know the reasoning types, we uniformly sample example demonstration from each reasoning type. ## related work prompting methods prompting was introduced as a way to test the reasoning capabilities of large language models @xcite . in followup works @xcite @xcite prompting techniques have been used as a mechanism to supervise the model decision with few demonstrations as a conditioning context to guide its predictions on an unseen example. works like chain-of-thought reasoning @xcite especially focus on compositional questions where they provide a chain of reasoning as demonstrations. in concurrent work, least-to-most prompting @xcite takes a similar view as ours to break down the problem into sub-problems. however, in successive prompting the question decomposition and answering stages are interleaved, unlike least-to-most where the problem is first reduced into subproblem and then executed in a sequence. in our method, the next question prediction has access to previously answered sub-questions, which is useful in questions that need long chain referencing. other contemporaneous works @xcite use very large language models (more than twice the size we used) and show better few-shot generalization. works like @xcite have shown the importance of having the right in-context examples for downstream performance leading to works that learn to retrieve relevant in-context examples @xcite . non-symbolic methods most non-symbolic methods are sequence-to-sequence models trained on a large amount of question answering data @xcite . ## conclusion we present a way to successively decompose complex questions into simple qa pairs, which allows for modular qd and qa systems that can be trained and queried independently. when performing in-context learning, we showed that successive prompting yields an improvement of 4.6 f1 over chain-of-thought prompting. when replacing just the in-context qa module with a fine-tuned one, which is adept at handling list type questions, we further improve the overall performance by 9.5 f1. we believe that modular systems that decompose and delegate tasks to the most appropriate model, whether that is a large lm or a tailored component, are more effective at solving complex tasks than trying to have a large lm solve the entire task on its own. successive prompting shows one way this decomposition and delegation can be done. there is also significant difficulty in choosing a level of granularity for decomposition. if a large pretrained model can directly answer a question as complex as "what was barth's second field goal?", we should let the model answer the question instead of trying to decompose it further. the right granularity for the decomposition thus depends on the capabilities of the underlying model, and those capabilities are rapidly changing as newer and larger pretrained models are released. there is the possibility that newer model iterations will not need any decomposition to answer the complex questions covered by our synthetic data generator, making that generator obsolete. however, it seems unlikely that pretrained models will be able to handle all complex scenarios in the near future, so the ideas of successive prompting and generating synthetic data to bridge reasoning gaps should still be applicable even when our particular application of them becomes obsolete. this method also increases the computational requirements for answering complex questions, as instead of making one query to a large lm, successive prompting makes many queries to answer a single question.