Chain-of-Questions Training with Latent Answers for Robust Multistep Question Answering (2023)

Abstract: We propose Chain-of-Questions, a framework that trains a model to robustly answer multistep questions by generating and answering sub-questions. We obtain supervision for sub-questions from human-annotated question decomposition meaning representation (QDMR), but QDMR does not include annotated answers to sub-questions. To overcome this technical challenge, we treat sub-answers as latent variables and infer them with a novel dynamic mixture of Hard-EM and MAPO. Chain-of-Questions is effective and robust, greatly outperforming strong neuro-symbolic methods by 9.0 F1 on a DROP contrast set and GPT-3.5 by 24.3 F1 on a HotpotQA adversarial set.

## introduction multistep question answering (qa) poses a reasoning challenge that current state-of-the-art qa models have not fully addressed. strong finetuned qa models like unifiedqa @xcite can achieve impressive results on various qa tasks through multitask training, but exhibit subpar performance on multistep reasoning. moreover, because some multistep reasoning benchmarks contain annotation artifacts or reasoning shortcuts @xcite , dedicated models trained on these benchmarks often have much lower f1 performance on contrast sets @xcite and adversarial sets @xcite , indicating their lack of robustness. prior research has attempted to tackle this challenge with various question decomposition strategies to explicitly incorporate reasoning chains into the question answering process. however, as we show in our experiments, existing methods @xcite ) that perform explicit reasoning steps still suffer from robustness issues. moreover, multi-step reasoning methods are often engineered for a specific domain or type of multistep qa @xcite , and thus cannot be easily extended to other multistep qa settings. prompting methods @xcite have shown promise in generating multistep solutions to questions, but they require very large language models (lms) as well as careful prompt engineering, and still lag behind fine-tuned methods @xcite . to develop a robust multistep qa system, we propose a novel framework, chain-of-questions training with latent answers. our framework trains a model to generate sub-questions and their corresponding sub-answers one at a time, as shown in fig. 1 , then aggregates those sub-answers to answer the original question. to define an appropriate set of sub-questions, we use question decomposition meaning representation (qdmr), an existing dataset with human-annotated sub-questions for questions from multiple multistep qa benchmarks. while qdmr is helpful, it only contains annotated sub-questions, not sub-answers, which makes training a qa system to generate sub-answers technically challenging. we view the sub-answers in the intermediate steps as latent variables, and apply hard-em @xcite to optimize these latent variables during training. to further improve performance, we use a memory buffer to store trajectories with high f1 score, inspired by memory-augmented policy optimization (mapo; @xcite , previously used for semantic parsing. because starting with mapo alone does not converge well, we design a dynamic loss function that combines the hard-em and mapo objectives for fast improvement at the beginning and better final convergence. we conduct experiments on drop @xcite , hotpotqa @xcite , and their contrast and adversarial sets to evaluate the performance of our proposed chain-of-questions framework. on the contrast set of drop, chainof-questions outperforms neuro-symbolic baselines by 9.0 on f1 score, and outperforms chainof-thought on gpt-3.5 by 16.8 despite using a much smaller model (t5-large, 770m parameters). on the adversarial set of hotpotqa, chain-of-questions outperforms longformer by 5.5 on f1 score, and outperforms chain-of-thought on gpt-3.5 by 24.3. our experimental results demonstrate that chain-of-questions successfully leverages existing qdmr annotations to train an effective and robust multistep qa model. ## background and related work we introduce the multistep qa benchmarks we use, as well as other methods using question decomposition during training and prompting. multistep qa benchmarks. we focus on two popular multistep qa benchmarks-drop and hotpotqa. drop @xcite focuses on questions that require discrete and symbolic reasoning. most of its questions require multiple steps of retrieval and numerical execution. hotpotqa @xcite ) contains 2-hop questions over 10 paragraphs. other work has constructed contrast and adversarial sets to evaluate the robustness of models trained on these datasets. @xcite created a contrast set for drop (drop-cs) by modifying test instances in ways that often change the correct answer. hotpotqa-adv @xcite adds adversarial paragraphs that do not change the correct answer but fool models that rely too heavily on reasoning shortcuts. we experiment on drop, hotpotqa, and their robustness evaluation sets. training with question decomposition. @xcite introduce qdmr, a humanannotated question decomposition format and dataset. since qdmr's for each question were annotated without looking at evidence passages, the qdmr dataset does not include sub-answers to any sub-questions within each qdmr. subsequent work has used qdmr to help train qa models. teabreac @xcite uses the qdmr decomposition graph to generate a large multistep qa dataset with synthetic contexts for pretraining. we show that teabreac has complementary benefits with chain-of-questions, which adds explicit multistep reasoning at inference time. @xcite train a model to generate qdmr's and provide them as context to a single-step qa model. unlike this model, our model explicitly attempts to generate sub-answers of qdmr sub-questions. we compare with a single-step baseline similar to @xcite , and show that learning to generate sub-answers improves performance. other work on multistep qa, such as decom-prc @xcite , onus @xcite and rerc @xcite , generate a decomposition with one model and the answers for the subquestions with another model, although none of these use qdmr. these methods require a singlestep qa model trained with other qa data, whereas our approach does not. moreover, they rely on entity matching to decompose questions; such approaches do not naturally extend to tasks requiring forms of multistep reasoning that are not entitycentric, such as numerical reasoning. neuro-symbolic methods such as bert-calculator @xcite and nerd @xcite generate functional programs for multistep numerical reasoning. however, they require the model to generate accurate programs in a single run, without observing the results of intermediate stages of computation. this process can make them more susceptible to learning simple reasoning shortcuts, compared with chain-of-questions. question: which side suffered more casualties, the federales or the americans? context: 12 us troops … question: which side suffered more casualties, the federales or the americans? context: 12 us troops … run #1: input output [qdmr] return casualties of the federales [qdmr-ans] {45} [qdmr] return casualties of the federales [qdmr-ans] {45} [qdmr] return casualties of the americans [qdmr-ans] {12} run #2: question: which side suffered more casualties, the federales or the americans? context: 12 us troops … [qdmr] return casualties of the federales [qdmr-ans] {45} run #3: [qdmr] return casualties of the americans [qdmr-ans] {12} training ⓵ sample top-k candidates, select the best to compute the hard-em loss replay buffer ⓶ select out-of-memory examples in top-5, sample at most 5 in-memory examples from the buffer, compute the mapo loss ⓷ from the top-k samples, add those with f1 no less than 0.8 to the buffer beam search [qdmr] return which side was more of {45}, {12} [end-qdmr] [qdmr-ans] {federales} append append figure 2: chain-of-questions framework and its training process with hard-em and mapo. in the left panel, the blue box contains the input question and context, the pink box shows the intermediate sub-questions and sub-answers, the orange box is the final sub-question and final answer. words in red braces {} are sampled as latent variables, and the other words are learned with supervision. in the right panel, during training, we first select the best of top-k candidates to compute the hard-em loss, then we combine these candidates with samples from the buffer to compute the mapo loss. finally, we add high-reward candidates into the buffer. ## problem formulation we define the notations related to the question decomposition annotation qdmr, as well as the multistep qa training and testing setups. ## proposed methods we discuss how to use the qdmr annotation and the qdmr parser, as well as how to train the qa model with hard-em and reinforcement learning in this section. we combine question decomposition and latent variable learning to construct a generalizable multistep reasoning framework. ## experiments we present our experimental setup and show coq outperforms other multistep fine-tuning or prompting baselines over multiple benchmarks. ## discussion we present chain-of-questions (coq), a robust sub-question generation and answering framework that shows strong performance on drop and hot-potqa. coq uses a combination of hard-em and mapo for training, effectively optimizing the latent variables associated with sub-answers of intermediate questions. we envision multiple directions for future work. coq requires supervision from qdmr; other families of rl methods we did not explore may be used to reduce our reliance on this supervision, and instead allow the model to learn appropriate decompositions from scratch. on the other hand, we could also explore using different question decompositions, such as ones generated by llms like gpt-3.5. either approach could help us extend coq to other multistep reasoning datasets with no qdmr annotation. similar to drop, finqa @xcite consists of numerical reasoning questions over financial data. in a similar format as hot-potqa, musique @xcite ) contains 3-hop and 4-hop retrieval questions. ropes @xcite requires complex multistep reasoning between a background context paragraph and situation context paragraph. we could either train models on these datasets if we can eliminate our reliance on qdmr data, or test whether models trained with coq can transfer well to these other datasets.