Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts (2022)

Abstract: Question-answering datasets require a broad set of reasoning skills. We show how to use question decompositions to teach language models these broad reasoning skills in a robust fashion. Specifically, we use widely available QDMR representations to programmatically create hard-to-cheat synthetic contexts for real questions in six multi-step reasoning datasets. These contexts are carefully designed to avoid common reasoning shortcuts prevalent in real contexts that prevent models from learning the right skills. This results in a pretraining dataset, named TeaBReaC, containing 525K multi-step questions (with associated formal programs) covering about 900 reasoning patterns. We show that pretraining standard language models (LMs) on TeaBReaC before fine-tuning them on target datasets improves their performance by up to 13 F1 points across 4 multi-step QA datasets, with up to 21 point gain on more complex questions. The resulting models also demonstrate higher robustness, with a 5-8 F1 point improvement on two contrast sets. Furthermore, TeaBReaC pretraining substantially improves model performance and robustness even when starting with numerate LMs pretrained using recent methods (e.g., PReasM, POET). Our work thus shows how to effectively use decomposition-guided contexts to robustly teach multi-step reasoning.

## introduction multi-step question answering (qa) is a complex problem that requires a wide variety of reasoning skills. in addition to basic reading comprehension (rc), models must connect multiple pieces of information, sometimes employ numerical and other forms of discrete reasoning, and compose these skills as needed for the question. however, question -decomposition pair teabreac instance repeat n times sample reasoning pattern questions with decompositions sample (question, decomposition) typed program executable (e.g. in python) figure 1 : teabreac dataset construction: we leverage widely available question decomposition annotations (qdmrs) for real questions from a broad range of datasets to carefully construct synthetic contexts such that answering the resulting question requires proper multi-step reasoning. these questions are further rebalanced to help teach a broad set of reasoning skills. how, then, can we teach models broad multistep reasoning skills? one way is to have greater control over the distribution of reasoning patterns and the types of input contexts models see during training-contexts that don't allow models to easily succeed via shortcuts. we observe that questions in existing datasets (henceforth referred to as "real questions") already cover a wide variety of reason-ing patterns. the challenge, then, is to teach these reasoning patterns robustly, even when they are relatively rare (e.g., 4-6 step reasoning). as a means to this end, we turn to synthetic context generation for real questions. specifically, we propose to construct contexts for real questions synthetically from scratch (instead of perturbing existing contexts), resulting in much greater control over reasoning shortcuts. further, context generation also enables us to balance out the distribution of reasoning patterns, e.g., by synthesizing additional contexts (and thereby examples) for questions from the long-tail of underrepresented reasoning patterns. our use of synthetic contexts to reliably teach broad skills is inspired by three strands of recent rc qa research. one strand has shown that skills learnt over synthetic data can indeed transfer to real datasets @xcite @xcite . a second strand has shown that perturbing the existing (natural) contexts of rc instances in a targeted fashion can reduce artifact-based reasoning @xcite . a third strand has shown that carefully constructing contexts (for synthetic questions) to have sufficient distractors can reduce artifacts @xcite . building upon these three strands, we introduce teabreac, foot_0 a teaching dataset that includes carefully constructed synthetic contexts for a broad set of real multi-step questions sourced from six existing datasets. teabreac was designed with the goals of strong control over cheatability and balanced coverage of reasoning patterns. to identify the intended reasoning, we leverage question decomposition annotations, specifically question decomposition meaning representation or qdmr annotations which are widely available for a broad set of datasets @xcite . figure 1 shows the overview of our construction process for teabreac. our approach relies on treating a question decomposition as an unambiguous typed program that can be used to generate a synthetic context and can be executed to provide an answer. to this end, we first turn natural language qdmrs into a precise typed program. we then construct a synthetic context by asserting a set of facts that relate to various parts of the multi-step question. we do this by grounding the predicates of qdmr (e.g., field goals of shayne graham in fig. 1 ) with randomly generated entities. we also add distractor statements to the context to ensure that bypassing reasoning steps results in an incorrect answer. the resulting contexts are hard to cheat on and thereby force models to learn the intended reasoning. we then add an outer loop around this process that ensures that the reasoning patterns-as measured by the program signatures of the questions-remain balanced in the final dataset. this forces models to learn a broad range of reasoning patterns instead of focusing on the few dominant ones. finally, similar to prior work @xcite , we also add simpler single-step questions to teach individual primitive skills underlying our formal programs. our experiments demonstrate that pretraining foot_1 large language models (lms) on teabreac before fine-tuning on target multi-step qa datasets results in significant improvements on multiple in-distribution evaluation sets (drop @xcite , tat-qa @xcite , iirc @xcite ), numglue @xcite by up to 13 f1 points, as well as on two contrastive evaluation sets of drop by 5-8 points. furthermore, even if we start with numerate lms already pretrained on similar past work @xcite @xcite , teabreac provides further improvement by up to 11 f1 points. interestingly, teabreac is substantially more beneficial for more complex questions (those with more reasoning steps), improving the t5-large model by about 20 f1 points on questions with 5 or more steps. more generally, we expect teabreac to be most valuable for datasets that require complex aggregation operations and their diverse compositions. in summary, we make three contributions: ## related work question decompositions have been used to build stronger models @xcite @xcite and challenge evaluation sets by modifying the questions @xcite . in contrast, our goal in this work is to use decompositions to teach broad multi-step reasoning skills to any text-to-text model by creating challenging contexts for real questions. building synthetic datasets to teach requisite skills has been considered in prior work, but limited to only numeric reasoning skills @xcite or few templated multistep reasoning patterns @xcite . even pretraining on program executions (arithmetic, logic-based, and sql-based) has been shown to help on multi-step qa tasks @xcite . in this work, we use real questions from a wide variety of datasets and show larger gains than these prior models. we even improve these prior models by fine-tuning on our dataset. we create more robust models by teaching reasoning skills via a dataset carefully designed to avoid shortcuts. past work often focuses on identifying lack of robustness via analysis @xcite or challenge evaluation sets @xcite . lastly, we define new conditions for constructing contexts for real questions with minimal reasoning shortcuts. this differs from prior work that only provides conditions to measure reasoning shortcuts in existing datasets @xcite . the "musique condition" of @xcite targets the construction of new non-cheatable multi-step datasets. we enforce this condition in teabreac and introduce two additional ones that are especially pertinent to our construction. appendix a includes additional discussion. ## teaching broad-coverage reasoning skills in a robust fashion multi-step questions come in a wide variety. some involve numeric operations @xcite , some involve assessing whether complete information is present or not @xcite , some involve tables and text @xcite , and so on. one way to surface the reasoning needed for answering these questions is to look at their decomposition into smaller reasoning steps. e.g., consider the question in fig. 1 , from what yard-line did shayne kick two field goals?. this can be decomposed as follows: list the field goals by shayne graham, identify the yard-lines for each of them, map each yard-line with the field goal and count them, and select the yard-line with two field goals. while questions in multi-step qa datasets are authored with the intent that such multi-step reasoning will be used to answer them, the context associated with the questions often allows models to cheat by taking shortcuts @xcite @xcite . e.g., if the context mentions field goals only by shayne graham and no one else, models can ignore the player name and still succeed. our key observation is that the decomposition of a question can be leveraged to carefully design a synthetic context for it that is hard to cheat, thereby allowing us to teach models a broad range of reasoning skills in a robust fashion. to achieve this, we procedurally create a large pretraining rc dataset, teabreac, by using real multi-step questions (from existing datasets) and their decompositions (available in the form of qdmrs), and carefully building synthetic contexts. qdmr or question decomposition meaning representation @xcite is a common way to represent the reasoning in many types of multi-step questions as a structured decomposition graph. qdmr has standardized operators (represented as nodes) such as select, project, group, etc., that transform their input. these are connected together to a final node which produces the answer. figure 1 shows the above example question paired with its qdmr graph. importantly, qdmrs are already available for several multi-step qa datasets. briefly, our method involves the following main steps; these are described in more detail in §4. making qdmrs more precise. to create qa instances that teach the precise reasoning in qdmrs, we need a precise and formal representation of reasoning captured in qdmrs. qdmrs, although structured, don't quite do so, as they are written in natural language and don't specify the datatypes of their inputs/outputs. since this is crucial for our approach, we convert qdmrs into formal programs with over 44 executable primitive operations along with their input/output types ( § 4.1). teaching robust compositional skills. past work has shown that compositional questions don't necessitate multi-step reasoning as datasets often have reasoning shortcuts @xcite ; chen and durrett, 2019; @xcite . to teach the reasoning reflected our formal programs robustly, our qa instances must be such that models cannot bypass the reasoning steps and still arrive at the correct answer. to achieve this goal, we create a synthetic qa instance from a questionprogram pair, where the question is the same as the original question, but the context is procedurally constructed by grounding the predicates in qdmr in a careful way such that models can't cheat their way to the correct answer. ## teabreac dataset construction the overview of teabreac construction pipeline is shown in fig. 2 . we discuss the qa instance generator in § 4.1 and the dataset generator in § 4.2. ## experiments to test the effectiveness of teabreac pretraining, we compare models directly fine-tuned on target datasets with models first pretrained on teabreac foot_7 before fine-tuning. datasets. we evaluate in-distribution performance using drop @xcite , tat-qa @xcite , iirc @xcite , and numglue @xcite . for iirc, we consider two settings: iirc-g uses only gold supporting sentences as context while iirc-r uses paragraphs obtained using a retrieval marginalization method @xcite . we evaluate robustness using the drop contrast set @xcite and the drop bpb contrast set @xcite 10 . to do this, we directly evaluate drop fine-tuned models on contrast sets. ## conclusions despite large lms' impressive reading abilities and the availability of large scale multi-step qa datasets requiring a rich set of reasoning skills, lmbased qa models do not reliably learn to use such skills for answering complex questions. in this work, we show that the greater control that synthetic contexts offer can be leveraged to create a teaching dataset where models can learn a broad range of reasoning skills in a reliable manner, especially for more complex questions.