Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts (2022)

Abstract: Explicit decomposition modeling, which involves breaking down complex tasks into more straightforward and often more interpretable sub-tasks, has long been a central theme in developing robust and interpretable NLU systems. However, despite the many datasets and resources built as part of this effort, the majority have small-scale annotations and limited scope, which is insufficient to solve general decomposition tasks. In this paper, we look at large-scale intermediate pre-training of decomposition-based transformers using distant supervision from comparable texts, particularly large-scale parallel news. We show that with such intermediate pre-training, developing robust decomposition-based models for a diverse range of tasks becomes more feasible. For example, on semantic parsing, our model, DecompT5, improves 20% to 30% on two datasets, Overnight and TORQUE, over the baseline language model. We further use DecompT5 to build a novel decomposition-based QA system named DecompEntail, improving over state-of-the-art models, including GPT-3, on both HotpotQA and StrategyQA by 8% and 4%, respectively.

## introduction answering questions often involves making educated guesses: we do not necessarily have accurate facts but can use common sense to understand what most questions are asking and what kinds of knowledge are needed. for example (see fig. 1 ), we can understand the question "is albany, ga more crowded than albany, ny?" involves comparing the size and population of two cities without knowing the specific numbers to compare. it is often desirable to make such a decomposition because a city's population is usually much easier to acquire than a direct answer to the original question. existing approaches to end-to-end questionanswering (qa) assume that pre-trained language is albany, ga more crowded than albany, ny? tokyo is a crowded city. … …tokyo is a small city that faces over-population issues… doc group a doc group b parallel news distantly-supervised model for query understanding related to the area size related the number of people better latent representation for decomposition albany, ga has 72,000 people. albany, ga is 55 mi 2 in size… answer: no actual question figure 1: an example of how parallel news documents can be used to train a model that is capable of making educated guesses on what the question is asking, and how it may help to derive a better answer. in this paper, 1 we attempt to bridge the gap of reporting biases, which hinders lms from learning implicit connections between questions and decompositions (e.g., "crowded" and "population"). we do this through intermediate pre-training on distant supervision, following recent attempts to distill common sense into transformers via distant supervision @xcite . specifically, we use collections of article pairs with parallel descriptions of similar news events from different angles as our distant supervision. as illustrated in fig. 1 , large collections of comparable texts ( §3.1) contain a wide variety of commonsense implications needed for decomposition. we extract 2.6 million sentence pairs ( §3.2) for this purpose, and then train decompt5 ( §3.4), a t5 @xcite model that is further-pre-trained on our distant supervision instances. in §5, we show that decompt5, while simple, serves as a more effective model than the base language model on general question understanding through experiments on overnight @xcite and torque @xcite semantic parsing tasks, achieving 22-32% absolute improvements. since smaller language models cannot sufficiently memorize facts (e.g., the exact population of albany), they are often used in conjunction with external knowledge retrieval for more complicated tasks such as qa. to bridge this gap, we design a novel qa pipeline using decompt5 at its core ( §4). the full model and pipeline, called decompentail, first generates explicit question decompositions, then makes factual corrections on the decomposed statements with gpt-3. as a final step, decompentail employs an entailment model that derives the final answer with the generated decomposition as the premise and the question and candidate answer as the hypothesis. in §7, we show that decompentail, despite its relatively small size, can generate good decomposition chains and outperforms gpt-3 on both strate-gyqa and a binary portion of hotpotqa by 4% and 8%, respectively. this shows that we can improve baseline language models or even much larger reasoners with explicit decomposition, which has the advantage of enhanced interpretability and transferability. on the other hand, decompt5 only relies on supporting fact annotations instead of explicit reasoning steps, which is more common in datasets and can be better applied for joint learning. contributions. in summary, our contributions are three-fold: 1) we collect distant supervision from parallel news to encourage robust semantic understanding for question decomposition, 2) we train a general decomposition model called decompt5 with our collected distant supervision that significantly improves over the baseline language models on intrinsic evaluations, and 3) we propose a decomposition-based qa pipeline called decompentail that relies on decompt5 at its core. we show that decompentail has improved performance over several baselines on decomposition-based qa. ## related work our work relates to the literature on multi-hop reasoning @xcite , which has recently produced new annotation schemes (e.g., qdmr from @xcite and strategy question decomposition annotations from @xcite ) and datasets for complex reasoning that target explicit model decomposition @xcite @xcite . we take inspiration from systems that build explicit reasoning paths, such as semantic parsers @xcite , and their modern variations @xcite @xcite . @xcite ; @xcite aim to build general question decomposition models, however, focusing on simpler tasks than our study. our work is also related to sentence-pair datasets collected from comparable texts @xcite @xcite . compared to most of these works, our extraction does not use human annotation, and produces clean and diverse signals for question understanding. previous work has also discussed using largescale further pre-training to improve language models @xcite @xcite . we follow a similar general scheme with novel extraction sources and focus on a general representation for questions, which resembles some idea in existing work @xcite . ## distant supervision for decomposition in §3.1, we describe our intuitions on why question decomposition is important and what is missing from existing pre-trained language models for them to do well. following that, we describe how we collect distant supervision signals to improve the process of learning to decompose in §3.2. in §3.4, we propose decompt5, a t5-based model that is further pre-trained on the collected distant supervision using standard seq-to-seq training objectives. ## decomposition-based qa pipeline our proposed model decompt5 has two uses: it can be directly fine-tuned on tasks that require query understanding and decomposition, as we later show in §5. it can also be applied in a pipeline fashion to produce meaningful decompositions that help with more complicated tasks that require external knowledge, such as general question answering. this section focuses on the design challenges and choices for such a qa pipeline. we first explain the intuitions in §4.1, then describe and propose decompentail in §4.3. we evaluate our proposed qa pipeline in §7. ## intrinsic experiments in this section, we conduct two intrinsic experiments with decompt5 that directly evaluate its general decomposition capability through finetuning task-specific input/output sequences. we compare with t5-large as it is the base lm, and such a comparison reveals how much we improve through pre-training with parallel news distant signals. we do not compare our model with gpt-3 because few-shot learning might not be enough for it to learn the complete grammar of different tasks' decomposition. this is an advantage of fine-tuning relatively small but capable models over directly using much bigger ones in few-shot settings. all experiments use a 5e-5 learning rate, and they are repeated and averaged with three seeds. foot_4 and the formal parsing, it annotates an intermediate "canonical" form with semi-formal language, which has recently been used for work on textbased semantic parsing with transformers @xcite that we take inspiration from. for example, the annotated intermediate form of "biggest housing unit that allows dogs" is "housing unit that has the largest size and that allows dogs". we evaluate the performance of mapping natural language queries to such intermediate forms with three domains that contain 3.8k training instances and 972 test cases. both models are trained with three epochs. we use the same inference for both t5large and decompt5, which generates ten candidates using beam search. following previous work, the generation is also constrained by possible "next words", that is, we assume that we know controlled output space beforehand. ## sanity check experiments in this section, we describe the details of the sanity check experiment mentioned and analyzed in §4.1. ## decomposition qa experiments we detail two experiments that evaluates the qa pipeline decompentail proposed in §4.3. ## conclusion this work proposes a novel method that extracts distant and incidental signals from parallel news to facilitate general question representation. such parallel news signals intuitively bridge the reasoning gap in pre-trained language models due to reporting biases. to support this intuition, we train a model named decompt5 on such distant supervision and show that it improves 20%-30% on two semantic parsing benchmarks, namely overnight and torque, that directly evaluate query understanding. with decompt5 as the basis, we design a well-motivated question-answering pipeline decompentail that follows a decomposition, correction, and entailment scheme. we show that decompentail improves on strategyqa and hotpotqa by 3.7% and 8%, respectively.