Is a Question Decomposition Unit All We Need? (2022)

Abstract: Large Language Models (LMs) have achieved state-of-the-art performance on many Natural Language Processing (NLP) benchmarks. With the growing number of new benchmarks, we build bigger and more complex LMs. However, building new LMs may not be an ideal option owing to the cost, time and environmental impact associated with it. We explore an alternative route: can we modify data by expressing it in terms of the model’s strengths, so that a question becomes easier for models to answer? We investigate if humans can decompose a hard question into a set of simpler questions that are relatively easier for models to solve. We analyze a range of datasets involving various forms of reasoning and find that it is indeed possible to significantly improve model performance (24% for GPT3 and 29% for RoBERTa-SQuAD along with a symbolic calculator) via decomposition. Our approach provides a viable option to involve people in NLP research in a meaningful way. Our findings indicate that Human-in-the-loop Question Decomposition (HQD) can potentially provide an alternate path to building large LMs.

## introduction with the advent of large lms, we have achieved state-of-the-art performance on many nlp benchmarks @xcite @xcite . our benchmarks are evolving and becoming harder over time. to solve new benchmarks, we have been designing more complex and bigger lms at the cost of computational resources, time and its negative impact on the environment. building newer lms for solving new benchmarks may not be an ideal and sustainable option over time. inspired by humans, who often view new tasks as a combination of existing tasks, we explore if we can mimic humans and help the model solve a new task by decomposing @xcite it as a combination of tasks that the model excels at and already knows. as nlp applications are increasingly more and more popular among people in their daily activities, it is essential to develop methods that involve humans in nlp-powered applications in meaningful ways. our approach attempts to fill this gap in lms by providing a human-centric approach to modifying data. solving complex qa tasks such as multi-hop qa, and numerical reasoning has been a challenge for models. question decomposition (qd) has recently been explored to empower models to solve these tasks with the added advantage of interpretability. however, previous studies on qd are limited to some specific datasets @xcite such as drop @xcite and hot-potqa @xcite . we analyze a range of datasets involving various forms of reasoning to investigate if "a question decomposition unit all we need?" figure 1 shows the schematic representation of a qd unit. the original question is difficult for a model to answer. however, it becomes easier for the model when a human decomposes the question into a set of simpler questions. we manually decompose randomly selected 50 samples of each dataset. the decompositions we perform are purely based on intuitions to reduce the complexity of the question, inspired by the success of task-level instruction decomposition @xcite in improving model performance. we experiment with @xcite and roberta @xcite fine-tuned on squad 2.0 @xcite and find that hqd significantly improves model performance (24% for gpt-3 and 29% for roberta-squad along with a symbolic calculator). here, the evaluation happens on unseen tasks on which the model is not fine-tuned. our findings indicate that humanin-the-loop question decomposition (hqd) can context: storm corrosion is the self-titled... divorce via her facebook page. ## related work a recent methodology to reason over multiple sentences in reading comprehension datasets is to decompose the question into single-hop questions @xcite . @xcite decompose questions from hot-potqa using span predictions based on reasoning types and picks the best decomposition using a decomposition scorer. @xcite generate decompositions by training a bart model on question generation task by providing context, answers and hints. @xcite crowd-sourced annotations for decompositions of questions. @xcite , on the other hand, uses the unsupervised mechanism of generating decomposition by mapping a hard question to a set of candidate sub-questions from a question corpus. @xcite answer a question sequentially using a neural semantic parsing framework over crowdsourced decompositions for questions from wikitablequestions. decomposition using text-to-sql query conversion has also been studied @xcite . also, knowledge graphs are combined with neural networks to generate decompositions @xcite . recently, @xcite presented another use case where decompositions can be used to probe models to create explanations for their reasoning. ## experimental setup models we use @xcite to generate answers for original and decomposed questions. to show that qd significantly improves performance even on simpler models, we use roberta-base finetuned on squad 2.0 dataset (i.e., roberta-squad). additionally, we use roberta-base finetuned on boolq dataset @xcite (i.e., roberta-boolq) for original and decomposed questions in strategyqa since they are true/false type questions. experiments to create baselines, we evaluate all models on the original question along with the context. we evaluate all models on the manually decomposed questions in the proposed method. we carry out all experiments in gpt-3 by designing prompts for each dataset 2 . for robertabased models, we use roberta-squad for mul-tirc, break, hotpotqa and drop datasets, since squad 2.0 is designed for a reading comprehension task. for strategyqa, we use two roberta-base models: (1) roberta-boolq, which is used to answer the final boolean type of questions, and (2) roberta-squad which is used to answer the remaining decomposition questions. for svamp, we use the roberta-squad model to extract the necessary operands using decomposed questions and then we use the computational module to perform various operations. in all experiments, we use decomposition to get to the final answer sequentially. metrics for all our experiments, we use rouge-l @xcite , f 1 -score and exact match (em) as the evaluation metrics. ## results and analysis here, we divide our datasets into four categories: (1) rc: hotpotqa, drop, multirc, and break in reading comprehension (rc), (2) math: mathqa and svamp in mathematical reasoning , (3) mc: qasc in multi-choice qa (mc) , and (4) sr: strategyqa in strategy reasoning (sr). all results presented in this sections are averaged over tasks for each category. ## conclusion the recent trend of building large lms may not be sustainable to solve evolving benchmarks. we believe that modifying data samples can significantly help the model improve performance. we study the effect of question decomposition (qd) on a diverse set of tasks. we decompose questions manually and significantly improve model performance (24% for gpt3 and 29% for roberta-squad along with a symbolic calculator). our findings indicate that human-in-the-loop question decomposition (hqd) can potentially provide an alternate path to building large lms. our approach provides a viable option to involve people in nlp research. we hope our work will encourage the community to develop human-centric solutions that actively involve humans while leveraging nlp resources.