Generating Complex Question Decompositions in the Face of Distribution Shifts (2025)

Abstract: Question decomposition has been found to help large language models’ (LLMs) performance on complex question answering (QA) by breaking these questions into simpler sub-questions for answering. Nonetheless, performance on the task remains dominated by supervised approaches, suggesting room for making LLMs better decomposers. One way of improving LLM training and fine-tuning is to leverage synthetic training data, but the superior performance of supervised approaches collapses in the face of distribution shifts, making them unsuitable for generating synthetic data across new domains and at scale. To address this, we propose an approach to generate synthetic decomposition data with only five annotated examples; we do this by (i) extending recent advancements in using LLM-as-judge and for reranking in novel ways, as well as (ii) using a panel of smaller-sized LLMs for data generation instead of resource-intensive larger models. Through careful validation of our approach over two benchmark datasets, we show that our data generation and modelling approaches bring consistent improvements over using few-shot prompting with LLMs for the task. Our code and models can be found at https://github.com/hankelvin/complex_question_decomposition.

## introduction the ability of large language models (llms) to perform on tasks not seen at training, especially reasoning-related ones, is a keen subject of research recently @xcite @xcite . however, a "compositionality gap" remains as models scale in size, i.e. even as llms with larger parameter sizes show improvements on answering single-hop questions, their improvements towards answering complex multihop questions lag meaningfully behind the former @xcite . to address this, decompositionbased approaches @xcite @xcite have been proposed, whereby an llm is prompted to break a complex question or task into smaller subproblems that are incrementally solved, foot_0 the benefits of which include: (i) these simpler tasks are performed relatively well by llms, and (ii) they facilitate extensions into retrieval, functions calling and tool usage that can aid the search for the solution @xcite . when these sub-questions are used with llms to answer the complex question, it has been referred to as 'chain-of-questions' (coq) @xcite and there are indications of increased internal reasoning consistency when llms are made to answer complex queries in this way @xcite . despite the promising performance of decomposition-based complex question answering (qa) approaches using llms, the current state-ofthe-art (sota) for the associated task of question decomposition (i.e. to produce a sequence of sub-questions such as the one in footnote 1) is achieved with supervised approaches. on the other hand, as we discuss in in this work, we focus on the task of decomposing english complex questions for machine reading comprehension (mrc), into a sequence of subquestions @xcite . our contributions are: (i) an approach to generate decompositions with smaller-sized models -of between three and nine billion parameters, or "compact" llms (cllms) as we refer to them in this work -that are more resource-efficient; (ii) novel extensions of recent advances in using llm-as-judge @xcite and llm-as-rankers @xcite , together with panel voting @xcite , in our approach that allows us to derive synthetic training data with only five annotated decomposition examples; and (iii) an extensive evaluation comparing against supervised approaches and fewshot prompting of llms, showing that using the generated data to fine-tune cllms brings about question decomposition performance comparable with or better than larger models. our work is novel in the using of panels of cllms to both produce the synthetic complex question data and also to rank the generated decompositions for quality. being able to do so with cllms is meaningful as being able to create a human-annotated decomposition dataset at scale requires substantial resources and comes with restric-tions. 2 being able to do so with only five annotated examples for a given dataset allows the automatic creation of question decomposition data for new question types and new domains. synthetic data for improving llms recent improvements in llm training for instructionfollowing and alignment with human preferences @xcite @xcite have brought about new opportunities to use llms for generating synthetic data, and research has shown @xcite @xcite @xcite that training on such synthetic data drive further model performance. the current approach is to produce such synthetic data using a single llm of the largest sizes (e.g. gpt-4, llama 70b/405b, palm @xcite ); however, this entails significant resources (compute and runtime) and, as we show, do not yet bring about substantially better question decomposition performance over smaller-sized llms. some work has focused on reducing the resourceintensiveness of the process via knowledge distillation @xcite @xcite from large models to smaller ones. initial efforts have also mainly focused on generating class-conditioned text data used for classification tasks (e.g. movie review sentiment analysis) @xcite @xcite and for instruction-following training data @xcite . for qa, @xcite used a 540b-parameter model to generate multilingual questions that can be answered from a single passage; whereas the datasets in our work are multihop across two or more documents. in computer vision, @xcite used templates to generate decompositions of complex visual questions as a way to obtain training data to improve llm complex visual qa performance; though the complex questions they work with are focused on a narrow context (a visual chart) unlike the open-domain nature of the complex questions encountered in the mrc datasets we investigate. our approach and findings align with contemporaneous work @xcite showing that under a fixed compute budget constraint, the use of smaller (and weaker) llms for generating synthetic training data to fine-tune models for math reasoning tasks leads to outcomes that outperform doing the same using costly larger models. it can also be seen as generalising existing methods that generate synthetic data by sampling repeatedly @xcite ) from the same model to obtain wider coverage and diversity of generations before selecting the optimal solution. llm as judges and rankers @xcite examined the use of a single large chat llm for judging llm outputs. @xcite recently extended this to using the majority vote of a panel of three smaller-sized llms to judge and identify the best candidate for a task, with the motivation that doing so alleviates the need for using a single resource-intensive llm for evaluation. the tasks they examined include single-and multi-hop qa, and they found that all of the largest-sized llms they investigated (gpt-4, command-r and haiku) give results that have lower correlations with human judgements of answer correctness compared to using a panel of cllms. another line of research has shown that llms show promise for use in ranking tasks in information retrieval @xcite , finding that it is possible to use few-shot prompting of llms, for ranking a collection of documents for their relevance to a search query, to obtain performance that is competitive with supervised approaches. ## approach in this section, we describe (i) our approach for generating synthetic decomposition data; and (ii) how we fine-tune cllms with this data to bring about more robust decomposition performance. the task we explore involves decomposing a complex question q c (without any additional context) into an ordered sequence of simpler subquestions q s p q s @xcite . depending on the structure of q c , each q s may contain a referring variable (i.e. a placeholder) that points to the answer of a preceding question. in this way, by successively substituting the obtained answers and answering each q s in sequence, the final answer obtained gives the same as answering q c directly. ## data we focus our work on mrc datasets (examples in appendix b) whose complex questions have the answers to their sub-questions distributed across multiple documents, which are harder to answer and whose open-domain nature has the most potential benefit from decomposition-based qa with llms (compared to complex questions answerable from a single document that can already be performed well by existing reader models). ## evaluation existing work on decomposition-based approaches with llms @xcite focus on the qa aspect of the task and treat decomposition (in the form of cot, coq, tree-of-thought etc) as a means for additional computation to allow the model to derive better performance on the answering of q c ; as such they focus evaluation on the token f1 of the final answer obtained for q c . we evaluate on the development sets of break and musique (section 6) and include qa (section 5.2) and we also investigate the generated decompositions (section 5.1). ## conclusion we propose an approach for producing synthetic complex question decomposition data using only five annotated examples and novel extensions of llm-as-judge/rankers as well as llm panel voting for the task of ranking complex question decompositions. we then fine-tune smaller-sized "compact" llms (cllms) with the generated data and show -with validation over two benchmark datasets and comparisons against supervised models and few-shot llm prompting -that it leads to consistently improved question decomposition performance. notably, the improvements give the cllms performance on the decomposition task that is comparable to or better than larger llms. having cllms that decompose questions better improves their abilities for decomposition-based qa, which has been shown to be promising for closing the "compositionality gap" that they face in answering complex questions. it can also serve as a means to produce synthetic data that can be used to train and fine-tune cllms (potentially for larger ones as well) for further improvements.