When Do Decompositions Help for Machine Reading? (2023)

Abstract: Answering complex questions often requires multi-step reasoning in order to obtain the final answer. Most research into decompositions of complex questions involves open-domain systems, which have shown success in using these decompositions for improved retrieval. In the machine reading setting, however, work to understand when decompositions are helpful is understudied. We conduct experiments on decompositions in machine reading to unify recent work in this space, using a range of models and datasets. We find that decompositions can be helpful in zero or limited-data settings, giving several points of improvement in exact match. However, we also show that when models are given access to around a few hundred or more examples, decompositions are not helpful (and can actually be detrimental). Thus, our analysis implies that models can learn decompositions implicitly even with limited data.

## introduction much past work has examined and improved models' ability to answer complex questions that require multiple steps of reasoning @xcite @xcite @xcite . a consistent theme in these works is to break the main complex question down into a series of sub-questions to be solved, which is referred to as question decomposition. these methods generally represent decompositions as a human would, with explicit natural language sub-questions that build together to the final answer. despite the large amount of research in decompositions for multi-step question answering, the majority of it has focused on using question decomposition for both information retrieval and question 1 we publicly release all code at https://github.com/ weikangda/question-decomposition ## results full-data experiments we see the main results in figure 2 with results for drop on the left and hotpotqa on the right. bars are colored-coded according to their method. all bars are the mean of three random seeds and error bars indicate the standard deviation. we see that most methods perform nearly the same, except for two that underperform: explicit and explicit + implicit. note that both of these have issues with training end-to-end, as an explicit decomposition is not differentiable through all decomposition steps. thus, we only end up differentiating through the last step of the explicit decomposition steps, leaving the model unable to learn as effectively. all other approaches to decomposition perform comparably, given random seed variance (e.g. t5-base drop implicit decomp. is 33.7% exact @xmath0.9% vs no decomp 34.0% ± 0.4%). in fact, in this full data setting, the no decomp method performs better or statistically similar to every other method according to two-sample t-tests with the bonferroni correction @xcite , across all datasets and models. we show the two more recent and larger models (llama and alpaca) in size experiment is the no decomp method always the same or better than the decomposition methods, or can decompositions help in the lowdata regime? we answer this question in figure 3 , by varying the amount of training data, comparing it to the zero-shot explicit method that is typically used @xcite . 3 as we need the zero-shot model to be fine-tuned on qa to handle the decomposed questions, foot_1 we use a modified version of t5-base (valhalla/t5-basesquad from huggingface) that was fine-tuned on squad @xcite . this places the no decomp method at a disadvantage, as it was not trained on squad and started from t5-base. despite that, however, we see that as the amount of data increases, the fine-tuned models perform better; eventually surpassing the zero-shot method between 100-250 examples for unifiedqa-v2 and 250-1000 examples for t5. 5 error analysis why does the zero-shot explicit method perform worse than the fine-tuned no decomp? in ## related work decompositions in qa decompositions for qa have a long history in complex question answering @xcite @xcite with recent interest in using them for large language models @xcite @xcite . two of the most related works to ours include @xcite , who focus on decompositions in the zero-shot setting only and show improvements (which aligns with our results), and other work @xcite that shows that decompositions help on the full drop dataset but which doesn't include a comparable non-decomposition baseline on the same data. our analysis complements and unifies our understanding of decompositions by identifying when decompositions help with respect to dataset size. decompositions in other fields our results for decompositions in textual qa helps to unify results across machine learning, as similar conclusions (e.g. decompositions being less effective then endto-end methods with large data) can be seen from scores on computer vision visual qa leaderboards @xcite . ## conclusion our work explored when decompositions are helpful for machine reading. we showed that decompositions are helpful when there is limited data available, or when parameters cannot be tuned. however, when enough data exists (empirically around a few hundred instances) and parameters can be fine-tuned, it is best to let the model learn the decompositions implicitly through end-to-end training. furthermore, we show that limitations of not fine-tuned decomposition approaches include error propagation of intermediate steps while also introducing more possibilities for annotator error. we hope that our work will help to inform readers as they create new datasets and select methods to use for complex question answering.