Making Long-Context Language Models Better Multi-Hop Reasoners (2024)

Abstract: Recent advancements in long-context modeling have enhanced language models (LMs) for complex tasks across multiple NLP applications. Despite this progress, we find that these models struggle with multi-hop reasoning and exhibit decreased performance in the presence of noisy contexts. In this paper, we introduce Reasoning with Attributions, a novel approach that prompts LMs to supply attributions for each assertion during their reasoning. We validate our approach through experiments on three multi-hop datasets, employing both proprietary and open-source models, and demonstrate its efficacy and resilience. Furthermore, we explore methods to augment reasoning capabilities via fine-tuning and offer an attribution-annotated dataset and a specialized training strategy. Our fine-tuned model achieves competitive performance on multi-hop reasoning benchmarks, closely paralleling proprietary LMs such as ChatGPT and Claude-instant.

## introduction the field of long-context modeling has garnered significant attention due to its importance in applications that demand extensive comprehension and generation capabilities @xcite . techniques for long-context modeling @xcite @xcite have been proposed with encouraging results on established benchmarks @xcite . nevertheless, we have identified a gap in the performance of these models when it comes to multi-hop reasoning tasks, where a model must navigate and synthesize information from disparate sources to answer complex questions. evidence from key benchmarks such as longbench @xcite , as well as our experimental results in section 4.3, indicate that these long-context lms underperform compared to leading multi-hop reasoning systems @xcite . the reasons for this shortfall in multi-hop reasoning effectiveness are not yet fully understood. we contend that the limitations in multi-hop reasoning observed in long-context lms stem from two main issues: the inability to discern pertinent information within noisy contexts @xcite and the struggle to incorporate knowledge within the context effectively, particularly for smaller-scale models @xcite . to address these challenges, we introduce reasoning with attributions, a methodology that compels lms to substantiate their reasoning by linking assertions to relevant context segments, such as citations @xcite or direct quotations @xcite . this approach not only guides lms to perform targeted information retrieval to identify the position of relevant contexts, thereby reducing noise, but also ensures their responses are wellgrounded in the source material. our preliminary and comprehensive experimental findings, detailed in sections 2.1 and 4.3, confirm the efficacy and resilience of this method across various multi-hop reasoning benchmarks. despite these advancements, smaller longcontext lms exhibit continued difficulties in reasoning. we explore the potential for these models to improve through learning to reason and attribute simultaneously. utilizing @xcite to annotate the multi-hop reasoning dataset musique @xcite , we create a specialized dataset musique-attribute for fine-tuning models in this dual capacity. we propose a potent learning strategy that leverages multi-task learning and data augmentation to fully exploit these annotations. our experiments with five long-context lms across three multi-hop reasoning datasets and two general instruction-following datasets reveal that our fine-tuned vicuna-7b model @xcite surpasses similar-scale baselines by a substantial margin, i.e., more than 20 points on average, and even outperforms chatgpt and claudeinstant on musique, albeit with a slight trade-off in other capabilities. this study illuminates a promising avenue to enhance the multi-hop reasoning capabilities of long-context lms through a focus on attributions. our contributions can be summarized as follows: ## learning to attribute in reasoning one intuitive approach to enhancing the multi-hop reasoning capabilities of lms is to fine-tune them on our curated musique-attribute, thereby teaching them to integrate attribution into their reasoning processes, specifically to generate coc. despite the simplicity of this method, our subsequent analysis in section 4.4 demonstrates that this direct approach fails to produce robust results. multi-task learning. beyond simply finetuning lms on the musique-attribute to learn to attribute in reasoning (denoted as la), we propose three auxiliary tasks that serve as simplified analogs of la. these tasks are designed to train lms in conjunction with la to enhance their proficiency in attribution-based reasoning: ## conclusion this study demonstrates that long-context lms face challenges with multi-hop reasoning within noisy contexts. we introduce a reasoning paradigm that incorporates attributions, which significantly improves the reasoning capabilities of long-context lms. alongside, we contribute a new dataset annotated with attributions and study training strategies tailored for multi-hop reasoning. our comprehensive experiments across five models and five benchmarks validate the superiority of our approach in enhancing multi-hop reasoning performance.