Joint Enhancement of Relational Reasoning for Long-Context LLM s (2025)

Abstract: Despite significant progress, large language models (LLMs) still struggle with long contexts due to memory limitations and their inability to tackle complex and long-context tasks. Additionally, LLMs often suffer from a lack of transparency and are prone to producing hallucinations. To address these challenges, we propose JERR , a novel framework designed to enhance long-context comprehension via graph-based reasoning in LLMs. JERR integrates three key components: synopsis extraction, graph construction, and relational reasoning. First, synopsis is extracted by chunking text strategically, allowing the model to summarize and understand information more efficiently. Second, we build a directed acyclic graph (DAG) to resolve redundancy, ensuring logical consistency and clarity. Finally, we incorporate Monte Carlo Tree Search (MCTS) to help the model navigate complex reasoning paths, ensuring more accurate and interpretable outputs. This framework provides a novel solution that enables LLMs to handle extended contexts and complex reasoning tasks with improved reliability and transparency. Experimental results show that JERR consistently outperforms all baselines on the ROUGE and F1 metrics, achieving the highest scores on the LLM-Rater evaluation.

## introduction in recent years, large language models (llms) have achieved significant achievements in natural language processing @xcite . however, transformer-based llms still face limitations in high-performance reasoning with long-context inputs due to constraints on memory usage and context window size @xcite . another challenge is that, although these models are pre-trained on extensive text corpora to respond coherently and appropriately to user inputs, they * corresponding author still struggle with complex knowledge-reasoning tasks @xcite @xcite . we consider that the capability to handle longcontext and graph-related tasks is critically important. firstly, llms often struggle to provide accurate answers when key information is deeply embedded within lengthy contexts @xcite or when specialized knowledge extends beyond the pre-training corpora @xcite , particularly with up-to-date information. secondly, llms lack interpretability, transparency, and accountability, which increases the risk of producing hallucinations @xcite . thirdly, few llms or frameworks operate in a human-like manner @xcite . they do not think and respond in the thoughtful, deliberative way humans do when faced with challenging questions. to address these three issues, researchers have proposed many methods to solve these problems recently, which can be roughly divided into two categories. the first approach focused on graph-related tasks by integrating llms with knowledge graphs @xcite @xcite . these methods fine-tune baseline models using large datasets of generated triplets and reasoning paths @xcite . the second approach emphasizes training-free frameworks, which deliver high benchmark performance and broader generalizability, particularly for longcontext processing, without compromising inference capabilities @xcite . these frameworks also simplify knowledge updating @xcite , allowing users to add new information to the knowledge base without retraining the model @xcite . given these challenges, we aim to design a generalized framework to enhance llms' capacity for processing long contexts and reasoning effectively. our approach simulates the way humans figure 1 : the overall framework of jerr consists of three main steps: 1) given a question and a long context, llms are first prompted to segment the context into chunks. 2) next, synopsis is generated through llms' prompts, followed by the construction of the graph. 3) finally, reasoning is performed using relevant nodes, identified through the mcts algorithm on the question and graph, along with the synopsis to generate the final answer. for synopsis extraction, we leverage llms' summarization prompts to strategically segment the context @xcite . in graph construction, we employ algorithms that address issues like entity redundancy @xcite and attribute recurrence @xcite . this phase also involves building a directed acyclic graph (dag) @xcite to allow models to explore complex reasoning paths while maintaining logical coherence. finally, in the relational reasoning phase, monte carlo tree search (mcts) @xcite @xcite enables precise retrieval of relevant nodes on the graph, enhancing the model's capacity to deliver accurate answers to complex queries. to this end, we propose jerr (figure 1 ), a joint enhancement of relational reasoning for long-context llms. ## methodology this section details the design and implementation of our framework, jerr, an agent that integrates graph structures with long-context processing capabilities. ## experiment in this section, we describe the experimental setup and present our main results with an ablation study, case study of our proposed approach (cost analysis is in §c.2). ## limitations our current framework jerr has so far been validated exclusively on quality, musique and narrativeqa. experimental results demonstrate that jerr has high performance on such longcontext benchmarks. however, jerr has limitations mainly on three aspects: for task scalability, although jerr performs well on the selected datasets, its ability to generalize to other reasoning tasks in different knowledge domains remains uncertain. further validation on a broader range of datasets is necessary to assess its scalability. for knowledge graph construction, the framework relies on the qwen api to extract graph nodes, which helps automate knowledge representation. however, constructing a refined knowledge graph still requires a complex pipeline which can make large-scale and high-precision graph building a challenging task. for task complexity, jerr applies a graph-based approach and monte carlo tree search (mcts) to improve knowledge reasoning and retrieve relevant text more effectively. however, its effectiveness on simpler tasks has not been systematically evaluated. future studies should compare its performance on both simple and complex tasks to provide a more complete assessment of its capabilities. algorithm 1 mcts relevant nodes extraction algorithm input: graph g, query text q, start nodes s, number of simulations n output: top nodes based on keyword relevance 1: initialize root node r with state s[0] 2: extract keywords k from q as set of lowercase words 3: for @xmath0: selection: set current node n ← r 5: while n has children do 6: select the best child of n based on win/visit ratio 7: if n has no selectable children then 8: break 9: end if 10: end while 11: expansion: 12: if n has no children then 13: expand children for n using neighbors in g 14: end if 15: simulation: 16: if n has children then 17: set current node n to best child of n 18: end if 19: initialize score score ← 0 20: for depth @xmath1: if n exists in g then 22: increment score by matching keywords k with state of n 23: set n to best child of n 24: if n has no children then 25: break 26: end if 27: end if 28: end for 29: backpropagation: 30: while n is not null do 31: update visits and wins for n based on score 32: set n to parent of n 33: end while 34: end for 35: sort root children by win/visit ratio and return top nodes 36: return top relevant nodes based on mcts exploration expansion if the selected node s next has not reached a terminal state or attained a sufficient visit count, the expansion step continues by adding its corresponding graph node v @xmath2. the appropriate set of nodes for expansion is defined as: simulation in this stage, a random simulation is conducted from one of the newly expanded nodes s to estimate the potential reward along this path. the reward for state s is evaluated based on its relevance to the query, which is determined by the maximum simulation depth d and a set of keywords extracted from the query. the reward function is given as: backpropagation the final step of mcts is the backpropagation phase, where the reward r (s ) obtained from the simulation is propagated back through each node along the traversal path, updating both the total reward w (s) and visit count n (s) for each node: as a result, the set of topk, ranked by relevance, is used as the set of retrieved nodes r: in summary, the entire mcts procedure comprises five stages, as illustrated in algorithm 1. the final retrieval of relevant nodes is expressed as: quality foot_3 is a four-way multiple-choice qa challenge comprising text data from diverse sources, evaluated based on accuracy. the dev set of quality includes 230 long-context samples with corresponding questions, answer options, and other details. musique foot_4 serves as a multi-hop long-context qa benchmark, with an average of 15.5 thousand tokens across 200 samples. musique's questions include various content types, such as narratives, expository texts, and factual material. this diversity challenges models to generalize across various genres and structures, strengthening their capacity for comprehensive language understanding and reasoning. we also include the narrativeqa foot_5 dataset as a single-hop long-context benchmark, averaging 29.7 thousand tokens across 200 samples. this dataset focuses on narrative comprehension in a long-form context, requiring models to answer questions based on an understanding of entire narratives, such as books or movie scripts, rather than isolated text segments. this differentiates narra-tiveqa from typical qa datasets, which generally focus on shorter text segments.