HAWK : Highlighting Entity-aware Knowledge for Alleviating Information Sparsity in Long Contexts (2025)

Abstract: As the textual data given as the context of various tasks lengthens, having necessary information scattered throughout makes it more difficult for large language models (LLMs) to capture relevant details. This challenge is particularly prominent in tasks such as question answering (QA), where key information is often not evenly distributed within the context. This problem of information sparsity has led to the attempts of various approaches, such as direct context adjustment and retrieval-based methods. However, these approaches typically leverage compressed contexts, which increases the risk that key information may be contained in the dropped portions. Therefore, research from the perspective of addressing the information sparsity while not losing key details in contexts is required. To address this issue, we propose Highlighting entity-AWare Knowledge (HAWK) framework. HAWK consists of three main steps: i) entity extraction, ii) entity-aware subcontext selection, and iii) triplet construction. The core mechanism of HAWK is to highlight key information in a context and structuralize it in an entity-aware manner, facilitating knowledge-enhanced generation. Through extensive experiments and comprehensive analysis, HAWK confirms significant improvements in QA tasks with long contexts, achieving up to a 27.6-point F1 score increase and at least an average win rate of 76.75% over existing methods.

## introduction the increasing demand for processing longer textual data in the real world has led to extending the length of context windows that large language models (llms) can understand @xcite . despite the growing interest in the context lengths, longer contexts still pose an obstacle to adequate grounding and generation @xcite et al., 2023a; @xcite . in particular, addressing the salient information scattered throughout a long context is a challenge that needs to be solved @xcite . this challenge is especially prominent in tasks such as question answering (qa), where key information is often unevenly distributed within the context. ## experimental setup in appendix a, additional experimental settings, including hyperparameters, dataset statistics, model details, and prompt templates, can be found. also, the taxonomy of baseline methods for comparison with hawk is described. datasets. this study adopted the natural questions (nq) dataset @xcite as the raw source data, which is widely used in many qa studies. for experiments on long context scenarios, a subset of examples with context lengths of 16k or more was curated. additionally, to further investigate the generalizability of our method in multi-hop qa scenarios, we adopt the musique dataset @xcite . metrics. the evaluation is carried out in two ways: through existing automated metrics, including f1 score, and an llm-based evaluation. first, automated metrics, which primarily evaluate lexical overlap with the ground-truth answer, are adopted, including f1, rouge-l, bleu (@xmath0, 2, 4) @xcite scores, and bertscore @xcite . additionally, to evaluate the generation results from various perspectives, we employed the llm-as-a-judge approach @xcite , which calculates the win-tie-lose rates by evaluating the generation results in a pair-wise manner. ## results and analysis additional results and qualitative analysis examples not covered in the main body can be found in appendix b. ## conclusion despite the increasing length of the context window of llms, challenges such as the 'lost in the middle' phenomenon @xcite continue to hinder the effective capture and retention of critical information within lengthened contexts. this study focuses on alleviating the information sparsity problem, which is challenging when utilizing long contexts. many existing studies have adopted approaches that coarsely truncate or compress the given context; however, these methods cannot completely eliminate the risk of omitting crucial information. therefore, we propose hawk, a knowledgedensified framework that highlights and structures key information in an entity-aware manner while preventing the loss of important contents. extensive experiments demonstrate that our knowledge augmentation approaches can enhance problemsolving ability in long-context qa scenarios.