Semantic Masking in a Needle-in-a-haystack Test for Evaluating Large Language Model Long-Text Capabilities (2025)

Abstract: In this paper, we introduce the concept of Semantic Masking, where semantically coherent surrounding text (the haystack) interferes with the retrieval and comprehension of specific information (the needle) embedded within it. We propose the Needle-in-a-Haystack-QA Test, an evaluation pipeline that assesses LLMs’ long-text capabilities through question answering, explicitly accounting for the Semantic Masking effect. We conduct experiments to demonstrate that Semantic Masking significantly impacts LLM performance more than text length does. By accounting for Semantic Masking, we provide a more accurate assessment of LLMs’ true proficiency in utilizing extended contexts, paving the way for future research to develop models that are not only capable of handling longer inputs but are also adept at navigating complex semantic landscapes.

## introduction many state-of-the-art large language models (llms) have recently claimed to have extended the input context window to 128k or above. (e.g., @xcite , llama-3.1 @xcite , etc.) such extensions significantly boost these models' abilities to take on a wider range of tasks as they enable them to take longer documents such as story outlines or even full stories as their input. specifically, they can aid authors in creative writing. in the flower and hayes model @xcite , writing is viewed as a network of three main cognitive processes: planning, translating and reviewing. as we extend the context window, llms can not only aid authors in the planning stage through brief writing prompts, but also help them in the translating stage by taking in and expanding on the story outline; or in the reviewing stage by taking in and refining the full story. however, the effectiveness of the extended context window remains questionable, as the evaluation metric those modifications are based on are mostly about language modeling ability, which does not necessarily capture how well the models utilize context in various downstream tasks -tasks that require understanding and interpretation of the context, especially in creative writing. in addition to those language-modeling-oriented metrics such as perplexity @xcite , many recent works on long text processing have turned to the needle-in-a-haystack pressure test @xcite , which is a more retrieval-oriented evaluation that inserts a statement (the needle) into a larger piece of text (the haystack) and asks llms or llm-based retrieval augmented generation (rag) models to retrieve it. alternatively, one can generalize information retrieval to free-form question answering in order to test how well the long input context is understood. however, one important factor that has been more or less ignored is the semantic masking effect the haystack may have on the needle or the question. in the original work, the haystacks are chosen solely by the length of the document in a random process. this process, although easily adaptable to different context window sizes, does not represent the practical usage of long context window in downstream tasks well. in practice, the long context provided, such as stories or books, is often semantically coherent, meaning that each sentence or paragraph should be more semantically related to its neighbours compared to the needle and the haystack chosen randomly. semantic masking in this case denotes the interference the surrounding text may impose on the needle, which effectively acts like a mask that hides the information in the needle. in this work, we will demonstrate how semantic masking effect might be a more important factor that impacts llm's long text capabilities than text length. based on the findings, we also propose an evaluation pipeline that assesses llm's long text capabilities through question-answering with the needle-in-a-haystack approach while taking account for the semantic masking effect. we select a subset of questions and their corresponding stories from narrativeqa @xcite . we name this pipeline as needle-in-a-haystack-qa test. the main contributions of our work are: ## method in this section, we will first discuss the setup of the needle-in-a-haystack-qa test. based on the test, we will list a few experiments that utilize this test to demonstrate what role the semantic masking effect and text length play in demonstrating llm's long text capabilities. ## conclusion in this study, we proposed the needle-in-ahaystack-qa test to assess llm's long text capabilities. through the experiments we have drawn 2 major conclusions: 1) length is not the primary factor that affects the difficulty of tests that follow the needle-in-a-haystack approach; 2) highly related haystack may impose semantic masking effect on the needle which exerts a more profound influence on llm performance. through these two conclusions, we wish to challenge the conventional emphasis on context length and suggest a more nuanced approach to evaluating llm's long text capabilities. we also propose a difficulty assessment framework that can be generalized to any questionanswering dataset in assessing question difficulty. this framework is also essential in validating the meaningfulness of experiments designed from the needle-in-a-haystack-qa test. there are also other factors we suspect may have an impact on the difficulty of the test, such as the position of needle insertion relative to the haystack. we will test these factors in subsequent experiments. in conclusion, our work advocates for a more nuanced approach to evaluating and enhancing the long text capabilities of llms. by incorporating semantic masking considerations into evaluation metrics, we pave the way for the development of models that are not only proficient in handling extensive contexts but also adept at extracting and interpreting relevant information within them.