Beyond the Haystack: Sensitivity to Context in Legal Reference Recall (2025)

Abstract: Reference retrieval is critical for many applications in the legal domain, for instance in determining which case texts support a particular claim. However, existing benchmarking methods do not rigorously enable evaluation of recall capabilities in previously unseen contexts. We develop an evaluation framework from U.S. court opinions which ensures models have no prior knowledge of case results or context. Applying our framework, we identify an consistent gap across models and tasks between traditional needle-in-a-haystack retrieval and actual performance in legal recall. Our work shows that standard needle-in-a-haystack benchmarks consistently overestimate recall performance in the legal domain. By isolating the causes of performance degradation to contextual informativity rather than distributional differences, our findings highlight the need for specialized testing in reference-critical applications, and establish an evaluation framework for improving retrieval across informativity levels.

## introduction language model (lm) benchmarks @xcite include evaluations which measure the longcontext recall performance of models. this has been done primarily through variations of the needle in a haystack format, in which a short string of text (the needle) is inserted in a larger context window (the haystack). the extent to which niah recall scores transfer to specialized domains such as law remains an open question. existing benchmarks often conflate retrieval with downstream reasoning, and lack systematic approaches to avoiding data contamination @xcite @xcite . we present a framework for closed-domain legal recall benchmarks using court opinions published after model training cutoffs to test retrieval from unseen texts. our matched needle task requires models to recover real legal references and facts, revealing a consistent performance gap between standard niah tasks and true legal recall, even after controlling for context by using shuffled texts. we find that models likely do not use recall to succeed in standard long-context benchmarks; instead, they process inputs via sentence-to-sentence cues on texts. we observe a j-shaped curve in model performance as shuffle window increases, with performance decreasing for local shuffles and increasing to a maximum for global shuffles. furthermore, we perform position ablations by inserting the needle in different locations and, finding no correlation between position and accuracy, show that positional bias does not affect performance. by emphasizing this distinction between reading and recall ability that is often conflated by standard long-context benchmarks, we highlight the need for new long-context benchmarks that properly isolate recall. legal benchmarks. having benchmarks that evaluate legal understanding and citation retrieval is crucial to assess how capable models are of being deployed in the legal field. these include lexglue @xcite , @xcite , and @xcite @xcite .however, these works often blend retrieval with reasoning components and do not ensure evaluation independence from training data. long context recall. many recall benchmarks test token extraction, these include needle-in-a-haystack (niah) @xcite and its variants, including needlechain @xcite and recall benchmarks in reasoning and recall-based applications @xcite @xcite . however, these works often permit preexisting mentions of the needle fragment in training data @xcite . there also exist several studies which conduct recall within a specialized domain. @xcite specifically tests model retrieval from legal contexts, while fan et al. @xcite assess long-context recall in the medical domain. however, these works omit causal analysis and do not guarantee a lack of data contamination. ## results our results show that model recall follows a jshaped trajectory as contextual information is disrupted. as shown in figure 3 , interfering with local semantics via small shuffle windows significantly impairs performance, hindering the model's "reading" process. however, completely destroying the context via a global shuffle forces the model into its "recall" mode, boosting accuracy to its peak. ultimately, every model tested performed much worse in the standard setting compared to the contextually uninformative global shuffle (niah setting), suggesting that such benchmarks can be unreliable. one potential counterfactual to consider is that position alone can explain the drops in accuracy. we eliminate this possibility through needle position ablation tests. as seen in the appendix, these tests show little correlation between position and accuracy and indicate that position alone cannot explain the gap. the performance gap between standard and niah contexts is exacerbated by document length. while a model's niah performance is stable, its ability to recall from a coherent legal text degrades sharply as the text gets longer. as seen in figure 4 , the error rate in a standard context increases 3.1x faster than in a niah task. this widening gap demonstrates that high niah scores are unreliable predictors for performance, especially on long domain-specific documents. specific accuracy scores for each model across the three key conditions-standard (reading), local shuffle (disrupted reading), and niah (recall)-are detailed in ## conclusion reference retrieval is critical for many applications in the legal domain, for instance in determining which case texts support a particular claim. however, existing benchmarking methods do not rigorously enable evaluation of recall capabilities in previously unseen contexts. we develop an evaluation framework from u.s. court opinions that ensures models have no prior knowledge of case results or context. applying our framework, we identify an consistent recall gap across models and tasks based on context. we show that standard needle-in-a-haystack benchmarks consistently overestimate recall performance in the legal domain. moreover, we isolate the causes of performance degradation to contextual informativity rather than distributional differences, and find a j-shaped curve suggesting complex reading behavior. thus, our findings highlight the need for specialized testing in reference-critical applications, and establish an evaluation framework for improving retrieval across informativity levels.