Systematic Evaluation of Long-Context LLM s on Financial Concepts (2024)

Abstract: Long-context large language models (LC LLMs) promise to increase reliability of LLMs in real-world tasks requiring processing and understanding of long input documents. However, this ability of LC LLMs to reliably utilize their growing context windows remains under investigation. In this work, we evaluate the performance of state-of-the-art GPT-4 suite of LC LLMs in solving a series of progressively challenging tasks, as a function of factors such as context length, task difficulty, and position of key information by creating a real world financial news dataset. Our findings indicate that LC LLMs exhibit brittleness at longer context lengths even for simple tasks, with performance deteriorating sharply as task complexity increases. At longer context lengths, these state-of-the-art models experience catastrophic failures in instruction following resulting in degenerate outputs. Our prompt ablations also reveal unfortunate continued sensitivity to both the placement of the task instruction in the context window as well as minor markdown formatting. Finally, we advocate for more rigorous evaluation of LC LLMs by employing holistic metrics such as F1 (rather than recall) and reporting confidence intervals, thereby ensuring robust and conclusive findings.

## introduction recently, there has been a growing interest in extending the context window sizes of large language models (llms) to produce long-context llms (lc llms) @xcite @xcite . this is especially promising as it allows to extend the "working memory" of llms. real world use cases need lc llms to be able to follow increasingly complicated instructions while reasoning over their long context length windows with high degree of reliability. figure 1 : an overview of our framework, made of three financial concepts resulting into four real-world practical tasks of varying difficulty levels. this diagram shows an illustration of constructing one test record for task t 4 : company+sentiment (cs). our framework allows complete control over the sampling and injection of needles, hard negatives and other haystack articles into the context window of an lc llm. current benchmarks test abilities of lc llms in several ways. for example, gemini team (2024) follow the "needle-in-a-haystack" @xcite ; @xcite analysis to evaluate gemini-pro on retrieval-based long-context synthetic tasks, measuring only surface-level retrieval capabilities. frequently, there is observed dissimilarity (or heterogeneity) between the needles and haystack in benchmarks like @xcite , making it artificially easier to retrieve the needle. other recent work by @xcite includes tasks such as arranging shuffled text segments in the correct order, while flenqa @xcite creates balanced true/false datasets inspired by @xcite to test models' abilities on chaining facts and simple inductions -deviating substantially from real-world usage of llms. other popular long-context benchmarks like l-eval @xcite beyond a fixed number of tokens, and hence not applicable to the more recent lc llms supporting 128k tokens context window. finally, metrics like recall remain prevalent @xcite ; kamradt et al. (2024); gemini team (2024); @xcite , which however provides only a partial signal on lc llms' performance. in this work, we try to address the limitations in existing benchmarks by creating a systematic evaluation framework that is based on real world financial news and tasks of practical interest. our framework allows the flexibility to manipulate several dimensions crucial for a comprehensive evaluation of long-context llms ( our findings suggest: ## methodology real world use of llms, commonly requires them to locate, reason over and synthesize relevant information across their context window, while accounting for constraints specified in the prompt. our methodology is inspired to mimic such realistic use-cases of lc llms. we now outline the pivotal characteristics of our tasks, experimental setup, prompts and evaluation strategy. ## results to delve into the factors affecting the performance of lc llms, our results center on three key research questions presented in sec. 1. ## towards better evaluation recall is not a reliable metric for difficult retrieval tasks at longer contexts. most research in long-context retrieval primarily report their results using recall as the evaluation metric @xcite @xcite , since the basic goal is to assess if lc llms can "remember" key information in long contexts. we argue that a simple recall metric is often artificially inflated (fig. 7 ) and hence of limited pragmatic value in real-world systems. for this reason, we also report bootstrap median f1-score throughout this work. ## conclusion our methodical framework characterized by realworld financial news concepts allows for flexible configurations to setup a range of different complexity tasks. our study reveals that long context retrieval and reasoning is still a challenging task for long-context llms. our dataset requires 32k tokens to challenge state-of-the-art gpt-4 models on easier tasks, and only 16k tokens on difficult tasks. models perform poorly on 75% of their context window, with performance declining sharply as input length and task complexity increases. this limitation underscores the need to reliably test the effectiveness of long context windows against model provider claims. moreover, lc llms also succumb to minor prompting variations, underpinning the importance of using robust metrics.