TAIL : A Toolkit for Automatic and Realistic Long-Context Large Language Model Evaluation (2024)

Abstract: As long-context large language models (LLMs) are attracting increasing attention for their ability to handle context windows exceeding 128k tokens, the need for effective evaluation methods for these models becomes critical.Existing evaluation methods, however, fall short: needle-in-a-haystack (NIAH) and its variants are overly simplistic, while creating realistic benchmarks is prohibitively expensive due to extensive human annotation requirements. To bridge this gap, we propose TAIL, an automatic toolkit for creating realistic evaluation benchmarks and assessing the performance of long-context LLMs.With TAIL, users can customize the building of a long-context, document-grounded QA benchmark and obtain visualized performance metrics of evaluated models.TAIL has the advantage of requiring minimal human annotation and generating natural questions based on user-provided long-context documents. We apply TAIL to construct a benchmark encompassing multiple expert domains, such as finance, law, patent, and scientific literature. We then evaluate four state-of-the-art long-context LLMs using this benchmark. Results show that all LLMs experience varyingdegrees of performance degradation as contextlengths increase.

## introduction the rise of long-context large language models (llms) has opened new possibilities for applications requiring comprehensive understanding and processing of extensive input context @xcite @xcite @xcite . however, the evaluation of long-context llms poses unique challenges. a line of research involves directly inserting specific document-irrelevant information into lengthy documents and querying about them, i.e., needlein-a-haystack (niah) and its variants @xcite figure 1: tail's workflow begins by constructing a long document from user-provided source data. it then identifies multiple "gold paragraphs" at various depths within this haystack. using these gold paragraphs, tail generates high-quality question-answer pairs through its qa generation module, ensuring that each pair corresponds to a single identified paragraph. these pairs are then verified by the quality check module. the rag-based filter further refines the collection by removing qa pairs that can be answered using multiple paragraphs within the test examples. finally, tail assembles a benchmark from the remaining high-quality qa-test document pairs. 2024; @xcite . however, the content and style of the inserted text differ significantly from the original document. these substantial distribution differences do not reflect real-world scenarios when dealing with long contexts and could influence the evaluation of the llms' long-context capabilities. furthermore, the niah method is too simplistic that current models easily achieve nearly 100% in the test. another line of research, such as longbench @xcite and lv-eval @xcite , follows the traditional evaluation protocols and directly extends the context lengths of test data. however, the documents included are typically limited to a maximum of 40k tokens and require expensive human anno-tation, making it challenging to extend to longer contexts based on user-specific needs. to address the aforementioned limitations and build upon the existing lines of research, we develop tail, a toolkit for automatic and realistic long-context llm evaluation creating reliable and high-quality evaluation benchmarks for longcontext llms automatically. tail is designed to generate natural and reliable qas at specific depths of long documents for creating high-quality evaluation benchmarks. the main contributions of our work are as follows: ## related work needle in a haystack (niah) benchmarks, requiring models to retrieve randomly inserted sentences or facts within a long sequence, provide an automatic way to build a benchmark @xcite . besides the vanilla niah task, advanced niah techniques are further developed, including techniques of multiple needles @xcite , confusing facts @xcite , counting needles @xcite and simple reasoning @xcite . however, with automatically-generated qa pairs, current niah benchmarks suffer from the problem that questions are often irrelevant to the rest of the contexts and might be solved simply by retrieval instead of long-context reasoning and understanding abilities @xcite . compared to niah benchmarks, realistic benchmarks comprise a wider range of tasks relevant to real-world needs by including tasks like summarization @xcite , numerical reasoning @xcite , and multihop reasoning @xcite and spanning over multiple domains like code @xcite , medical @xcite , novel @xcite , legal, finance, and etc. @xcite . there also exist comprehensive realistic benchmarks encompassing multiple tasks like l-eval @xcite , longbench @xcite , loogle @xcite , ∞bench @xcite and bamboo @xcite . due to the irregularity of question types, realistic benchmarks are usually either not long enough (less than 100k tokens), or expensive to collect and annotate. tail seeks to address both realistic needs and reduce human annotation when creating long-context benchmarks. ## the tail toolkit this section provides an overview of the tail workflow foot_0 , highlighting the main components and their interactions as illustrated in figure 1 . tail consists of three key components: tail then extracts multiple paragraphs at designated depths from the composed long documents as 'gold paragraphs' and generates qas based on those paragraphs. then, to ensure high-quality qa pairs, tail uses a quality checker to filter out qas that cannot be correctly answered even if given the gold paragraph as references ( §3.2). we now have a long document and qa pairs (together with their golden paragraphs containing the answer) at different locations in the long sequence. this long document has a maximum specified length. finally, tail has an extraction module to further extract test data of different lengths from the long document, e.g., from 4k to maximum length set by users ( §3.3). note that these test data of different lengths coming from the same long-context documents share exactly the same qas, thus ensuring control of variables when assessing long-context abilities. ## experiments and results next, we demonstrate how tail is utilized to evaluate nine long-context llms across four specialized domains: finance, patents, legal, and scientific papers. we present the results for these nine evaluated llms and provide a detailed analysis. ## conclusion the emergence of long-context llms has highlighted the need for more effective evaluation tools. in this paper, we propose tail, an automatic and realistic toolkit for long-context large language model evaluation. tail can generate benchmarks end-to-end with the given source documents. moreover, tail offers evaluation modules for testing and results visualization. we demonstrate tail's capabilities by creating a cross-domain benchmark, illustrating its effectiveness in both benchmark development and llm performance evaluation. we believe that the tail will serve as a useful toolkit for evaluating long-context llms.