S 3 E val: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Model (2024)

Abstract: The rapid development of Large Language Models (LLMs) has led to great strides in model capabilities like long-context understanding and reasoning.However, as LLMs are able to process longer contexts, it becomes more challenging to evaluate whether they have acquired certain capabilities, since the length of text (e.g., 200K tokens) they can process far exceeds what humans can reliably assess in a reasonable duration.In this paper, we propose using complex synthetic tasks as a proxy evaluation method, and present S3Eval, a Synthetic, Scalable, Systematic evaluation suite for LLMs evaluation.The synthetic nature of S3Eval provides users full control over the dataset, allowing them to systematically probe LLM capabilities by scaling text length and varying task difficulty across diverse scenarios.The strong correlation between S3Eval and real-world benchmarks demonstrates the soundness of using S3Eval for evaluation of LLMs.S3Eval provides a flexible and infinite long-context data generation method. We have generated a comprehensive dataset called S3Eval-Standard, and experimental results have shown that it poses significant challenges for all existing LLMs.

## introduction large language models (llms) have greatly propelled significant advancements in natural language processing (nlp), such as openai @xcite , llama @xcite , @xcite , and others. these models perform well in many nlp tasks and claim to have made progress in advanced capabilities such as reasoning, long-context understanding, real task performance and so on. however, existing benchmarks @xcite often fail when it comes to evaluating extremely long-context llms or analysing the controllable characteristics and limitations of llms. for long-context understanding, previous work has often evaluated llms using the scope of language modeling metrics (i.e., perplexity) @xcite or the performance on simple artificial tasks @xcite @xcite . there is a widely used evaluation method known as needlein-a-haystack (kamradt, 2023), as shown in figure 1 . in this method, a vital piece of information is concealed within a lengthy document, resembling a haystack, and the model's objective is to locate and retrieve this hidden key information. however, these evaluation tasks tend to lack complexity and are narrowly focused on simple comprehension, which is misaligned with the sophistication required for real-world downstream applications. while recent work has made great progress on building evaluation benchmarks at longer context lengths with real-world use cases (e.g, question answering) @xcite , these manually annotated datasets often lack the scale and diversity to thoroughly assess performance on extended context lengths. for example, existing benchmarks struggle to effectively evaluate llms that claim an ability to process contexts up to 100k tokens, due to the limited capacity of human annotation for very long text. developing more scalable and diverse evaluation datasets, potentially leveraging automated supervision, remains an open challenge. for reasoning analysis @xcite @xcite , conducting both qualitative and quantitative analysis of answers and reasoning processes provides important insights. however, existing benchmarks lack the ability to precisely control the distribution of the dataset, limiting their utility for in-depth research analysis. in other words, the nature of these benchmarks makes it challenging for developers to identify the specific weaknesses of their llms. more configurable and granular benchmarks are needed to enable detailed analysis of model performance. in addition, these benchmarks often draw their evaluation data from nlp tasks that have been extensively studied and are likely to be used in the training corpus of llms. the potential data leakage makes the evaluation less convincing. in this paper, we propose a new evaluation suite called s3eval, which addresses the aforementioned issues by using a complex synthetic task -sql execution -as a proxy for the performance of llms on realistic reasoning tasks. as shown in figure 2 , inspired by the work of tapex @xcite , s3eval is based on the sql execution task. specially, given a randomly generated table and a random sql query, s3eval evaluates whether llms can return the correct execution results. s3eval has three notable characteristics: (1) it is synthetic, with no table or sql query present in the llm training corpus. the tasks use complex, grammatically correct sql syntax, making them very challenging. (2) it is scalable, allowing users to customize the benchmark to any length and difficulty. (3) it is systematic, containing diverse reasoning types and operations. this enables comprehensive evaluation of llm capabilities. with these powerful features, developers can extend the context to really long lengths and generate meaningful sql statements using s3eval. we conducted comprehensive multi-perspective experiments on several popular llms using s3eval. experimental results demonstrated that the performance of llms on s3eval aligns closely with their performance on mainstream llm bench-marks. while llms have shown impressive capabilities, our work reveals limitations in their ability to leverage long contexts, since we observe performance degradation of almost all llms in long-context settings. by carefully studying experimental results, we can work to pinpoint situations where llms tend to fail and summarize valuable insights. in the era of rapid llm development, the most significant contribution of s3eval lies in its effectiveness as a method for long-context evaluation. capable of generating evaluation data of infinite length, it ensures that assessments are not only reasonable but also sufficiently challenging. ## synthetic: suite and benchmark in this section, we introduce the details of the s3eval evaluation suite (as shown in figure 2 ) and the new benchmark we proposed. ## correlation with realistic benchmark in this section, we describe the details of synthesizing the evaluation data (section 2.1) and verify the correlation between our synthetic suite s3eval and real-world benchmark results. ## scalable: unlimited evaluation resources s3eval provides a unique capability to generate infinite number of examples (section 4.1) with infinite length (section 4.2). ## systematic suite: controllable analysis s3eval provides a comprehensive framework that empowers developers to synthesize diverse evaluation examples for systematically assessing llms from multiple perspectives. in this section, inspired by the work of lost in the middle (liu et al., 2023b), we first analyze the impact of answer position on performance (section 5.1). then we evaluate llms from different viewpoints, and we have conducted some initial explorations on the reasoning types analysis (section 5.3). last, we provide some insights by analyzing llms on three selected sql templates (section b.2). these experiments reveal counter-intuitive performance trends and new discoveries that may inspire further research and extension of the work. ## related work evaluating large language models (llms) has garnered significant interest in the nlp community @xcite . this allows us to gain a deeper understanding of the specific capabilities and limitations of llms while guiding further research. researchers proposed mmlu @xcite to measure the knowledge acquired by a language model during pre-training. in recent years, with the development of llms, a series of general evaluation benchmarks have emerged. for instance, bbh @xcite and agieval @xcite assess the reasoning ablitities. gsm8k @xcite evalutes the math reasoning, humaneval (chen et al., 2021a) and mbpp (austin et al., 2021) measure code capalities. our work aims to provide an evaluation suite for measuring reasoning ability. many previous works on long-text modeling rely on the perplexity @xcite or performance on simple artificial tasks @xcite @xcite . concurrently, zero-scrolls @xcite , l-eval @xcite and @xcite are proposed as evaluation benchmarks for long-text modeling. however, these benchmarks are built from existing public datasets and have fixed evaluation types. in contrast, s3eval can effectively assess comprehension of infinitely long-context. furthermore, s3eval allows customization of settings to generate evaluation data that meets specific needs, enabling effective evaluation of model deficiencies and discovery of new insights into llms. ## conclusion in this paper, we have introduced s3eval, a novel synthetic evaluation suite for llms using sql execution. s3eval represents a scalable and systematic approach to evaluate llms on a dynamic task. our experiments demonstrate strong correlation between s3eval and traditional evaluation benchmarks. the key innovations of s3eval are its flexibility, allowing unlimited context length and unlimited evaluation examples, and its fine-grained, systematic nature which enables detailed analysis of model capabilities and flaws. most importantly, for long-context evaluation, s3eval can generate evaluation data of infinite length. this type of task is not only challenging but also rooted in real-world tasks. considering the rapid development of llms, even as llm lengths extend significantly, s3eval can serve as a valuable benchmark for llm development and contribute to the community.