D yna C ode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation (2025)

Abstract: The rapid advancement of large language models (LLMs) has significantly improved their performance in code generation tasks. However, existing code benchmarks remain static, consisting of fixed datasets with predefined problems. This makes them vulnerable to memorization during training, where LLMs recall specific test cases instead of generalizing to new problems, leading to data contamination and unreliable evaluation results. To address these issues, we introduce DynaCode, a dynamic, complexity-aware benchmark that overcomes the limitations of static datasets. DynaCode evaluates LLMs systematically using a complexity-aware metric, incorporating both code complexity and call-graph structures. DynaCode achieves large-scale diversity, generating up to 189 million unique nested code problems across 4 units of code complexity and 16 types of call graphs. Results on 12 latest LLMs show an average performance drop of 16.8 to 45.7 compared to MBPP+, with performance progressively decreasing as complexity increases. This demonstrates DynaCode’s ability to effectively differentiate model performance based on code complexity and how different parts of a program interact. Our benchmark and evaluation code are available at https://github.com/HWH-2000/DynaCode.

## introduction the performance of large language models (llms) in code generation has garnered significant attention @xcite . with their powerful language comprehension abilities, llms are now capable of autonomously generating high-quality code and, to some extent, addressing complex programming challenges. these advancements have not only accelerated the software development process but have also had a profound impact on enhancing developer productivity @xcite . however, as llms advance, reliable benchmarks for code generation have become increasingly crucial for evaluating and selecting suitable llms. currently, the evaluation of llms' code generation capabilities primarily relies on standardized benchmarks such as humaneval @xcite , mbpp @xcite , codexglue @xcite , and classeval @xcite . these benchmarks provide an initial reference for assessing the performance of llms in code generation by evaluating the functionality and correctness of the generated code. moreover, recent work such as evalplus @xcite , bigcodebench @xcite , cruxeval @xcite , and evoeval @xcite aim to enhance evaluation quality by expanding test cases and employing techniques like prompt transformation to convert prompts into more appropriate ones for more precise evaluation. despite these developments, existing bench-marks exhibit two notable limitations: data contamination. existing benchmarks are static and small-scale, making them easily accessible during training and allowing models to memorize test cases instead of generalizing to unseen problems. meta-llama-3-8b-instruct @xcite and phi-2 (microsoft, 2024) have been reported to exhibit data contamination @xcite , suggesting that the model may "memorize" specific test cases or code snippets, which could compromise the accuracy and fairness of the evaluation process. uncontrollable complexity. existing benchmarks lack systematic complexity control, making it challenging to evaluate llm performance across different task complexities. while some works @xcite define code complexity using simple metrics such as lines of code and time complexity, these measures fail to capture deep nesting and complex execution dependencies, thereby leaving a critical gap in assessing real-world code generation capabilities. to address these limitations, we propose dyna-code, a novel dynamic evaluation framework that automatically creates python code benchmarks by classifying code problems based on complexity and forming nested problems using call graphs. this benchmark provides a more comprehensive and fair evaluation of code generated by llms. specifically, dynacode categorizes code problems into multiple code problem units and, for each unit, constructs call-graph structures of varying complexity. in doing so, it establishes complexity-aware metrics along two dimensions: code complexity and call-graph complexity. compared to traditional static benchmarks, dynacode offers a significantly more diverse and complex evaluation. as shown in figure 1 , meta-llama-3-8b-instruct (meta ai, 2024a), a model that exhibits data contamination, shows a larger performance drop from mbpp and mbpp+ to dynacode. dynacode generating approximately 189 million unique code generation tasks across 4 units of code complexity and 16 call-graph structures. by assessing llms from both code complexity and call-graph complexity perspectives, dynacode provides a structured and scalable evaluation framework while mitigating data contamination. to further investigate llm limitations, we analyzed 4279 error examples, categorizing errors into 3 distinct types. our analysis reveals that llms perform well on sequential call graphs but struggle with complex, multi-branch dependencies, highlighting their difficulty in handling deeply nested execution flows and long-range function interactions. in summary, our major contributions are listed as follows: ## methodology in this section, we present the construction approach and process of dynacode, as shown in figure 2. specifically, we first introduce the dynamic evaluation strategy based on call graphs in sec. 3.1, explaining how they capture the relationships and dependencies within a program. in sec. 3.2, we introduce a complexity matrix that measures code and graph complexity, essential for evaluating tasks of varying difficulty and capturing program interactions. finally, we describe the process of generating benchmarks in dynacode in sec. 3.3. ## conclusion we present dynacode, a dynamic, complexityaware benchmark designed to systematically evaluate llms on code generation tasks. by integrating code complexity with call-graph structures, dy-nacode generates nested code problems that capture varying levels of code complexity and interfunction dependencies. we evaluated 12 of the latest llms, and the results reveal significant performance drops as both unit and graph complexity increase, highlighting dynacode's ability to systematically assess llms. notably, dynacode also addresses the challenge of data contamination and demonstrates its capacity to mitigate this limitation. in summary, dynacode provides a new perspec-tive for examining and analyzing llms, offering a scalable framework for more comprehensive and reliable llm evaluation.