L ib E volution E val: A Benchmark and Study for Version-Specific Code Generation (2025)

Abstract: Recent advancements in code completion models have primarily focused on local file contexts. However, these studies do not fully capture the complexity of real-world software development, which often requires the use of rapidly-evolving public libraries. To address this gap, we introduce LibEvolutionEval, a comprehensive study that emphasizes the need to understand library evolution to perform accurate in-line code completions. LibEvolutionEvaloffers a version-specific code-completion task across eight libraries as they evolve over the years, along with an in-depth analysis of the evolution of two widely used and well-maintained public libraries: PyTorch and Matplotlib. We evaluate several popular models and find that public library evolution significantly affects their performance. To mitigate this, we explored how retrieving version-specific library documentation and prompt-based techniques can enhance model capability in dealing with these fast-evolving packages. This suggests a promising path forward for better handling fast-evolving libraries. Our tasks will be made publicly available upon acceptance.

## introduction large language models for code (a.k.a. code llms) @xcite @xcite have significantly advanced developer productivity through improved code completion tasks. these models are pivotal not only in code completion, but also in debugging, code summarization, and language translation for software development @xcite @xcite . these models are usually evaluated either with code contest dataset @xcite or with a focus on local files for context to enhance the completion of the function @xcite @xcite @xcite . however, these studies do not fully encompass the complexities of real-world software development, which requires public libraries. complexity of code completion with public library apis increases, as the apis often evolve-some apis change their signature, some gets deprecated, while many new apis surfaced in this evaluation process @xcite . while some works perform code completion involving public libraries @xcite , use documentation of the library for prediction @xcite , and show that zero shot code completions suffer from hallucinations @xcite , these works do not focus on the rapidly evolving nature of public libraries. large language models are trained on extensive corpora of open-source code, which likely includes public libraries. consequently, while llmgenerated code may appear reasonable, it might not be accurate for the specific version of the library being used, leading to version-dependent performance issues. figure 1 shows that code llm's generation is correct for v2.2 but incorrect for v1.2. this variability is significant since developers often work with different library versionsnewest versions for current projects and older ones for legacy code maintenance. therefore, the developer's experience with coding assistants depending on llms for code completion can vary greatly depending on their specific use case. existing benchmarks and studies do not fully capture evolution, revealing a gap in our current evaluation and understanding of code llms. this work focuses on the following research questions: (1) does the performance of code llms change as the library evolves? (2) if yes, can retrieving versionspecific meta-data like library documentation mitigate the impact of library evolution on code completion? (3) with the evolution of libraries, new relationships between apis are introduced, while existing ones are altered. can code llms effectively adapt to these evolving relationships between apis? (4) does the introduction, modification, and deprecation of apis as libraries evolve to make it more challenging for code llms to perform accurate code completions? to investigate these research questions, we introduce libevolutioneval, a benchmark and detailed study specifically designed to understand the impact of public library evolution on code completion. libevolutioneval offers version-specific code-completion tasks spanning multiple years for eight librariestorch, torchvision, scipy, pil, tqdm, pyyaml, matplotlib, and pandas. libevolutioneval also performs a detailed analysis of the evolution of two popular libraries (torch and matplotlib) by providing version-specific meta-data, documentation retrieval tasks, and codecompletion tasks. it requires code llms to perform version-specific code completions under both realistic (where the evaluation examples are sampled from permissively licensed github repositories that is publicly available ) and controlled scenarios (where a template uses api documentation to create evaluation examples that is not publicly available ). it provides version-specific api documentation to investigate the impact of library evolution on embedding models during retrieval. it offers tasks based on completion type to compare: (1) completions guided by import statements and clear library prefixes with (2) completions that are object-oriented references and do not have a library-defined prefix. we call them direct and indirect code completions (see figure 3 ), and such tasks evaluate models' ability to adapt to evolving relationships between apis. furthermore, libevo-lutioneval also offers tasks based on granularity that compares overall developer experience with performance on specific apis that have been newly introduced, modified, or deprecated as the library evolves, providing us with insights on the impact of evolution against overall code-completion performance. we conducted a comprehensive evaluation using widely used code llms @xcite @xcite and embedding models @xcite to report the following insights. providing version-specific api documentation as a context improves code completion performance but does not entirely address inherent version-based bias in versionspecific code completions. • code llms perform indirect api completions better than direct api completions, demonstrating an understanding of evolving relationships between apis. • introduction, modification, and deprecation of apis make it harder for code llms to perform code completion where new models might forget old deprecated apis while old models cannot predict the latest apis. ## libevolutioneval: version-specific code completions each code completion example in libevolu-tioneval consists of code prompts ending at a position where the llm is tasked to complete the missing expression, typically involving one or more api calls to the public library under consideration as shown in figure 1 . the uniqueness of this dataset lies in its emphasis on version-specific api usage, reflecting scenarios where developers use llms to perform code completions for different versions of the same library. we evaluate code completions under two scenarios: realistic (github based) and controlled (documentation based). ## experimental setup models we benchmark public code llms: mistral (jiang et al., 2023), starcoder2 @xcite , gpt-4o-mini (openai, 2024) and codegen 1.0 @xcite . we benchmark versionspecific retrieval tasks using codesage @xcite and openai-ada-002 (openai, 2022). lastly, we conducted scaling experiments with star-coder @xcite (1b, 3b, 7b), starcoder2 (3b, 7b, 15b), and codesage (small, large). evaluation metrics for code completion, we concentrate on the correctness of apis called by calculating the f1 score @xcite . for documentation retrieval, to evaluate the performance of embedding models by using mean reciprocal rank (mrr), assessing how well they retrieve version-specific documentation and whether their performance varies with library evolution. ## related works large language models for code excel in various software development tasks @xcite @xcite , facilitating the developments of coding assistants. similarly, developments in code embedding models used for retrieval @xcite @xcite have fur-ther enhanced llms' capabilities. in this journey, evaluation benchmarks have played a pivotal role with numerous works developing benchmarks to evaluate code llms @xcite @xcite @xcite @xcite @xcite . these studies typically assess code completion abilities given local file contexts, both in-file @xcite @xcite and repository-level @xcite @xcite . however, they do not fully encompass the complexities of real-world software development, which requires extensive use of public libraries. some works have explored code completion involving public libraries @xcite @xcite , but they do not address the rapidly evolving nature of these libraries. to fill this gap, we introduce libevo-lutioneval that evaluates the performance of llms on code completion across multiple versions of public libraries, capturing their evolution and reflecting real-world scenarios where developers interact with different versions of the same library. ## conclusion in this paper, we introduced libevolutioneval, a comprehensive benchmark specifically designed to assess the performance of code large language models (code llms) in code completion tasks as public libraries evolve. our results demonstrate significant variability in llm performance based on the api version, highlighting the challenges of handling library evolution. the findings underscore the necessity for future advancements in code completion technologies to consider the dynamic nature of public libraries, aiming to improve developer productivity and accuracy in real-world settings.