Turning the Tide: Repository-based Code Reflection (2025)

Abstract: Code large language models (LLMs) enhance programming by understanding and generating code across languages, offering intelligent feedback, bug detection, and code updates through reflection, improving development efficiency and accessibility. While benchmarks (e.g. HumanEval/LiveCodeBench) evaluate code generation and real-world relevance, previous works ignores the scenario of modifying code in repositories. Considering challenges remaining in improving reflection capabilities and avoiding data contamination in dynamic benchmarks, we introduce , a challenging benchmark for evaluating code understanding and generation in multi-file repository contexts, featuring 1,888 rigorously filtered test cases across 6 programming languages to ensure diversity, correctness, and high difficulty. Further, we create , a large-scale, quality-filtered instruction-tuning dataset derived from diverse sources, used to train through a two-turn dialogue process involving code generation and error-driven repair. The leaderboard evaluates over 40 LLMs to reflect the model performance of repository-based code reflection.

## introduction code large language models (llms) @xcite @xcite @xcite represent a significant advancement in comprehending and producing code across numerous programming languages. fueled by extensive training on massive code repositories, llms empower developers by offering intelligent feedback, identifying potential bugs, and updating code snippets from human instructions. code reflection refers to the ability of llms to examine and modify their previous responses. using reflection, llms can streamline the development process, boost efficiency, and make programming more accessible to a wider number of developers. most previous works primarily focus on generating and evaluating functionally correct code from human instructions illustrated in figure 1 , such as humaneval/mbpp @xcite and multipl-e @xcite . more recent llms (e.g. claude3.7 @xcite and qwen3 @xcite ) and code benchmarks (e.g. livecodebench @xcite , @xcite , and big-codebench @xcite ) aim to increase the difficulty, complexity, and real-world relevance by introducing more libraries, algorithms, and programming languages. the recently proposed benchmark aider-polyglot @xcite ) covering 6 languages measures the ability of llms to apply code changes to source files without human intervention, reflecting a more realistic development workflow. however, the aider benchmark is limited by a repository of programming exercises, where the repositories have been included in the pre-training data. there is still a lack of knowledge on how to improve the repository-based code reflection capability of llms and build dynamic benchmarks to measure the actual llm capabilities to avoid data hacking. to explore the llm capability of repositorybased code reflection, we propose an automatic creation pipeline to dynamically update the largescale instruction corpus and the evaluation benchmark to avoid data hacking. this paper introduces livereporeflection, a challenging benchmark for evaluating code understanding and generation in repository-style multi-file contexts, drawing inspiration from the aider benchmarks and exercism problems. to ensure data consistency and facilitate realistic evaluations, livereporeflection comprises coding problems organized with a defined repository structure, including problem definitions, reference answers, code signatures, unit tests, and environment support files. unlike the potentially redundant data from exercism, livereporeflection employs an optimized and streamlined file structure. furthermore, the paper details the construction of reporeflection-instruct, a large-scale instruction tuning dataset derived from diverse sources and filtered for quality, which is used to train reporeflectioncoder, a code-focused large language model. the data generation process for reporeflection-instruct involves a sophisticated multi-turn dialogue simulation encompassing code generation, error-driven repair, and style standardization, aiming to enhance the model's ability to handle complex coding tasks and iterative interactions. the contributions are summarized as follows: ## automatic and dynamic pipeline for repository-based code reflection repository code data file structure following aider code edit benchmark @xcite and aider polyglot benchmark @xcite , we collect 702 coding problems from exercism foot_0 for c++, go, java, javascript, python, and rust, available at different repositories foot_1 . then we remove the 225 coding problems in the aider polyglot benchmark @xcite tests and some erroneous data, and we retain about 473 coding problems to keep the file structure of newly generated code problems the same as these code problems. however, this code data may contain redundancy, as the purpose of these coding problems is to help people complete courses and become proficient in programming languages. illustrated in figure 7 in appendix a, we design a standardized repository code data file structure and streamline the file structure to the minimum size while ensuring normal evaluation compared to them. these data are used to guide the llms to generate the repository code file structure format of the newly generated coding problems but do not participate in guiding the content of the new coding problems illustrated in figure 2 . ## livereporeflection benchmark selection of executable program using our automated pipeline, we generate 100k coding program repository cases. we run them in a sandbox, including environment setup, compilation and testing, discarding any case that all llms can passe unit tests for too easy or exceeded 180s running time for too slow. finally, we minimized same topic overlap per language to maximize diversity and retained 10k high-difficulty, high-correctness cases. ## training of reporeflectioncoder source data creation and quality-based reject sampling we collect approximately 500,000 code examples (including 400,000 newly automatically and dynamically generated ones) to construct the reporeflection-instruct corpus. to ensure data quality, strict rejection sampling was applied to all examples, and only repositories meeting all five of the following criteria were retained as a highquality fine-tuning dataset: 1) at least one unit test file; 2) at least one reference answer file; 3) the ## analysis 6.1 performance comparison between livereporeflection and aider polyglot benchmark figure 5 reveals that while pass@1 and pass@2 scores on livereporeflection correlate moderately with the older aider polyglot benchmark (r 2 ≈ 0.65 for pass@1 and 0.78 for pass@2), nearly all points fall below the @xmath0, indicating consistently lower absolute pass rates on livere-poreflection and thus greater task difficulty. wellformedness remains highly consistent between the two datasets (r 2 ≈ 0.83 full-file, 0.70 diff-based), showing that syntactic and structural constraints are enforced similarly. in stark contrast, fix-weight exhibits very weak alignment (r 2 ≈ 0.21 and 0.04), underscoring divergent repair performance and demonstrating that livereporeflection more reliably challenges and discriminates model debugging capabilities. moreover, because the aider polyglot benchmark has been publicly available since mid-2024 and may have been subject to overfitting by recent model releases, our freshly curated liv-ereporeflection, compiled from never-before-seen repositories through may 2025, provides a more robust and less biased evaluation of real-world code generation and repair. all models exhibit strictly increasing performance with diminishing marginal gains beyond @xmath1; patch-based editing yields more uniform improvements across samples, whereas full-file generation achieves higher absolute pass rates. ## conclusion this paper proposes livereporeflection, a highdifficulty code reflection benchmark for multi-file repository scenarios. by combining automated pipelines with manual verification, we built a total of 1,888 strictly screened test cases covering six programming languages. based on multi-source high-quality corpora and multi-round dialogue generation strategies, we built the reporeflection-instruct instruction set and trained reporeflec-tioncoder, achieving significant performance improvements. experimental results show that liv-ereporeflection can truly and effectively measure the model's reflection and repair capabilities in cross-file dependency and iterative repair scenarios, providing a solid foundation for subsequent research.