Skeleton-Guided-Translation: A Benchmarking Framework for Code Repository Translation with Fine-Grained Quality Evaluation (2025)

Abstract: Code translation benchmarks are essential for evaluating the accuracy and efficiency of LLM-based systems. Existing benchmarks mainly target individual functions, overlooking repository-level challenges like intermodule coherence and dependency management. Recent repository-level efforts exist, but suffer from poor maintainability and coarse evaluation granularity. We introduce Skeleton-Guided-Translation, a framework for benchmarking Java-to-C# translation at the repository level, featuring fine-grained quality evaluation. It follows a two-step process: first translating repository “skeletons”, then refining the entire repository guided by these skeletons. Based on this, we present TRANSREPO-BENCH , the first test-driven benchmark of high-quality Java repositories paired with C# skeletons, unit tests, and build configurations. Our adaptive unit tests support multiple and incremental translations without manual tuning, enhancing automation and scalability. We also propose fine-grained metrics that evaluate translation quality per test case, overcoming limitations of binary metrics in distinguishing build failures. Evaluations using TRANSREPO-BENCH reveal issues like broken cross-file references, showing that our structured approach reduces dependency errors and preserves interface consistency.

## introduction large language models (llms) are reshaping software development, driving system modernization and legacy code migration. for example, migrating c to rust improves safety @xcite , and frameworks like tensorflow require synchronized multi-language updates. evaluating llms in migration tasks is key to assessing reliability. benchmarks provide quantitative insights for comparison and improvement, but existing ones focus on function-level tasks or competition-style problems @xcite @xcite , ignoring real-world complexities. repository-level translation is essential for managing dependencies, structure, and interconnected components @xcite , requiring reliable benchmarks to assess model performance. a major challenge in repository-level code translation is the absence of a systematic framework that enables fine-grained control over maintainability. for instance, updating parts of a java-based sdk often requires re-translating large portions of the corresponding c++ codebase, making small changes costly. without fine-grained control, maintainability suffers. another challenge is the lack of repository-level parallel corpora, complicating automated verification. line-by-line metrics like codebleu @xcite lack functional validation, and automatic test generation remains unreliable @xcite . a practical alternative is translating unit tests from the source library for systematic validation. however, ensuring test accuracy and consistency with translated code interfaces is crucial for reliable verification. the third challenge is that current metrics often miss nuanced translation outcomes, reducing usability. repotransbench @xcite , for example, uses a binary build success metric, ignoring partial successes. as figure 1 shows, this oversimplifies performance by neglecting cases where some components translate correctly while others fail. schaeffer et al. @xcite warn that threshold-based metrics can create misleading performance leaps. in contrast, continuous metrics, such as the percentage of successfully translated modules (e.g., 66.7%), improve usability by identifying failures, guiding fixes, and providing smoother, more reliable insights. ## motivation in this section, we use an example to illustrate the challenges involved in building a repositorylevel code translation benchmark and explain our solutions more effectively. ## transrepo-bench benchmark as shown in figure 3 , users receive the source repository and target skeleton, guiding llms to generate a complete target repository. correctness is verified using the target's unit tests within the testing environment. this section presents the benchmark content, details transrepo-bench 's construction, and introduces our fine-grained evaluation design. ## evaluation we first analyze llm performance on our benchmark, then highlight our framework's effectiveness in using repository skeletons for translation and fine-grained evaluation. ## conclusions we present skeleton-guided-translation and the transrepo-bench benchmark to address the challenges of evaluating llm-based repositorylevel code translation. we provide the "skeletons" that preserve file structures and interfaces, enabling fair and consistent evaluation of different llms with unified unit tests. moreover, we offer finegrained evaluation through detailed error localization, moving beyond simple pass/fail outcomes. our evaluation of transrepo-bench , a set of java repositories covered by high-quality tests, shows that even sota llms reach only 26.65% accuracy. using skeletons prevents partial errors from affecting correct modules, providing a finegrained assessment of build success and test pass rates. we also demonstrate the benefits of using skeletons to maintain interface consistency, enable fine-grained quality assessment, and support incremental translation.