C ode R eview QA : The Code Review Comprehension Assessment for Large Language Models (2025)

Abstract: State-of-the-art large language models (LLMs) have demonstrated impressive code generation capabilities but struggle with real-world software engineering tasks, such as revising source code to address code reviews, hindering their practical use. Code review comments are often implicit, ambiguous, and colloquial, requiring models to grasp both code and human intent. This challenge calls for evaluating large language models’ ability to bridge both technical and conversational contexts. While existing work has employed the automated code refinement (ACR) task to resolve these comments, current evaluation methods fall short, relying on text matching metrics that provide limited insight into model failures and remain susceptible to training data contamination.To address these limitations, we introduce a novel evaluation benchmark, \textbf{CodeReviewQA} that enables us to conduct fine-grained assessment of model capabilities and mitigate data contamination risks.In CodeReviewQA, we decompose the generation task of code refinement into \textbf{three essential reasoning steps} : \textit{change type recognition} (CTR), \textit{change localisation} (CL), and \textit{solution identification} (SI). Each step is reformulated as multiple-choice questions with varied difficulty levels, enabling precise assessment of model capabilities, while mitigating data contamination risks. Our comprehensive evaluation spans 72 recently released large language models on \textbf{900 manually curated, high-quality examples} across nine programming languages. Our results show that CodeReviewQA is able to expose specific model weaknesses in code review comprehension, disentangled from their generative automated code refinement results.

## introduction the proficiency of state-of-the-art large language models (llms) in code generation has garnered significant attention (zhuo et al., 2025), demonstrating their ability to follow explicit instructions to author code. however, their competency in realworld software engineering environments remains limited (pornprasit and tantithamthavorn, 2024), particularly in collaborative tasks involving colloquial and complex forms of communication. a quintessential example is code reviewing, where review comments (yang et al., 2023; efstathiou and spinellis, 2018) represent natural communication between developers with a shared mental model, often resulting in under-specified, ambiguous, and implicit expressions of intent. for example, this comment "for all of the fuzz tests, does it make sense to have versions for 'len_prefixed' both 'true' and 'false'?" is a rhetorical question that expresses an intended code change without explicitly detailing the literal change requirement and instructions. as a result, the ability to resolve code review comments not only requires proficiency in understanding and generating code, but also the ability to comprehend the communicative intent behind natural language code reviews in relation to the source code that they address. therefore, assessing how llms resolve code review comments serves as a crucial testbed for their proficiency in understanding and following implicit, conversational instructions in software development. success in this domain would significantly advance automated software development assistance, potentially reducing developer workload and improving code quality. to evaluate model capability in resolving code review comments, prior work has explored the automated code refinement (acr) task using both small scale neural language models (tufano et al., 2022; thongtanunam et al., 2022) and llms (guo et al., 2024; pornprasit and tantithamthavorn, 2024), aiming to automatically revise source code based on code review comments. while these efforts have advanced this direction, several critical challenges remain unaddressed. firstly, current automatic evaluation approaches rely heavily on metrics such as exact match and bleu @xcite , which merely capture surface-level token similarities between the generated output and the ground truth, without reflecting intermediate comprehension capabilities. secondly, as these evaluation benchmarks typically use popular github projects, they risk data contamination from training data in llms @xcite , potentially obfuscating true model capabilities. as a result, there are currently no suitable approaches for assessing the capabilities of state-of-the-art llms in automated code refinement. to address these challenges, we introduce a novel evaluation benchmark that enables comprehensive assessment of automated code refinement capabilities. our benchmark decomposes the original one-step generative task into three underlying reasoning steps: change type recognition (ctr), change localisation (cl), and solution identification (si). these components represent essential cognitive processes required for understanding code review comments, and subsequently generating the required code revision. by reflecting explicit intermediate reasoning steps, our benchmark provides fine-grained feedback of model failures to support future model development. to mitigate potential data contamination, we formulate each reasoning step as a multiple-choice question answering (mcqa) probe with synthetic answers. this approach transforms the original task into unfamiliar formats with unseen solutions, demanding proficiency in code review comprehension, rather than sequence memorisation of contaminated data @xcite . furthermore, we leverage mcqa's flexibility to introduce distractor variation strategies, enabling assessment of model understanding across different difficulty levels. to avoid the pervasive issue of noisy examples present in past benchmarks (tufano et al., 2024) and ensure high-quality evaluation data, we manually verify and curate 900 valid code refinement examples that cannot be automated by traditional software engineering tools. these examples are sourced from 199 repositories, reflecting nine of the most popular programming languages on github. finally, we evaluate 72 state-of-theart code-intelligent llms, providing an extensive benchmark to facilitate future research. our contributions can be summarised as follows: ## codereviewqa: mcqa probes effective acr relies heavily on the ability to comprehend r nl under the context of h pre . rather than focusing this task as a sequence-to-sequence translation problem like the prior works, we argue that the model must be able to: 1) reason about the type of change r nl is requesting and 2) identify the relevant lines of code in h pre that is the subject of the change; and 3) formulate the required code changes from a wide action space of potential code edits that can be performed on h pre , before generating the code revision h post . the inability to perform the final code generation step may be caused by any failure point amongst this multi-step reasoning process. additionally, any failure within the intermediary reasoning steps might be propagated from a failure in a prior reasoning step, which obfuscates the specific incompetencies of the model. to assess the proficiency of llms in acr, we propose three mcqa probes, each representing a specific intermediate reasoning step. see figure 2 for an example of the mcqa structure. below, we describe the construction of each mcqa probe. ## dataset preparation data source. we built our benchmark based on the most recently published automated code refinement dataset (guo et al., 2024). this multilingual dataset was constructed from code reviews that occurred after january 1, 2022. to ensure that we have a sizable amount of clean data for each of the programming languages in our benchmark, we only include the nine most popular programming languages on github i.e. c, c++, c#, go, java, javascript, php, python, ruby. these 9,367 examples were mined from 259 repositories, filtered from a list of the most starred github projects. ## results overview to compare how models of different scales perform on codereviewqa, we conducted experiments using all 72 models. due to space limitations, complete results on acr and all three mcqa probes are presented in appendix f. in acr vs mcqa probes. ## insights from probing acr failures what does mcqa probes reveal about model failures in acr? we use our mcqa probes to investigate acr failure cases of the top performing models in the results are presented in on the other hand, we also analysed the successful cases where the model achieves an exact match. intuitively, if a model can achieve exact match on an example, it should be able to fully comprehend the code review, thus correctly answering all probes. however, the models could not accurately answer all probes for 49.0%-99.6% of their successful cases. this trend shows a strict inverse relationship with model size, where llama-3.1-70b-instruct and qwen2.5-coder-3b-instruct could not consistently complete all of the probes for 49.0% and 99.6% of their successful cases, respectively. the ability to complete these acr examples verbatim without a prerequisite understanding of the intent behind the code reviews allude to prior exposure and rote memorisation of these examples. ## evaluating data contamination to what extent is codereviewqa resistant to data contamination? we utilise two canonical metrics for measuring data contamination, perplexity @xcite and n-gram accuracy @xcite . perplexity is an information-theoretic metric, which quantifies the uncertainty of a language model in a token sequence @xcite , which can be formulated as: for this experiment, we use the largest and newest models that we can support from the most popular model families, as they are most likely to exhibit memorisation (kiyomaru et al., 2024). we use base versions of models as instructiontuning optimises for responses to prompts rather than completing sequences verbatim. to test the effectiveness of mcqa reformulation in mitigating data contamination, we compare our benchmark in mcqa probe form with the original acr form, as well as the most widely used acr benchmarks, i.e. codereviewer (li et al., 2022) and codereview-new (guo et al., 2024). we find that codereviewqa in the original acr format yields similar results to codereview-new, which is within expectation as one is simply a curated subset of the other. however, when reformulating into the mcqa probe format, our benchmark yields significantly higher perplexity than all past benchmarks with lower 5-gram accuracies, despite using the same examples. therefore, we find that mcqa reformulation with synthetic questions and answers does mitigate the effects of data contamination, allowing for the reuse of code reviews that may have been previously included in the training corpus. coinciding with our experimental results, models that perform well on acr with only memorisation can be exposed when evaluated with mcqa probes on the same examples. ## conclusion in this study, we focus on evaluating recent large language models' capabilities in automated code refinement, a challenging task that requires an understanding of the intended code revisions behind natural language code reviews, before subsequently performing them. we addressed two key limitations in existing work, the inability of text matching metrics to provide fine-grained feedback on intermediate model failures and the potential for benchmark contamination. to this end, we propose codereviewqa, which consists of 900 manually curated high-quality code review examples. we reformulated the generative task of automated code refinement into three intermediate reasoning probes, which are presented in the multiple choice question and answering format. our experimental results across 72 state-of-the-art large language models revealed capability differences that traditional evaluation metrics failed to capture. additionally, our evaluation of data contamination demonstrated that task reformulation effectively mitigates these concerns, exposing cases of memorisation without comprehension.