CRS core: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells (2025)

Abstract: The task of automated code review has recently gained a lot of attention from the machine learning community. However, current review comment evaluation metrics rely on comparisons with a human-written reference for a given code change (also called a diff ). Furthermore, code review is a one-to-many problem, like generation and summarization, with many “valid reviews” for a diff. Thus, we develop CRScore — a reference-free metric to measure dimensions of review quality like conciseness, comprehensiveness, and relevance. We design CRScore to evaluate reviews in a way that is grounded in claims and potential issues detected in the code by LLMs and static analyzers. We demonstrate that CRScore can produce valid, fine-grained scores of review quality that have the greatest alignment with human judgment among open-source metrics (0.54 Spearman correlation) and are more sensitive than reference-based metrics. We also release a corpus of 2.9k human-annotated review quality scores for machine-generated and GitHub review comments to support the development of automated metrics.

## introduction code review is an essential quality control tool for software engineers to ensure that source code is free of bugs and upholds standards @xcite . software engineers prefer lightweight, asynchronous review processes, as enabled through github's review comment feature, over formal, in-person reviews @xcite . this has led to the creation of benchmarks for automated generation of natural language (nl) review comments @xcite . however, these benchmarks use reference-based evaluation metrics like bleu @xcite , which have been shown to have low validity (re-1 https://github.com/atharva-naik/crscore iter, 2018; @xcite , especially when paired with limited and low-quality references. code review is fundamentally a one-tomany problem, where a given diff can have multiple possible issues that a review can tackle. having a limited number of reference reviews (e.g., one per diff in codereviewer @xcite ) leads to unfairly low scores with reference-based metrics. for example, for the diff shown in figure 1 , the ground truth review focuses on whether the tohexstring() function could cause a performance issue. however, the model-generated review focuses on tohexstring().equals("0000000000000000") being an odd condition with a scenario where its being triggered is unlikely, which is also a valid review for the diff. however, the bleu score value for the model-generated review is very low, specifically at 0.0458, due to poor n-gram overlap. additionally, the references can also be low-quality due to missing context, as shown in motivated by these drawbacks, we propose crscore, an automated but reference-free evaluation metric that uses dimensions of review quality from prior work @xcite . in particular, comprehensiveness -does the review convey all the necessary information? conciseness -does the review only convey the necessary information in an efficient way and relevance -is all the information on topic. we operationalize our metric through a two-step process: 1 generate a list of pseudo-references spanning information like possible claims, issues, and implications of a code change, and 2 use semantic textual similarity (sts) to align parts of the review to the pseudo-references. to generate the pseudoreferences, we use a neuro-symbolic approach that combines large language models (llms) and code analysis tools (cats) that can detect formatting errors, faulty design patterns (code smells rasheed et al. 2024), etc. we combine these methods for more exhaustive pseudo-references and to overcome drawbacks of each method (section a.1). finally, we demonstrate the validity of crscore by human evaluation of the quality of pseudoreferences and by measuring the alignment of our metrics with human judgment. we show that a large number (82.6%) of pseudo-references generated by crscore are correct, and that it has the greatest alignment with (spearman correlation 0.5431) and sensitivity to (figure 8 ) the human judgment of review quality, as compared to that of the reference-based metrics. contributions. (1) we propose an automated reference-free metric that combines the advantages of llms and cats to measure review quality along fine-grained dimensions. (2) we collect human annotations of pseudo-reference quality and review conciseness, comprehensiveness, and relevance to validate our approach. we plan to make these scores publicly available as a resource for continued development of automated metrics. (3) we benchmark several llms of code for code review on the codereviewer dataset using crscore and compare it with reference based metrics. the ground truth and model-generated reviews focus on different topics, like the performance of the added check, and how likely it is to be triggered. however, a reference-based metric like the bleu score assigns this review a low score of 0.0458. ## related work in this section, we summarize the limitations of reference-based evaluation, the need for better code review evaluation metrics, inspiration from reference-free evaluation for other tasks, and how code smells can be leveraged to evaluate reviews. ## operationalizing crscore motivated by the one-to-many nature of code reviews, noisy references, and the pitfalls of reference-based automated metrics, we develop crscore -a reference-free, quality dimensionbased automated metric. as shown in figure 2 , instead of relying on explicit references, our metric generates "pseudo-references" from the code change spanning claims, implications, and issues/smells that hurt maintainability -in other words, some topics that a review should address @xcite . then we use semantic textual similarity (sts) measures to quantify how much these topics are addressed by a code review as shown in figure 2 through the lens of three quality dimensions: conciseness, comprehensiveness, and relevance. they capture review quality, similar to precision, recall, and f-score used for classification and retrieval. we describe the three main components of our framework -the quality dimensions, pseudo-reference generation, and similarity measurement -below. ## validating crscore to be a valid metric, crscore needs to satisfy a few properties. firstly, the generated pseudo-references should have few errors and unverifiable claims while being exhaustive. additionally, we want our dimension-level scores (con, comp, and rel), especially rel, to correlate with human judgment for each dimension. the most important property, is arguably the ability of our metric (especially rel) to rank review generation systems in the same order as humans. in the subsequent sections, we describe how we design the experiments to collect annotations for these properties (section 4.1, 4.2), choose a set of systems to be rated by humans (section 4.4), and set up reference-based metrics (section 4.5) for comparison crscore. ## discussion in this work we identify issues with current code review evaluation benchmarks like codereviewer @xcite , which fail to capture the oneto-many nature of code review and contain noisy references. to enable auditing current evaluation metrics and aid the development of a better metric, we propose three review quality dimensions -conciseness, comprehensiveness, and relevance based on current literature on reference-free evaluation @xcite @xcite . to ground these dimensions in topics that reviews should address @xcite , we propose an automated pseudo-reference generation pipeline that leverages llms and code smell detectors. we validate the quality of these pseudoreferences via human evaluation. based on these dimensions, we develop reliable guidelines for coding review quality using pseudo-references and collect annotations for 9 review generation systems and "ground truth" reviews for the codereviewer dataset spanning python, java, and javascript. the collected annotations show that current reference-based metrics except for llm-as-ajudge with powerful closed source models like gpt-4o fail to capture human preferences, which is further compounded by humans preferring some models over the references. we propose crscore as a metric to capture the three dimensions using the pseudo-references generated by open source llms and static analysis tools like code smell detectors through sts models like sentence transformers @xcite . our approach has the second greatest alignment with human preferences, lagging only behind laaj-gpt and the greatest alignment among open source metrics. it achieves the best review quality correlation scores (0.4577 τ and 0.5425 r s ), system ranking correlations (0.95 r s and 0.8889 τ ), as well as the greatest sensitivity (figure 8 ) among open source metrics. additionally, it greatly improves over using the magicoder llm directly as a judge, while also being more efficient (since the llm needs to be run only once to produce the pseduo-references as compared to the llm-as-a-judge, where it needs to be re-run for every review model evaluated). however, we also note that despite its reproducibility, efficiency and greater alignment with human preferences, crscore exhibits only moderate correlations with review quality, and systematically underestimates or overestimates it in some cases. ## conclusion our work takes the first steps towards addressing the challenges involved in evaluating the quality of code reviews. we propose useful dimensions for capturing review quality, and offer crscore as an automated, efficient, open-source and reproducible metric for capturing them. we collect a dataset of human judgment of review quality scores to show the validity of crsore. we compare crscore with 7 reference-based metrics, for 9 review generation systems using the collected annotations. our metric achieves the best alignment with human preferences among open source metrics and is the most sensitive. however, there is scope for improvement by developing better pseudo-reference generation and sts matching methods to try and match the performance of closed-source models like gpt-4o in judging review quality.