Identifying Aspects in Peer Reviews (2025)

Abstract: Peer review is central to academic publishing, but the growing volume of submissions is straining the process. This motivates the development of computational approaches to support peer review. While each review is tailored to a specific paper, reviewers often make assessments according to certain *aspects* such as Novelty, which reflect the values of the research community. This alignment creates opportunities for standardizing the reviewing process, improving quality control, and enabling computational support. While prior work has demonstrated the potential of aspect analysis for peer review assistance, the notion of aspect remains poorly formalized. Existing approaches often derive aspects from review forms and guidelines, yet data-driven methods for aspect identification are underexplored. To address this gap, our work takes a bottom-up approach: we propose an operational definition of aspect and develop a data-driven schema for deriving aspects from a corpus of peer reviews. We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis. We further show how the choice of aspects can impact downstream applications, such as LLM-generated review detection. Our results lay a foundation for a principled and data-driven investigation of review aspects, and pave the path for new applications of NLP to support peer review.

## introduction peer review is an essential part of academic publishing. it is a complex, multifaceted process that requires a range of competencies including paper understanding, domain-specific knowledge, and critical thinking @xcite . ensuring review quality, especially among novice reviewers, is an open challenge @xcite . the increasing volume of publications puts further strain on the process, which motivates the development of computational approaches to support different stages of peer review, from reading the paper to the final decisionmaking by the program committees @xcite @xcite @xcite @xcite . peer reviews are the central component of the reviewing process. while individual reviews can vary widely, reviewers within the same community tend to focus on a specific set of general quality categories, or aspects, such as clarity and novelty. these aspects can be found in review forms, instructional materials, guidelines, and the resulting review texts. aspects allow comparison of submissions across different dimensions of quality. a comprehensive set of aspects shared among reviewers is critical for ensuring reviewing quality and consistency, and prior work has demonstrated the potential of aspect-based tools to support the review writing process @xcite . yet, several open questions remain. first, what is an aspect? while most prior work derives aspects from review forms and guidelines, the lack of an operational definition of aspect prevents the comparison of aspect schemata across studies. second, what aspect granularity is appropriate for different tasks? while prior work operates with top-down, coarse-grained aspect schemata derived from review forms and guidelines, it is unclear whether they are comprehensive or provide sufficient granularity for nlp applications. third, how can fine-grained aspect analysis support peer review? while aspects have been applied to certain tasks, the tasks that require a higher level of granularity are underexplored. to address these questions, this work introduces an alternative, data-driven approach to peer review aspect analysis. we propose an operational definition of aspect grounded in its role within the evaluation process. we develop a semi-automatic approach that leverages a state-of-the-art large language model (llm) to identify aspects in reviews, and apply it to a large collection of peer reviews to extract aspects in a bottom-up fashion. building on these results, we develop a multi-level taxonomy of aspects and present a novel dataset of peer reviews augmented with their corresponding aspects. our dataset facilitates the exploration of two tasks: predicting the review aspects that should be focused on given a paper (paper aspect prediction), and identifying the aspects that a review focuses on (review aspect prediction). based on these tasks, we conduct a detailed empirical study of aspect at the community level. furthermore, we demonstrate that a comprehensive, fine-grained set of aspects allows for a new dimension in comparing reviews, offers a nuanced assessment of review quality in terms of specificity, and can be used for detection of automatically generated reviews. our work paves the path for data-driven analysis of aspects and enables new nlp applications to support peer review. our method offers a bottomup perspective that complements existing top-down schemata, and can be integrated with them by, for example, allowing domain experts to refine llmderived aspects into higher-level, domain-specific categories. together, these perspectives enable a comprehensive aspect-based analysis of peer reviews. to summarize, we contribute the following: ## aspect set construction 3.1 definition of aspect while aspects are outlined in review guidelines and used in related work, a formal definition of aspect is lacking. to address this, we propose an operational definition of aspect as a characteristic of a paper that a reviewer makes a judgment on when evaluating the paper, which is later used to compare the manuscripts to each other and to the quality standards provided by the review guidelines. from this definition, we do not consider terms such as acceptance decision as aspects, since they are not characteristics of a paper. an aspect can be general, such as soundness, or specific, such as missing citations on controlled generation. however, aspects that are either too general or too specific are difficult to use for comparing manuscripts to each other or to the publication standards. therefore, an aspect taxonomy that accommodates different levels of granularity is needed. we assume that an aspect is expressed in a review sentence, and we allow multiple aspects per sentence. we treat the aspects of a review as a set. formally, for a paper p, let r p denote the set of reviews for p. for each review @xmath0, we derive a r , the set of aspects in r. ## aspect prediction our fine-grained approach to aspect analysis enables two tasks: predicting the aspects that should be focused on given a paper (paper aspect prediction, pap), and identifying the aspects that are covered in the review (review aspect prediction, rap). these tasks are formalized as follows: ## practical applications we now demonstrate how a comprehensive, finegrained aspect set and the proposed tasks enable new types of nlp assistance in the peer review process. we picked the trained roberta model on the coarse label set with the best f1 score and obtained predicted aspects using it. this model achieves an f1 score above 0.75 for 10 out of 17 labels (see ## conclusion in this paper, we have introduced a data-driven approach to peer review aspect analysis. we have provided an operational definition of aspect, and developed an semi-automatic approach to identify aspects from peer reviews. we have proposed a taxonomy that involves a comprehensive set of aspects with different granularity, and introduce a new dataset of peer reviews augmented with aspects. we introduced two tasks, paper aspect prediction and review aspect prediction, and have shown how they contribute to a detailed empirical study of aspects. our results demonstrate that finegrained, data-driven aspects complement coarse aspects from review guidelines, and allow for more nuanced review comparison and new interpretable approaches for llm-generated review detection.