Automated Peer Reviewing in Paper SEA : Standardization, Evaluation, and Analysis (2024)

Abstract: In recent years, the rapid increase in scientific papers has overwhelmed traditional review mechanisms, resulting in varying quality of publications. Although existing methods have explored the capabilities of Large Language Models (LLMs) for automated scientific reviewing, their generated contents are often generic or partial. To address the issues above, we introduce an automated paper reviewing framework SEA. It comprises of three modules: Standardization, Evaluation, and Analysis, which are represented by models SEA-S, SEA-E, and SEA-A, respectively. Initially, SEA-S distills data standardization capabilities of GPT-4 for integrating multiple reviews for a paper. Then, SEA-E utilizes standardized data for fine-tuning, enabling it to generate constructive reviews. Finally, SEA-A introduces a new evaluation metric called mismatch score to assess the consistency between paper contents and reviews. Moreover, we design a self-correction strategy to enhance the consistency. Extensive experimental results on datasets collected from eight venues show that SEA can generate valuable insights for authors to improve their papers.

## introduction with the rapid pace of scientific advancement, there has been a significant increase in the volume of research publications @xcite @xcite . nevertheless, it poses considerable challenges for traditional scientific feedback mechanisms @xcite . on one hand, it exacerbates the pressure on the peer review process @xcite ; on the other hand, the disparate quality of these numerous publications can negatively affect the scientific research milieu @xcite . consequently, the paper is not well written and hard to follow. incorporating additional visual illustrations and explanatory texts might make the paper more understandable. the paper is not well written and hard to follow. the paper would benefit from extra visual illustrations. ## sea this section details three major modules (i.e., standardization, evaluation and analysis) of sea, and the overall framework is illustrated in figure 2 . ## conclusion in this paper, we present sea, a novel framework for automated paper reviewing. specifically, we propose a new paradigm for constructing a standardized review dataset. based on this dataset, we fine-tune an llm to generate high-quality reviews. moreover, we propose a new evaluation metric to measure the consistency between papers and generated reviews. comprehensive experimental results demonstrate that the sea framework can generate feedback that aligns with human reviews. in summary, we emphasize that the initial motivation for the field of automated paper review stems from the time-consuming and labor-intensive nature of traditional peer reviewing. automated peer reviewing can provide timely feedback, enhancing research quality and accelerate the progress of scientific development. therefore, we anticipate that the sea framework will help researchers improve the quality of their work and shed light on the field of automated scientific reviewing.