AutoRef: Generating Refinements of Reviews Given Guidelines (2024)

Abstract: When examining reviews of research papers, we can distinguish between two hypothetical referees: the maximally lenient referee who accepts any paper with a vacuous review and the maximally strict one who rejects any paper with an overly pedantic review. Clearly, both are of no practical value. Our interest is in a referee who makes a balanced judgement and provides a review abiding by the guidelines. In this paper, we present a case study of automatic correction of an existing machine-generated or human review. The {\tt{AutoRef}\ } system implements an iterative approach that progressively “refines” a review by attempting to make it more compliant with pre-defined requirements of a “good” review. It implements the following steps: (1) Translate the review requirements into a specification in natural language, of “yes/no” questions; (2) Given a (paper,review) pair, extract answers to the questions; (3) Use the results in (2) to generate a new review; and (4) Return to Step (2) with the paper and the new review. Here, (2) and (3) are implemented by large language model (LLM) based agents. We present a case study using papers and reviews made available for the International Conference on Learning Representations (ICLR). Our initial empirical results suggest that {\tt{AutoRef}\ } progressively improves the compliance of the generated reviews to the specification. Currently designed specification makes {\tt{AutoRef}\ } progressively generate reviews which are stricter, making the decisions more inclined towards “rejections”. This demonstrates the applicability of $AutoRef $ for: (1) The progressive correction of overly lenient reviews, being useful for referees and meta-reviewers; and (2) The generation of progressively stricter reviews for a paper, starting from a vacuous review (“Great paper. Accept.”), facilitating authors when trying to assess weaknesses in their papers.

## introduction the modern practice of peer-review of scholarly articles is attributed to william whewell's proposal in 1831 that manuscripts submitted to the transactions of the royal society be subject to a review by two other members of the royal society. the referees-as they were called-would submit a written report on the claims of the paper. the process immediately ran into trouble, as the very first paper reviewed in this manner resulted in a strong disagreement between the two referees (whewell was one of them). while conflicting opinions about a paper are not new, the number of such occurrences is greatly increased in the modern day, where 100s of 1000s of manuscripts are received for peer-review. 1 this high demand for refereeing effort inevitably results in researchers being unable to write the kind of comprehensive review that they would have liked and authors expect. it is in this context that we investigate the possibility of machine-assistant capable of refining reviews-by way of correction or completion-given some pre-defined notion of the ingredients of a "good" review. a natural extension of the machine-assisted correction of a review is machine-assisted generation of a review (viewed as a refinement of some default trivial review). in this paper, we demonstrate the working of the autoref system that is capable of both aspects. the rest of the paper is organized as follows. in section 2 we describe the autoref system. related work is in section 3. a case study of using autoref using conference submissions and reviews for the international conference on learning representations (iclr) is presented in section 4. section 5 highlights some limitations of the current autoref implementation and concludes the paper. ## the autoref system we first present autoref conceptually, referring to the main computations that are performed. ## related work the recent work on the peer-reviewing system is through introducing new datasets, novel tasks, and analysis. disapere is a labeled dataset with fine-grained annotation of reviews and rebuttal sentence-wise that showcases the understanding of argumentative strategies used by the reviewers and authors @xcite . argscichat is a dataset of argumentative dialogues between researchers addressing both exploratory and argumentative interactions on the scientific paper @xcite . nlpeer is a multidomain corpus of 5k papers and 11k review reports from five venues with unified data representations @xcite . qasa is a question-answering dataset on scientific articles in ai and ml. the questions require full-stack reasoning through a 3-step process: associative selection of paragraphs followed by evidential rational generation, which is utilized to compose the answer systematically @xcite . aries (aligned, review-informed edits of scientific papers) is a dataset of computer science papers (drafts & revised versions, reviewer feedback, and authors' responses to the review (d' @xcite . aries introduced the novel task of editing scientific papers using peer reviews. peersum introduces a dataset for generating meta-reviews of scientific papers with rammer (relationship-aware multi-task meta-review generator) to generate meta-reviews @xcite . similarly, @xcite attempts to generate meta-reviews through the reviews and rebuttal. all these datasets have been designed for fine-grained analysis of review data, generation of meta-reviews, or updating the paper based on the review. our task is a review generation task and none of these datasets can be directly used for this task. moreover, all of these datasets contain information sourced from the open web, which is dated earlier than the data employed to train the llm utilized in our experimentation. there is a possibility that this data has been memorized by the llm, making it unsuitable to showcase the effectiveness of our approach. hence, we synthesized our own dataset elaborated in section 4.2. reviewergpt provided an exploratory study on reviewing papers using llm, showing that llm can identify errors and verify checklists, but llms struggled to discern a good paper against a relatively bad paper, with a naive prompt-based approach @xcite . @xcite empirically analyze if large language models can provide useful feedback on research papers. a monolithic prompt is fed to the gpt-4 @xcite to generate reviews for a scientific paper. this monolithic prompt only focuses on these four aspects: significance, novelty, potential reasons for acceptance or rejection, and suggestions for improvement. marg (multi-agent review generation) (d'arcy et al., 2024) proposes a multi-agent system to generate reviews for a scientific paper. the system has three multi-agent groups, each one of them focusing on different aspects: experiments & evaluation, clarity & reproducibility, and novelty process in autoref manifest itself in changes to reviews?". we will also provide some preliminary observations on the question: "can autoref be used to improve the reviews generated by human reviewers?". ## concluding remarks we present autoref system, which implements an iterative approach that progressively "refines" a machine-generated or human provided review of a research paper. our case study with iclr papers demonstrates the capability of autoref to make the reviews more compliant with a review requirement specification, pre-defined in terms of "yes/no" questions. we observe a decrease in the 'accepts' over the reviews generated over iterations of autoref , leading to the fact that autoref is generating stricter reviews. this demonstrates the applicability of autoref towards progressive correction of overly lenient review or vacuous reviews, being useful for referees, meta-reviewers, as well as authors to detect the weaknesses of the paper. further comparison of the outcomes of autoref generated reviews with meta-reviewers decisions yields lower precision as compared to outcomes of human reviews with comparable recall. in the attempt to generate stricter reviews along with converting falsely detected 'accepts' to 'rejects' (reducing fps), autoref is also converting correctly detected 'accepts' (reducing tps), further reducing the precision as well as recall. this hints at the need for improvement in autoref to ensure better precision. currently, the review decision is taken at the final leaf node of the task orderings based on the outcomes of the parent tasks. reviewer agent being an llm, this decision can be hallucinated. instead, a more deterministic algorithm can be designed to make this decision based on llm-generated outcomes of the parent tasks. moreover, the current scoring system is very crude (+1 for each "yes" and 0 for each "no"). clearly, several alterations could be made like increasing the weightage of more important questions and "negative marks" for incorrect answers. the questions themselves are fairly shallow and only examine superficial aspects of the review. instead of just checking the relevancy of the review to the paper with respective vector representations, there is a need for having the paper in context for questionnaire-based evaluation of the review. more in-depth questions are possible, but it will need the language model to be able to detect those subtleties in the paper. we have also not utilized several features of language models, like using "few-shot" instances of good and bad reviews or the ability to generate multiple reviews when called repeatedly. consequently, the current implementation should be seen as a first step in an automated review generation system. however, we do expect that the system's specification has been sufficiently abstracted to allow scope for significant improvements. a tasks