Approche générative de la conformation pragmatique : une étude de cas de l’analyse d’une conférence (2025)

Abstract: La relecture en double aveugle est centrale dans les conférences scientifiques, mais des biais persistent. OpenReview a introduit plus de transparence en rendant publics les articles, les évaluations et les décisions. Ce travail explore l’utilisation des grands modèles de langage (LLMs) pour assister différentes étapes du processus de relecture : production de méta-revues, détection de biais et de subjectivité dans les évaluations. L’étude s’appuie sur les données ICLR de 2017 à 2022 et inclut des analyses quantitatives et des évaluations humaines à l’aveugle. Les résultats visent à encourager une relecture scientifique plus efficace et équitable.

## introduction the progress of contemporary science is significantly dependent on the peer review mechanism, enabling authors to publish and disseminate their research findings. therefore, it is crucial that the peer review process maintains fairness and impartiality, especially for emerging researchers whose career paths are significantly influenced by paper acceptance. regrettably, the peer review process has recently come under scrutiny. due to the overwhelming volume of submissions, many platforms have faced an unprecedented demand for reviewers, resulting in numerous complaints from authors regarding unfair or substandard reviews. as an example, the neurips experiments @xcite 1 assigned a different set of reviewers to the same submissions, and found the reviewer ratings are often significantly different. then, @xcite analyzed 128,454 articles in nature-branded journals and found that authors from less prestigious academic institutions are more likely to choose double-blind review as opposed to single-blind review. in this context, llms have attracted attention of various domains to analyze the intrinsic bias associated to corpus @xcite and investigated their capabilities to make decisions of diverse nature based on natural language @xcite . in this paper, we question whether one could use the zero-shot capabilities of llms to assist the peer review process in terms of acceptance and meta-reviewing capabilities in addition to detection of problematic reviews. for this purpose, we utilize a collection of submissions of the international conference on learning representations (iclr) and propose a series of quantitative and qualitative evaluations. contributions. our contributions are summarized as follows (1) we evaluate the capability of several open-source llms for submission acceptance based on original reviews, meta-review and automatically generated meta-reviews. (2) we evaluate an automatic and comprehensive protocol to assess objectivity and bias in a peer review process. (3) we evaluate the zero-shot capabilities of llms to detect problematic reviews. ## related work the peer review dynamics have recently attracted attentions in the nlp community. as a first instance, dataset of papers have been recently created from acl, neurips, and iclr venues @xcite , @xcite . another dataset @xcite has focused on investigating the effect of rebuttals in nlp conferences. then, many recent research has been proposed to investigate the bias present in the review process @xcite , argument mining @xcite , automatic review generation @xcite , improved review process @xcite , and review explanation @xcite . in this context, evaluating the capability of automatic decision of llms in this review process hasn't been investigated so far. pragmatics and discourse analysis hold pivotal positions within linguistics @xcite . specifically, pragmatics delves into the contextual nuances of meaning that are often overlooked in the formulation of content or logical structure @xcite . a recent work @xcite compares human and model comprehension, revealing language models' struggles with humor, irony, and conversational maxims but seems to well capture discourse structure and basic common-sense. notably, this paper constitutes a novel exploration of pragmatic aspects within conference reviews. as a second contribution, we analyse the capabilities of large language model to detect such flaws during the scientific review process in addition to acting as an impartial decision maker based on original reviews, meta-reviews and generated meta-reviews. as far as our knowledge goes, none of the current related works has investigated such capabilities through the lens of large language models. ## iclr papers corpus in this study, we utilize a subset of an iclr corpus of papers. to compile this dataset, we use the openreview api foot_0 to gather data from 10, 289 submissions to the international conference on learning representations (iclr) spanning the years 2017 to 2020. the gathered data encompasses (i) submissions and their associated metadata ; (ii) double-blinded review and rebuttal data ; and (iii) decisions made by area chairs and program chairs. in this work, no information about authorship and associated institutions are considered. the review data is comprised of ratings, confidence levels, and textual reviews and rebuttals. the review and rebuttal data includes each reviewer's comments on the submission, a numerical score, a confidence rating, and the authors' rebuttal. lastly, we binarize the paper decision by categorizing diverse acceptance designations, i.e spotlight, poster, short talk, into two groups : accept and reject. upon paper assignment, a minimum of three reviewers independently evaluate a paper. following the rebuttal phase, reviewers can access the authors' responses and other peer reviews, enabling them to revise their reviews accordingly. the program chairs then compile a meta-review for each paper, making the final accept/reject decision based on the three anonymous reviews. each official review primarily consists of a review score, a reviewer confidence level, respectively from 1 to 10 and 1 to 5, and detailed review comments. the final dataset we are sampling from is composed with 5,527 submissions and 16,853 official reviews from iclr 2017-2020 venues on the openreview platform. desk-rejected and withdraw submissions have been pruned. ## large language models for reviews leveraging this iclr corpus, we aim at answering the following questions : (rq1) how can llms be used to generate accurate acceptance decisions directly from the list of reviews, meta-review and generated meta-review from the set of original reviews ? (rq2) how do llms compare to existing dedicated models of sentence-level polarity, bias and subjectivity detection for conference review evaluation ? (rq3) can llms be used to identify reviews that require special attention in regard of subjectivity and bias ? importantly, our approach leverages the capabilities of llms to analyse discourse structures and contents. as a consequence, we assume that the model requires only limited knowledge about the actual subject of the paper to establish the feedbacks and decisions while remaining informative, objective and impartial. for each experiment, the prompt is given in appendix. for experiments involving annotators, the participants have been asked to rate the model's decision on each review with a discrete value of {-1, 0, 1} to express their accordance with the model's decision. a score of -1 indicates disagreement, 0 indicates neutrality, and 1 indicates agreement. ## limitations and ethical concerns despite the promising results of our study on the evaluation of large language models (llms) for writing meta-reviews, deciding on paper acceptances, and detecting problematic reviews, several limitations should be acknowledged. our evaluation is based on a specific dataset of academic papers and reviews within the research of machine learning, which may limit the generalizability of our findings to other disciplines with distinct writing styles and review standards. the quality and representativeness of the training data significantly influence the model's outputs, potentially leading to biases in model performance. evaluating the quality of academic reviews involves judgments that llms may not fully capture, potentially overemphasizing certain aspects of reviews while underestimating others. then, identifying problematic reviews remains challenging due to the subjective nature of such determinations, and llms might not consistently identify all problematic elements accurately. the use of llms also raises ethical concerns about transparency, accountability, and fairness, necessitating responsible and ethical implementation. furthermore, llms require substantial computational resources, which may limit their accessibility and scalability. future research should aim to address these limitations by expanding the diversity of datasets, improving model robustness. role : assistant to verify is a conference paper review is problematic. instruction : analyze the provided conference paper reviews and estimate if they are problematic. steps : 1. read and understand the provided conference paper reviews. 2. identify any instances of bias, lack of understanding of the subject matter, subjective opinions presented as facts, and any immoral or unethical statements. 3. provide a detailed and objective analysis of the reviews, citing specific examples where necessary. 4. summarize the overall estimation of the reviews, indicating whether they are problematic or not. end-goal : to provide a comprehensive and objective analysis of the conference paper reviews, indicating whether they are problematic or not, and to support the analysis with specific examples from the reviews. narrow : ## annotation instructions for human evaluators participants in this annotation task are provided the following instructions for evaluating models used in the automated assessment of conference reviews and meta-reviews : task overview : your participation is crucial for evaluating a small set of language models (llms) across two tasks. each participant is contacted individually to maintain independence. the task involves reviewing and filling excel sheets with numeric evaluations. evaluation criteria : each evaluation criterion uses the scale : {1, 0, -1}. second experiment evaluate meta-reviews generated by llms based on the following criteria : ## conclusion this paper evaluates large language models for academic paper reviewing and meta-reviewing. we assess their ability to decide paper acceptance, generate meta-reviews, and identify problematic reviews. our findings show that llms can effectively support and augment the review process, improving the quality and reliability of assessments. our results aim to foster more research in this direction.