Overview of P rag T ag-2023: Low-Resource Multi-Domain Pragmatic Tagging of Peer Reviews (2023)

Abstract: Peer review is the key quality control mechanism in science. The core component of peer review are the review reports – argumentative texts where the reviewers evaluate the work and make suggestions to the authors. Reviewing is a demanding expert task prone to bias. An active line of research in NLP aims to support peer review via automatic analysis of review reports. This research meets two key challenges. First, NLP to date has focused on peer reviews from machine learning conferences. Yet, NLP models are prone to domain shift and might underperform when applied to reviews from a new research community. Second, while some venues make their reviewing processes public, peer reviewing data is generally hard to obtain and expensive to label. Approaches to low-data NLP processing for peer review remain under-investigated. Enabled by the recent release of open multi-domain corpora of peer reviews, the PragTag-2023 Shared Task explored the ways to increase domain robustness and address data scarcity in pragmatic tagging – a sentence tagging task where review statements are classified by their argumentative function. This paper describes the shared task, outlines the participating systems, and summarizes the results.

## introduction scholarly communication lies at the heart of scientific discovery @xcite and is argumentative by nature. scientific publications present results, interpret them, justify the experimental setup, and substantiate the claim for new knowledge @xcite . peer review reports, in turn, assess the validity, novelty and impact of the underlying publication and argue for or against its acceptance. peer review is a key component of scientific quality assurance. it is a complex process prone to heuristic behavior @xcite and bias (e.g. @xcite . a growing area of nlp + aux data diso iscb case rpkg scip + secret zero low full avail. data domain 3 1 2 test 5 + aux data 4 figure 1 : pragtag-2023 overview. given a mixeddomain corpus of peer reviews annotated with pragmatic tags, the participants submit systems trained with varying amounts of training data (1-3) with optional use of unlabeled auxiliary data (4). the systems are evaluated in each of the five domains (section 3.1), as well as on a previously unseen secret domain (5). for peer review analysis aims to investigate those issues by analyzing argumentation in peer review reports (e.g. @xcite @xcite @xcite . the resulting systems have numerous potential applications, incl. facilitating metascientific analysis of reviewing practices, helping authors and program chairs aggregate information from multiple reviews, and supporting junior reviewers in giving thorough, objective and helpful feedback. standards and practices of scholarly communication vary across research communities. yet, to date, nlp for peer review has focused on data from machine learning conferences @xcite @xcite , and the applications outside of this domain remain under-investigated. this over-focus on one domain can be attributed to data scarcity -while some communities make their reviewing public, peer reviews are generally hard to obtain and legally clear for research use @xcite the authors address the issue of... this idea reminded me of the work by... please compare your method to... the discussion is superficial. the paper is original and sound. 2022). in addition, due to the technical nature of peer review texts, they are expensive to annotate. measuring the effects and mitigating the impact of domain shift and data scarcity are important and under-researched questions in nlp for peer reviews. ## pragmatic tagging task. pragmatic tagging is a sentence classification problem where given the sequence of sentences s r 1 , ..., s r n from a review report r, a model should predict the pragmatic label for each sentence l r 1 , ..., l r n from the label set l. we adopt the label set proposed by @xcite , illustrated in figure 2 . the label set was evaluated in an annotation study and shown to be well-applicable across different research fields and communities while yielding good inter-annotator agreement of approx. 0.7 krippendorff's α. the core sources of disagreement are the coarse granularity of the schema (necessary for generalization), sentencelevel analysis (necessary to avoid discrepancies due to differences in sub-sentence splitting), and the natural ambiguity of the classes (e.g. weakness vs todo). evaluation. @xcite provide the data, but do not specify metrics for evaluating nlp systems for pragmatic tagging. in pragtag-2023, we evaluate system performance via the f1 score. since the label distribution is skewed, we opt for the macro-averaged f1 within each domain for evaluation. we then compute scores for each domain individually and use the mean across all domains as the final leaderboard score (section 4). baselines. to contextualize the submission scores, we implemented two baselines. the supervised roberta baseline is a roberta-base model @xcite fine-tuned for 20 epochs on the training data available for a given experimental condition (section 4.2). the majority baseline directly assigns the most frequent pragmatic tag from the training data to the input sentence. ## data the participants of the shared task were given two types of data (figure 4 ). the smaller-scale core data contains peer review texts labeled with pragmatic tags on the sentence level. core data is used for training and evaluating the systems. the large-scale auxiliary data consists of two unlabeled text collections. it can be used to enhance the systems' robustness to low-data conditions in the train test test 5 domains f1000rd secret coling-20 f1000raw arr-22 core data aux data figure 4: pragtag-2023 data overview. in addition to labeled core data from f1000rd and coling-20, the parcitipants are provided with two unlabeled collections: a large multi-domain corpus of unstructured peer reviews (f1000raw), and smaller collection of semistructured peer reviews in the nlp domain (arr-22). ## submissions out of over 20 teams that signed up for the competition, five teams have made it to the final submission. the submitted systems explore a wide range of techniques and architectures for multi-domain pragmatic tagging in low-resource scenarios. we summarize the main ideas behind each submission below and refer to the system papers for details. catalpa_edunlp @xcite investigated a wide array of approaches. for the full-and low-data setting, this includes supervised sentence labeling via roberta @xcite augmented with additional features (domain, position, context, word normalization), as well as iob-style sequence tagging using long-document transformers and nearest-neighbor-based labeling using sbert @xcite . in the zero-shot setting, the team experimented with labeling test instances based on their similarity to class defini-mean case diso iscb rpkg scip secret deepblueai 84.1 82.9 84.1 82.8 86.0 89.0 80.1 nus-ids 83.2 83.8 85.4 83.3 84.8 87.8 74.1 milab 82.4 84.0 83.7 80.1 85.4 86.5 74.9 suryakiran 82.3 82.0 82.8 81.8 82.8 86.5 77.9 catalpa 81.3 80.8 82.0 81.1 82.5 82.5 78.8 ensemble 84.4 84.0 85.2 83.3 87.3 88.7 78.0 roberta 80.3 80.3 80.8 79.9 83.1 83.8 73.7 majority 8.0 9.3 7.3 7.5 8.6 7.9 7.3 deepblueai @xcite focused their approach on increasing the robustness of pre-trained models in the sentence labeling setting. the experiments were conducted using three models -roberta, deberta @xcite and xlm-roberta @xcite . the participants augmented the model via max pooling and attention pooling, introduced adversarial training via fast gradient method, and reported comparative performance of the models trained under different settings via cross-fold validation, showing that the modifications lead to variable performance gains. the authors report that the deberta model consistently outperforms the other two models on the task. to tackle the secret test set in the final phase of the competition, the authors used a voting approach combining a range of models trained in different configurations and selecting the label with the maximum vote, stressing the benefits of fusing different types of models for prediction. nus-ids (gollapalli et al., 2023) explored multiple approaches to the task for each experimental condition. in the zero-shot no-data condition, the participants proposed two methods: a questionanswering model that selects passages from the peer review based on a set of questions derived from peer reviewing guidelines of nlp conferences, and a prompting-based approach based on the flan-t5 (chung et al., 2022) model. for the low-and full-data setting, the participants experimented with fine-tuning pre-trained language models, additionally exploring ensembling and data augmentation techniques by tentatively labeling the auxiliary shared task data. the results indicate that prompting via flan-t5 outperforms questionanswering based approach in the no-data setting; in low-and full-data data, fine-tuning a t5 model @xcite on tentatively labeled auxiliary data followed by fine-tuning on the core task data performs best. milab @xcite approached the problem of data scarcity and domain shift via data augmentation. in particular, to compensate for the lack of data, the team applied an ensemble of roberta-based classifiers to label auxiliary data from f1000raw and arr-22. apart from majority labeling, the authors explored a novel recall labeling technique: the models assign tentative labels to the unlabeled instances in the decreasing order of recall on a validation set, while labeling the residual instances as other. additionally, the authors experimented with diversifying the data by applying off-the-shelf synonym generation followed by bertscore filtering @xcite . the results indicate that the proposed data augmentation techniques combined with ensembling improve the model performance on the task, especially in the no-data condition. suryakiran @xcite explored the use of unsupervised pre-training on f1000raw auxiliary data to increase domain robustness of the pragmatic tag classifier. in particular, the participants pre-trained the deberta model on f1000raw using masked language modeling objective @xcite , and later used an ensemble of five models further fine-tuned on different training data splits to make the test set prediction. their results demonstrate that pre-training via masked language modeling leads to improved performance only in some cases; the authors attribute this to the vocabulary discrepancies between the domains. the team submitted their system only to the final evaluation. ## main results the final leaderboard of pragtag-2023 is shown in ## analysis access to all the participating system's predictions at once allows additional insights into the task. given the broad range of approaches proposed by the pragtag-2023 participants, a natural question arises if these approaches are complementary. we investigate this by combining the predictions of the best-performing submissions via majority vote. the results show that a majority ensemble indeed outperforms every individual system on average ( ## discussion a high-level picture of the submissions to the prag-tag competition reveals several trends. despite the advances in llm development, fine-tuning of bert-family lms was still used by most participants, although some have experimented with prompting. while our rules prohibited the use of commercial llms, new open llms like llama @xcite have been released. investigating the performance of these models for our task is a promising avenue for future studies. while some submissions focused on modifying the model architecture and pre-training regime, others explored data augmentation and creative adaptations of the task, e.g. by casting it as a questionanswering task or labeling the instances based on the similarity to guideline class definitions. most participants used auxiliary data as an unlabeled substrate for pseudo-labeling or language model pre-training. we note the wide use of model ensembling across the submissions, and believe that such techniques will remain relevant in the age of llms. pragtag-2023 was designed to accommodate var-ious approaches to the task: pragmatic tagging can be cast as sentence labeling and as sequence labeling, and can can be approached via prompting. while the participants have experimented with many of these options, in-context learning @xcite remained under-explored. we deem such exploration promising. the ongoing adaptation of the field to the lastgeneration llms presents new challenges to the benchmarking and shared task methodology. the technical requirements of pre-training and finetuning llms put the teams without access to massive data and compute at disadvantage. the opaqueness of the llm pre-training for commercial models introduces the risk of model exposure to the test data or related datasets. pragtag-2023 attempted to mitigate these issues by explicitly limiting the competition to the models for which open weights are available and pre-training procedure is known, and by prohibiting the use of any additional pre-training sources apart from the core and auxiliary data provided with the task. an alternative solution could be to limit the competition to several open llm instances, inference-only. this, however, would limit the scope of methods the participants can explore to prompting-based approaches. we leave the search for flexible, fair and reproducible benchmarking methodology in the age of llms to future work. ## conclusion this paper has introduced pragtag-2023: the shared task in low-resource multi-domain pragmatic tagging of peer reviews. we have described the rationale behind the task, introduced the data and outlined a range of experimental conditions under which the competition took place. the shared task participants proposed a wide range of techniques for increasing the robustness of pragmatic tagging across domains and data availability scenarios. the results of the competition underline the importance of evaluating pragmatic tagging systems across different domains and in different data availability conditions. the arguably most important gain from an organized competition is not finding the best-performing system for the task, but the accompanying exploration of approaches to solving the problem at hand. to this end, we hope that the ideas and observations from the pragtag-2023 submissions foster future progress in pragmatic tagging, and in cross-domain and low-data processing of peer reviews in general.