NUS - IDS at P rag T ag-2023: Improving Pragmatic Tagging of Peer Reviews through Unlabeled Data (2023)

Abstract: We describe our models for the Pragmatic Tagging of Peer Reviews Shared Task at the 10th Workshop on Argument Mining at EMNLP-2023. We trained multiple sentence classification models for the above competition task by employing various state-of-the-art transformer models that can be fine-tuned either in the traditional way or through instruction-based fine-tuning. Multiple model predictions on unlabeled data are combined to tentatively label unlabeled instances and augment the dataset to further improve performance on the prediction task. In particular, on the F1000RD corpus, we perform on-par with models trained on 100% of the training data while using only 10% of the data. Overall, on the competition datasets, we rank among the top-2 performers for the different data conditions.

## introduction peer review is employed across various subject domains to assess the quality of research documents such as grant proposals, journal manuscripts, and conference proceedings. peer reviews are performed by independent researchers with expertise on the relevant topic for purposes such as awarding grants or publishing latest research for the advancement of science. review text reports, the result of these peer assessments, are brief summaries describing the document's main contributions, its strengths and weaknesses, along with other revision related comments and constructive feedback @xcite . though standards and practices may vary across different subject domains and even across venues within the same domain, the main objective of the peer review process is to ensure the advancement of quality research @xcite . to this end, alleviating the reviewing burden and supporting the diverse nature of reviewer expertise becomes vital @xcite and motivates the on-going research on developing tools to assist and improve the peer reviewing process @xcite @xcite . in particular, a significant direction towards developing ai-assisted peer reviewing models involves the compilation of relevant datasets to support the meta-analyses of reviews @xcite @xcite . from the perspective of language and nlp research, review reports provide a rich ground for investigation for various argument mining problems @xcite including classification tasks such as paper acceptance prediction and sentence labeling @xcite . the pragtag shared task foot_0 at the 10th workshop on argument mining at emnlp-2023 comprises one such sentence labeling task in which every sentence from a review report is assigned a label from one of the pragmatic categories: { recap, strength, weakness, todo, other, structure}. due to space constraints, we refer our readers to @xcite and @xcite for the precise definitions of the pragmatic categories and the f1000rd corpus which forms the basis for the datasets used in the pragtag-2023 competition. ## proposed methods in this section, we briefly describe the various models we employed for the pragmatic tagging task under the three data conditions. no-data setting: we studied two approaches for predicting pragmatic tags under the no-data condition. in the first "semantic search" approach, we simply use a list of "questions" to find sentences in the review texts that best answer the question. this list was curated based on the typical questions employed during the peer review process of nlp conferences and augmented to cover labels such as "recap". 2 example questions include "how original are the results described in the paper?" and "what is the main finding of this paper?". we used the state-of-the-art sentence transformer models 2 complete list shared as part of the code distribution trained for semantic search for this method @xcite . 3 recent breakthrough research has shown that large language models (llms) can be trained "to act in accordance with the user's intentions" and as a consequence be "prompted" to perform a range of nlp tasks @xcite @xcite . for our second approach, in keeping with this recent direction, we designed a multiple-choice question prompt along with the task description provided in the competition for use in instruction fine-tuned language models @xcite . our prompt is listed in with the objective to utilize the unlabeled data provided in the competition as means to overcome the scarcity of labeled data in the low-data settings, we employed traditional semi-supervised approaches-self-training and voting, to combine predictions from multiple learners foot_3 and obtain tentative labels for the unlabeled data @xcite . the "tentatively labeled" unlabeled data is incorporated via two methods in our models. in the pretraining approach (pt), we simply pretrain our classifier on the tentativelylabeled unlabeled data before fine-tuning on the labeled data whereas in the combined approach, the augmented dataset is used to train a model. ## related work sentence classification tasks are well-studied in nlp research with deep learning models comprising the state-of-the-art @xcite . some recent sentence-level classification tasks include identification of complex linguistic phenomena in texts such as emotions, empathy, humor, sarcasm, and dialog acts @xcite @xcite . recently, efforts are underway for collecting relevant datasets for designing assistive automation aids for peer review @xcite @xcite @xcite . in this context, @xcite introduced pragmatic tagging for labeling sentences of peer reviews using a schema that applies across different research fields and communities. we borrow from the latest nlp advances such as prompt-based models and combine them with unlabeled data on precisely this task. ## conclusions and future work we presented our approaches for the pragmatic tag prediction task for peer reviews as part of the prag-tag shared task @ argmining workshop 2023. in particular, we studied prompt-based fine-tuning as a viable alternative to traditional learning methods for this task and showcased how unlabeled data may be utilized via multiple learners to improve performance in the low-data settings. in future, we would like to address the generalizability of our proposed models across various subject domains as well as extend our approaches to related tasks such as paper acceptance prediction @xcite .