MILAB at P rag T ag-2023: Enhancing Cross-Domain Generalization through Data Augmentation with Reduced Uncertainty (2023)

Abstract: This paper describes our submission to the PragTag task, which aims to categorize each sentence from peer reviews into one of the six distinct pragmatic tags. The task consists of three conditions: full, low, and zero, each distinguished by the number of training data and further categorized into five distinct domains. The main challenge of this task is the domain shift, which is exacerbated by non-uniform distribution and the limited availability of data across the six pragmatic tags and their respective domains. To address this issue, we predominantly employ two data augmentation techniques designed to mitigate data imbalance and scarcity: pseudo-labeling and synonym generation. We experimentally demonstrate the effectiveness of our approaches, achieving the first rank under the zero condition and the third in the full and low conditions.

## introduction peer review is a fundamental procedure for assessing the quality of academic manuscripts @xcite . most reviews are characterized by concise argumentative feedback, wherein reviewers highlight both strengths and weaknesses while offering suggestions for revision. this observation has led researchers to frame the structures of peer reviews as a subset of argument mining @xcite @xcite . parallel to these insights, efforts have been made to automate the peer review process @xcite . the automation of this process yields two primary advantages: it facilitates authors by distilling the main feedback from reviews and helps reviewers by aggregating information from multiple reviews. recently, @xcite introduced a novel task, pragmatic tagging for peer review, wherein each sentence of a scientific review is classified into one of six predefined pragmatic categories. the proposed task is tailored for a multi-domain scientific corpus, where certain domains might employ specific terminologies that are not prevalent in others or require a unique evaluative perspective during the review process @xcite . furthermore, the nature of scientific review necessitates profound domain knowledge and careful examination by the reviewer, thereby posing challenges in large-scale data collection. such challenges, referred to as cross-domain generalization @xcite , have been the subject of intensive investigation within natural language processing. to address these challenges, we propose two approaches to enhancing the generalization of the model over multiple domains: pseudo-labeling and synonym generation. under full and low conditions, we finetune bert @xcite based classifiers using the training data and pseudo-label auxiliary data through an ensemble approach to ensure label quality. in the zero condition, we exploit the existing sections of the arr dataset and inject intrinsic characteristics of pragmatic tags without utilizing any large language models. our method accomplished the highest performance in the zero condition as well as the third in the full and low conditions. training data phase 2: synonym generation on training data unlabeled auxiliary data tep tep pseudo-labeled auxiliary data 1 pseudo-labeled auxiliary data 2 phase 3: recall labeling on auxiliary data tep phase 1: majority labeling on auxiliary data test data sentence label ? ? ? test data sentence label strength recap other inference: majority vote on test data tep unlabeled data a unlabeled data b pseudo-labeled data @xmath0: overview of our proposed approach to pragmatic tagging in the full condition. phase 1: pseudo-labeler models are trained using provided training data and subsequently utilized to label unlabeled auxiliary data. phase 2: training data are augmented by a synonym generator. phase 3: augmented data from phase 1 and 2 are used to finetune the labeler. models reapply tagging to the auxiliary data with increased certainty. phase 4: classifier trained with the labeled data from phase 3 are ensembled to predict the labels of the test data. data augmentation data augmentation is widely exploited to enrich and generalize datasets @xcite . a sentence can be expanded through the utilization of rule-based techniques and interpolation @xcite . furthermore, in the case of unlabeled datasets, a trained model can assign pseudo-labels to the unlabeled data, thereby facilitating supplementary training @xcite . ## dataset task data the data for the task is sourced from f1000rd @xcite , which is a comprehensive multi-domain collection of both reviews and their pragmatic labels. under the low condition, only 20% of the full task dataset is employed. detailed statistics of the six tags across five distinct domains are described in auxiliary data the auxiliary data is comprised of two datasets: f1000raw and arr-22 @xcite . the former, f1000raw, is an extensive, unlabeled corpus originating from the same source as f1000rd. conversely, arr-22 represents a col- ## methodology the efficacy of an individual model can be influenced by various hyperparameters throughout the training process, which could potentially lead to inaccurate predictions. therefore, we opt for an ensemble approach for our task, as depicted in figure 1 . from the entire training data, we set aside 18 reviews to constitute a validation subset. this subset excludes reviews that belong to the low condition dataset. the validation subset is consistently applied across all scenarios for the selection of hyperparameters and models. majority consensus f1000raw 0.8454 0.8333 f1000raw+arr 0.8263 0.8251 ## results experiment results over different conditions and domains are presented in ## conclusion in this study, we have empirically demonstrated the effectiveness of data augmentation methodologies, particularly in scenarios characterized by limited data availability. our findings pinpoint that strategies such as pseudo-labeling and synonym generation are instrumental in leveraging unlabeled auxiliary data, therefore amplifying the generalization capacity of the classifier. furthermore, our exploration of an ensemble approach for pseudolabeling, aimed at maximizing certainty, suggests promising avenues for enhancing the efficacy of pragmatic tagging processes.