D eep B lue AI at P rag T ag-2023:Ensemble-based Text Classification Approaches under Limited Data Resources (2023)

Abstract: Due to the scarcity of review data and the high annotation cost, in this paper, we primarily delve into the fine-tuning of pretrained models using limited data. To enhance the robustness of the model, we employ adversarial training techniques. By introducing subtle perturbations, we compel the model to better cope with adversarial attacks, thereby increasing the stability of the model in input data. We utilize pooling techniques to aid the model in extracting critical information, reducing computational complexity, and improving the model’s generalization capability. Experimental results demonstrate the effectiveness of our proposed approach on a review paper dataset with limited data volume.

## introduction peer review stands as a fundamental pillar of the scientific process, yet it presents formidable challenges that could greatly benefit from automation and support. at the heart of peer review are review reports -concise, argumentative documents in which reviewers assess research papers and offer recommendations for improvement. automating the analysis of argumentation within peer reviews @xcite holds vast potential, ranging from facilitating meta-scientific investigations into review practices to consolidating insights from multiple reviews and aiding less experienced reviewers. text classification is a significant and challenging task. however, when relying on relatively small datasets, traditional machine learning methods may encounter issues such as overfitting and poor generalization performance. in such cases, pre-trained models serve as powerful tools that offer robust solutions for addressing data scarcity. pre-trained models, particularly those based on the deep learning transformer @xcite architecture, have demonstrated significant success in natural language processing tasks. roberta (a robustly optimized bert pretraining approach) @xcite , xlm-roberta @xcite and deberta (deep bert with disentangled attention) @xcite are pre-trained models based on the transformer architecture that have garnered widespread attention in the field of nlp. through fine-tuning these pre-trained models, exceptional performance can be achieved on smaller datasets, mitigating overfitting issues and improving generalization performance. this paper focuses on the application of roberta, xlm-roberta and deberta to address text classification problems within a peer review dataset. ## related work pragmatic tagging of peer reviews is, in fact, a classification task, and in common classification tasks. in the field of text classification, models like recurrent neural networks (rnn) @xcite , and long short-term memory networks (lstm) @xcite introduced more nonlinear factors, enabling them to automatically learn feature representations from data and achieving remarkable results. however, deep learning methods may face overfitting issues on small datasets and require a substantial amount of labeled data for training. to address these issues, the development of pre-trained models has become a groundbreaking direction. pre-trained models are trained on large-scale unlabeled corpora, learning rich language representations that enable them to better capture semantic relationships between words, as seen in models like bert @xcite . subsequently, these pre-trained models can be fine-tuned for specific tasks to exhibit exceptional performance. in the application of pre-trained models, researchers have introduced various techniques to further optimize model performance. techniques such as k-fold cross-validation better evaluate the model's stability and generalization ability. adversarial training methods like fast gradient method (fgm) @xcite enhance the model's robustness, preventing it from being disrupted by adversarial attacks. pooling techniques such as max pooling, min pooling and attention pooling allow models to understand text information at different levels. additionally, model ensemble techniques combine predictions from multiple models, improving overall classification performance. ## task description the goal of this task is to perform automatic analysis of argumentation in peer review. our input data consists of each sentence in the argumentation, and the output results are the corresponding label categories for each sentence. the competition is divided into multiple stages, each providing two datasets @xcite : "low_data" and "full_data". the training set provided in the "low_data" comprises a total of 34 review articles, 793 sentences in total. the objective is to classify each sentence into one of the six categories. we have conducted a statistical analysis for each category in the dataset, and the results are presented in the training set provided in the "full_data" consists of a total of 118 review articles, 2324 sen-tences in total. the objective remains the classification of each sentence into one of the six categories. similar to the previous dataset, we conducted a statistical analysis for each category in the dataset, and the results are presented in ## conclusion in this paper, we have presented a comprehensive approach for text classification tasks on smallscale peer review datasets. by combining attention pooling, max pooling, and adversarial training (fgm), we achieved significant performance improvements. through experimental validation, we have demonstrated the superiority of our method on small datasets. in the evolving era of deep learning, our approach amalgamates various techniques, providing an effective solution for text classification on small datasets. it overcomes the challenges posed by data scarcity, enhancing both model performance and robustness, offering novel insights and methodologies for addressing text classification challenges on small datasets.