Detecting generated scientific papers using an ensemble of transformer models (2022)

Abstract: The paper describes neural models developed for the DAGPap22 shared task hosted at the Third Workshop on Scholarly Document Processing. This shared task targets the automatic detection of generated scientific papers. Our work focuses on comparing different transformer-based models as well as using additional datasets and techniques to deal with imbalanced classes. As a final submission, we utilized an ensemble of SciBERT, RoBERTa, and DeBERTa fine-tuned using random oversampling technique. Our model achieved 99.24% in terms of F1-score. The official evaluation results have put our system at the third place.

## introduction state-of-the-art natural language processing (nlp) tools generate high-quality texts that could hardly be distinguished from human-written texts. this represents a remarkable achievement in modern science, but raises challenges in terms of detecting machine-generated texts. detection of automatically generated texts is crucial for many nlp tasks, in particular, for prevention of spreading fake scientific publications and citations @xcite . here we focus on the task of detecting automatically generated scientific excerpts as a part of the third workshop on scholarly document processing shared tasks. the source code that we used for fine-tuning our models as well as additional data generated by us are freely available 1 . the work is based on the participation of our team in the dagpap22 shared task. the objective of the task is to detect automatically generated papers in terms of a binary classification task. this task is challenging due to the developing models for text generation and wide spread-ing of untruthful content on the internet. to date, language models for generating texts are widely used in the scientific domain, for example for producing long and short summaries @xcite @xcite , citation texts @xcite , keyphrases @xcite , peer reviews @xcite . the scientific community has held several machine learning competitions to identify machine-generated texts in different domains @xcite . the paper is organized as follows. we provide the dataset and task description in section 2. in section 3, we describe our experiments during the development phase and report the official results. section 4 concludes this paper. ## conclusion in this work, we have explored the application of bert-based models to the task of detecting machine-generated scientific texts. we have evaluated several techniques for handling imbalanced data and compared three models in a variety of settings. our results on the test data showed that the ensemble of different transformer-based models outperforms other our submissions and strong baselines. moreover, our final model ranked third in this task. a further study could explore the state-of-the art-in detecting automatically generated papers for other languages and multilingual corpora. another future direction is to continue our experiments with generating new data to improve the classification performance.