Utilizing an Ensemble Model with Anomalous Label Smoothing to Detect Generated Scientific Papers (2024)

Abstract: Generative AI, as it becomes increasingly integrated into our lives, has brought convenience, though some concerns have arisen regarding its potential impact on the rigor and authenticity of scientific research. To encourage the development of robust and reliable automatically-generated scientific text detection systems, the “DAGPap24: Detecting Automatically Generated Scientific Papers” competition was held and shared the same task with the 4th Workshop on Scholarly Document Processing (SDP 2024) to be held at ACL 2024. In the DAGPap24 competition, participants were tasked with constructing a generative text detection model that could accurately distinguish between the human written fragment, the synonym replacement fragment, the ChatGPT rewrite fragment, and the generated summary fragment of a paper. In this competition, we first conducted a comprehensive analysis of the training set to build a generative paper detection model. Then we tried various language models, including SciBERT, ALBERT, DeBERTa, RoBERTa, etc. After that, we introduced an Anomalous Label Smoothing (ALS) method and a majority voting method to improve the final results. Finally, we achieved 0.9948 and 0.9944 F1 scores during the development and testing phases respectively, and we achieved second place in the competition.

## introduction with the rapid development of nlp technology, especially with the emergence of chatgpt 1 , there is an increasing amount of text generated by nonhuman entities. however, machine-generated text may sometimes contain errors that are not easily discernible by humans, leading to a decline in the rigor and credibility of scientific papers. current nlp generation technology has nearly reached parity with human writing, which poses 1 https://chatgpt.com/ the challenge of distinguishing them @xcite . in response to this issue, @xcite propose a detection method that does not require training a classifier or collecting additional datasets. they argue that model-generated text is more sensitive to minor rewriting perturbations than human-written text. to further distinguish the sentences with various lengths, researchers in @xcite propose a length-sensitive multiscale positive-unlabeled loss that can improve the ability of detection of bert @xcite and roberta @xcite . to further understand the achievements, this survey @xcite provides an overview of existing techniques for detecting llm-generated text. in this paper, we mainly detail our work during the shared task dagpap2024. we are tasked with detecting automatically generated papers. we can abstract this task into a token-level multiclassification problem. there are three challenges in this task. 1) difficulty in efficiently bridging the gap between word-level tokenization and subwordlevel tokenization more rationally. the dataset provided in the competition is tokenized in word-level, while most of the existing language models using the subword-level tokenization method. if the two tokenized methods are not accurately mapped, it will not only lead to a waste of useful information but may also lead to misalignment of the prediction results and the test set. 2) difficulty in ensuring that the distribution of the generated prediction results is consistent with the label distribution of the training set. we found that only relying on the model's predictions will lead to inconsistency in the distribution between the training set and test set. for example, there will be several other labels predicted in the middle of a continuous text. it leads to inconsistency in the distribution between training set and development set. 3) difficulty in determining which model performs best on tokenlevel classification tasks. different models, each with unique training parameters, yield varying predictions for this task, resulting in different label prediction outcomes for the same token. the selection of the final result directly impacts the f1 score. to overcome the aforementioned challenges and enhance our performance in the competition, we propose the following solutions: ## task overview to enforce the reliability of scientific papers, we focus on detecting automatically generated scientific papers. we should distinguish human-written fragments, synonym-replaced fragments, chatgptgenerated fragments, and summarized fragments, which can be abstracted as a token-level classification task. fig. 1 provides a visualization of the fragments extracted from the training set. ## conclusion in this paper, we mainly introduce the automatically generated papers detection and detail our solution for the dagpap2024 competition. firstly, based on the analysis of the data, we adopted two different tokenization ways to ensure that the pre-dicted results can accurately correspond to the original text set. then, we introduced an anomalous label smoothing method to ensure that the distribution of predicted results is consistent with the label distribution of the training data set without the loss of information. finally, we used model fusion to maximize the performance of different models. the above efforts ensured that we achieved a high f1 result in this competition, which was 0.9948 and 0.9944 on f1 score during the development and testing phases, and we achieved second place in the competition.