Papilusion at DAGP ap24: Paper or Illusion? Detecting AI -generated Scientific Papers (2024)

Abstract: This paper presents Papilusion, an AI-generated scientific text detector developed within the DAGPap24 shared task on detecting automatically generated scientific papers. We propose an ensemble-based approach and conduct ablation studies to analyze the effect of the detector configurations on the performance. Papilusion is ranked 6th on the leaderboard, and we improve our performance after the competition ended, achieving 99.46 (+9.63) of the F1-score on the official test set.

## introduction a series of the dagpap shared tasks addresses mitigation of the risks associated with misusing language technologies in the scientific domain, such as fabricating research publications @xcite and affecting the peer review ecosystem at the leading ai conferences @xcite . dagpap22 @xcite focuses on a well-established task formulation in the artificial text detection field @xcite , which requires to determine if a given text is human-written or ai-generated. dag-pap24 foot_0 offers a less explored task formulation, which is framed as a token-level classification problem. developing robust and reliable ai-generated scientific text detectors promotes analyzing scientific text collections at scale and assisting nonexpert and expert users in identifying the fabricated content @xcite . this paper proposes papilusion foot_1 , an aigenerated text detection system for tagging text segments, which are produced by a generative language model (lm) or modified through synonym replacement and summarization. the token-level ai-generated text detection accounts for a more practical scenario, where a generative lm is used as a writing assistant. papilusion is an ensemble of encoder-only sequence taggers that leverages the strengths of multiple independently finetuned lms to enhance the accuracy of detecting ai-generated text. we conduct ablation studies to analyze the impact of various hyperparameter and lm configurations on the target performance. papilusion is ranked 6th among 30 participating teams at dagpap24, with 89.83 of the f1-score on the leaderboard. after the competition ends, we improve our solution achieving 99.46 (+9.63) of the f1-score on the official evaluation set. ## background the dagpap24 dataset is a collection of humanmachine mixed scientific texts, where the text segments can be (i) human-written, (ii) modified through an nltk-based synonym replacement @xcite , (iii) produced by chat-gpt, and (iv) summarized. the organizers provide 5000, 5000, and 20000 training, development, and test examples, respectively. the average number of tokens foot_2 is 5591. task formulation the task is to assign one of the four corresponding labels to each token in a given text, as it was described above. the example for a text with a synonym replacement is shown below. here, "1" stands for a synonym replacement, and "0" refers to the human-written segment: performance metric the performance is evaluated using the macro-averaged f1-score. ## papilusion the papilusion pipeline involves several key steps (see figure 1 ): although based on the same architecture, the models were fine-tuned with different hyperparameters to create a more diverse ensemble. each model was trained on a sequence labeling task, predicting labels for every token in the input sequence. by using encoder models fine-tuned for text classification and applying a majority voting mechanism, we aimed to improve the accuracy and robustness of our classification system. ## experiments overview we conducted a wide series of experiments with a family of debertav3 models @xcite . we investigate experiments with bottom layers freezing @xcite and the size of the input sequence as the most important hyperparameters for the efficient model selection. we used longer sequences to fit in more context for a model. additionally, we believe that there are some dependencies between the size of the model and effectiveness of a layer freezing and using various input sequences lengths. in our experiments we conducted a detailed research and comparison of various setups. post-competition study after the end of competition we identified a tokenization issue in the code: the chunk function truncated sequence by the number of words, not tokens which resulted in information loss. we fixed our training scripts with proper tokenization in chunk function, along with some minor adjustments in the prediction function. these modifications significantly improved the models' performance, resulting in a score increase of +9.63, and enabled us to conduct a detailed study of various models and hyperparameter variations. the results are shown in the we fixed other hyperparameters (learning rate, scheduler, seed, validation dataset size) to perform fair comparison. in this case, we can see how changing previously mentioned hyperparameters affects the result. additionally, we did a series of ensemble mixtures based on majority voting technique for our best models to improve our final score. ## results competition phase during the competition we fine-tuned base deberta models with different number of frozen bottom layers and then applied majority voting for 3 models. this brought us to the 6th place (see ## conclusion in this paper, we proposed papilusion, an aigenerated text detection system designed for the dagpap24 shared task. our system ranked 6th in the competition, achieving an f1 score of 89.83. post-competition enhancements, including fixing tokenization errors and optimizing model parameters, increased f1 score to 99.46. through comprehensive ablation studies, we identified the most impactful hyperparameters and model configurations, leading to substantial performance improvements. while our experiments demonstrate that larger models achieve the best results when resources are not limited, we also found that even the small models (deberta-xsmall) exhibit promising performance metrics with minimal computational resources.