Multi-head Span-based Detector for AI -generated Fragments in Scientific Papers (2024)

Abstract: This paper describes a system designed to distinguish between AI-generated and human-written scientific excerpts in the DAGPap24 competition hosted within the Fourth Workshop on Scientific Document Processing. In this competition the task is to find artificially generated token-level text fragments in documents of a scientific domain. Our work focuses on the use of a multi-task learning architecture with two heads. The application of this approach is justified by the specificity of the task, where class spans are continuous over several hundred characters. We considered different encoder variations to obtain a state vector for each token in the sequence, as well as a variation in splitting fragments into tokens to further feed into the input of a transform-based encoder. This approach allows us to achieve a 9% quality improvement relative to the baseline solution score on the development set (from 0.86 to 0.95) using the average macro F_{1} -score, as well as a score of 0.96 on a closed test part of the dataset from the competition.

## introduction modern advances in the field of text generation models provide high quality artificial texts that are hardly distinguishable from human-written texts at fluent reading. text generation systems such as llama @xcite , chatgpt (ray, 2023), mistral @xcite are increasing the list of successfully solved problems in updated benchmarks @xcite with the release of each newer version of the product. although the progress of such models is impressive, it poses new challenges for scientists, as the development of these systems implies the emergence and spread of generated fragments in texts of the scientific domain. in the field of natural language processing there are already a large number of methodologies for detecting texts generated by machine learning models @xcite , including approaches for the scientific domain @xcite . therefore, the improvement of artificial text detection techniques occurs simultaneously with the improvement of text generation methods. in order to prevent the gap between quality generation and precision of detection from growing, it is necessary to periodically update and modernise existing detection approaches with newly generated fragments. indeed, increasing the appearance of generated fragments in scientific papers carries with it a potential increase in plagiarism @xcite , fake studies @xcite and missinformation. the dagpap24 competition allowed us to perform a further cycle of updates and make a reliable ai-generated scientific text detection system. the challenge is to build a detection system that is robust to generated fragments from fundamentally different frameworks and diverse scientific domains. in this paper, we present a solution that was developed for the dagpap24 competition by our team. we propose a method to find artificially generated fragments at the token level using a multitask architecture. our model has two classifiers for each token, that are trained jointly and allow the model to have a higher generalisation capability and to process the text sequences arriving at the input faster. we introduce the aforementioned architecture in this paper and conduct experiments to tune its components to obtain the best result. ## data and task description the dagpap24 competition has been formulated as one classification task for the english language. participants had to identify intervals in the scientific text that were human-written or machinegenerated accordingly. the intervals are not fixed in advance and have to be labelled automatically by the detection algorithm based on the context. the problem has been proposed to be solved by token classification approaches from the competition organisers. a large number of works on artificial text detection is dealing with the classification problem at sequence level @xcite . however, the approach of token classification and their subsequent grouping into intervals of the same class is also able to show a competitive performance in this task, in some cases producing better results than the established approach @xcite . according to the organisers the task contains scientific texts whose tokens should be classified into one of four classes: human, generated by chatgpt, generated using synonyms from nltk @xcite , generated using an unnamed summarisation model. ## approach 3.1 model there are several ways to solve the given problem, we have chosen the token classification approach. one of its subfields is grammatical error correction (gec) @xcite , which aims to correct as many errors as possible in a given passage. it correlates with our task, where, for example, the junction from one class to another can be in an uncommon place and of any length, so we need to identify the boundaries very precisely. one state-of-the-art solution is the approach described by @xcite . their gector model has a custom multi-head transformer-based architecture with and solves the gec problem with sequence tagging. we decided to use a similar architecture with our extension in the machinegenerated fragments detection domain where we need to solve the problem at the token level. the architecture we use is shown in figure 1 . ## experiments the organisers of the competition presented a basic solution: distilbert @xcite and @xcite . both models were finetuned with the classical architecture and pipeline of the token classification task. considering the limited time of the competition, we planned experiments with the architecture described in the previous section: varying the input sequence length, changing bert-like encoders and selecting hyper-parameters. ## results the results of the experiments can be seen in ## conclusion in this paper we present a descriptive overview of the solution approach that was used by our team to solve the dagpap24 competition for detecting ai-generated fragments in scientific documents. a token classification approach was chosen to map each token to one of four pre-defined classes. in our solution, we used a bert-based encoder and two linear layers to process the output vectors of the encoder for each token in sequence. one of the layers is responsible for binary classification and the other for the predefined 4 classes. exploring shared training, the classification quality becomes better for identical model settings, and on the provided development dataset, this method helped to improve the quality by 9% on the macro f 1 -score metric. in the future work we would be interested in analysing the distribution of the metric across classes and based on this we would like to add weights to the loss function during pre-training for class balancing. in addition, it would be interesting to observe this approach on a larger dataset, where classes can be defined by more common text generation methods.