D eep P avlov at S em E val-2024 Task 8: Leveraging Transfer Learning for Detecting Boundaries of Machine-Generated Texts (2024)

Abstract: The Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection shared task in the SemEval-2024 competition aims to tackle the problem of misusing collaborative human-AI writing. Although there are a lot of existing detectors of AI content, they are often designed to give a binary answer and thus may not be suitable for more nuanced problem of finding the boundaries between human-written and machine-generated texts, while hybrid human-AI writing becomes more and more popular. In this paper, we address the boundary detection problem. Particularly, we present a pipeline for augmenting data for supervised fine-tuning of DeBERTaV3. We receive new best MAE score, according to the leaderboard of the competition, with this pipeline.

## introduction recently, there has been a rapid development of auto-regressive language models, for example, @xcite , @xcite , and llama2 @xcite . these models are trained on enormous amounts of data and are able to produce coherent texts that can be indistinguishable from human-written texts @xcite . the semeval-2024 task 8 competition suggests to tackle the problem of detecting machinegenerated texts. this problem has become more relevant recently due to the release of chatgpt foot_0 , a model by openai that simplified the access to the large language models (llm) and their usage. for example, llm can be maliciously used to generate fake news @xcite . there are also some concerns raised among scientists @xcite and educators @xcite that the usage of llms will devalue the process of learning and research. the commonly used approach to formulate the task of detecting machine-generated texts is a binary classification task @xcite . in this case, a text can be attributed to either a human or a llm. otherwise, the task can be formulated as a multiclass classification or an authorship attribution task @xcite , where it is needed to determine which one of the k authors is the real author of the given text. finally, the trend toward human-ai collaborative writing is rising, which highlights the importance of the boundary detection task. in this setup, text contains consecutive chunks of different authorship, and it is required to detect where the boundaries between chunks lie and who is the author of every chunk. due to its complexity, it is usually assumed the text has a human-written prefix and the rest of the text is aigenerated @xcite @xcite . our main contributions are three-fold: additionally, we've made the code of augmentation publicly available. 2 2 related work most of approaches @xcite for machine-generated text detection are based on calculating linguistic @xcite , stylometric, and statistical features, as well as on using classical machine learning methods like logistic regression, random forest, and gradient boosting as classifiers. among commonly used features are word and n-gram frequencies @xcite , and tf-idf @xcite . an alternative strategy is to use zero-shot techniques based on the internal metrics of the texts. for example, token-wise log probability can be evaluated by models like grover @xcite or gpt-2 @xcite . a probability threshold is established to distinguish writings produced by machines from those written by humans. moreover, rank @xcite or log-rank @xcite can be calculated for each token and compared for consistency with the prior context. it's shown by @xcite that featurebased methods are inferior to methods based on using encoders of pretrained language models like bert @xcite as a basis for fine-tuning on the selected domain. representations from auto-regressive language models can be used as input for the classification head. such transformerbased methods require supervised detection examples for further training. among the models commonly used for fine-tuning are roberta @xcite and xlnet @xcite . fine-tuned roberta has shown the state-of-the-art results on the problems of authorship attribution @xcite . to tackle the problem of mixed human-machine writing, @xcite introduces the real or fake text (roft) tool, where humans were asked to detect the sentence where the text transitions from human-written text to machine-generated text. one of the possible formulations of this task is multilabel classification @xcite , where the boundary detector needs to determine the first generated sentence, and the number of this sentence is considered the label of the text. in that work, each sentence is processed separately with shallow classification and regression models based on roberta and sroberta @xcite . that solution perform well in an in-domain setup, but is limited in an out-of-domain setup. any solution for tasks about detecting machine-generated texts should be robust to domain change. the organisers of the semeval-2024 task 8 competition claimed to have added new domains to the test set for testing the robustness of participants' solutions. there are several works on performance of detecting methods on out-of-domain setup, such as @xcite and @xcite . @xcite conclude that perplexitybased and topological features appear to be helpful in case of domain shift. ## proposed method the provided train dataset consists only of 3,649 texts. it's well known that an abundance of indomain training data is crucial for classifier performance @xcite . however, during the competition period, it was prohibited to use any external data for training, and thus we were limited to working only with the provided dataset. in this case, getting more training data with some kind of augmentation plays a crucial role. we designed an augmentation pipeline and ran all our experiments on two sets of data: the one provided by the organisers of the competition, described in section 3, and our augmented data. ## experimental setup we have used pretrained longformer-4096-base and longformer-4096-large with default hyperparameters to fine-tune longformer. for experiments with roberta, we have chosen two models: roberta-base and roberta-large. the models were fine-tuned with the set of custom hyperparameters, taken from the original paper @xcite . finally, for experiments with deberta, we have also chosen two models: deberta-v3-base and deberta-v3-large. the models were finetuned with the set of custom hyperparameters taken from @xcite . all custom hyperparameters are listed in appendix b. for all models we set the maximum length of context in tokenizer equal to 512 as there are only few text items in both train and test set with tokenized text length greater than 512. additionaly, we've used the early stopping method for all of our experiments to get rid of epochs dependency. ## conclusion in this paper we describe the system submitted for semeval2024-task 8, the subtask dedicated to hybrid human-machine writing detection. we present a simple yet effective augmentation pipeline. we explore how adding this pipeline to the process of fine-tuning can significantly increase the performance on the task, and provide an analysis of performance of various models with and without our augmentation pipeline. the best model, which is debertav3-large fine-tuned on a large set of augmented data, receives a new best score according to the leaderboard of the competition. other finetuned models achieve competitive results, ranking in the upper half of the leaderboard and beating the organisers' baseline. as the provided data was limited to english language only, future work might include training multilingual boundary detection solution by mixing training data of different languages and using a multilingual encoder @xcite . such a system can be used for hybrid aiwriting detection as a standalone solution or can be integrated into existing nlp frameworks like deeppavlov @xcite . the first text contains no signs of a flawed generation, the second and third texts have some flaws. in the second text, the model begins to generate from the capital letter, and its generation is incoherent with the human-written prefix. in the third text, the model starts to repeat the end of the human part, which is a glaring sign of machine-generated text. model i noticed that in figure 2 , the two quantization factors for quantized layers are missing labels. it would be helpful for the reader to understand which layers are being quantized in the figure . hi authors, you seem to have submitted two of the same paper? pls advise could you please clarify if this is a mistake or if there are any differences between the two submitted papers? there has been prior work on semi-supervised gan, though this paper is the first context conditional variant. the novelty of the approach was the novelty of the approach was leveraging in-painting using an adversarial loss to generate contextually relevant images.