G roningen team D at S em E val-2024 Task 8: Exploring data generation and a combined model for fine-tuning LLM s for Multidomain Machine-Generated Text Detection (2024)

Abstract: In this system description, we describe our process and the systems that we created for the subtasks A monolingual, A multilingual, and B forthe SemEval-2024 Task 8: Multigenerator, Multidomain, and Multilingual Black-Box MachineGenerated Text Detection. This shared task aimsat detecting and differentiating between machinegenerated text and human-written text. SubtaskA is focused on detecting if a text is machinegenerated or human-written both in a monolingualand a multilingual setting. Subtask B is also focused on detecting if a text is human-written ormachine-generated, though it takes it one step further by also requiring the detection of the correct language model used for generating the text.For the monolingual aspects of this task, our approach is centered around fine-tuning a debertav3-large LM. For the multilingual setting, we created an ensemble model utilizing different monolingual models and a language identification toolto classify each text. We also experiment with thegeneration of extra training data. Our results showthat the generation of extra data aids our modelsand leads to an increase in accuracy.

## introduction the semeval-2024 shared task focuses on multigenerator, multidomain, and multilingual blackbox machine-generated text detection. the shared task is split into three different subtasks. each subtask is monolingual except for the first subtask, which has a monolingual (english) and a multilingual track. the languages covered in this shared task include english, chinese, russian, urdu, indonesian, italian, german, and arabic. this paper presents our the systems that we created for the shared task. the paper provides an overview of our research strategies and results for subtasks a and b. subtask a focuses on the detection of machineversus human-written text, we differentiate between mono-and multilingual data. our approach involves fine-tuning llms, deberta-v3 (large) in particular. we experimented with different parameters for the model, searching for the best performance possible. subtask b extends the challenge presented in subtask a, we now attempt to recognize the specific language model used for text generation. we do this in addition to distinguishing between human and machine-generated text. we again use to classify the data. to optimize model accuracy, we fine-tune hyperparameters. additionally, we generate extra wikipedia articles to further expand the training data. we hypothesize that extra data will lead to better model performance, and thus better applicability to realworld applications. our research focuses on finding both the best possible language model settings to recognize machine-and human-written text and distinguish between different language generation models. our code and the additionally generated data can be found on github foot_0 2 related work train dev test subtask a-mono 119757 5000 34272 subtask a-multi 172417 4000 42378 en 136589 0 28200 ar 0 1000 2103 ru 0 2000 0 zh 11934 0 0 id 5995 0 0 ur 5899 0 0 bg 12000 0 0 de 0 1000 6000 it 0 0 6075 subtask b 71027 3000 18000 langid.py (lui and baldwin, 2012) is a supervised language identification tool trained using a naive bayes classifier. langid.py has the following advantages: fast, usable off-the-shelf, unaffected by domain-specific features (e.g. html, xml, markdown), single file with minimal dependencies, and flexible interface. langid.py was applied in our system to identify the multi-language training set of subtask a and we found that it can identify languages with very high accuracy. ## data the dataset provided by the shared task creators originates from the benchmark m4 @xcite . m4 is a comprehensive dataset encompassing machine-generated text from diverse generators, domains, and languages. m4 focuses on the development of automated systems for detecting machine-generated text and identifying potential abuse. the dataset comprises text samples sourced from various platforms, including wikipedia, reddit, wikihow, peerread, arxiv, chinese qa, urdu news, russian ruatd, indonesian news, and arabic wikipedia. it spans multiple languages and domains, presenting a rich and diverse collection of machine-generated text for analysis and classification. ## system overview this section presents an overview of the methods we employed for subtask a, both the monolingual and multilingual data setting, as well as subtask b. we follow previous work on a similar topic @xcite , by fine-tuning llms, predominantly deberta @xcite , on this task. we were further stimulated to explore this model specifically, as deberta is developed as an improvement over the roberta model @xcite , the latter being employed by the task organizers as a baseline. specifically, we looked at using both the base and large variants of deberta-v3, as this improved version of deberta is reported to significantly outperform previous iterations on numerous tasks. as the goal of this task was to create systems that can discriminate between human-written and machine-generated text regardless of generator, textual domain, or language, we opted not to preprocess our data any further than what the task organizers already did. this will keep our data as close to instances that can be encountered in realworld scenarios as possible. we fine-tuned these pre-trained language models using the transformers library from huggingface @xcite . ## results and analysis in this section, we show and analyze the results achieved for each of the subtasks. ## discussion/conclusion in conclusion, we think our participation in the shared task resulted in some valuable insights into the challenges of machine-versus human-written text. despite our efforts, our systems unfortunately fell short of surpassing the baseline scores established by the task organizers. across the different subtasks, our models showed varying performance. for subtask a monolingual, our models achieved some promising results on the development set, with an accuracy of 87.80%. however, our model did not manage to generalize enough, leading to an accuracy of 63.68% on the test set. for the multilingual part of subtask a, our model reached 65.90% on the development set. in this case, the model did manage to generalize the data, leading to an accuracy of 71.79% on the test set. however, this was still below our expectations, and the baseline accuracy of 80.88%. in subtask b, our models struggled to identify the specific language model used for text generation accurately, with accuracies of 72.80% on the development, and 61.50% on the test set. despite optimizing hyperparameters and training on both original and additional data, our models failed to outperform the baseline accuracy of 74.60%. we think our analysis revealed several points for improvement. our models tended to misclassify human-written text, indicating a potential bias towards machine-generated content. furthermore, the models seemed unable to generalize, leading to worse performance on the test set for monolingual task a. moving forward, we think there are many improvements to be made. future research could focus on using other model architectures or exploring other data augmentation techniques. also, training the model in more languages could improve the performance of multilingual models. of course, using larger pre-trained models could also lead to an easy increase in performance, although it does require significant resources. lastly, our findings also show generating extra training data is essential for improving model performance. therefore, a promising direction for future work is to explore new data sources and methods to create richer and higher-quality training data to further improve the performance and generalization ability of the model. morgan funtowicz, joe davison, sam shleifer, patrick platen, clara ma, yacine jernite, julien plu, canwen xu, teven scao, sylvain gugger, and alexander rush. 2020. transformers: state-of-theart natural language processing. pages 38-45, 01.