DUT h at S em E val 2024 Task 8: Comparing classic Machine Learning Algorithms and LLM based methods for Multigenerator, Multidomain and Multilingual Machine-Generated Text Detection (2024)

Abstract: Text-generative models evolve rapidly nowadays. Although, they are very useful tools for a lot of people, they have also raised concerns for different reasons. This paper presents our work for SemEval2024 Task-8 on 2 out of the 3 subtasks. This shared task aims at finding automatic models for making AI vs. human written text classification easier. Our team, after trying different preprocessing, several Machine Learning algorithms, and some LLMs, ended up with mBERT, XLM-RoBERTa, and BERT for the tasks we submitted. We present both positive and negative methods, so that future researchers are informed about what works and what doesn’t.

## introduction llms are becoming more and more part of our everyday lives due to their easy accessibility and their remarkably fluent responses in different fields like news, healthcare and education. this extensive usage can lead to unintended consequences. specifically, llms could replace humans, provide sometimes false, incomplete or even misleading information, risk the critical thinking of students and progressively of the whole society. so, it is of high importance to find a way to identify if a text was written by a human or by a machine. since all these complex models are trained on large datasets and have achieved generating texts that are so human like, it is difficult for a person to identify who generated a text. here comes the importance of the automatic models, capable of differentiating between human written texts and machine generated texts, by exploiting patterns invisible to a human. in this paper we describe the duth participation in semeval 2024 task 8: multigenerator, multidomain, and multilingual black-box machine-generated text detection. the task features three directions: binary human-written vs. machine-generated text classification, multi-way machine-generated text classification and human-machine mixed text detection. the first two refer to the scenario where a system must classify input texts which are fully written by a human or a machine. one detail regarding the second scenario is that we are also provided with the specific language model which generated the input text. in the third scenario we are presented with a text which is half human and half machine written and we have to determine the boundary, where the change occurs. in all subtasks the text data are coming from different sources and different generators, and we are not allowed to use any external data except for the ones given from the organizers. the sub-tasks can be briefly described as follows: subtaska monolingual: given only english texts, we need to determine whether a text is humanwritten or machine-generated. subtaska multilingual: given texts from 8 different languages (english, arabic, chinese, indonesian, urdu, german, bulgarian, russian), we need to determine whether a text is human-written or machine-generated. subtaskb: given only english texts, we need to determine whether a text is human-written or machine-generated and which is the specific generator. subtaskc: given only english mixed texts, where the first part is human-written and the second part is machine-generated, we need to determine the boundary. our team participates by submitting on sub-taska (both monolingual and multilingual) and subtaskb. during the competition we examine several methods, especially on subtaska monolingual, like different preprocessing techniques on the text data, several machine learning algorithms, some ensembling methods and llms. we ended up submitting llms to all subtasks. the models we choose are mbert for subtaska monolingual, xlm-roberta for subtaska multi-lingual and bert for subtaskb, as they achieve better performance on average. all these pre-trained models have been proven to be powerful for different nlp tasks. our proposed system for every subtask is a classifier based on a fine-tuned large language model. during the training process, our model is provided with a text as input and a label regarding whether this input is human or machine generated. additionally, this paper provides a comparative study of different llms fine tuned in this task. in this context, we also provide results regarding the use of classic machine learning algorithms trained to tackle this task. ## system overview transformers have achieved state-of-the-art @xcite performances on several natural language processing tasks such as text classification. this is why all final models submitted are llms. here we present the submitted model on each subtask. we have all the hyperparameters for tuning the models submitted in the appendix section 6. ## conclusion based on the results we have on our new validation set and the results on the test set, we assume that this drop occurs since our model cannot generalize well. we believe that if the test sets had texts coming from the same domains and generators as the training texts, and had the same languages, then our models would have achieved better results. future work could focus on either training larger language models or trying to improve generalization of ours, possibly with some preprocessing like data augmentation. we look forward to further research on these tasks, hoping for better results.