G roningen Team F at S em E val-2024 Task 8: Detecting Machine-Generated Text using Feature-Based Machine Learning Models (2024)

Abstract: Large language models (LLMs) have shown remarkable capability of creating fluent responses to a wide variety of user queries. However, this also comes with concerns regarding the spread of misinformation and potential misuse within educational context. In this paper we describe our contribution to SemEval-2024 Task 8 (Wang et al., 2024), a shared task created around detecting machine-generated text. We aim to create several feature-based models that can detect whether a text is machine-generated or human-written. In the end, we obtained an accuracy of 0.74 on the binary human-written vs. machine-generated text classification task (Subtask A monolingual) and an accuracy of 0.61 on the multi-way machine-generated text-classification task (Subtask B). For future work, more features and models could be implemented.

## introduction recent large language models (llms), such as chatgpt, have shown remarkable capability of creating fluent responses to a wide variety of user queries. this, in combination with increased accessibility to these models, has lead to an increase of machine-generated content over various channels. however, these llms come with concerns regarding the potential misuse of such tasks, like spreading misinformation and misuse within the education system @xcite . therefore, detecting whether a text is human-written or machinegenerated is extremely important. unfortunately, humans perform only slightly better than chance at this task, as found by @xcite . this introduces the need for developing automatic systems that can identify machinegenerated texts, in order to mitigate their potential misuse @xcite . while previous work has been done on identifying machine-generated texts, they either focused on only one or two particular languages, or focused on detecting machine-generated text for a specific llm or a specific domain @xcite . for instance, @xcite note that most of the research on machine-generated text detection uses systems that were trained on english datasets and that prior works show that detectors fine-tuned on english data fail to generalize to other languages. this can be seen in the results from @xcite , where a decrease is seen from 0.946 auc roc to 0.537 when working with german data. in addition, @xcite get similar results. meanwhile, @xcite address the issue of detection systems not generalizing across different generation models and domains. they mention that most previous works often overlook that detection systems would be applied to a broad variety of domains, writing styles, and generation models. thus, the goal of task 8 of semeval 2024 @xcite is to take a first step into creating a mono-or multilingual system that is able to detect machine-generated text created by different llms in different domains. the task is divided into three subtasks, of which we participate in two. these subtasks are described in section 2. in order to tackle this task, we have created multiple feature-based models. the features on which the models have been trained will be described in section 3. our decision for creating these featurebased models is supported by our beliefs that a model with carefully crafted features is computationally less expensive than fine-tuning a llm, while it may be able to achieve equal or better performance because of its ability to generalize @xcite , which is something that llms tend to struggle with @xcite . we focused on creating monolingual models for subtask a and b. eventually, we created a model that was ranked 95th out of 137 teams for the monolingual track of subtask a with an accuracy of 0.69. for subtask b, we ranked 50th out of 77 participating teams with an accuracy score of 0.61. all of our code for this project can be found on our github repository. foot_0 ## background the goal of this shared task is to create machinelearning models that are able to detect machinegenerated text across multiple languages, generators and domains. the datasets we used to train and test were provided by the task organizers. we took part in two subtasks: a and b. ## system overview we experimented with four different kind of supervised models and the features that we created ourselves. we will describe each model and feature one by one. ## experimental setup in this following section, we will describe how we created the models and crafted the features. ## conclusion to conclude, while we still gained quite good results, our models do not outperform the state-ofthe-art models. we achieved an accuracy of 0.74 on subtask a and an accuracy of 0.61 on subtask b. both scores are lower than the roberta baseline. we discovered that sentence tense, sentence voice and the ratio between pronouns and named entities seemed to be effective for the classification task, while sentiment did not have that much influence. in future research, there could be more experimentation with different kinds of machine learning models and more features could be created to further improve the models.