G roningen Group E at S em E val-2024 Task 8: Detecting machine-generated texts through pre-trained language models augmented with explicit linguistic-stylistic features (2024)

Abstract: Our approach to detecting machine-generated text for the SemEval-2024 Task 8 combines a wide range of linguistic-stylistic features with pre-trained language models (PLM). Experiments using random forests and PLMs resulted in an augmented DistilBERT system for subtask A and B and an augmented Longformer for subtask C. These systems achieved accuracies of 0.63 and 0.77 for the mono- and multilingual tracks of subtask A, 0.64 for subtask B and a MAE of 26.07 for subtask C. Although lower than the task organizer’s baselines, we demonstrate that linguistic-stylistic features are predictors for whether a text was authored by a model (and if so, which one).

## introduction the semeval-2024 task 8 is aimed at the detection of machine-generated texts across different domains, languages, and generators. the challenge of distinguishing machine-generated from human written texts has become increasingly relevant with the rapid improvement and coinciding widespread usage of large language models (llms) such as chatgpt. detection of machine-generated text can be important to uncover the purposeful spreading of misinformation on social media, or fraudulent articles and papers in the context of journalism and academics @xcite . to this end, the task organizers collect human data from wikipedia, reddit, wikihow, peerread, and arxiv abstracts in english, chinese, urdu, russian, indonesian, arabic, and bulgarian @xcite . consequently, @xcite prompted 5 different generative llms to write the corresponding posts or abstracts based on the titles. the shared task consists of 3 subtasks. subtask a is the task of classifying between human and machine-generated texts, subtask b pertains pointing out which llm (or human) generated the text specifically, and lastly, subtask c is about determining the boundary where a text switches from human written to machinegenerated. the focus for our submission to the shared task is to investigate and compare the linguistic-stylistic characteristics of various llms, given that previous literature has shown that text produced by generative llms contain linguistic-stylistic anomalies @xcite . additionally, we explore ways to combine features with the power of a pretrained language model (plm). although a system inspired by linguistic-stylistic features may not achieve the greatest scores, it may perform well across domains and is highly interpretable. additionally, the performance with linguistic-stylistic features may differ per llm and per domain, for which they possibly yield interesting insights and contribute to scientific knowledge regarding what llm-generated anomalies consist of. ultimately, our system yields passable results, coming in at 110 and 41 for subtask a mono-and multilingual respectively, and ranking at 46 for subtask b and 20 for subtask c. contrary to expectations, the system appears to be relatively poor at generalizing between domains, but markedly better at dealing with multiple languages, as indicated by the increase in accuracy as well as ranking between the mono-and multilingual conditions in subtask a. ## background to investigate which features contribute to the detection of machine-generated text, we collected 20 metrics from previous research which seem relevant. these features can be broadly divided into 6 categories, which will each be presented in this section. ## system overview the backbone of our system is combining a featuredriven approach with the state-of-the-art in text classification, namely plms. to accomplish this, we tested three main system architectures for this shared task. firstly, we augment distilbert with the feature set. similarly, to achieve token-level classification, we perform the same for longformer, and lastly, we use a random forest classifier with distilbert embeddings in conjunction with the feature set. for comparison the features were also used separately and in combination with unigrams in a random forest classifier. the following sections will go into detail on each of these individually. ## experimental setup much of the experimental setup is similar to the format dictated by shared task organizers @xcite . in particular, the provided train and dev sets were used as-is. however, some relevant aspects for our specific system will be presented in this section. first and foremost, the features are extracted using a variety of external libraries, including spacy and fasttext, and subsequently put into json format. furthermore, in the multilingual track, we recognize the language in question using stanza and fasttext (how and why these models were used is explained in appendix b.1), and apply languagespecific feature extraction methods accordingly. features that only work for english (e.g., dale chall) are not included in the multilingual track. these features were all assigned a value of -1. an overview of the features and how they were calculated can be found in appendix b. to compare the features, we measured importance using mean decrease in impurity (mdi), which computes the average change in homogeneity in random forest nodes for each feature. the systems will be evaluated using the official metrics of the task: accuracy for subtasks a an b and mean absolute error for subtask c. ## conclusion unfortunately none of our systems performed better than the baseline but our experiments did give some insight in how features can be used. our final systems producing the most satisfactory results for the test set were: the augmented distilbert system for subtasks a and b, resulting in accuracies of 0.63 and 0.77 respectively, and the augmented longformer for subtask c, obtaining a mae of 26.07. contrary to the literature, it does not appear that the feature-based methods are better at generalizing. during our experiments we saw that when the unseen test data is from the same distribution (held from the training data), the feature-based approach performs far better in terms of accuracy (a mono: 0.88; a multi: 0.79; b: 0.71; for a full overview, see appendix e). this is a clear indication that our proposed stylistic-linguistic features contain sufficient predictive power to distinguish humanfrom machine-written text. in particular, we find type-token ratio, mtld, the amount and length of lexical chains, and the average number of syllables per word to be noteworthy features for the given task. future research into (different) features, especially for multilingual tasks could therefore be fruitful.