Team art-nat- HHU at S em E val-2024 Task 8: Stylistically Informed Fusion Model for MGT -Detection (2024)

Abstract: This paper presents our solution for subtask A of shared task 8 of SemEval 2024 for classifying human- and machine-written texts in English across multiple domains. We propose a fusion model consisting of RoBERTa based pre-classifier and two MLPs that have been trained to correct the pre-classifier using linguistic features. Our model achieved an accuracy of 85%.

## introduction after rapid developments in large language models (llms) and generative ais in the last years, the detection of machine-generated content has become one focus of study as deepfakes, machinegenerated lawyer statements and even libel suits (superior court of gwinnett county) concerning language machines stress the importance of detecting texts not written by humans. the semeval shared task 8 in 2024 aims at multi-and monolingual machine-generated text (mgt) detection from various domains by multiple models. for the monolingual english data in subtask a @xcite we propose a fusion model built using pre-trained roberta word embeddings specialized for ai-generated text detection and correction mlp classifiers, supported by the additional computation of linguistic, stylistic and probabilistic features selected based on their informational value. with this system design, our model ranked at position 25 out of 124 with an 0.855 accuracy score on the task. the only data used for training was the one provided by the organizers without further data augmentation. because of the different distributions of the data in the development and test data sets several strategies were tested and a fusion model was chosen as the best strategy. ## background over the last years, numerous approaches have been proposed to tackle the task of mgt detection. some models, such as detectmgt @xcite , focus on detecting texts from a specific source, such as gpt-family llms, while other approaches are specialized in texts from a specific genre, such as @xcite for toefl essays. other architectures, like ensemble models combining different classifiers @xcite have been successfully used for machine-generated text detection to improve out-of-distribution performance @xcite . @xcite show that, overall, deeplearning approaches, and in particular a robertabased-detector, are one of the best individual models for mgt detection. the roberta-baseddetector was shown to be particularly robust against oov scenarios in both chinese and english, compared to a machine learning model. moreover, @xcite and @xcite conducted large-scale bench-marking on existing approaches for mgt detection across multiple domains, models, and languages and concluded that the roberta language model, especially the variants that have been optimized for ai detection tasks, consistently outperforms most other methods across evaluation metrics. ## system overview our system is based on a pre-classifier that is a roberta model fine-tuned for ai generated text detection. in order to correct the predictions of the pre-classifier, two correction classifiers have been trained that are based on linguistic, stylistic and probabilistic features. an overview of the system setup is given in figure 1 . our roberta pre-classifier is based on the roberta base openai detector 1 @xcite , a roberta model fine-tuned for ai generated text detection. this model has been further fine-tuned on 10% of the training data. a prediction was then generated for each text in the train, dev and test set. in order to improve the predictions, two correction multi-layer perceptron (mlp) classifiers, one for each label, were trained on the training and development data of their respective label (see figure 1 ) as well as on a range of features outlined in section 4.2.1. to generate the final classification, all texts were classified again by the correction mlp that corresponded to the label predicted by the roberta pre-classifier. this provided an opportunity for the more specialized classifier to correct the initial prediction. ## experimental setup the m4 dataset consists of both machine (label 1) and human-generated texts (label 0). the dataset features texts from six different llm generators 1 https://huggingface.co/openai-community/ roberta-base-openai-detector (davinci, chatgpt, dolly, cohere, bloomz and gpt4) and five different genres (reddit, wikihow, arxive, wikipedia, and peerread). participants were provided first with a train and dev set and later with a test set with 119,757, 5,000, and 34,272 texts in total, respectively. roughly 53% of the documents in the train set are machine generated (davinci: 14,343, chatgpt: 14,339, dolly: 14,046, cohere: 13,678), and 47% are human-written. in the dev set, exactly half of the texts were machine-generated by the bloomz model, the other half was human-written. the test set contains 18,000 (53%) machine-generated texts from davinci, chatgpt, dolly, cohere, bloomz and gpt4 (3,000 texts each) and 16,272 (47%) human-written texts. an overview of the data is provided in ## results although the pre-classifier performed fairly well on the dev data (accuracy: 0.89, for more details see a closer look at the recall and precision for the two classes 'human' and 'machine' reveals that the fusion model balances out the problems of the pre-classifier and the mlp. the high precision and low recall of the pre-classifier for 'human' and vice versa for 'machine' indicate that it is biased towards the 'machine' class. accordingly the relatively high precision and low recall of the mlp classifier for 'machine' and vice versa for 'human' indicate that it is biased towards the 'human' class. in contrast, the fusion model shows equally high precision and recall for both classes. as described in section 4.1, the pre-classifier is obtained from the roberta base openai detector by further training. comparing its performance to the original model (see ## conclusion and limitations overall, this study has highlighted the benefit of using a fusion architecture consisting of a preclassifier and linguistically informed correctional classifiers. by adding syntactic, stylistic, sentiment, frequency-and word difficulty-based features, we were able to improve the performance of a finetuned pre-trained roberta model for ai generated text detection and adjust the bias towards the machine label. because our fusion model uses a pre-trained roberta model, all computations for this paper can be run locally or, in the case of the roberta fine-tuning, using a free google colab account. this means that our model can be easily expanded and leaves a smaller environmental footprint. future studies could expand our fusion model by incorporating more semantic-level or complex features such as contextual predictability, as well as fine-tuning the pre-classifier using more, balanced data. our code is available on github