M ason T igers at S em E val-2024 Task 8: Performance Analysis of Transformer-based Models on Machine-Generated Text Detection (2024)

Abstract: This paper presents the MasonTigers entryto the SemEval-2024 Task 8 - Multigenerator, Multidomain, and Multilingual BlackBox Machine-Generated Text Detection. Thetask encompasses Binary Human-Written vs.Machine-Generated Text Classification (TrackA), Multi-Way Machine-Generated Text Classification (Track B), and Human-Machine MixedText Detection (Track C). Our best performing approaches utilize mainly the ensemble ofdiscriminator transformer models along withsentence transformer and statistical machinelearning approaches in specific cases. Moreover, Zero shot prompting and fine-tuning ofFLAN-T5 are used for Track A and B.

## introduction in academia and beyond, machine-generated content is proliferating across news platforms, social media, forums, educational materials, and scholarly works. breakthroughs in large language models (llms), like gpt-3.5 and gpt-4, facilitate the creation of fluent responses to diverse user queries. while this capability raises prospects of replacing human labor in various tasks, concerns arise about potential misuse, including the generation of deceptive misinformation @xcite and completing student assignments, which hinders the development of essential writing skills @xcite .this highlights the importance of developing automated systems to detect and mitigate the potential abuse of machine-generated content, as well as distinguishing between machine-written and human-generated text. additionally, prior studies (zerogpt 1 ; @xcite ) predominantly adopted a binary classification approach for machine-generated text (mgt), with a primary focus on english. however, there has been limited research addressing the amalgamation of human-written and mgt texts @xcite . in response to these limitations, semeval-2024 introduces a shared task: multigenerator, multidomain, and multilingual black-box machine-generated text detection @xcite . this task comprises three subtasks, each targeting different aspects of machine-generated text complexity.subtask a focuses on binary human-written vs. mgt classification, involving two tracks: monolingual and multilingual. subtask b tackles multi-way machine-generated text classification to identify the source of a given text. subtask c involves detecting the transition point within a mixed text, determining where it shifts from human-written to machine-generated. the data provided for this task is an expansion of the m4 dataset @xcite and benchmark evaluation of @xcite . . in conducting these tasks, we conduct a range of experiments and observe that ensemble methods outperform individual models significantly in classification tasks, e.g., @xcite , @xcite . our weighted ensemble approaches achieve accuracies of 74%, 60% and 65% in subtask a monolingual; multilingual tracks and subtask b respectively, given that we have used different models for both tasks. in subtask c, we explore different setups, ensembling which results in mean absolute error (mae) of 60.78. for the classifications, we utilize zero-shot prompting and fine-tuning of flant5 foot_0 , while adhering to the restriction of no data augmentation in this task. ## related works the difficulties of separating human-written text from large language models and the significance of previous work in determining mgt from humanwritten ones include higher order n-grams @xcite , utilizing linguistic patterns @xcite , curvature-based criterion @xcite , tweaking with multiple variables @xcite , fine-tuning transformerbased models e.g., @xcite . very recently, @xcite puts forward llm-detector, offering a fresh method for identifying text at both document and sentence levels by employing instruction tuning of llms. to tackle challenges of this field, several datasets have been released, e.g., multitude @xcite , m4 @xcite . additionally, there have been multiple shared tasks organized related to this topic @xcite @xcite . despite several collective findings and techniques, as argued by @xcite , there remains a critical need for the creation of reliable detection methods capable of accurately distinguishing between human and machine-generated text, a requirement essential across both english and other languages. wang et al. (2024d) collects datasets from a variety of sources, including wikipedia (the march 2022 version), wikihow (koupaee and wang, 2018), reddit (eli5), arxiv, peerread @xcite (for english), and baike (for chinese). they employ web question answering for chinese, news content for urdu, indonesian, and ruatd @xcite for russian language. ## datasets the method of prompting machine-generated text (mgt) has been extensively outlined in @xcite . subtask a, binary human-written vs. machine-generated text classification, in the monolingual track involves a same-domain cross-generator experiment, where instances are exclusively in english and gathered from five distinct sources with two labels: 0 and 1. human-generated texts receive a label of 0, while machine-generated texts from four different llms (chatgpt, cohere, davinci-003, and dolly-v2) are labeled as 1. the distribution of train and validation datasets, both in terms of labels and sources, along with the number of test instances, is detailed in tables 1. during the test phase, there are 16,272 instances labeled as 0 and 18,000 instances labeled as 1. on the other hand, subtask a in the multilingual track entails a cross-domain same-generator experiment. instances are sourced from nine different sources during the training phase, including four different languages, while the validation dataset comprises three different languages as indicated in subtask b, multi-way machine-generated text classification, represents another cross-domain same-generator experiment. in contrast to subtask a, subtask c involves six labels: 0 for human, 1 for chatgpt, 2 for cohere, 3 for davinci-003, 4 for bloomz, and 5 for dolly. these labels correspond to instances sourced from five different sources. however, it's noteworthy that the sources for the training and validation data differ, and this distinction is outlined in tables 3 and 4 . ## results subtask a and b are evaluated based on accuracy, as specified by @xcite , while subtask c employs mean absolute error (mae) as the evaluation metric 4 . ## error analysis in the monolingual track of subtask a, the final model demonstrates proficiency in accurately identifying machine-generated text. nonetheless, there is a notable presence of false positives, indicating instances where the model incorrectly identifies human-written texts as machine-generated. despite this, the model effectively detects machinegenerated text without omission. similarly, in the multilingual track of subtask a, the ultimate model excels in accurately distinguishing machinegenerated text. however, false positives are prevalent, indicating numerous cases where humanwritten texts are inaccurately classified as machinegenerated. additionally, the model encounters instances where it fails to predict machine-generated texts. in subtask b, the model excels in accurately predicting chatgpt-generated texts. however, its performance declines notably for davinci-generated text, often misclassifying it as chatgpt generated. additionally, the model's accuracy is lower for dolly-generated and human-written texts, indicating a discrepancy in handling machine-generated versus human-written content. for subtask c, mae is higher due to the presence of outliers because the dev mae was significantly lower than the test mae. to handle this issue, it is essential to address the preprocessing of data, handling outliers, selecting appropriate features, optimizing model complexity, improving data quality, and ensuring model stability through proper tuning and evaluation procedures. this can be the future scope of research in this specific domain. for a clearer understanding, refer to the visual evaluations in figure 2 , 3, 4 of appendix. ## conclusion in our investigation of semeval-2024 task 8, we applied a diverse set of methodologies, encompassing statistical machine learning techniques, transformer-based models, sentence transformers, and flan t5. subtask a involved binary classification, where the monolingual track focused on cross-generator scenarios within the same domain, and the multilingual track addressed cross-domain scenarios within the same generators. subtask b dealt with multi-label classification, requiring the discrimination of human-generated text from five distinct language models. subtask c centered on human-machine mixed text detection, employing tf-idf, ppmi, and roberta with linear regression and elasticnet for prediction. the outcomes of three subtasks highlighted the efficacy of ensemble methods, showcasing specific models excelling in each subtask. additionally, we explored the applicability of zero-shot prompting and fine-tuning flan-t5 for tracks a and b. in summary, our approach harnessed a blend of transformer models, machine learning methodologies, and ensemble strategies to tackle the complexities presented by semeval-2024 task 8. the paper underscores the imperative need for robust detection methods to effectively navigate the growing prevalence of machine-generated content.