Fine-tuning Language Models for AI vs Human Generated Text detection (2024)

Abstract: In this paper, we introduce a machine-generated text detection system designed totackle the challenges posed by the prolifera-tion of large language models (LLMs). Withthe rise of LLMs such as ChatGPT and GPT-4,there is a growing concern regarding the po-tential misuse of machine-generated content,including misinformation dissemination. Oursystem addresses this issue by automating theidentification of machine-generated text acrossmultiple subtasks: binary human-written vs.machine-generated text classification, multi-way machine-generated text classification, andhuman-machine mixed text detection. We em-ploy the RoBERTa Base model and fine-tuneit on a diverse dataset encompassing variousdomains, languages, and sources. Throughrigorous evaluation, we demonstrate the effec-tiveness of our system in accurately detectingmachine-generated text, contributing to effortsaimed at mitigating its potential misuse.

## introduction large language models (llms) are becoming mainstream and easily accessible, bringing in an explosion of machine-generated content over various channels, such as news, social media, questionanswering forums, educational, and even academic contexts. recent llms, such as chatgpt and gpt-4, generate remarkably fluent responses to a wide variety of user queries. the articulate nature of such generated texts makes llms attractive for replacing human labor in many scenarios. however, this has also resulted in concerns regarding their potential misuse, such as spreading misinformation and causing disruptions in the education system. since humans perform only slightly better than chance when classifying machine-generated vs. human-written text, there is a need to develop automatic systems to identify machine-generated text with the goal of mitigating its potential misuse. the advent of sophisticated large language models (llms), including chatgpt and gpt-4, has catalyzed a surge in artificially generated text across myriad domains, from news media to social platforms, educational resources, and scholarly publications. these neural network models exhibit an unprecedented capacity to produce natural language, enabling the automation of written content creation. however, the human-like fluency of llm outputs has concurrently raised serious concerns surrounding potential misuse. with syntactically coherent and topically relevant text, llms could plausibly disseminate misinformation, plagiarize or falsify documents, and automate persuasion-based attacks on a massive scale. the integration of models like chatgpt into education has additionally ignited fierce debate; while proponents highlight opportunities for personalized instruction, critics argue llms enable academic dishonesty and undermine human knowledge acquisition. amidst this controversy, institutions urgently seek policies to uphold academic integrity. alarmingly, humans perform only marginally better than random chance at distinguishing machine-generated versus human-authored text. developing reliable technical systems to automatically detect ai content has therefore become a research priority. the goal is to provide educators, moderators, and end users tools to identify llm outputs, thereby mitigating potential dangers from increasingly accessible, human-like models. constructing robust llm detectors demands interdisciplinary collaboration, combining machine learning advances with insights from fields like ethics, media studies, and education. with judicious coordination across stakeholders, experts aim to actualize benefits of llms for automation while curtailing risks of misinformation, deception, and cheating. ## dataset the dataset provided by the organizers of this shared task @xcite comprises a diverse collection of texts encompassing various domains, languages, and sources. the dataset is structured to address the three subtasks outlined: binary human-written vs. machine-generated text classification (subtask a), multi-way machine-generated text classification (subtask b), and human-machine mixed text detection (subtask c). for subtask a (binary classification), the dataset consists of a balanced corpus of humanwritten and machine-generated texts. the humanwritten texts are sourced from various publications, academic papers, forums, and social media platforms. the machine-generated texts are generated by state-of-the-art language models such as chat-gpt, gpt-4, cohere, davinci, bloomz, and dolly. these texts cover a wide range of topics to ensure diversity and representativeness. for subtask b (multi-way classification), the dataset includes texts generated by each of the six specified language models: chatgpt, cohere, davinci, bloomz, dolly, and human-written texts. the texts are annotated to indicate their respective sources, enabling the classification task to determine the origin of each text accurately. for subtask c (human-machine mixed text detection), the dataset contains texts where the first part is human-written, and the subsequent part is machine-generated. annotations demarcate the boundary where the transition from human to machine-generated text occurs. this allows for training and evaluating models on detecting the boundary between human and machine-generated segments within a single text. the dataset is preprocessed to remove noise, standardize formatting, and ensure consistency across texts. ## methods we employed a fine-tuning approach using the xlm-roberta base model @xcite for the task of machine-generated text detection. we chose roberta-base as our base model for fine tuning as xlm-roberta is a multilingual language model optimized for classification tasks. it is pretrained on massive multilingual data, and has a robust architecture and performance enable efficient fine-tuning across diverse text classification problems with state-of-the-art accuracy the fine-tuning process involves initializing the roberta base model with pre-trained weights and then fine-tuning it on our specific dataset for the tasks of binary human-written vs. machinegenerated text classification (subtask a), multiway machine-generated text classification (subtask b), and human-machine mixed text detection (subtask c). during fine-tuning, we optimize the model's parameters using stochastic gradient descent (sgd) with backpropagation. we employ task-specific loss functions, such as cross-entropy loss for classification tasks and mean squared error (mse) for mixed text detection. additionally, we utilize techniques such as dropout regularization to prevent overfitting and gradient clipping to stabilize training. the roberta base model is fine-tuned separately for each subtask, with hyperparameters tuned using grid search or random search techniques. we split the dataset into training, validation, and test sets to facilitate model training and evaluation, ensuring that the model generalizes well to unseen data. ## results we present the performance metrics achieved by our machine-generated text detection system on each of the subtasks: binary human-written vs. machine-generated text classification (subtask a), multi-way machine-generated text classification (subtask b), and human-machine mixed text detection (subtask c). the evaluation metrics include f1 score (macro and micro) and accuracy. ## conclusion in this study, we proposed a machine-generated text detection system capable of addressing three subtasks: binary human-written vs. machinegenerated text classification (subtask a), multiway machine-generated text classification (subtask b), and human-machine mixed text detection (subtask c). our system leverages the roberta base model, fine-tuned on a diverse dataset comprising texts from various domains, languages, and sources. through extensive experimentation and evaluation, we achieved promising results @xcite across all subtasks. for subtask a, our system demonstrated robust performance in distinguishing between humanwritten and machine-generated texts, achieving high f1 scores and accuracy across multiple epochs. similarly, in subtask b, where the classification involves identifying the source language model among multiple candidates, our system achieved competitive performance, indicating its effectiveness in multi-way classification scenarios. in subtask c, where the objective is to detect boundaries between human-written and machinegenerated segments within a single text, our system showed reasonable performance, albeit with some room for improvement. future work could focus on refining the model architecture and exploring additional features to enhance the system's performance in this challenging task. overall, our study highlights the importance and feasibility of developing automatic systems for detecting machine-generated text, contributing to efforts aimed at mitigating the potential misuse of large language models in various contexts.