Sharif- MGTD at S em E val-2024 Task 8: A Transformer-Based Approach to Detect Machine Generated Text (2024)

Abstract: In this paper, we delve into the realm of detecting machine-generated text (MGT) within Natural Language Processing (NLP). Our approach involves fine-tuning a RoBERTa-base Transformer, a robust neural architecture, to tackle MGT detection as a binary classification task. Specifically focusing on Subtask A (Monolingual - English) within the SemEval-2024 competition framework, our system achieves a 78.9% accuracy on the test dataset, placing us 57th among participants. While our system demonstrates proficiency in identifying human-written texts, it faces challenges in accurately discerning MGTs.

## introduction recent advancements in large language models (llms) have endowed them with an impressive capability to generate written text that closely resembles human writing @xcite . however, this technological progress brings along significant challenges, as the proliferation mgt poses various threats in digital environments. mgts have been implicated in spreading misinformation in online reviews, eroding public trust in political or commercial campaigns, and even facilitating academic fraud (crothers et al., 2022; song et al., 2015; tang 1 https://semeval.github.io/semeval2024/ et al., 2023). the identification of mgt remains a pressing concern, as distinguishing between human-written and machine-generated content is often challenging for humans. consequently, there is a growing imperative to develop automatic systems capable of discerning mgt @xcite . in this study, we address this challenge within the english language context using the dataset provided by @xcite . as highlighted in @xcite overview paper on the task, recent approaches to mgt detection predominantly employ binary classification methods. existing literature highlights the superior performance of transformerbased methods over alternative approaches @xcite . however, a significant challenge in utilizing these models lies in the requirement for gpu hardware and computational resources. our study aims to address this challenge within the constraints of limited hardware capacity. keeping this in mind, we propose a system that leverages fine-tuning of the roberta transformer model @xcite to automatically classify input text as either human-written or machine-generated. our system architecture involves augmenting the roberta-base model with a classifier head. the embeddings component facilitates contextual understanding of texts, while the encoder component processes input texts in parallel, and the classifier head performs binary classification by linearly outputting a single value. our proposed system achieves an accuracy of 78.9% on the test data, surpassing the average results provided by the task's baseline and ranking 57th among 140 participants. the area under the roc curve (auc) metric is measured at 0.69. while the roc curve analysis demonstrates our model's capability to classify substantial portions of positive cases, its proximity to the diagonal line indicates room for further improvement. notably, our primary challenge stemmed from computational constraints, which limited our ability to implement larger token sizes or batch sizes. further discussions reveal that our system encounters difficulties in accurately detecting mgts. to facilitate reproducibility and further research in this area, the code for our system is available on github foot_0 . ## system overview this section presents an overview of our system's architecture, highlighting implementation details and challenges. drawing on the preceding works discussed above, which showed the efficacy of fine-tuning roberta models, our system aims to attain peak performance in mgt detection while optimizing configurations for limited hardware resources. the decision to employ the transformer architecture for detecting synthetic texts is motivated by its capacity to capture intricate dependencies within textual data. this choice seems logical considering that such texts often exhibit semantic features that can be harnessed for fact-checking, cohesion, coherence, and other properties that may unveil their origin @xcite . in contrast to traditional architectures, the transformer model overcomes the constraints of fixed window sizes or sequential processing, enabling it to utilize contextual information from the entire input sequence. additionally, the self-attention mechanism empowers the model to selectively focus on pertinent segments of the input, rendering it highly effective for tasks necessitating long-range dependencies and contextual comprehension. as for roberta, it is specifically chosen for its extensive training duration, broader dataset coverage, ability to handle longer sequences, and focus on natural language understanding tasks, making it more suitable than other bert-based models. additionally, a wealth of research, such as the recent study of @xcite , has further highlighted the inherent potential of roberta for this specific task. ## results using the official accuracy metric of semeval-2024 task 8 @xcite , our system achieved the following accuracy scores on different data splits: a direct comparison of our results with prior works is challenging due to the unique nature of our research. to the best of researchers' knowledge, the most comprehensive benchmark on supervised mgt detection is presented by @xcite using the m4 dataset and employing roberta, xlm-r, gltr-lr, gltrsvm, stylistic-svm, and nela-svm. however, our primary objective was to determine strategies for addressing limited hardware resources as discussed in appendix a. as a contribution to this field, through repeated experiments, we identified that among hyperparameters, token size plays a slightly more significant role in model accuracy. while the system's accuracy is influenced by increasing the token size, drawing meaningful scientific conclusions necessitates further controlled experiments. additionally, the expansion of token size is restricted by hardware limitations, requiring a detailed investigation with robust computational resources like gpu or tpu. considering the constraints of google colab's 3 free runtimes, we opted for a token size of 512 as a balance between hardware limitations and time constraints. consequently, based on the official accuracy metric of semeval2024 task 8 @xcite , our system achieved the following accuracy scores on various data splits: the evaluation of our model also included analysis of the area under the curve (auc), a crucial metric that reflects the discriminative power of a binary classification model. our fine-tuned roberta model demonstrated an auc of 0.69, suggesting its ability to effectively distinguish between positive and negative instances. figure 1 illustrates the receiver operating characteristic (roc) curve, depicting the model's capability to accurately classify a significant proportion of positive cases. however, the proximity of the curve to the diagonal line suggests opportunities for further enhancement. interestingly, analysis of the confusion matrix, as depicted in figure 2 , revealed notable patterns in our model's classification tendencies. while our system effectively identified human-written documents with low false positives, it exhibited difficulties in correctly identifying mgts. this observation suggests potential areas for refinement, particularly in enhancing the model's ability to detect subtle cues and characteristics unique to machine-generated content. overall, our study contributes to the ongoing efforts in the field of nlp by showcasing the effectiveness of fine-tuned transformer models, particularly roberta, in mgt detection tasks. moving forward, future research directions could explore novel approaches to mitigate computational costs and further improve the performance of mgt detection systems, ultimately advancing the capabilities of nlu models in real-world applications. ## conclusion in summary, our study focused on fine-tuning a roberta-base transformer model for binary classification, specifically in distinguishing humanwritten from mgt. while our system showed promise in identifying human-written text, it faced challenges with accurately classifying machinegenerated content. as discussed in appendices a and b, we recommend exploring larger token sizes to improve model performance, albeit with awareness of computational costs. additionally, we advocate for the development of low-cost algorithms capable of efficient processing across hardware platforms. our findings contribute to advancing mgt detection, with implications for combating misinformation and enhancing cybersecurity in the digital age. a hyper-parameter tuning . we were unable to use premium runtime accounts due to financial issues arising from iran sanctions. therefore, we couldn't change our model's token size to larger than 512 due to the 12-hour time limit in free colab. to understand the impact of increasing token size, we aimed to experiment on a local laptop gpu. during the experiments aimed at finding the proper token size, we encountered the "cuda error: device-side assert triggered" frequently, which was resolved by restarting the session. our experiments were conducted using an rtx 2060 mobile with 6 gb of vram. throughout all experiments, we maintained fixed parameters, including number of @xmath0, train @xmath1.7, and learning @xmath2. increasing the max length from 512 to 1024 in this experimental setup resulted in an improvement in test accuracy by at least 2%. however, this enhancement came at the cost of a nearly 15-fold decrease in training speed, making it challenging to implement on limited hardware. additionally, this requires plenty of controlled experiments by researchers to shed light on finding the proper hyper-parameters.