NCL - U o R at S em E val-2024 Task 8: Fine-tuning Large Language Models for Multigenerator, Multidomain, and Multilingual Machine-Generated Text Detection (2024)

Abstract: SemEval-2024 Task 8 introduces the challenge of identifying machine-generated texts from diverse Large Language Models (LLMs) in various languages and domains. The task comprises three subtasks: binary classification in monolingual and multilingual (Subtask A), multi-class classification (Subtask B), and mixed text detection (Subtask C). This paper focuses on Subtask A & B. To tackle this task, this paper proposes two methods: 1) using traditional machine learning (ML) with natural language preprocessing (NLP) for feature extraction, and 2) fine-tuning LLMs for text classification. For fine-tuning, we use the train datasets provided by the task organizers. The results show that transformer models like LoRA-RoBERTa and XLM-RoBERTa outperform traditional ML models, particularly in multilingual subtasks. However, traditional ML models performed better than transformer models for the monolingual task, demonstrating the importance of considering the specific characteristics of each subtask when selecting an appropriate approach.

## introduction large language models (llms) are sophisticated natural language processing (nlp) models extensively trained on vast textual datasets @xcite . these models demonstrate an impressive proficiency in generating human-like text based on the input they receive. however, using llms for generating texts has raised concerns about potential misuse, such as disseminating misinformation and disruptions in the education system @xcite . thus, urgent development of automated systems to detect machine-generated texts is essential @xcite . recently, several llms have been developed such as chatgpt 1 @xcite , cohere foot_1 , davinci foot_2 , bloomz 4 @xcite , and dolly foot_4 @xcite . the versatility of these models extends across various domains, such as news, social media, educational platforms, and academic contexts, in multiple languages not only english @xcite . this wide application poses a challenge in developing an automated system capable of detecting machinegenerated texts from various generators, across multiple domains and languages. to tackle this challenge, semeval-2024 task 8: multigenerator, multidomain, and multilingual black-box machine-generated text detection @xcite introduces the task of detecting machine-generated texts obtained from different llms, in various domains and languages. this task consists of three subtasks: subtasks a, b, and c. subtask a involves binary classification of text as either human-written or machine-generated, with two tracks: monolingual (english only) and multilingual. subtask b focuses on multi-class classification of machine-generated text, aiming to identify the source of generation, whether human or a specific language model. subtask c addresses the detection of human-machine mixed text, requiring the determination of the boundary where the transition from human-written to machine-generated occurs in a mixed text. this paper focuses on subtasks a and b. to tackle these tasks, we propose two approaches: (1) classical machine learning, leveraging nlp techniques for feature extraction, and (2) fine-tuning llms for the classification of human-written and machine-generated texts. ## related work researchers have employed a variety of methods and tools to detect ai-generated texts. broadly, these approaches can be categorized into two main types: black-box and white-box detection methods @xcite . black-box detection relies on api-level access to llms, utilizing textual samples from both human and machine sources to train classification models @xcite . the study by @xcite integrated existing questionand-answer datasets and leveraged fine-tuning of pre-trained models to investigate the characteristics and similarities between human-generated and ai-generated texts. as for white-box detection, @xcite introduced a novel approach involving the embedding of watermarks in the outputs of llms to facilitate the detection of ai-generated text. additionally, a variety of tools and methodologies, including xgboost, decision trees, and transformerbased models, have been evaluated for their efficacy in detecting texts produced by ai @xcite . these techniques incorporate multiple stylistic measurement features to differentiate between ai-generated and human-generated texts @xcite . specific tools and techniques in this domain include the gltr tool developed by @xcite , which analyzes the usage of rare words in texts to distinguish between those generated by the gpt-2 model and human writers. the detectgpt method posits that minor rewrites of llm-generated texts tend to reduce the log probability under the model, a hypothesis that has been explored in depth @xcite . furthermore, intrinsic dimension analysis, including methods like the persistent homology dimension estimator (phd), has been applied to distinguish between authentic texts and those generated artificially @xcite . detectors specifically designed for certain llms, such as the grover detector for the grover model @xcite and the roberta detector using the roberta model @xcite , also play a significant role in this field. in summary, the combination of statistical analysis with advanced language models is being employed by researchers to more effectively differentiate between content generated by humans and machines. the continuous evolution and refinement of these techniques reflect the dynamic nature of the field and the complexities involved in distinguishing between the increasingly nuanced outputs of llms and human-authored texts. ## methods to tackle these tasks, we employ two distinct strategies. the first is classical machine learning, tailored for natural language preprocessing (nlp). the second approach involves transformer-based llms, with an emphasis on lora (low-rank adaptation of large language models) fine-tuning @xcite . we then enhance our results by integrating these methods through ensemble techniques. ## https://huggingface.co/distilbert-base-multilingualcased 4 experiments in our study, subtask a focuses on distinguishing between human-written (label 0) and machinegenerated text (label 1), offered in both monolingual (119,757 train, 5,000 dev, 34,272 test) and multilingual versions (172,417 train, 4,000 dev, 42,378 test), across various sources and languages are given in ## conclusions the results showed that transformer models, particularly lora-roberta and xlm-roberta, performed exceptionally well in most text classification tasks. distilmbert represented a more streamlined transformer approach and was also proven to be efficient, especially in multilingual task. contrary to popular belief, traditional ml models such as multinomialnb and xgboost can outperform transformers in monolingual tasks. these findings highlight the importance of carefully considering the characteristics of the task and the trade-offs between model complexity and performance when selecting an appropriate approach. our results contribute to the understanding of model selection strategies for text classification and emphasize the need for a nuanced approach that takes into account the specific demands of each subtask. future research could explore the development of hybrid models that combine the strengths of traditional ml techniques and transformer architectures, as well as the design of more efficient and lightweight transformer models for resourceconstrained environments. these findings reflected the dynamic nature of nlp tools and the importance of selecting models based on the specific requirements of the task.