L3i++ at S em E val-2024 Task 8: Can Fine-tuned Large Language Model Detect Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text? (2024)

Abstract: This paper summarizes our participation in SemEval-2024 Task 8: Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection. In this task, we aim to solve two over three Subtasks: (1) Monolingual and Multilingual Binary Human-Written vs. Machine-Generated Text Classification; and (2) Multi-Way Machine-Generated Text Classification. We conducted a comprehensive comparative study across three methodological groups: Five metric-based models (Log-Likelihood, Rank, Log-Rank, Entropy, and MFDMetric), two fine-tuned sequence-labeling language models (RoBERTA and XLM-R); and a fine-tuned large-scale language model (LS-LLaMA). Our findings suggest that our LLM outperformed both traditional sequence-labeling LM benchmarks and metric-based approaches. Furthermore, our fine-tuned classifier excelled in detecting machine-generated multilingual texts and accurately classifying machine-generated texts within a specific category, (e.g., ChatGPT, bloomz, dolly). However, they do exhibit challenges in detecting them in other categories (e.g., cohere, and davinci). This is due to potential overlap in the distribution of the metric among various LLMs. Overall, we achieved a 6th rank in both Multilingual Binary Human-Written vs. Machine-Generated Text Classification and Multi-Way Machine-Generated Text Classification on the leaderboard.

## introduction the rise of large language models (llms) has led to a significant step forward in producing remarkably controllable, fluent, and grammatical text, triggering a surge in machine-generated content across diverse platforms such as news, social media, question-answering forums, educational, and even academic contexts. notably, recent llms like chatgpt 1 and gpt-4 (openai, 2023) exhibit a remarkable ability to generate coherent and contextually appropriate responses to a wide array of user queries. unfortunately, use and abuse come hand in hand. although the fluency of these generated texts positions llms as potential candidates for replacing human labor in numerous applications, this has 1 https://chat.openai.com/ also raised concerns about their potential for misuse, particularly in spreading misinformation and causing disruptions within the education system. given that humans struggle to distinguish between machine-generated and human-written text, it becomes imperative to develop automated systems capable of identifying machine-generated text to curb the risks associated with its misuse. in this paper, as the participants in semeval-2024 task 8: multigenerator, multidomain, and multilingual black-box machine-generated text detection @xcite , we investigate the feasibility of training a classifier that can reliably differentiate between text generated by humans and text that appears human-like but is generated by machines in two paradigms: to address these problems, we explore the performance of diverse methodologies, which can be divided into three categories, including: ## related work the success of llms in various downstream nlp tasks @xcite @xcite leads to the overuse and abuse of the information generated by llms. however, it is essential to acknowledge that the outputs generated by llms are not always accurate, giving rise to the issue of hallucination @xcite . consequently, there is a need for clear differentiation in addressing this concern. to address these issues, researchers have developed several automatic detection methods @xcite @xcite that can identify the machine-generated text from the human-written text, which initially can be divided into two categories, i.e., metric-based methods and model-based methods. ## data we work on two datasets provided by semeval-2024 task 8: multigenerator, multidomain, and multilingual black-box machine-generated text detection @xcite , whose statistics covering the number of examples for each source and each label are presented in tables 1 and 2 for both versions of subtask a, data were collected from diverse sources, leading to label imbalances. for example, in the monolingual subtask a training set, there is a notable scarcity of samples from peerread compared to the other sources. conversely, in subtask b, the dataset is balanced. ## methodology this section tackles the problem by formulating it as supervised classification tasks. we then introduce our proposed solution architecture for each task, covering the models used, and present how we fine-tuned them with hyperparameter configurations, and how we assessed their performance. ## conclusions in conclusion, this paper outlines our contribution to the first two subtasks of semeval-2024 task 8: multigenerator, multidomain, and multilingual black-box machine-generated text detection, namely monolingual and multilingual binary human-written vs. machine-generated text classification and multi-way machine-generated text classification. we conducted a comprehensive comparative study across three methodological groups: five metric-based models (log-likelihood, rank, log-rank, entropy, and mfd-metric), two fine-tuned sequence-labeling language models (roberta and xlm-r); and a fine-tuned large-scale language model (ls-llama). our findings suggest that our llm outperformed both traditional sequence-labeling lm benchmarks and metric-based approaches. furthermore, our fine-tuned classifier excelled in detecting machinegenerated multilingual texts and accurately classifying machine-generated texts within a specific category, (e.g., chatgpt, bloomz, dolly). however, they do exhibit challenges in detecting them in other categories (e.g., cohere, and davinci). this is due to potential overlap in the distribution of the metric among various llms. overall, we ranked 6 th in both multilingual binary human-written vs. machine-generated text classification and multi-way machine-generated text classification on the leaderboard. in future work, we would like to take a step further to evaluate whether our classifier is robust enough against adversarial attacks (e.g., paraphrasing, random spacing, adversarial perturbation) as well as investigate how to make our model more interpretable and explainable, which is important, but insufficiently addressed when detecting machinegenerated contents.