S em E val-2024 Task 8: Multidomain, Multimodel and Multilingual Machine-Generated Text Detection (2024)

Abstract: We present the results and the main findings of SemEval-2024 Task 8: Multigenerator, Multidomain, and Multilingual Machine-Generated Text Detection. The task featured three subtasks. Subtask A is a binary classification task determining whether a text is written by a human or generated by a machine. This subtask has two tracks: a monolingual track focused solely on English texts and a multilingual track. Subtask B is to detect the exact source of a text, discerning whether it is written by a human or generated by a specific LLM. Subtask C aims to identify the changing point within a text, at which the authorship transitions from human to machine. The task attracted a large number of participants: subtask A monolingual (126), subtask A multilingual (59), subtask B (70), and subtask C (30). In this paper, we present the task, analyze the results, and discuss the system submissions and the methods they used. For all subtasks, the best systems used LLMs.

## introduction the proliferation of large language models (llms) has led to a significant increase in the volume of machine-generated text (mgt) across a wide range of domains. this rise has sparked concerns regarding the potential for misuse in fields such as journalism, education, academia, etc @xcite . moreover, it poses challenges to maintaining information integrity and ensuring accurate information dissemination. as such, the ability to accurately distinguish between human-written content and machinegenerated content has become paramount for identifying potential misuse @xcite @xcite . in response to these challenges, we are introducing a shared task that focuses on the detection of machine-generated text across multiple generators, domains, and languages. we are providing largescale evaluation datasets for three subtasks with the primary goals of fostering extensive research in mgt detection, advancing the development of automated systems for detecting mgt, and reducing instances of misuse: subtask a: human vs. machine classification. the goal of this subtask is to accurately classify a text as either produced by a human or generated by a machine. this is the basic, but one of the most common use-cases of mgt detection systems for preventing the misuse of llms. this task is divided into two tracks: (i) the monolingual track, which focuses solely on english texts; and (ii) the multilingual track, which involves texts in a variety of languages, thereby expanding the diversity and complexity beyond existing benchmarks. ## background detecting machine-generated text is primarily formulated as a binary classification task @xcite @xcite , naively distinguishing between human-written and machine-generated text. in general, there are two main approaches: the supervised methods @xcite @xcite @xcite and the unsupervised ones, such as zero-shot methods @xcite @xcite @xcite . while supervised approaches yield relatively better results, they are susceptible to overfitting @xcite . meanwhile, unsupervised methods may require unrealistic white-box access to the generator. in the following, we provide background information on each subtask, respectively. ## dataset and metrics in this section, we describe the datasets and evaluation metrics for all subtask tracks, including the size, domains, generators, and language distribution across training, development, and test splits. ## participating systems in this section, we first summarize common features for all teams based on the information they provided in the google docs. then, we delve into team name ranking small plm llm gpt fine-tuning zero-shot few-shot (@xmath0 the methods employed by the top 3 teams, accompanied by brief descriptions of the approaches utilized by the other top 10 teams. the approaches of all teams are presented in appendix a. ## conclusion and future work we have described semeval-2024 task 8 on multigenerator, multidomain, and multilingual machine-generated text detection. the task garnered significant interest from researchers, with 126, 59, 70, and 30 teams submitting entries for subtask a monolingual, subtask a multilingual, subtask b, and subtask c, respectively. additionally, we received 54 system description papers before finalizing this submission. overall, subtasks a and b were relatively easier, with all systems showing improvements over the baseline. however, subtask c proved to be significantly more challenging. fewer teams participated, and many struggled to surpass our baseline results set in @xcite . in future work, we plan to extend our focus beyond machine-generated text detection to other modalities such as image, speech, and video detection. additionally, we intend to develop an opensource demonstration system capable of distinguishing between ai-generated content and humanproduced content.