G en AI Content Detection Task 1: E nglish and Multilingual Machine-Generated Text Detection: AI vs. Human (2025)

Abstract: We present the GenAI Content Detection Task 1 – a shared task on binary machine generated text detection, conducted as a part of the GenAI workshop at COLING 2025. The task consists of two subtasks: Monolingual (English) and Multilingual. The shared task attracted many participants: 36 teams made official submissions to the Monolingual subtask during the test phase and 27 teams – to the Multilingual. We provide a comprehensive overview of the data, a summary of the results – including system rankings and performance scores – detailed descriptions of the participating systems, and an in-depth analysis of submissions.

## introduction the success and popularity of large language models (llms) have led to the proliferation of generative artificial intelligence (genai) content, which is now widely applied across numerous aspects of daily life. however, this widespread adoption has brought several concerns to light, including challenges to the integrity of student assignments and the potential for fabricated content to mislead individuals @xcite . as generative llms continue to advance rapidly, it is becoming increasingly difficult for humans to distinguish machine-generated content from authentic humanauthored text. consequently, developing effective methods to address these challenges is crucial. to this end, we propose a genai content detection task, with task 1 focusing specifically on the detection of machine-generated text in both english and 1 https://github.com/mbzuai-nlp/ coling-2025-workshop-on-mgt-detection-task1 multilingual contexts. this task is the continuation of semeval shared task 8 @xcite . the new task introduces a broader range of languages and domains while incorporating updated generators that leverage the latest llms. the task consists of two subtasks: monolingual (english) subtask a and multilingual subtask b. the data for the shared task covers various domains and llm generators. the data for english subtask covers diverse domains, including peer reviews, student essays, scientific papers, news articles, social media, emails, speech content and so on, similar for multilingual subtask data, with the test set involving more than 8 domains. to construct the data for the shared task, we produced machine-generated texts (mgts), using state-ofthe-art llms, including @xcite , llama-3.1 @xcite , vikhr-nemo @xcite , qwen-2 @xcite , etc. multilingual subtask data encompasses 21 unique languages. the task attracted 36 participants who made official submissions during the test phase for the monolingual subtask a and 27 participants who made official submissions to the multilingual subtask b. ## related work this section discusses prior work about machinegenerated text detection methods, datasets and shared tasks. ## participants' submissions in this section, we first describe ranking, macro-f1 and accuracy of participants, followed by a brief description of all submitted systems. we classify sets -arabic, bulgarian, catalan, chinese, czech, dutch, english, german, indonesian, italian, portuguese, russian, spanish, ukrainian, and urdu. to describe systems participating in the english and multilingual subtasks separately, in the text we add the subscript english:rank to participants in the english subtask and the subscript multi:rank to participants in the multilingual subtask. for example the team fraunhofer sit is ranked 3rd in the english subtask and referred to as fraunhofer sit english:3 while it is ranked 10th in the multilingual subtask and thus referred to as fraunhofer sit multi:10 . ## analysis based on the test set, we analyze submitted systems by comparing the detection accuracy on (1) in-domain vs. out-of-domain, (2) seen vs. unseen languages, and (3) generations produced using normal prompts vs. prompts attempting to fill the gap between human and machine based on observations in manual annotations. ## conclusion in this work, we presented the dataset, baseline, participating systems and a detailed analysis across various detection methods for genai shared task 1: binary machine generated text detection. we explored both english and multilingual settings with diverse domains, llm generators, and languages. all submitted systems show good performance on domains and languages that are seen during training, while witness the significant declines on unseen domains and languages. moreover, detectors show remarkable vulnerability when machinegenerated text is adapted to mimic humans, either by introducing typo, link, and tags, or by using fill-human-machine gap prompts. we expect our task can attract more researchers to develop robust and generalized detection models, and our analysis insights can provide a direction for future work, advancing research in machine-generated content detection.