G en AI Content Detection Task 2: AI vs. Human – Academic Essay Authenticity Challenge (2025)

Abstract: This paper presents a comprehensive overview of the first edition of the Academic Essay Authenticity Challenge, organized as part of the GenAI Content Detection shared tasks collocated with COLING 2025. This challenge focuses on detecting machine-generated vs human-authored essays for academic purposes. The task is defined as follows: “Given an essay, identify whether it is generated by a machine or authored by a human.” The challenge involves two languages: English and Arabic. During the evaluation phase, 25 teams submitted systems for English and 21 teams for Arabic, reflecting substantial interest in the task. Finally, five teams submitted system description papers. The majority of submissions utilized fine-tuned transformer-based models, with one team employing Large Language Models (LLMs) such as Llama 2 and Llama 3. This paper outlines the task formulation, details the dataset construction process, and explains the evaluation framework. Additionally, we present a summary of the approaches adopted by participating teams. Nearly all submitted systems outperformed the n-gram-based baseline, with the top-performing systems achieving F1 scores exceeding 0.98 for both languages, indicating significant progress in the detection of machine-generated text.

## introduction the rapid progress in artificial intelligence (ai) and the proliferation of generative content produced by llms have introduced transformative opportunities across various domains -yet they also pose profound challenges @xcite . one such challenge lies in the detection and prevention of misuse of llms in contexts such as fake news, misinformation, disinformation, and academic dishonesty @xcite . for instance, the volume of ai-generated news on misinformationprone websites surged by 457% between january 1, 2022, and may 1, 2023, with a corresponding increase of 57.3% on mainstream platforms @xcite . these issues pose substantial barriers to the broader adoption of llms, thereby limiting their potential across various applications. effectively detecting llm-generated content is crucial for leveraging the capabilities of these models while mitigating associated risks. researchers have responded to these challenges through a variety of approaches. previous methods include classification algorithms designed to distinguish between ai-generated and human-authored text @xcite , as well as watermarking techniques @xcite @xcite . these watermarking approaches strategically embed imperceptible signatures within generated texts, enabling modelspecific identification while maintaining humanindistinguishable quality. other recent efforts have focused on the creation of question-answering datasets such as m4 @xcite , generated by humans and chatgpt in both english and chinese and the associated shared task @xcite . within academic settings, concerns surrounding the potential misuse of llms have intensified, particularly regarding academic dishonesty involving ai-assisted essay writing and problem-solving. recent research has made considerable progress in the development of datasets and benchmarking efforts to address these issues. for instance, @xcite introduced the cheat dataset, which focuses on abstracts from ieee xplore, while @xcite developed a comprehensive multilingual dataset. additionally, @xcite presented a robust dataset designed to address the challenge of detecting machine-generated text. despite these efforts, large-scale initiatives in academic contexts remain limited. hence, this shared task aims to bridge this gap by tackling the task of distinguishing ai-generated essays from human-authored ones. the challenge attracted substantial interest, with 99 teams registered to access the dataset and 56 teams actively participating in the development and evaluation phases. in the evaluation phase, 25 teams submitted systems for english, and 21 teams participated for arabic. furthermore, five teams submitted system description papers. the majority of participating systems employed transformer-based models, while one team utilized state-of-the-art llms such as llama 2 and llama 3. notably, most submissions outperformed the traditional n-gram-based baseline, signaling substantial progress in ai-generated content detection methodologies. the subsequent sections of this paper are structured as follows: section 2 provides a comprehensive review of related work. section 3 presents the task formulation and dataset setup. section 4 presents empirical results and offers a comprehensive overview of participating systems. finally, section 5 concludes with a summary of findings and future directions. ## related work the detection of ai-generated text relies on analyzing statistical patterns and linguistic features that distinguish human and machine writing styles. @xcite highlight that ai-generated text tends to use repetitive sentence patterns and a limited vocabulary, prioritizing clarity over the nuanced variations of human writing. similarly, @xcite report that such texts often exhibit lower syntactic complexity and reduced lexical diversity, making them identifiable through these markers. additionally, @xcite report that higher predictability in word n-gram is a key indicator of machine generated text. machine learning approaches have become central to ai-generated text detection. darda et al. (2023) explored traditional classification algorithms such as support vector machines (svm) and random forest. @xcite propose a multimodal approach that uses bert to analyze syntactic and semantic features of text and cnn architectures for image. @xcite investigated using stylometric features and transformerbased models. their findings showed that ensemble techniques, particularly those employing majority voting, outperformed individual classifiers. there has also been effort to combine different machine learning approaches. for instance, deep learning architectures can extract features from text, while traditional classifiers make predictions based on these features, leveraging the strengths of both techniques @xcite . incorporating user feedback further enhances hybrid models, enabling them to adapt to real-world usage patterns @xcite . despite advancements in detection methodologies, significant limitations persist. @xcite reveal that many detection tools struggle with high rates of false positives and false negatives, indicating a need for further refinement. according to @xcite , humans naturally incorporate varying sentence lengths and structures in their writing, creating what researchers call "burstiness"-a key feature that distinguishes human-authored content from ai-generated text. this variation in writing style, along with occasional grammatical inconsistencies and stylistic irregularities, represents the natural "imperfections" that make human writing unique. interestingly, @xcite found that texts with lower levels of perplexity and coherencecharacteristics often found in writing by non-native english speakers-are more likely to be flagged as human-authored. another challenge in ai-generated content detection is the lack of transparency in models' predictions, reducing their applicability in real-life scenarios, particularly in high-stakes contexts such as academia and forensic applications. thus, a number of researchers worked on developing explainable ai (xai) methods for ai generated text detection. for instance, @xcite develop an xai model using stylistic features. @xcite proposes a hybrid approach that combine statistical analysis with machine learning techniques. additionally, the integration of user feedback into hybrid models may facilitate the development of more adaptive systems that can learn from usage patterns @xcite . ## results and overview of the systems in team integrityai (al-smadi, 2025) fine-tuned electra-small for english and araelectrabase for arabic to balance high performance with computational efficiency. stylometric features, including word count, sentence length, and vocabulary richness, were incorporated to enhance detection capabilities. the lightweight models achieved f1-scores of 0.985 for english and 0.984 for arabic, demonstrating the effectiveness of combining transformer-based architectures with stylometric analysis. the system was further optimized for deployment on gpus with moderate memory capacity, ensuring both efficiency and accessibility. larger models, such as electra-large, were also tested, achieving an f1-score of 0.997 for english, demonstrating the potential for even greater accuracy with additional computational resources. team cmi-aigcx @xcite proposed a method leveraging the llama-3.1-8b model as a proxy to capture the semantic feature of each token in the text. these token representations were subsequently used to train a model. instead of fine-tuning an llm, they leveraged multilingual knowledge and trained a model to enhance detection performance. their approach demonstrated that using a proxy model with diverse multilingual knowledge can effectively detect machinegenerated text across multiple languages, regardless of model size. for english, an f1 score of 0.999 was achieved, securing first place out of 25 teams. for arabic, an f1 score of 0.965 was obtained, which ranked fourth among 21 teams. team tesla (indurthi and varma, 2025) extracted a comprehensive set of features encompassing style, language complexity, bias, subjectivity, and emotion. these features were used to train four machine learning algorithms: logistic regression, random forest, randomized decision trees (extra trees), and xgboost, leveraging diverse approaches to optimize detection performance. their methods ranked 6th on the leaderboard for the english subtask, achieving an f1-score of 0.986. team essaydetect @xcite proposed a fusion model by integrating pre-trained language model embeddings with stylometric and linguistic features to improve classification accuracy. the contributions were threefold: (i) lime was utilized to identify and highlight highly discriminative features, (ii) focal loss was employed to address class imbalance, and (iii) layer-wise freezing was implemented during fine-tuning to preserve core linguistic representations in the lower layers while enabling the higher layers to capture task-specific stylistic differences in essays. team ra (gharib and elgendy, 2025) fine-tuned several models for english, including roberta, xlm-roberta, mbert, and deberta. similar performance was observed across all models on the validation set, except for mbert, which exhibited slightly lower performance. for arabic, arabert, arbert, and marbert were fine-tuned on the full dataset. arabert consistently demonstrated superior performance in terms of f1-score across both languages. the models consistently exceeded both the mean and median scores across tasks, achieving an f1-score of 0.969 in classifying ai-generated essays in english and 0.957 in arabic. ## conclusion and future work we presented an overview of the shared task on the academic essay challenge. the task attracted significant attention, with a total of 56 teams registering to participate in the development and evaluation phases. of these, 21 teams submitted official results on the test set for arabic, and 25 teams did so for english. finally, five teams submitted task description papers. most systems fine-tuned transformer-based language models; however, several teams also incorporated additional features, such as style, language complexity, bias, subjectivity, and emotion. for both languages, the topperforming teams achieved f1 scores above 0.98.