RA at G en AI Detection Task 2: Fine-tuned Language Models For Detection of Academic Authenticity, Results and Thoughts (2025)

Abstract: This paper assesses the performance of “RA” in the Academic Essay Authenticity Challenge, which saw nearly 30 teams participating in each subtask. We employed cutting-edge transformer-based models to achieve our results. Our models consistently exceeded both the mean and median scores across the tasks. Notably, we achieved an F1-score of 0.969 in classifying AI-generated essays in English and an F1-score of 0.957 for classifying AI-generated essays in Arabic. Additionally, this paper offers insights into the current state of AI-generated models and argues that the benchmarking methods currently in use do not accurately reflect real-world scenarios.

## introduction large language models (llms), as advanced artificial intelligence (ai) systems, have been trained on vast amounts of text data and can generate, summarize and comprehend human languages with impressive fluency @xcite . as these models are based on deep learning architectures, primarily transformer, they can learn complex language patterns and respond end-to-end with impressions very similar to human interaction. recently, models such as gpt-4, mistral @xcite , llama @xcite , etc have enabled significant advances in ai language processing for natural language understanding and generation tasks. their promise, however, raises serious ethical and social concerns about honesty, transparency, and misuse @xcite . a major area where ai could create change is education. ai offers personalized learning, providing students with tailored resources that enhance effectiveness. however, the accessibility of writing tools questions academic integrity, as students may bypass essential learning processes that promote critical thinking, creativity, and problem-solving skills. addressing these challenges is crucial for maximizing ai's benefits while minimizing its risks. the academic essay authenticity 2024 shared task @xcite focuses on creating systems to distinguish human-written text from aigenerated content and provides a validation dataset. our model builds on recent efforts by fine-tuning multiple language models using an ensemble approach. this paper covers data analysis, pipeline, results, related work, and future directions, highlighting real-world improvements. ## related work this section examines recent advancements in detecting content generated by large language models (llms). with the rapid adoption of llms and associated challenges, researchers have increasingly focused on this area. numerous studies have introduced systems employing both deep learning and traditional machine learning to address the authenticity and reliability concerns of ai-generated content across various fields. the widespread use of ai-generative models has fueled the development of methods to detect text generated by these models, especially to safeguard integrity in domains such as education. broadly, classification methods fall into two categories: white-box and blackbox approaches. white-box methods require direct model access and include techniques like wordlevel probability analysis, perturbations @xcite , and local rank perturbations. in contrast, black-box methods work without model access and include supervised learning with linguistic features (prova, 2024), supervised learning with pretrained lms @xcite , and regeneration techniques. recent years have also seen the creation of various datasets aimed at advancing aigenerated text detection @xcite , alongside shared tasks dedicated to developing novel, robust approaches @xcite @xcite . however, a research gap persists in detecting ai-generated text in arabic. the datasets for arabic are often sourced from various resources, lacking cohesion and presenting limited challenge. notably, a specialized dataset for arabic was created to aid model development but contains only 1,000 examples @xcite . ## dataset & task description the shared task on academic essay authenticity challenge 1 consists of two main subtasks. each subtask will be discussed in details in the following subsections. the provided dataset primarily comprises essays created either by a human or through promopting a generative language model. the subsequent subsections will present an overview of the distribution for each dataset, emphasizing the challenges posed by imbalances and a complete description of each dataset. ## conclusion this paper introduces an approach for detecting academic authenticity using an ensemble of language models. despite its simplicity, the method achieves high performance after only a few epochs. while this is advantageous, it also has drawbacks. the straightforward nature of the approach, when trained on benchmark datasets, may not accurately represent its performance in real-world scenarios. we discuss several factors that could challenge the model's effectiveness and call on researchers to address these challenges in future work.