I ntegrity AI at G en AI Detection Task 2: Detecting Machine-Generated Academic Essays in E nglish and A rabic Using ELECTRA and Stylometry (2025)

Abstract: We present a robust system for detecting machine-generated academic essays, leveraging pre-trained, transformer-based models specifically tailored for both English and Arabic texts. Our primary approach utilizes ELECTRA-Small for English and AraELECTRA-Base for Arabic, fine-tuned to deliver high performance while balancing computational efficiency. By incorporating stylometric features, such as word count, sentence length, and vocabulary richness, our models excel at distinguishing between human-written and AI-generated content. Proposed models achieved excellent results with an F1- score of 99.7%, ranking second among of 26 teams in the English subtask, and 98.4%, finishing first out of 23 teams in the Arabic one. Main Contributions include: (1) We develop lightweight and efficient models using ELECTRA-Small and AraELECTRA-Base, achieving an impressive F1-score of 98.5% on the English dataset and 98.4% on the Arabic dataset. This demonstrates the power of combining transformer-based architectures with stylometric analysis. (2) We optimize our system to maintain high performance while being computationally efficient, making it suitable for deployment on GPUs with moderate memory capacity. (3) Additionally, we tested larger models, such as ELECTRA-Large, achieving an even higher F1-score of 99.7% on the English dataset, highlighting the potential for further accuracy gains when using more computationally intensive models.

## introduction since the launch of chatgpt in november 2022, research on developing models for artificial intelligence (ai)-generated text detection has increased. this increase reflects growing concerns about maintaining academic integrity in the face of advanced generative ai (genai) tools capable of producing human-like text @xcite . @xcite were among the first working on this topic by developing a dataset named the "human chat-gpt comparison corpus (hc3)" out of nearly 40,000 questions from different datasets along with their answers provided by humans. they also generated responses to these questions by chatgpt and used the combined dataset to train detectors in both english and chinese. the developed models included machine and deep learning based models like roberta @xcite and demonstrated decent performance across different scenarios. another paper focused on detecting chatgptgenerated text written in english and french @xcite . the english model was trained using the hc3 dataset. the authors also translated some of its english content to french and included additional small french out-of-domain dataset of 113 french responses from chatgpt and 116 from binggpt. they fine-tuned two pretrained models, camembert @xcite and camemberta @xcite , using the french dataset, and roberta @xcite and electra @xcite models using the english one. they also used xlm-r @xcite as multi-language model for the combined datasets of both languages. research results showed that all models demonstrated good performance in identifying machine-generated content within the same domain, but when tested on out-of-domain content, their results dropped. @xcite used transformer-based models, namely araelectra @xcite and xml-r @xcite to solve the challenges of machine-generated arabic text identification. the authors focused on the influence of diacritics on detection model performance. their method showed great accuracy on the airabic benchmark dataset. other research utilized stylometric features to detect machine-generated content. for instance, @xcite introduced a machine learning model with stylometry for identifying "contract cheating", the act of students depending on others to complete academic assignments on their behalf, by detecting deviations from a learner's distinctive writing style, which achieved excellent accuracy in their research. @xcite developed a data-driven model named "styloai" trained with 31 stylometric features to detect machine-generated content. "styloai" performance outperformed other models on the same dataset. wee and reimer (2023) discovered that ai identification technologies classified human-written writings translated from non-english languages as ai-generated, which raised worries among nonnative english speakers @xcite this research aims at addressing the challenge of ai-generated text. the rest of this paper is organized as follows: section 2 discusses the research methodology, section 3 presents the findings, and section 4 concludes the study and highlights future directions. ## results and findings to evaluate the impact of the stylometric features on the model performance, we trained the models without features. the results demonstrate that excluding these features leads to a decrease in model performance, with a 1.5% and 2.4% drop in f1 score for araelectra and electra models, respectively. this indicates that extracted features enhanced model predictions. despite the modest decline, the impact underscores the importance of these features for better generalization. the results of training vs. validation loss values after each epoch of models training in figure 3 , show that the training loss rapidly declined from the first epoch and then quickly stabilized to run in parallel with the validation loss. both values of training and validation loss kept decreasing smoothly until the end of models training epoch without any sign of overfitting, as the validation loss remains close to the training loss throughout the training process. this was also maintained by enabling the option of early_stopping during the models training. moreover, this also indicates that both models generalizes very well when confronted by new unseen data. the rapid stabilization of loss values may indicate that more complex model architectures might achieve even better results. therefore, we trained the electra_large instead of the elec-tra_small model for the english subtask for 10 epochs as well. as, expected the electra_large achieved better results with f1 score of 99.7%. for more information on the results of other participating teams in the task, the reader is redirected to @xcite . ## conclusion and future work this study demonstrates the efficacy of transformerbased models for identifying machine-generated academic articles. using electra-small for english and araelectra-base for arabic, paired with stylometric characteristics, our models produced remarkable f1-scores of 98.5% and 98.4%, respectively. experiments using electra-large for english revealed the possibility of even better f1-score, reaching 99.7%, but at a larger computing cost. our proposed models offer an adaptable solution that balances performance and efficiency and is appropriate for a variety of hardware setups. to improve robustness, future study might focus on real-time detection, expanding to new academic areas, and extending language coverage.