T ur QU az at G en AI Detection Task 1:Dr. Perplexity or: How I Learned to Stop Worrying and Love the Finetuning (2025)

Abstract: This paper details our methods for addressing Task 1 of the GenAI Content Detection shared tasks, which focus on distinguishing AI-generated text from human-written content. The task comprises two subtasks: Subtask A, centered on English-only datasets, and Subtask B, which extends the challenge to multilingual data. Our approach uses a fine-tuned XLM-RoBERTa model for classification, complemented by features including perplexity and TF-IDF. While perplexity is commonly regarded as a useful indicator for identifying machine-generated text, our findings suggest its limitations in multi-model and multilingual contexts. Our approach ranked 6th in Subtask A, but a submission issue left our Subtask B unranked, where it would have placed 23rd.

## introduction the rapid proliferation of large language models (llms) has brought both remarkable advancements and significant challenges to the field of natural language processing (nlp). while these models enable unprecedented levels of fluency and coherence in generated text, their potential for misuse-ranging from generating misleading information to creating plagiarized content-necessitates robust detection methods. distinguishing between human-written and machine-generated text has thus become a critical area of research, especially in multilingual and multi-model contexts. task 1 of the genai content detection shared tasks @xcite , addresses these challenges by focusing on the development of robust classifiers capable of identifying ai-generated text. this task is divided into two subtasks: subtask a, which deals with english-only datasets, and subtask b, which extends detection to multilingual datasets. in this paper, we present our approaches for subtask a and subtask b. for subtask a, we relied on a fine-tuned version of the xlm-roberta model, achieving competitive performance. for subtask b, we explored the integration of additional features such as perplexity and term frequency-inverse document frequency (tf-idf). while perplexity has traditionally been considered a valuable metric for identifying machine-generated text (varshney et al., 2020), we found its effectiveness limited in complex, multilingual scenarios. our approaches were ranked 6th place out of 35 participants in subtask a. unfortunately, due to a submission-related issue, our entry for subtask b was not officially ranked. however, it would have placed 23rd out of 26 participants. ## related work the advance of llms has necessitated the development of robust methods for detecting machinegenerated text. this has become a critical research area due to the potential misuse of llms for generating misleading or plagiarized content @xcite . early zero-shot detection methods, relying on statistical features like log probability and rank @xcite @xcite , are computationally efficient but often lack robustness as models improve in generating text that closely resembles human writing @xcite . more sophisticated zero-shot approaches have been proposed to address these shortcomings. de-tectgpt @xcite use the concept of probability curvature, comparing the log probability of a text with perturbed versions to identify machine-generated content. supervised detection, training classifiers on labeled data, has also been investigated @xcite . however, these methods suffer from limitations related to data requirements and generalization across domains and llms. similarly, watermarking techniques @xcite , while promising, have shown vulnerability to paraphrasing attacks @xcite . ## proposed approach for both subtask a and subtask b, we fine-tuned the facebook xlm-roberta base model 123 @xcite . in subtask a, we submitted the labels from the bare fine-tuned classifier, as we were unable to improve its performance using additional features including perplexity. for subtask b, we aimed to enhance the classifier's performance by incorporating additional features including tf-idf, the source language as a one-hot encoded feature, and perplexity values derived from the llama 3.2 1b model @xcite . these combined features were fed into an xgboost classifier to improve overall performance. while our goal was to evaluate the effectiveness of perplexity as a predictive metric, we found it to be a suboptimal measure in this multi-source, multi-language context. ## conclusions in this paper, we presented our approach to the genai content detection shared tasks, focusing on distinguishing ai-generated from human-written text in both monolingual and multilingual contexts. our primary findings indicate that while fine-tuned xlm-roberta models can achieve competitive performance, the incorporation of additional features such as perplexity scores did not yield significant improvements in detection accuracy. this was particularly evident in the multilingual context, where perplexity's effectiveness varied considerably across different languages and source models.