AI -Monitors at G en AI Detection Task 1: Fast and Scalable Machine Generated Text Detection (2025)

Abstract: We describe the work carried out by our team, AI-Monitors, on the Binary Multilingual Machine-Generated Text Detection (Human vs. Machine) task at COLING 2025. This task aims to determine whether a given text is generated by a machine or authored by a human. We propose a lightweight, simple, and scalable approach using encoder models such as RoBERTa and XLM-R We provide an in-depth analysis based on our experiments. Our study found that carefully exploring fine-tuned parameters such as i) no. of training epochs, ii) maximum input size, iii) handling class imbalance etc., plays an important role in building an effective system to achieve good results and can significantly impact the underlying tasks. We found the optimum setting of these parameters can lead to a difference of about 5-6% in absolute terms for measure such as accuracy and F1 measure. The paper presents crucial insights into optimal parameter selection for fine-tuning RoBERTa and XLM-R based models to detect whether a given text is generated by a machine or a human.

## introduction large language models (llms) like gpt-4, claude 3.5, and gemini 1.5-pro have rapidly become mainstream tools, offering highly fluent and articulate text generation across a wide range of applications, from social media and news to academic and educational content. these models are capable of producing human-like responses to various queries, making them increasingly attractive for replacing human labor in tasks such as content creation, customer support, and even academic writing. this challenge @xcite underscores the need for automated systems designed to detect machine-generated content. as llms become more sophisticated and pervasive, developing robust detection methods is critical to preventing misuse. these systems could help mitigate the risks of misinformation, ensure academic integrity, and provide safeguards against the over-reliance on machine-generated material in sensitive contexts. this work is part of a larger project where we are building solutions for emerging plagiarism detection, llm-based response generation detection for academic studies, and assignments. one major issue is the difficulty humans face in distinguishing machine-generated text from human-written content. this has posed significant challenges when analyzing and evaluating student assignments and open-book answers, where there are potential issues with using llm-generated answers and challenges in accurately detecting them. this problem calls for the development of effective automated systems for grading, assessment, and identifying whether a text is generated by a human or a machine. additionally, it is important to identify instances where machine-generated content has been post-edited by humans to avoid detection by automatic systems. in this paper we describe the work carried out by our team ai-monitors on the binary multilingual machine-generated text detection (human vs. machine) task at coling 2025 @xcite . towards building a fast scalable system which can auto-train and learn with more data, we focused on exploring roberta and xlm-r models as they have shown to perform well for this task @xcite . ## related work automatic detection of machine-generated text is typically framed as a binary classification task, where, for a given input, we must classify whether it is generated by a machine or a human @xcite . two main approaches are commonly used: one relies on supervised techniques, which require large training datasets, while the other relies on unsupervised approaches using common detection models such as roberta @xcite , xlm-r @xcite , or stylistic features. jawahar et al. (2020) provides a critical survey of the pros and cons of various alternatives for detecting machinegenerated text. given sufficient training data, some prior works @xcite have experimented with fine-tuning the roberta language model for the detection task. @xcite found that roberta established stateof-the-art performance in identifying web pages generated by the largest gpt-2 model, achieving an accuracy of approximately 95%. @xcite demonstrated that the roberta detector also set the state-of-the-art in accurately distinguishing machine-generated tweets from humanwritten tweets, outperforming both traditional machine learning models (e.g., bag-of-words) and complex neural network models (e.g., rnn, cnn) by a large margin. this promising result suggests that the roberta detector can generalize well to previously unseen publication sources, such as twitter. the automatic detection systems comprise approaches that are used to detect domains including the one that ustc-bupt has developed for semeval-2024 task 8 with the help 1 https://github.com/aimonitors25/machine-generatedtext-detection of dated, llam, tle, and audm for monolingual as well as multilingual detection targets (guo et al., 2024). further, black-box machinegenerated text detection where llms are finetuned with parameter-efficient smaller llms and per-language classification-threshold calibration was proposed and was proved to perform well in semeval-2024 task 8 (spiegel and macko, 2024). another valuable contribution is genaios' llm ixtic system in which several llama-2 models operate within a transformer encoder framework; this system performed extremely well in the monolingual track and underlines the role of token-level probabilistic features for text classification (sarvazyan et al., 2024). ## methodology in the task of distinguishing between humanwritten and machine-generated text, we are performing finetuning of roberta and xlm-r. roberta @xcite , is a robust transformerbased language model, adapted to better capture the nuances in writing style and language patterns that differentiate human and machine-generated content. the model is trained on a labeled dataset consisting of both machine-generated and humanwritten articles to learn these distinctions. figure 1 shows an illustration of the roberta model. to build effective and efficient solution we explore different parameters as shown in there is a noticeable class imbalance in the training and development sets, with the machine class significantly outnumbering the human class. this could lead to challenges in model training, where the model might become biased toward the majority class (machine). hence in our experiments we explored weighing the minority class more as compared to equal weights for both the classes. the test set is relatively more balanced between the two classes, which is important for evaluating the model's ability to generalize to both machine and human instances. ## results we performed multiple parameters explorations as described in roberta vs xlm-r: we tested two models-roberta and xlm-r-and evaluated their the xlm-r model performed better in the dev set, achieving an accuracy of 92.68% after 1 epoch, compared to 86.61% for roberta. however, the roberta model saw greater improvements with more training, reaching 96.05% and then 97.82% accuracy on the dev set after 3 and 5 epochs, respectively. thus, we used roberta for further explorations of optimum parameters. this best solution was also the submission of the official leader board for the shared task. more details to follow in the later section. roberta finetuning -max input size explorations: roberta finetuning -no. of epochs: ## conclusion & future work a critical survey on automatic text detection (jawahar et al., 2020) provides a summary of key error categories made by these automated models, namely: fluency, brevity, factuality, spurious entries, contradictions, repetitions, common sense reasoning, typos, grammatical errors etc,. we plan to perform a similar error analysis on this task dataset and work towards building a hybrid pipeline that leverages techniques like those in @xcite . a summary of this pipeline on leveraging transformer encoder that incorporates token-level probabilistic features extracted from the llama models is shown in figure 2 and discussed in appendix a. in future, we aim to explore ensemble-based solutions, comprising a simple fine-tuned roberta pipeline alongside a richer pipeline as described in appendix a, that leverages multiple llms to better capture patterns and data distributions for detecting machine generated text.