Leidos at G en AI Detection Task 3: A Weight-Balanced Transformer Approach for AI Generated Text Detection Across Domains (2025)

Abstract: Advancements in Large Language Models (LLMs) blur the distinction between human and machine-generated text (MGT), raising concerns about misinformation and academic dishonesty. Existing MGT detection methods often fail to generalize across domains and generator models. We address this by framing MGT detection as a text classification task using transformer-based models. Utilizing Distil-RoBERTa-Base, we train four classifiers (binary and multi-class, with and without class weighting) on the RAID dataset (Dugan et al., 2024). Our systems placed first to fourth in the COLING 2025 MGT Detection Challenge Task 3 (Dugan et al., 2025). Internal in-domain and zero-shot evaluations reveal that applying class weighting improves detector performance, especially with multi-class classification training. Our best model effectively generalizes to unseen domains and generators, demonstrating that transformer-based models are robust detectors of machine-generated text.

## introduction the rapid advancement of large language models (llms) has made it increasingly difficult to distinguish between human-written and machinegenerated text. this challenge poses significant risks in areas such as misinformation dissemination, academic dishonesty, and the breach of trust in online communications. existing detection methods often rely on small datasets and struggle to generalize across different domains and generator models. we formulate the detection task as both binary and multi-class classification, and examine the use of class weighting to investigate the impact of class imbalance on detection performance. we build approved for public release under etrace approval id# 24-leidos-1113-28544 and evaluate four distil-roberta-base foot_0 based models trained with raid dataset @xcite , which contains over 6 million text samples from 11 generator models across 8 domains. to assess generalizability, we conduct additional crossdomain evaluations using the mgt detection task 1 dataset @xcite , which includes texts from 41 models not seen during training. our results demonstrate that incorporating class weighting improves detection accuracy and that our models perform effectively across both familiar and unfamiliar domains and generator models. according to the coling 2025 mgt detection challenge task 3 @xcite official evaluation, our models surpass commercial mgt detection tools and achieved top rankings. ## related work authorship attribution has a long history, and machine authorship is a recent focus. for example, iarpa's recent hiatus program (human interpretable attribution of text using underlying structure) foot_1 takes both human and machine authorship into consideration. according to leidos' experience in hiatus, we found that with sufficient training data, transformer-based encoder models can effectively learn features that discriminate authorship and consistently outperformed featurebased approaches. there are more efforts treating mgt detection as a classification problem. for example, xiong et al. 2024 addressed multilingual mgt detection in semeval-2024 task 8, which includes binary classification (human vs. machine) and model attribution. their study found that fine-tuned transformerbased models significantly outperformed traditional machine learning methods, demonstrating superior effectiveness in accurately detecting and at-tributing machine-generated content across various contexts. llm-detectaive @xcite categorizes machine-generated texts into four types: purely human-written, entirely machine-generated, machine-generated then humanized, and humanwritten then machine-polished. this nuanced approach is impactful in educational and academic settings where subtle llm edits may hide machine involvement. however, its reliance on the narrows scope of the m4gt-bench @xcite dataset may cause llm-detectaive to perform accurately within familiar domains but struggle with unfamiliar ones, leading to high false positive rates and reduced accuracy in diverse real-world scenarios. dugan et al. 2024 introduced raid, a comprehensive benchmark with over 6 million text samples from 11 models across 8 domains, incorporating adversarial attacks and diverse decoding strategies. evaluating 12 detectors under a fixed 5% false positive rate revealed that open-source detectors often misclassified human-written texts and lacked robustness against minor text modifications and adversarial attacks. the study highlighted that while detectors perform well on familiar data, they struggle to generalize to unseen domains and models, underscoring the need for more resilient mgt detection methods. ## method we approach mgt detection as a classification task using transformer-based models. our base model, distil-roberta-base 3 @xcite , is a parameter-efficient, distilled variant of roberta @xcite that enables robust detection with limited resources. we train four mgt detectors to evaluate both binary and multi-class classification, exploring the effects of class weighting to address dataset imbalance, to distinguish human-written from machine-generated text. for the bw and mw models, we compute balanced class weights using the following formula: where n is the total number of samples in the dataset, c is the total number of classes and n i is the number of samples in class i. this formula distributes weights evenly across classes by normalizing with the total number of classes, which helps prevent extreme weighting in cases of high imbalance. we select this method to balance the loss contributions across classes due to its simplicity and effectiveness in enhancing the model's generalizability. alternative strategies, such as oversampling underrepresented classes or using synthetic data augmentation techniques, can address class imbalance but add complexity and pose the risk of overfitting @xcite . another alternative is focal loss @xcite that dynamically adjusts the loss based on sample difficulty. this approach can be effective, but demands extensive hyper-parameter tuning, which is impractical given our resource constraints. our class weighting scheme, though static and less adaptable to extreme imbalance, is resource-efficient and easy to implement. baseline (od) as a baseline, we apply the roberta-large-openai-detector foot_2 , an opensource model that is also featured as a baseline on the shared task, directly to our test sets in a zeroshot manner. this enables us to gauge the models performance on our unofficial test sets. ## results this section presents two sets of results: 1) performances of our four mgt detectors against raid-derived test set, and 2) performances of the best-performing model according to 1) against outof-domain task 1 dataset to further demonstrate model's robustness. for each input example, a machine-likelihood score is calculated as @xmath0, representing the probability that the input sample x is machine-generated. the share task adopts the same evaluation metric in raid @xcite to measure how well each detector identifies machinegenerated text while only misclassifying 5% of human-written text. specifically, it is the true positive rate (tpr) after calibrating the decision thresholds to ensure a false positive rate (fpr) of 5%. if the fpr can be optimized to less than 5%, the evaluation script will attempt to do so. ## conclusion in this paper, we address the challenges of detecting machine-generated text (mgt) from multiple domains and different generators. four variants of the distilroberta-base model are trained on large and diverse raid dataset, all achieving highly promising results. specifically, the multi-class classifier with class weighting (mw) performs the best in both multi-domain and cross-domain evaluations despite of adversarial attacks. this suggests that our approach generalizes well across multiple different domains, unseen generator models, and text manipulations. our study demonstrates that transformer-based models with class weighting are effective for mgt detection, representing a significant step toward robust and generalizable detection techniques. however, the strong performance may be influenced by factors limiting true generalizability. specifically, shared training data among llm generators might lead detectors to recognize common patterns rather than genuinely generalize across different generators. additionally, the current evaluation method sets decision thresholds based on a 5% fpr during testing, whereas in practice, thresholds are learned and fixed during training. these aspects require further investigation to ensure detection reliability. in future work, we aim to address these limitations and extend experiments to quantify the impact of training data size.