AAIG at G en AI Detection Task 1: Exploring Syntactically-Aware, Resource-Efficient Small Autoregressive Decoders for AI Content Detection (2025)

Abstract: This paper presents a lightweight and efficient approach to AI-generated content detection using small autoregressive fine-tuned decoders (AFDs) for secure, on-device deployment. Motivated by resource-efficiency, syntactic awareness, and bias mitigation, our model employs small language models (SLMs) with autoregressive pre-training and loss fusion to accurately distinguish between human and AI-generated content while significantly reducing computational demands. The system achieved highest macro-F1 score of 0.8186, with the submitted model scoring 0.7874—both significantly outperforming the task baseline while reducing model parameters by ~60%. Notably, our approach mitigates biases, improving recall for human-authored text by over 60%. Ranking 8th out of 36 participants, these results confirm the feasibility and competitiveness of small AFDs in challenging, adversarial settings, making them ideal for privacy-preserving, on-device deployment suitable for real-world applications.

## introduction advancements in generative ai (genai) powered by large language models (llms) have significantly improved natural language generation (nlg) capabilities. ai-generated text, often indistinguishable from human writing, presents risks to information integrity, trust, and security @xcite @xcite . the widespread availability of opensource, user-friendly llms enables individuals with minimal expertise to conduct misinformation, disinformation, and phishing campaigns, highlighting the need for accurate ai content detection. however, most research rallies behind sophisticated, resource-intensive solutions, often overlooking the security and privacy aspect of ai content detection. most solutions require cloud connectivity or extensive computational resources, making them impractical for secure, on-device deployment. many existing approaches depend on complex architectures, including large pre-trained models (plms) like roberta-large and longformer @xcite , or leverage larger llms like llama @xcite through techniques such as instruction-tuning @xcite . others employ ensemble methods that combine multiple llms @xcite @xcite @xcite , that can significantly increase latency. in contrast, we focus on lightweight models that are optimized for secure, on-device deployment. on-device processing supports real-time analysis, essential for fast-paced environments, while also enhancing privacy by retaining sensitive data locally and reducing risks associated with transmitting information to external servers @xcite . this is particularly important for security-sensitive fields such as defense, healthcare, finance, and personal communications, where protecting data from unauthorized access is critical. this study evaluates the feasibility of small autoregressive fine-tuned decoders (afds) for efficient and secure ai-generated content detection. our approach is guided by three motivations: first, using slms (≤135m parameters) suitable for on-device deployment, enabling privacy, realtime processing, and accessibility; second, leveraging an autoregressive pre-training objective that mirrors the sequential nature of language production, enhancing syntactic awareness essential for detecting structural nuances to differentiate human and machine language; and third, employing a loss fusion strategy to learn from difficult examples and encourage class separation for bias mitigation. the proposed system, using smollm, achieved the highest macro-f1 score of 0.8186 on the test set, significantly outperforming the task baseline (macro-f1 of 0.7568) under similar settings, while also reducing the number of parameters by ≈60%. our submitted model (selected based on validation performance) attained a macro-f1 score of 0.7874, ranking 8th among 36 participants. these results underscore the potential of small afds for effective on-device ai-content detection. ## task description in the coling workshop on detection ai generated content, task 1 posed a binary classification problem: determining whether a given text is machine-or human-authored @xcite . our investigation focused on subtask a, which targets english-only mgt detection and extends the semeval shared task 8 (subtask a) @xcite . our approach centers on fine-tuning small autoregressive decoders combined with a loss fusion strategy to enhance classification performance. anticipating the presence of surprise domains and llms in the test set, we experimented with two distinct loss functions to optimize performance, particularly on challenging examples. ## experimental setup we evaluate five different slms (≤135m parameters), selected for their suitability for on-device deployment @xcite . these include smollm (allal et al., 2024), gpt2 @xcite , gpt-neo @xcite , opt @xcite and mobilellm @xcite . since the task baseline (roberta-large) has significantly more parameters than our selected models, we also include roberta-base as a baseline for fair pernote: %gain represents the performance improvement over our baseline (roberta-base), with arrows (↑/↓) indicating increase or decrease. test performance of the submitted system is marked with (*). formance comparison. to align with our objective of testing the feasibility of small afds, we adopt a simple architecture: a single linear layer added on top of the frozen afds for classification, with a dropout layer (dropout @xmath0.3) applied before classification. each text sample is represented using mean pooling of all token embeddings. maximum length is set to 512 tokens, with shorter samples padded and longer samples truncated to max length. we use a 50:50 split of the provided validation dataset for validation and testing. training employs early stopping (with @xmath1, retaining the model with the lowest validation loss. optimization is performed using the adamw optimizer with a linear warmup scheduler (10% warmup steps). the learning rate is set to 2 × 10 -5 , with a batch size of 32. although the maximum training epochs are set to 15, early stopping is usually triggered within 4 epochs in practice. mixed-precision training with gradient scaling is used to speed-up training. no further hyperparameter tuning is performed. we re-lease our data and source code for reproducibility foot_0 . ## results we evaluated 20 models across four loss configurations and five afds, with 15 achieving a positive gain over the baseline-a 75% success rate, validating our hypothesis and highlighting the effectiveness of small afds for ai-generated content detection. the highest performance was achieved with the smollm model, likely due to its training on the high-quality smollm-corpus @xcite , which includes a mix of human and synthetically generated text. to analyze performance imbalance between the human and llm classes, we examined their recall scores on the test set. while baseline models achieve high recall for the llm class, they perform poorly on the human class -confirming the bias reported in previous studies. in contrast, afd models show a slight decrease in llm recall but deliver a substantial improvement in recall for the human class -effectively mitigating bias. our best model increases human recall by 60% and 28% over our and task baselines, with minor llm recall reductions of 9% and 5%, respectively. focal loss consistently outperforms crossentropy, but the addition of contrastive loss does not always lead to performance improvements. while contrastive loss occasionally enhances the performance of plms, afds generally show a decline when it is applied. this can be attributed to the already well-separated nature of afd embeddings where further enforcing separation may over-penalize instances near class boundaries. this observation reinforces the suitability of afds, as their inherent syntactic awareness provides strong separability, making additional loss optimization redundant and reducing computational overhead. ## discussion given the test set's significant differences from the training set, namely, unfamiliar data sources, adversarially modified text, and unknown llms, it is imperative to analyze their impact on model performance. figure 2 presents a box-and-whisker plot of model performance by data source, with greater spread indicating variability. overall, the human class shows lower performance compared to the llm class, with variations across sources. mixset and cudrt have the highest misclassification rate for the llm class, likely due to the inclusion of adversarially perturbed text samples. for the human class, mixset, detectaive, and ieltsduck show the lowest performance. detectaive and ieltsduck contain ielts test takers' data, likely written by non-native english speakers, contributing to misclassifications. in mixset, llm-modified samples labeled as human may also lead to errors. the perturbation operations performed on each dataset are described in detection accuracy for each llm in the test set is depicted in figure 4 . variations of the chatfigure 3: performance across adversarial perturbations figure 4: model performance per llm in test set gpt family demonstrate stronger detection, likely due to the inclusion of related models from the same family in the training set. nevertheless, the generalization of performance across data sources underscores our model's effectiveness. moreover, chatgpt's detectability aligns with existing research @xcite , ensuring safety against misuse of most widely used chatbot. in contrast, lesser-known models like dolly and baichuan show lower detection rates, highlighting areas for improvement. ## conclusion this paper addresses the challenge of secure ondevice ai content detection by proposing a simple yet effective solution leveraging small afds. with their resource-efficient design and syntactic alignment enabled by autoregressive pre-training, the proposed approach-combining afds with loss fusion, particularly focal loss-outperforms larger, resource-intensive models by a large margin while reducing model size. our proposed approach mitigates bias, maintains generalization, and handles challenging data, including unknown domains, unseen llms, and adversarially modified text. these results highlight the potential of small afds as efficient backbones or ensemble components, especially in scenarios requiring data privacy and faster ai-content detection.