S zeged AI at G en AI Detection Task 1: Beyond Binary - Soft-Voting Multi-Class Classification for Binary Machine-Generated Text Detection Across Diverse Language Models (2025)

Abstract: This paper describes the participation of the SzegedAI team in Subtask A of Task 1 at the COLING 2025 Workshop on Detecting AI-Generated Content. Our solutions investigate the effectiveness of combining multi-class approaches with ensemble methods for detecting machine-generated text. This approach groups models into multiple classes based on properties such as model size or generative capabilities. Additionally, we employ a length-based method, utilizing specialized expert models designed for specific text length ranges. During inference, we condense multi-class predictions into a binary outcome, categorizing any label other than human as AI-generated. The effectiveness of both standard and snapshot ensemble techniques is evaluated. Although not all multi-class configurations outperformed the binary setup, our findings indicate that the combination of multi-class training and ensemble methods can enhance performance over single-method or binary approaches.

## introduction in recent years, machine-generated text has become increasingly sophisticated, with advances in generative models. nowadays, substantial amounts of ai-generated content are being produced, even in the academic and scientific literature @xcite . recent findings estimate that between 6.5% and 16.9% of peer review text submitted to ai conferences may be substantially modified by large language models, revealing subtle trends in the use of llms within academic settings @xcite . this trend is further supported by the fact that various models, such as gpt @xcite , mixtral @xcite , and gemini @xcite have become easily accessible to lay people, even those without a strong technical background. as a result, there is an increasing need for algorithms capable of predicting the difference between human-and machine-generated content. workshops have been organized to address this need, such as those discussed in @xcite and @xcite . in this paper, we present our solutions for task 1 in the workshop on detecting ai-generated content at coling 2025 @xcite . our focus was on subtask a, where the objective was to make a binary prediction indicating whether the text was human-written or ai-generated. the dataset for this subtask contained only english texts. to tackle this challenge, we experimented with various classification schemes to determine the most effective setup for detecting machine-generated text. binary, 3, 5, 6 and 41 class solutions were developed, with each configuration representing a different heuristic grouping of models based on parameters, model size, or the quality of generated text. standard ensemble methods with soft voting were applied across all class configurations to improve robustness and performance. an additional experiment involved using a snapshot ensemble @xcite , capturing multiple snapshots of the model during training to create a diverse but computationally efficient ensemble. this snapshot ensemble was created only for the 41-class configuration. in addition, a length-based solution was developed in which the texts were categorized by length. three intervals were created based on text length, with an expert model assigned to each interval. these texts were classified according to their length and directed to the corresponding expert model, optimizing performance by matching each text to a model specialized in its specific length range. our final submission, the 6-class solution, achieved a macro f1 score of 0.791, though it was not our highest performing model. our post-shared task evaluation revealed that the length-based approach achieved a macro f1 score of 0.827, while the 41-class standard ensemble scored slightly higher at 0.826. ## related work recent advancements in detecting machinegenerated text have highlighted the efficacy of ensemble learning methods, particularly those that employ soft voting strategies. in the semeval-2024 task 8 @xcite , @xcite developed a class-balanced soft-voting system that fine-tuned transformer-based models, including encoder-only, decoder-only and encoder-decoder architectures @xcite . their approach effectively addressed data imbalance and achieved state-of-the-art performance in multi-class classification tasks involving various text generators. building upon this, we explored the application of snapshot ensembles within transformer architectures for machine-generated text detection. this method captures multiple "snapshots" of a model at different training stages, offering computational efficiency over traditional ensembles. although this approach may not match the robustness of standard ensembles, it presents a viable alternative when resources are limited @xcite . another approach for detecting ai-generated text leverages a transformer encoder that combines probabilistic features from multiple llama-2 models @xcite . this model reached impressive accuracy in distinguishing machine-generated from human-written content. the study reveals that integrating probabilistic features from various language models can significantly boost detection precision, with llmixtic effectively emphasizing the unique features of the final tokens in a sequence. ## results our final submission was a 6-class soft voting, which secured 7th place. we evaluated our solutions in both the development set and the published test set. based on these evaluations, the 6-class approach was not the optimal choice. the best performing experiment was the length-based solution with soft voting, closely followed by the simple 41-class approach. the corresponding f1 scores are shown in ## conclusion our findings highlight that the binary approach may not be the most effective way to detect machinegenerated texts. instead, we recommend considering multi-class solutions, as they might improve classification by capturing distinctions among aigenerated texts. in our experiments, we found that different approaches performed well on different datasets: one solution showed strong results on the development set, while another excelled on the test set. for example, the length-based approach was not promising in the development set but performed well on the test set. to evaluate our approach comprehensively, we considered not only the results, but also the computational efficiency. we implemented a snapshot ensemble technique and also a data split approach with 3 models. both strategies demonstrated improved performance compared to a single model fine-tuned on the full training dataset while maintaining the same computational costs. however, the more expensive solutions still outperformed these. notably, the 3-class approach showed a drop in performance on the test set, likely due to the absence of weaker models in the test data. this suggests that certain multi-class configurations may be less effective when faced with a set of highperforming models only. however, even with these variations, the 41-and 5-class solutions outperformed the binary approach and provided a consistent gain over binary modeling. ultimately, our approach demonstrates that moving beyond binary classification to a multi-class strategy, especially when using ensemble soft voting, can yield better results in detecting machinegenerated text.