DCBU at G en AI Detection Task 1: Enhancing Machine-Generated Text Detection with Semantic and Probabilistic Features (2025)

Abstract: This paper presents our approach to the MGT Detection Task 1, which focuses on detecting AI-generated content. The objective of this task is to classify texts as either machine-generated or human-written. We participated in Subtask A, which concentrates on English-only texts. We utilized the RoBERTa model for semantic feature extraction and the LLaMA3 model for probabilistic feature analysis. By integrating these features, we aimed to enhance the system’s classification accuracy. Our approach achieved strong results, with an F1 score of 0.7713 on Subtask A, ranking ninth among 36 teams. These results demonstrate the effectiveness of our feature integration strategy.

## introduction in recent years, with the rapid development of large language models, distinguishing between machinegenerated text and human-authored text has become increasingly challenging. this issue can lead to several potential problems. low-quality generated text, when posted on social media, can reduce user experience, hinder the growth of platforms and high-quality content creators @xcite . generated text that lacks fact-checking can lead to the spread of rumors and misinformation @xcite , causing public panic and undermining government credibility. in academia, the presence of generated text raises ethical concerns regarding academic integrity @xcite . therefore, there is an urgent need to develop effective techniques for detecting machine-generated content @xcite . unlike typical machine-generated text, the data for this shared task are derived from multiple models and spans various domains @xcite . the human-authored texts in subtask a originate from over 20 specialized fields, including finance, medicine, arxiv, wikihow, imdb, and yelp. correspondingly, the machine-generated texts are produced by more than 40 different large language models. due to the diverse sources of this dataset, many simple yet effective statistical features are no longer viable, significantly increasing the challenge of the detection task. machine-generated text often exhibits certain characteristics, such as weaker emotional expression, fewer numeric details, simpler grammar and vocabulary, and the absence of word order or spelling errors. however, these characteristics can be mitigated through iterative prompt optimization, which makes detection less reliable. to address this, we aim to develop a more generalized detection method that minimizes the risk of counterdetection. since large models are pretrained on next-token prediction tasks, machine-generated text inherently exhibits high-probability characteristics. this feature remains consistent across texts generated by different models or under various prompt conditions. specifically, we leverage the [cls] vector of the roberta @xcite model as the semantic feature of the text and use llama3 @xcite model to calculate the difference between the probability of the actual next token and the predicted next token at each token position, obtaining a vector as the probabilistic feature. by concatenating these two feature sets and feeding them into a feedforward network for binary classification, as illustrated in figure 1 , we achieve robust detection performance. ## related works the task of detecting machine-generated text is essentially a binary classification problem. @xcite provide a comprehensive overview of the field of llm-generated text detection, thoroughly examining the necessity of this task. they categorize detection techniques into watermarking methods, statistical detectors, neural network-based detectors, and human-assisted approaches, and also list available data resources. suvra ghosal et al. (2023) conducted a similarly excellent review, focusing on the possibilities and limitations of text inspection. they categorize detection frameworks into a priori and post hoc detectors, as well as analyzing novel attack strategies for evading detection in machine-generated text. due to challenges in achieving consistency and widespread adoption of watermarking methods, post hoc detection is currently the only feasible approach for real-world detection tasks. this approach is further divided into zero-shot detection and fine-tuned model detection, with the latter being the category of this shared task. zero-shot detection does not require labeled datasets. typically, it involves calculating entropy, perplexity, n-gram frequency, or the average log probability per token of a given sequence, followed by thresholding. @xcite detect machine-generated text by examining the curvature of a language model's log probability function. they generate perturbations of a given text sample, compares their log probabilities with the original text, and identifies machine-generated text based on a higher discrepancy metric. yang et al. (2023) detects machine-generated text by truncating a text in the middle, regenerating the remainder using a language model, and then analyzing n-gram differences between the original and newly generated text segments. fine-tuned model detection, on the other hand, trains binary classifiers using features extracted from pretrained language models. petukhova et al. (2024) combine roberta-base embeddings with diversity features and resample the training set. @xcite calculate the log probability of tokens using a series of weaker language models, generates additional synthetic features through vector and scalar operations, and uses a logistic regression classifier to detect machine-generated text based on these features. ## method as shown in figure 1 , our model is divided into two stages. in the first stage, we perform supervised learning for binary classification using the roberta model, aiming to enhance the [cls] vector of the roberta model with features relevant to the task of detecting machine-generated text. in the second stage, we freeze the parameters of the roberta model and obtain the [cls] vector for each text as the semantic feature h s . for a given text @xmath0, . . . , x n ], where n is the token length of the text, we freeze the parameters of the llama3-8b-instruct model and compute the probabilistic features h @xmath1, . . . , h n ], where h i is calculated according to equation 1: that is, under model θ, the probability of predicting the next token x i+1 given the prefix @xmath2. v represents the entire vocabulary. for the different dimensions h i of the probabilistic features h p for the same text, we performed normalization, as shown in equation 2 : we compute the mean of the probabilistic features h p for a text. the distribution of the probabilistic features mean is illustrated in figure 2 , where we can observe that machine-generated text tends to follow high-probability sampling for the next token, whereas human-authored text does not exhibit this distinct characteristic. the semantic features h s and probabilistic features h p are first subjected to dimensionality reduction individually. these reduced vectors are then concatenated to form a unified representation. this concatenated representation is subsequently processed through a series of linear layers. finally, a softmax activation function is applied to produce the final label predictions. ## experiments as shown in figure 3 , the text lengths in the dataset are primarily concentrated around 500 words. in the first stage illustrated in figure 1 , we uniformly truncate texts to the first 512 tokens and experiment with four models: roberta, robertalarge, deberta @xcite , and debertalarge. we use the baseline script for training, with hyperparameters set as follows: a learning rate of 2e-5, batch size of 16, three epochs, and an l2 weight regularization of 0.01. on the validation set, roberta-large achieved the best performance, with comparative results shown in in the second stage, we select roberta-large to extract the [cls] vector with a dimension of 1024. the text is again truncated to the first 512 tokens and input into llama3-8b-instruct to compute the probabilistic feature vector with a dimension of 512. we then train a feedforward neural network with three hidden layers and relu activation functions. the first layer reduces both features to 128 dimensions, which are then concatenated. the second layer further reduces the dimensionality to 64, and the final layer reduces it to 2 classes. we use a learning rate of 1e-4 and a dropout rate of 0.5. this approach achieves a macro f1 score of 0.8980 on the validation set. our experiments were conducted using an nvidia geforce rtx 4090 24gb. it is evident that using the same generative model as the text source for computing the probabilistic features in the second stage would yield better results. however, on the one hand, the dataset for the competition does not originate from a single model, and on the other hand, in real-world scenarios, we cannot know the potential model source of the text. we chose to use llama3-8b-instruct for computing the probabilistic features because the llama series models have had a significant influence in the open-source model domain. many subsequent open-source models have been affected by it and may have been trained on the same general datasets, leading to similar probability distributions in text generation. additionally, llama3-8b-instruct performs exceptionally well in the english domain. due to the large scale of the competition dataset and our limited computational resources, we did not conduct comparative experiments using other large models for probabilistic feature extraction. although we did not participate in the final submission for subtask b, we conducted experiments on the validation set for this subtask. we used a combination of xlm-roberta @xcite and llama3-8b-instruct, achieving a score of 0.6766 compared to the baseline of 0.6546 for subtask b. this result suggests that probabilistic features can be helpful for detecting multilingual text, but the current model framework does not perform outstandingly. figure 3 : the x-axis represents the text length segmented by words, while the y-axis shows the probability density corresponding to each length. it can be observed that the text lengths in both the training and validation sets are primarily concentrated within 500 words. ## conclusion in this work, we proposed a two-stage detection system for machine-generated text. by integrating semantic features from roberta with probabilistic features from llama3, our system achieves a macro f1 score of 0.7713 on the test set, ranking ninth overall. our experiments confirmed the effectiveness and generalizability of this feature integration approach. compared to average results, our proposed system demonstrates robustness and strong generalization capability, which we aim to further enhance in future work.