I 2 C - H uelva at S em E val-2024 Task 8: Boosting AI -Generated Text Detection with Multimodal Models and Optimized Ensembles (2024)

Abstract: With the rise of AI-based text generators, the need for effective detection mechanisms has become paramount. This paper presents new techniques for building adaptable models and optimizing training aspects for identifying synthetically produced texts across multiple generators and domains. The study, divided into binary and multilabel classification tasks, avoids overfitting through strategic training data limitation. A key innovation is the incorporation of multimodal models that blend numerical text features with conventional NLP approaches. The work also delves into optimizing ensemble model combinations via various voting methods, focusing on accuracy as the official metric. The optimized ensemble strategy demonstrates significant efficacy in both subtasks, highlighting the potential of multimodal and ensemble methods in enhancing the robustness of detection systems against emerging text generators.

## introduction in the era of digital communication, ai-based text generators have become increasingly sophisticated, necessitating advanced detection methods to differentiate between human and machine-generated content @xcite . this paper addresses the challenge within the scope of english language texts, emphasizing the importance of reliable detection mechanisms in maintaining the integrity of digital discourse. the task at hand is crucial for various applications, including content moderation, misinformation prevention, and ensuring the authenticity of digital communication. the core strategy of this system lies in its robustness and the optimization of model training. by limiting the size of the training dataset, the approach prevents models from overfitting to specific text generators, thereby enhancing their generalizability to novel content. furthermore, the system leverages multimodal models that integrate traditional nlp techniques with numerical text features, such as lexical diversity and sentence structure, to enrich the detection capabilities. this is complemented by a rigorous exploration of ensemble methods and voting mechanisms to optimize model performance. specifically, our conception of multimodal entails the strategic fusion of traditional, fine-tuned language models like roberta with extracted numerical values from the text, such as number of grammatical errors and average sentence length, thereby enriching the language models with quantifiable text insights to enhance detection precision. participation in this task led to the system ranking 47th in the monolingual subtask a with an accuracy of 0.8079 and 18th in the multilabel subtask b with an accuracy of 0.789 @xcite . these outcomes affirm the system's robustness in handling diverse generative models. however, the primary challenge encountered was distinguishing between texts produced by similar generators. the system struggled to consistently differentiate between certain generators, often misattributing texts to one over another when faced with stylistically comparable outputs. this difficulty in discerning subtle variations between generator styles points to the need for further refinement in the detection algorithm, suggesting an area for future research to enhance the sensitivity and specificity of the model. as shown in figure 1 , the double bar graph compares the original label distribution and the predicted label distribution. the labels represent the model that generated each text of the samples, being 0 human, 1 chatgpt, 2 cohere, 3 davinci, 4 bloomz and 5 dolly. ## background the task of detecting machine-generated text has garnered significant attention due to the rapid advancement and widespread use of ai-based text generators. the input for this task consists of textual content, with the output being a classification decision indicating whether the text is human or machine-generated in subtask a and which generator created the text in subtask b. for this study, the dataset comprised english texts from diverse sources, including wikipedia (march 2022 version), wikihow, reddit (eli5), arxiv, and peerread @xcite . the machine-generated texts were produced using leading multilingual large language models (llms) such as chatgpt, textdavinci-003, llama, flant5, cohere, dolly-v2, and bloomz. these models were prompted to create content resembling the human-written texts from the mentioned sources, ranging from wikipedia articles to peer reviews and news briefs, ensuring a rich variety of genres and styles within the dataset. this richly varied dataset forms the foundation of the analysis, drawing on the comprehensive compilation of machine-generated texts as detailed in the work by @xcite . as shown in figure 2 , the training data distribution illustrates the sources and quantity of data used in the study. this work focuses solely on the english portion of the dataset, engaging in the monolingual classification track. the choice of english allows for a concentrated examination of the nuances in detecting machine-generated texts in a language with extensive generative model research and develop-ment. the task setup and dataset composition are pivotal in understanding the challenges and innovations presented in this study. this work builds upon foundational efforts in the field, such as "machine-generated text detection using deep learning" by raghav gaggar et al. @xcite , which emphasizes deep learning approaches for distinguishing ai-generated content. gaggar's methodology leverages traditional neural network architectures, providing a critical basis for understanding how machine learning can be applied to text detection challenges. similarly, "on the possibilities of ai-generated text detection" by souradip chakraborty et al. @xcite contributes to the discourse by establishing theoretical frameworks based on information theory, highlighting the nuanced differences between human and ai-generated texts and the implications for detection mechanisms. this paper underscores the importance of sample complexity and the robustness of detection systems to new and evolving text generators. "ghostbuster: detecting text ghostwritten by large language models" by vivek verma et al. @xcite methodology employs a series of weaker language models to compute token generation probabilities, offers a specialized perspective on model-agnostic detection. in contrast, this work extends the discourse by incorporating numerical text features alongside conventional nlp techniques within a multimodal framework, providing a more holistic analysis of text characteristics. this integration allows for a more nuanced distinction between human and ai-generated texts, addressing the challenges of style and generator diversity that single-model systems may struggle with. ## system overview the system is designed to detect machinegenerated text, combining an ensemble of finely-tuned transformer models such as roberta ('facebookai/roberta-base') @xcite , electra ('google/electra-base-discriminator') @xcite , albert (albert/albert-base-v2) @xcite , roberta-base-openai-detector ('roberta-base-openai-detector'), chatgpt-detectorroberta ('hello-simpleai/chatgpt-detectorroberta') and bert ('bert-base-uncased') @xcite with custom adaptations of roberta including a one-vs-all system, that independently predicts each label's presence, treating each label as a separate binary classification problem, and multimodal models . a random forest classifier is used to analyze numerical text features. this ensemble integrates outputs from each model by aggregating predictions and confidence levels through various voting mechanisms. the process identifies the best combination of models and voting method, optimizing the ensemble to achieve the highest detection accuracy and robustness. to optimize training samples for robustness, the number of samples used to train each model were systematically varied, aiming to find an optimal size that enhances robustness to new text generators while preventing overfitting. for instance, in the case of the albert model, training began with 500 samples, then the model was reset and trained again with increasing sizes: 1000, 2000, 5000 samples, and so on. this process revealed that smaller sample sizes increased the model's robustness. it was determined that the ideal average number of samples for binary classification was 10,000, whereas multilabel classification required a larger average of 48,000 samples to maintain high predictive accuracy without compromising robustness to unseen generators in the evaluation dataset. ## experimental setup in this study, the optimization of the training sample size was a critical preliminary step before proceeding with the standard division of the dataset for model training and evaluation. the objective was to determine the most effective training sample size that would enable the models to learn sufficiently from the data without overfitting. this involved iterative testing of various sample sizes to identify the optimal balance that maximized model performance on unseen data. once the ideal training sample size was established, it was then split following an 80-20 ratio, with 80% of the samples used for training and the remaining 20% for evaluation. the dev dataset served as the test set throughout the experiments, ensuring a consistent benchmark for evaluating the generalization ability of the models across different configurations and optimizations. fine-tuning the models was conducted with careful consideration of hyperparameters that directly influence model performance. the hyperparameters were determined by experimenting with a range of values and choosing those that led to better performance metrics. this approach aimed to enhance the model's ability to be robust to new text generators that were present in the evaluation dataset but not in the training dataset. the learning rate was set to 2e-5, a value chosen to ensure steady yet effective model updates without causing large fluctuations in model weights that could hinder learning. the batch size for both training and evaluation phases was maintained at 16, balancing computational efficiency with the need for granularity in gradient updates. the models underwent training for 3 epochs, a decision underpinned by the desire to minimize overfitting while allowing sufficient iterations for the models to converge to an optimal state. weight decay was applied at 0.01 to regularize the model and further mitigate overfitting. the training process incorporated an epoch-based evaluation and save strategy, enabling continuous monitoring of model performance and retention of the best-performing model state at each epoch's conclusion, as determined by evaluation metrics. the experimental framework utilized pytorch for implementing the transformer-based models, specifically leveraging the roberta-base model from the hugging face transformers library for both the multimodal models and the one-vs-all classification approach in the multilabel subtask. for numerical feature extraction from text, the nltk library was employed, enriching the model inputs with linguistic features that provide additional context and depth to the analysis. the numerical model, built using a random forest classifier, was optimized using the scikit-learn library, demonstrating the integration of traditional machine learning techniques with advanced nlp models for enhanced predictive performance. as depicted in figure 7 , the diagram illustrates the experimental setup for training, showcasing the steps and pipelines involved in the process. ## results the system demonstrated commendable performance in the task, adhering to the official evaluation metric of accuracy. in subtask a (monolingual classification), the system attained an accuracy of 0.8079, placing it at the 47th position in the competition. this ranking underscores the system's capability to effectively distinguish between human and machine-generated texts in a monolingual setting. for subtask b (multilabel classification), the system achieved an accuracy of 0.789, ranking 18th out of the total number of participants. this notable performance highlights the system's robustness and effectiveness in handling more complex multilabel scenarios, despite the inherently in a comprehensive evaluation using the evaluation dataset, tables comparing model performances shed light on the system's effectiveness. for subtask a, comparisons between various models in binary classification, and specifically between the roberta-base model and its multimodal extensions, reveal the somewhat superior performance of the multimodal models. these models, incorporating key numerical features, mostly outperformed other fine tuned classificators. a similar trend was observed in subtask b's multilabel classification, where multimodal models again demonstrated some enhanced accuracy. this data, while not from the final test set, underscores the potential of multimodal approaches in effectively distinguishing between human and machine-generated texts across different classification scenarios. ## conclusions this study showcased innovative techniques aimed at enhancing model robustness in the task of detecting machine-generated text, notably through the careful optimization of training sample size, the strategic assembly of diverse models into optimized ensembles, and the deployment of multimodal mod-els. these methodologies collectively facilitated a system that adeptly navigates the challenges of monolingual and multilabel classifications. the exploration of training sample sizes revealed a delicate balance between sufficient model training and the avoidance of overfitting, highlighting the importance of dataset optimization. the ensemble model's success, derived from combining models with varying strengths, emphasizes the value of diversity in model architecture for robust performance. moreover, the integration of multimodal models, blending traditional nlp techniques with numerical text features, showcased a sophisticated approach to capturing the nuanced distinctions between human and machine-generated texts. looking ahead, the focus will be on refining these novel techniques to further bolster model robustness. future work will explore more granular adjustments to training sample sizes and investigate the potential of dynamic ensemble configurations responsive to the nature of the text being analyzed. additionally, the extension of multimodal model frameworks to incorporate emerging linguistic and semantic features presents a promising avenue for enhancing detection capabilities. applying these advanced methodologies to other areas of model building could advance the landscape of machine learning, offering a blueprint for developing systems that are not only robust but also universally applicable across various nlp tasks and challenges.