Team QUST at S em E val-2024 Task 8: A Comprehensive Study of Monolingual and Multilingual Approaches for Detecting AI -generated Text (2024)

Abstract: This paper presents the participation of team QUST in Task 8 SemEval 2024. we first performed data augmentation and cleaning on the dataset to enhance model training efficiency and accuracy. In the monolingual task, we evaluated traditional deep-learning methods, multiscale positive-unlabeled framework (MPU), fine-tuning, adapters and ensemble methods. Then, we selected the top-performing models based on their accuracy from the monolingual models and evaluated them in subtasks A and B. The final model construction employed a stacking ensemble that combined fine-tuning with MPU. Our system achieved 6th (scored 6th in terms of accuracy, officially ranked 13th in order) place in the official test set in multilingual settings of subtask A. We release our system code at:https://github.com/warmth27/SemEval2024_QUST

## introduction large language models (llms) enable quick, coherent responses and content creation but also raise ethical concerns about misinformation and academic integrity @xcite . to differentiate between machine-generated and human-created content, previous study @xcite has been extensively discussed in industry and academic works. semeval 2024's task 8 @xcite encourages the participants to develop an automatic system for detecting ai-generated text by leveraging an extended version of the m4 dataset @xcite . we engaged in subtasks a and b, during which we encountered the challenges of overcoming linguistic differences, data scarcity, and inadequate cross-lingual generalization capabilities. furthermore, existing multilingual models are less discussed in detecting ai-generated text, compared to monolingual ones, which further exacerbates the difficulty in model selection. meanwhile, we found that the multilingual dataset in subtask a contains data from both monolingual data and subtask b, as shown in figure 1 . to enhance the diversity and scale of the text dataset, we performed back-translation on the multilingual training set to increase the volume of monolingual data and conducted data cleaning to improve data quality. in our approach to subtasks a and b, we initially applied deep learning methods for a swift assessment in subtask a and proceeded to fine-tune multiple pre-trained language models (plms), inspired by recent studies highlighting the efficacy of fine-tuning methods in text classification. however, the training cost of the fine-tuning method is relatively high. we further utilize the adapter @xcite to parameter efficiency fine-tune (peft) @xcite the model while preserving its performance. we also noted that despite the reduced training time, the performance of adapter models was not consistently stable, showing variability across experiments. given the critical importance of model performance, decided not to utilize adapters in the testing phase. to enhance model performance and generalizability, we adopted a stacking-based ensemble learning method, utilizing the logits from the top two performing models as inputs for a linear layer to generate final predictions. finally, our experimental results on the test set show that integrating 1 463 data augmentation and ensemble learning significantly improves model efficacy in task-specific settings. ## methodology upon obtaining the dataset, we conducted a comprehensive statistical analysis of its scale and distribution. we observed that the multilingual training dataset contains both monolingual data and subtask b-related data, along with an additional portion. to enhance the model's generalization capability, we opted to augment the training data through the following steps 2: firstly, we employed google translate to uniformly translate the multilingual training dataset into english. subsequently, considering the presence of monolingual datasets within the multilingual training data and to prevent leakage of validation data, we excluded the validation dataset for monolingual tasks and subtask b from the multilingual training dataset. in accordance with the multi-class nature of subtask b, we balanced the categories of the translated multilingual training dataset. we reorganized the dataset into multiple categories to better align with the data distribution of subtask b. to ensure data quality, we conducted thorough cleaning of the datasets. following data preprocessing, we separately fed the cleaned training datasets into the respective models for training. in order to leverage the strengths of different models and enhance classification accuracy, we contemplated performing stacking. we selected the top two models based on their performance on the validation dataset and saved their generated logits. finally, we stacked these logits to obtain the ultimate results. ## conclusion in conclusion, our team developed three distinct systems for semeval-2024 task 8, targeting the monolingual and multilingual aspects of subtask a and addressing subtask b. we achieve 8th place in the multilingual setting of subtask a. we leveraged back-translation to expand the training datasets for both monolingual and subtask b. the robertabase and xlm-r models, enhanced by the mpu framework, showed improved detection of short texts in both monolingual and multilingual settings. finally, the stacking method allowed us to combine the strengths of multiple models, improving our system's predictive accuracy. future efforts will consider incorporating advancements in crosslingual pre-trained models in our subsequent work to further enhance the model's understanding of texts across diverse languages and cultural backgrounds.