AISPACE at S em E val-2024 task 8: A Class-balanced Soft-voting System for Detecting Multi-generator Machine-generated Text (2024)

Abstract: SemEval-2024 Task 8 provides a challenge to detect human-written and machine-generated text. There are 3 subtasks for different detection scenarios. This paper proposes a system that mainly deals with Subtask B. It aims to detect if given full text is written by human or is generated by a specific Large Language Model (LLM), which is actually a multi-class text classification task. Our team AISPACE conducted a systematic study of fine-tuning transformer-based models, including encoder-only, decoder-only and encoder-decoder models. We compared their performance on this task and identified that encoder-only models performed exceptionally well. We also applied a weighted Cross Entropy loss function to address the issue of data imbalance of different class samples. Additionally, we employed soft-voting strategy over multi-models ensemble to enhance the reliability of our predictions. Our system ranked top 1 in Subtask B, which sets a state-of-the-art benchmark for this new challenge.

## introduction large language models (llms) have emerged as a foundational element for artificial intelligence (ai) applications. their text generation capabilities are impressive and have almost reached the humanlevel performance. however, their widespread use also poses risks. the use of llm-generated text can lead to the spread of inaccurate information, academic dishonesty, and privacy breaches. additionally, the machine-generated text may become trapped in a loop during the process of llms' own development, gradually replacing human-written training data and reducing the quality and diversity of subsequent models @xcite . to prevent the misuse of llms and improve the iterative refinement of ai tools, it is crucial to distinguish between machine-generated and humanwritten text. semeval-2024 task 8: multigener-ator, multidomain and multilingual black-box machine-generated text detection @xcite introduces the task of detecting machinegenerated text across various generators, domains and languages. our system focuses on subtask b, which is a multi-class classification task. it involves detecting text from multi-generators over multi-domains in english only. given a text, the system tells whether the text is written by a human or generated by a particular llm. it emphasizes not only the accuracy of detecting the in-domain texts, but also the generalization to identify other out-of-domain text sources. current research on llms text detection primarily focuses on chatgpt or a specific model in a limited domain. @xcite scientific writing between humans and chatgpt exclusively. @xcite detects ai-generated news by chatgpt. however, there are many other emerging llms that generate various domain texts that needed to be distinguished from those written by humans. @xcite presents a large-scale corpus generated by popular llms, including chat-gpt, cohere, davinci, bloomz, and dolly, across various domains such as wikihow, wikipedia, arxiv, peerread, and reddit. to address this complex scenario, we fine-tune transformer-based encoder-only, decoder-only, and encoder-decoder models.we then ensemble the model that performs best for a specific class and mitigate sample imbalance using a weighted loss function. our contributions can be summarized as follows: ## related work researchers have explored automatic detection methods for distinguishing machine-generated text from human-written text. these methods can be categorized into two distinct groups, i.e., metricbased methods and model-based methods @xcite . ## system overview based on the analysis of the task situation, we have carried out preliminary studies of several methods and integrated pre-trained language models finetuning, class-balanced weight loss function, and soft-voting model ensemble into our system. ## results and analysis to assess the efficacy of our proposed methods, we carried out multiple sets of experiments. on the baseline roberta-large, an ablation study was conducted. the performance comparison is shown in further, we fine-tuned a series of transformerbased models to select the most suitable base model. the results in at last, we conducted a model ensemble by soft voting method to ensure robustness and generalization and reduce the effect of noise. the selected single base fine-tuned model is chosen based on its performance in the specific class. we tested various combinations, and the results are shown in we attempted to combine various single base models, including 2, 3, and 4 types. compared to the best single model, the ensembled model showed significant improvement, even with the least number of ensemble types. furthermore, as the differences in ensemble models increased, the results improved even further. additionally, if the ensemble base models perform well individually in every class, the overall result is also improved. ## conclusion this paper presents a systematic study on detecting machine-generated text from multi-generators and multi-domains. we fine-tuned a series of transformer-based models and found that the encoder architecture is better suited for the task. we employed a weighted cross entropy loss function to address the sample imbalance. to improve robustness and generalization, various base models were ensembled by soft-voting method, and resulting in 99.46% accuracy on the dev set. in the final test, our system ranked 1st. moving forward, we plan to explore more widely used llms and work towards enhancing our capabilities in fewshot learning and transfer-learning for similar tasks.