USTC - BUPT at S em E val-2024 Task 8: Enhancing Machine-Generated Text Detection via Domain Adversarial Neural Networks and LLM Embeddings (2024)

Abstract: This paper introduces the system developed by USTC-BUPT for SemEval-2024 Task 8. The shared task comprises three subtasks across four tracks, aiming to develop automatic systems to distinguish between human-written and machine-generated text across various domains, languages and generators. Our system comprises four components: DATeD, LLAM, TLE, and AuDM, which empower us to effectively tackle all subtasks posed by the challenge. In the monolingual track, DATeD improves machine-generated text detection by incorporating a gradient reversal layer and integrating additional domain labels through Domain Adversarial Neural Networks, enhancing adaptation to diverse text domains. In the multilingual track, LLAM employs different strategies based on language characteristics. For English text, the LLM Embeddings approach utilizes embeddings from a proxy LLM followed by a two-stage CNN for classification, leveraging the broad linguistic knowledge captured during pre-training to enhance performance. For text in other languages, the LLM Sentinel approach transforms the classification task into a next-token prediction task, which facilitates easier adaptation to texts in various languages, especially low-resource languages. TLE utilizes the LLM Embeddings method with a minor modification in the classification strategy for subtask B. AuDM employs data augmentation and fine-tunes the DeBERTa model specifically for subtask C. Our system wins the multilingual track and ranks second in the monolingual track. Additionally, it achieves third place in both subtask B and C.

## introduction the burgeoning capabilities of large language models (llms), exemplified by chatgpt @xcite , @xcite and llama @xcite , have made machine-generated text * corresponding author: zhendong mao. more fluent and human-like, which has led to an increasing concern about the abuse of llms such as misinformation spread @xcite @xcite and disruption in education system @xcite . so far humans perform only slightly better than chance when distinguishing between text generated by llms and human @xcite , so it calls for an automatic system to identify machine-generated text. to this end, mbzuai nlp department holds semeval-2024 task8, which consists of three subtasks. subtask a focuses on determining whether a full text is human-written or machine-generated. the biggest challenge lies in the domain difference between the training set and test set while the multilingual track requires strong adaptation to text across various languages. it demands the model to have a strong capability of generalization in out-of-domain scenarios. subtask b aims at doing multi-way machine-generated text detection and brings a new challenge of identifying the text source without knowing the domains in the test set. subtask c proposes human-machine mixed text detection, which gives a text where the first part is human-written and the second part is machinegenerated. the goal is to robustly determine the boundary where the change occurs using a fairly small training set. a detailed description can be found in the task description paper @xcite . currently, training-based machine-generated text detection strategies such as fine-tuning roberta model @xcite underperform in the out-of-domain scenario. prominent zero-shot methods @xcite can only discriminate whether a text is produced by a specific llm or by a human. sniffer @xcite and seqxgpt @xcite appear to handle the problem of subtask b and c, but the claimed result is based on the assump-tion that we know the origins of the text. therefore, we propose our system, which consists of domain-adaptive text detection (dated), llm-powered language-aware model (llam), threestage llm embeddings (tle) and augmented deberta model (audm). it performs outstandingly in out-of-domain scenarios, especially in subtask a. in dated, we enhance performance by innovatively incorporating domain adversarial neural networks (dann) @xcite into the task of machine-generated text detection. dann consists of a feature extraction layer and a category predictor, forming the backbone network to predict classification labels. furthermore, the domain classifier is connected to the backbone network through a gradient reversal layer for classifying domain labels. this enables the model to learn transferable features between the training and development set, effectively overcoming challenges in out-of-domain scenarios. we achieve second place out of 126 participants. in llam, we handle text from various languages in a distinct manner. for english text, we employ llm embeddings, leveraging the powerful representation capabilities of llm by directly extracting embeddings from the last layer of a proxy llm, and we classify the text using a two-stage cnn. for text in other languages, we utilize llm sentinel, which reframes the classification task as a next-token prediction task. we win the first place in the track among 59 participants. in tle, we utilize the llm embeddings method mentioned above, with only a minor difference in the classification strategy. we rank third out of 70 participants. in audm, we fine-tune a deberta-base @xcite model with a linear layer for token classification. this is an easy but effective system and ranks third out of 30 submissions. in short, our contributions are as follows: ## system overview our system consists of four components: dated, llam, tle and audm. each of these components addresses one of the four tracks of the task respectively. in subtask a, we aim to do machine-generated text detection on mono/multilingual data. in the monolingual track, the domains in test set differ a lot from ones in training set. we innovatively apply dann to the detection in dated. this is achieved by adding a gradient reversal layer on top of the base model. additionally, besides category labels, we incorporate extra domain labels into the dataset (training set: 0, development set: 1), enabling the model to learn transferable features between the training and development set (section §3.1). in the multilingual track, another challenge is that the model is supposed to have robust gener-alization capabilities to adapt to the distinct characteristics of different languages, especially lowresource languages. we propose llam, which employs different methods for different languages. for text identified as english, we feed it into a proxy llm to extract embeddings from the last layer and subsequently pass it through a two-stage cnn for classification. in the case of non-english text, we redefine the classification task as a nexttoken prediction task (section §3.2). in multi-way text detection, texts originate from human, chatgpt, cohere, davinci, bloomz and dolly @xcite . the challenge lies in distinguishing texts generated by various llms. therefore we conduct a three-stage classification based on the llm embeddings method (tle) mentioned in the multilingual track to better fit the scenario of multi-classification (section §3.3). in human-machine mixed text detection, the target is to do fine-grained detection. inspired by the wide use of bert @xcite in sequence labeling task, we fine-tune a deberta model with data augmentation (audm) to classify each token in a text (section §3.4). ## conclusion in conclusion, this paper presents the development and performance of our system for the semeval-2024 task 8. our system wins the multilingual track and secures second place in the monolingual track. additionally, we attain third place in both subtask b and subtask c. we demonstrate the efficacy of incorporating dann, which significantly enhances out-of-domain accuracy by introducing a gradient reversal layer and integrating additional domain labels. leveraging llm embeddings proves to be a straightforward yet effective method, harnessing the representation capabilities of llm without fine-tuning. furthermore, our implementation of llm sentinel exhibits remarkable performance, especially in low-resource language scenarios. in the future, we plan to investigate the application of dann to multi-label classification scenarios and explore more effective strategies to leverage llm embeddings.