Werkzeug at S em E val-2024 Task 8: LLM -Generated Text Detection via Gated Mixture-of-Experts Fine-Tuning (2024)

Abstract: Recent advancements in Large Language Models (LLMs) have propelled text generation to unprecedented heights, approaching human-level quality. However, it poses a new challenge to distinguish LLM-generated text from human-written text. Presently, most methods address this issue through classification, achieved by fine-tuning on small language models. Unfortunately, small language models suffer from anisotropy issue, where encoded text embeddings become difficult to differentiate in the latent space. Moreover, LLMs possess the ability to alter language styles with versatility, further complicating the classification task. To tackle these challenges, we propose Gated Mixture-of-Experts Fine-tuning (GMoEF) to detect LLM-generated text. GMoEF leverages parametric whitening to normalize text embeddings, thereby mitigating the anisotropy problem. Additionally, GMoEF employs the mixture-of-experts framework equipped with gating router to capture features of LLM-generated text from multiple perspectives. Our GMoEF achieved an impressive ranking of #8 out of 70 teams. The source code is available on https://gitlab.com/sigrs/gmoef.

## introduction the advancements in large language models (llms) have made generating human-level text more accessible and cost-effective than ever before. these advancements, coupled with techniques such as chain-of-thought @xcite and instruction tuning @xcite , have enabled llms in producing high-quality text on various topics. however, in the real world, using llmgenerate text is not always acceptable. thus, there is an urgent need for an easy yet reliable way to detect llm-generated text. the semeval-2024 task 8 @xcite aims to find methods that can detect machinegenerated text. in this work, we followed the most common black-box detection paradigm, which regards such problem as a classification task. we argue that current methods all suffer from the following issues: (1) anisotropy of the text embeddings @xcite @xcite . using small pretrained language models (plms) to encode text is the very first step for all classification models, however, plm may suffer from anisotropy issue, which makes text embeddings clustering in a small cone in the latent space, and compromise the classification performance. (2) language style of llm-generated text is dynamic. as aforementioned, llm can generate text that accommodates various topics and contexts; different llm may have different optimization targets during pre-training w.r.t. text generation. in other words, finding a regular pattern for llm-generated text is difficult. to this end, we propose gated mixture-of-experts fine-tuning (gmoef) to tackle these problems. gmoef first uses the plm to encode the text, then employs parametric whitening transformation to normalize the embedding distribution, in order to mitigate the anisotropy issue; furthermore gmoef adopts mixture-of-experts equipped with gating router to capture features of llm-generated text from multiple perspectives. our gmoef achieved an impressive ranking of #8 out of 70 participating teams on subtask b. man writing @xcite , making it even challenging for humans to differentiate between llm-generated text and human-written text. consequently, there is a need to develop effective detectors to mitigate the potential misuse of llm @xcite . recently, owing to the construction of numerous high-quality benchmarks and innovations in detection methods, significant progress has been made in llm-generated text detection technology. high-quality datasets play a crucial role in advancing research on detecting llm-generated text. hc3 dataset @xcite , represents one of the pioneering open-source efforts aimed at comparing chatgpt-generated text with humanwritten text.the cheat dataset @xcite comprises academic abstracts written by humans sourced from ieee xplore, and is committed to detecting artificially generated deceptive academic content from chatgpt. additionally, there are numerous datasets containing text generated by various llms, such as monolingual datasets deepfaketext-detect-dataset @xcite , gpt-written dataset @xcite , and m4 @xcite , used in this competition. focusing on recently proposed detection methods, these primarily encompass zero-shot @xcite , fine-tuning lms @xcite , adversarial learning @xcite , and llms as detectors @xcite . detectgpt @xcite is dedicated to the detection of llm-generated text by analyzing the structural attributes inherent in the probability functions of llms. @xcite noted that fine-tuning roberta @xcite resulted in optimal classification outcomes across diverse encoding configurations. recent studies @xcite have additionally supported the outstanding performance of fine-tuned variants within the bert family, such as roberta, in discerning llm-generated text. @xcite conducted an adversarial data augmentation process on llmgenerated text, and the results showed that models trained with augmented data exhibited enhanced robustness. ## methodology in this section, we present our gmoef in details. we first introduce the overall architecture of the proposed gmoef, then give a comprehensive insight of the adopted parametric whitening and gated mixture-of-experts ## conclusion and future work in this work, we find out that current llmgenerated text detection methods may suffer from anisotropy issue, and they fail to capture the dynamic language features. to this end, we propose gmoef, which incorporates parametric whitening to mitigate the anisotropy issue. gmoef further adopts the mixture-of-experts equipped with gating router to model the pattern of llm-generated text from multiple aspects. our gmoef exhibits an impressive #8 out of 70 participating teams on the multi-model generated text detection subtask. extensive experiments show that our gmoef is suitable for complicated scenarios where texts are multi-lingual and may generated by multiple possible llms. in the future, we aim to extend our observations to other text classification tasks, and incorporate llm itself to detect machine-generated text.