T rust AI at S em E val-2024 Task 8: A Comprehensive Analysis of Multi-domain Machine Generated Text Detection Techniques (2024)

Abstract: The Large Language Models (LLMs) exhibit remarkable ability to generate fluent content across a wide spectrum of user queries. However, this capability has raised concerns regarding misinformation and personal information leakage. In this paper, we present our methods for the SemEval2024 Task8, aiming to detect machine-generated text across various domains in both mono-lingual and multi-lingual contexts. Our study comprehensively analyzes various methods to detect machine-generated text, including statistical, neural, and pre-trained model approaches. We also detail our experimental setup and perform a in-depth error analysis to evaluate the effectiveness of these methods. Our methods obtain an accuracy of 86.9% on the test set of subtask-A mono and 83.7% for subtask-B. Furthermore, we also highlight the challenges and essential factors for consideration in future studies.

## introduction recent advancements in large language models (llms) have facilitated a wide range of applications, notably in content generation @xcite . while llms offer creative and informative content generation capabilities, concerns such as misinformation, fake news, personal information leakage, legal and ethical issues have emerged @xcite @xcite . consequently, detecting machine-generated text has become a crucial task to address these aforementioned challenges. the identification of machine-generated text is still an open challenge because of its overlapping similarities with human-written text. the current text generation models produce text that is strikingly similar to human language in terms of grammaticality, coherency, fluency, and utilization of real-world knowledge @xcite @xcite . however, variations in sentence length, the presence of noisy data, and the generation of incomplete sentences are common indicators of machine-generated text. ## related work recent works on llm-generated foot_0 text detection has shown promising results. statistical methods are used to detect the llm-generated text by utilizing the entropy @xcite , and n-gram frequency @xcite . some other studies uses the fact that language models assign high probability for the repeated sentences which is often ai model generated and ranks the ai model generated sentence @xcite . in a study, openai has trained a classifier to detect llmgenerated text using the roberta-based model @xcite . some of the widely-used methods adopted the gpt detectors such as openai detection classifier foot_1 , gptzero foot_2 , and zerogpt foot_3 . another variant is detectgpt @xcite , which works on the assumption of llm-generated text lies in the negative curvature region of the log-likelihood. using this approach, detectgpt perturbs the input text using masked language models, such as bert @xcite , bart @xcite , t5 @xcite and compare the log probability of the text and masked filled variants. similarly, few works utilized the different decoding strategies including top-k, nucleus, and temperature sampling to generate the text from gpt2 and bert based models employed to perform binary classification to label text as human-written or machine generated @xcite . recently, watermarking methods have been used in enterprises to protect the intellectual properties and fair use of the generation models. however these techniques simplify the detection of the llmgenerated output text by synonym replacement over generated outputs and text level posthoc lexical substitutions @xcite , and soft watermarking was introduced in @xcite using green and red token lists. hidden space operations were also introduced by injecting secret signals into the probability vector of each target token @xcite . @xcite proposed a method which triggers when the text has common words randomly assembled as it is easier to find than identifying unique and rare tokens. sadasivan et al. (2023) focused on zero-shot ai text detection by using two clusters depending on watermarked or not. another study @xcite , proposed a benchmark framework consists of an input module, a detection module and an evaluation module for machine generated text detection against humanwritten text. in contrast to existing works, this study presents the multi-domain multi-lingual machine generated text detection techniques. ## datasets this section given an overview of the dataset utilized and the corresponding analysis. ## system overview this section offers various approaches employed to perform machine-generated text identification. our approaches are categorized into 1) statistical, 2) neural, and 3) pre-trained models. ## results and analysis this section provides a detailed analysis of the models utilized for subtasks a and b. our experiments aim to showcase the effectiveness of several machine-generated text detection techniques. ## conclusions the study explores different methodologies for detecting machine-generation text, leveraging statistical, neural, and pre-trained models. we observe that the ensemble models are more effective in classifying the mono-lingual data (subtask-a mono), while models trained on gpt2-text surpass other models in multi-class classification.