Human vs Machine: An Automated Machine-Generated Text Detection Approach (2024)

Abstract: With the advancement of natural language processing (NLP) and sophisticated Large Language Models (LLMs), distinguishing between human-written texts and machine-generated texts is quite difficult nowadays. This paper presents a systematic approach to classifying machine-generated text from human-written text with a combination of the transformer-based model and textual feature-based post-processing technique. We extracted five textual features: readability score, stop word score, spelling and grammatical error count, unique word score and human phrase count from both human-written and machine-generated texts separately and trained three machine learning models (SVM, Random Forest and XGBoost) with these scores. Along with exploring traditional machine-learning models, we explored the BiLSTM and transformer-based distilBERT models to enhance the classification performance. By training and evaluating with a large dataset containing both human-written and machine-generated text, our best-performing framework achieves an accuracy of 87.5%.

## introduction in recent years the applications of large language models (llms) such as chatgpt foot_0 , microsoft copilot foot_1 , google gemini foot_2 , etc. have been widely popular among high school students to aged professionals. people can perform any task using those llm applications such as answering a question, summarising a chapter, checking grammatical errors and many other things which elevates their overall productivity and saves time. however, a few users also misuse the power of llm applications by doing homework assignments with chatgpt or similar kinds of ai chatbots, paraphrasing an article from one's original article etc. which are likely to be unethical practices. moreover, with the increasing advancement of deep learning, generative ai and llms it is quite hard nowadays to distinguish whether a text is written by a human or an llm-integrated chatbot. the main objective of this paper is to identify a text whether it is written by a human or it is generated by some llm integrated chatbot (machine-generated text). we employed different schemes to classify human-written text from machine-generated text starting from simple textual features to a deep learning model to a combination of deep learning and textual features. the main contributions in this paper can be summarized as follows: ## related work although ai-generated text detection is a comparatively new domain in the field of nlp, a number of methods have been proposed by several researchers in the last couple of years. a statistical method based machine-generated text detention tool named 'gltr' was proposed by @xcite . @xcite extracted several textual features and experimented with three machine-learning models to classify human-written text from chatgpt-generated texts. ippolito et al. (2020), guo et al. (2023 bhattacharjee and liu (2023) employed chat-gpt with prompt engineering to identify humanwritten text and machine-generated text. @xcite proposed a zero-sort based machinegenerated text detection approach (detectgpt) using negative log probabilities. @xcite proposed another zero-sort based machine-generated text detection approach named 'fast-detectgpt' using conditional probability curvature. their proposed approach outperforms the performance of the detectgpt model and increases the machinegenerated text detection process by 340 times compared to detectgpt. @xcite proposed an ai text detection tool 'radar' using adversarial learning taking paraphrasing of text into account. @xcite proposed a llm generated text detection method using next-word probabilities. @xcite presented a survey paper in which the authors critically explored state-of-theart machine-generated text detention tools and performed in-depth error analysis. weber-wulff et al. (2023) explored 14 different types of ai-generated text detection tools including turnitin and plagia-rismcheck type of widely used tools in their study to measure their reliability and accuracy. a benchmark consisting of a set of methods to assess the robustness of the existing ai-generated text detection models was proposed by @xcite . the authors also introduce the "ai detectability index (adi)" to measure the ai detectability of different generations of llm models. ## data all the textual analysis and experiments were performed using the data collected from the "semeval-2024 task 8: multigenerator, multidomain, and multilingual black-box machine-generated text detection" foot_3 @xcite ) shared-task. in this shared task, the organizers released both monolingual (english) and multilingual data, however, we had chosen only monolingual data for this current experiment. this dataset contains approximately 1.2 lakh training examples, of which 63,351 were written by humans and 56,406 were generated by large language models such as chatgpt, cohere foot_4 , bloomz 6 etc. along with the training data, there were also development data and gold label testing data with a sample size of 5000 and 34,272 respectively. a distribution of all data is provided in ## methodology this section discusses the proposed methodologies for this work. given a text t , our main aim is to determine whether it was written by a human (humanwritten) or generated by llm models (machinegenerated). ## conclusion in this paper, we proposed machine learning and deep learning methods to classify the 'humanwritten' and 'machine-generated' texts leveraging textual features and transformer-based distilbert model. along with the distilbert framework we also experimented with three machine-learning models and a deep-learning bilstm model. our proposed 'distilbert with post-processing' scheme provides an accuracy of 87.5% which is a superior performance than the other proposed schemes. furthermore, it is observed that the distilbert framework identifies machine-generated texts in more better way whereas the bilstm based frameworks identify human-written texts more accurately than distilbert. however, the machine learning models didn't perform well and provided an unsatisfactory result. in future, we'll evaluate our proposed 'distil-bert with post-processing' framework on other datasets and with other models such as bert @xcite and roberta @xcite to justify our claims more accurately and to validate the robustness of the proposed framework.