Detecting AI -Generated Text with Pre-Trained Models Using Linguistic Features (2024)

Abstract: The advent of sophisticated large language models, such as ChatGPT and other AI-driven platforms, has led to the generation of text that closely mimics human writing, making it increasingly challenging to discern whether it is human-generated or AI-generated content. This poses significant challenges to content verification, academic integrity, and detecting misleading information. To address these issues, we developed a classification system to differentiate between human-written and AI-generated texts using a diverse HC3-English dataset. This dataset leveraged linguistic analysis and structural features, including part-of-speech tags, vocabulary size, word density, active and passive voice usage, and readability metrics such as Flesch Reading Ease, perplexity, and burstiness. We employed transformer-based and deep-learning models for the classification task, such as CNN_BiLSTM, RNN, BERT, GPT-2, and RoBERTa. Among these, the RoBERTa model demonstrated superior performance, achieving an impressive accuracy of 99.73. These outcomes demonstrate how cutting-edge deep learning methods can maintain information integrity in the digital realm.

## introduction the fast progress of large language models (llms), like openai's @xcite represent and other similar models developed by various organizations and start-up companies, has revolutionized the domain of natural language processing (nlp) (chowdhary and chowdhary, 2020). these models, pretrained on extensive text corpora, produce fluent and contextually appropriate writing. this aids in advancing several nlp activities, such as query translation, text categorization, and language translation. the ability to generalize without prior taskspecific training is particularly noteworthy. the llms' adaptability in producing a range of writ-ing styles, from academic to creative, without requiring domain-specific training is further shown by recent research by @xcite . due to their versatility can be used in various contexts, including automated content creation, chatbots, and virtual assistants. however, this capability also brings significant challenges. with ai-generated text becoming increasingly prevalent, verifying the authenticity of content becomes increasingly complex, raising concerns over academic integrity and the spread of misinformation. but moral dilemmas are associated with llms' sophisticated capabilities @xcite their ability to write intelligible and contextually relevant content makes it easier for them to be abused, including spreading false information and fake news. these dangers damage society's views and undermine public confidence. plagiarism, intellectual property theft, and the creation of false product evaluations are all issues that should worry businesses and customers alike @xcite . additionally, llms can maliciously change web information, affecting political debate and public opinion. however, this capability also brings significant challenges. as ai-generated text becomes more prevalent, verifying the authenticity of content becomes increasingly complex, raising concerns over academic integrity and the spread of misinformation @xcite . llms must be developed and implemented responsibly in light of these ethical issues. these approaches come with a complicated and diverse ethical environment. these problems must be addressed to realize the full benefits that appropriately deployed llms may bring to society. to do this, current research has shifted to developing detectors that can differentiate between text produced by computers and text written by people. these detectors act as a defense against possible llm abuse. our research aims to develop a robust classifier for differentiating human and ai-generated texts. we use the diverse hc3-english dataset, which contains several extracted features from both kinds of texts. we aim to analyze these features and train models to correctly identify the text's origin. extracted features include all the pos tags, vocabulary size, word density, active and passive voice usage, flesch reading ease score, gunning fog index, perplexity, and burstiness. we have taken more sophisticated models such as cnn_bilstm, rnn, bert, and roberta to evaluate their performance in this classification task. roberta was the most accurate for this task, making this model the most effective for our task. the key contributions of this paper are outlined as follows: the structure of this paper is as follows: section 2 reviews existing research in the field of text classification, focusing on methods for distinguishing between human-written and ai-generated text. this section also presents various classification techniques and discusses previous sota model results, offering a comparative analysis of both types of texts. section 3 outlines our approach to addressing the problem statement. it details the methods, including model selection, experimental setup, and implementation specifics. section 5 discusses the performance of our models, highlighting the superior performance of the roberta model. section 7 summarizes the findings and concludes the paper. section 6 suggests potential areas for further investigation and improvement based on the findings of this study. ## related work this chapter will review existing research on aigenerated text and explore the classification of texts generated by ai and humans. ## propsed methodology problem statement the sophistication with which ai-generated text proliferates has made it difficult to discern whether it is human or ai-generated text. this problem has real-world applications in domains where reliable text source classification is essential, such as spotting fraudulent emails and false news. given two text samples, the objective is to develop a binary classification model that can accurately identify which text is generated by ai and which is human-authored. this classification problem can be mathematically formalized as follows: let x i represent a text sample, where i denotes the index of the sample in the dataset. the objective is to categorize the text with a specific label y i to each x i , where: the problem then becomes one of estimating a function f : @xmath0, where x is the space of all possible text samples and @xmath1, 1} is the set of possible labels. ## results and discussions various dl and transformer-based models were utilized to evaluate training and testing accuracy, as demonstrated in the comparing baseline and proposed models as shown in several factors contributing to this success were identified. first, roberta's advanced architecture was recognized as a significant contributor to its effectiveness. the model was pre-training on a vast corpus of diverse texts, enabling it to identify various linguistic patterns and styles. this comprehensive pre-training provided a solid foundation, allowing the model to excel across different text types. ## future work in future research, we aim to gather additional data to expand the dataset to include a broader range of text types and sources. we also aim to explore other more effective feature engineering, investi-gating additional aspects that could optimize the model's architecture performance, such as syntactic and discourse features. furthermore, we will develop more sophisticated models, experimenting with advanced dl techniques like graph neural networks and transformer-based models. finally, we plan to deploy the model by developing a userfriendly interface to make the ai-generated text detection technology accessible to researchers, educators, and the general public. this step is crucial for improving the effectiveness and reliability of the technology across various application scenarios, enabling broader adoption and feedback-driven refinement. ## conclusion in today's network security landscape, public opinion monitoring, and news media, identifying aigenerated text is growing progressively vital due to dl technologies' rapid advancements and extensive adoption. this study introduces an innovative ai text detection model built on the roberta algorithm, providing a novel solution in this domain. our approach integrates feature extraction, model development, and evaluation, demonstrating its efficacy with a test accuracy of 99.73. these results highlight the model's capability to differentiate between human-written and ai-generated text accurately, showcasing its potential to contribute significantly to nlp and llm's responsible and ethical use. through rigorous training and advanced dl techniques, the model exhibits high accuracy and low loss on the training set and stable and commendable performance on the test set. these findings are crucial for enhancing aigenerated text detection technology and serve as a valuable resource for further research and practical applications in related fields.