Beyond Black Box AI generated Plagiarism Detection: From Sentence to Document Level (2023)

Abstract: The increasing reliance on large language models (LLMs) in academic writing has led to a rise in plagiarism. Existing AI-generated text classifiers have limited accuracy and often produce false positives. We propose a novel approach using natural language processing (NLP) techniques, offering quantifiable metrics at both sentence and document levels for easier interpretation by human evaluators. Our method employs a multi-faceted approach, generating multiple paraphrased versions of a given question and inputting them into the LLM to generate answers. By using a contrastive loss function based on cosine similarity, we match generated sentences with those from the student’s response. Our approach achieves up to 94% accuracy in classifying human and AI text, providing a robust and adaptable solution for plagiarism detection in academic settings. This method improves with LLM advancements, reducing the need for new model training or reconfiguration, and offers a more transparent way of evaluating and detecting AI-generated text.

## introduction in recent years, large language models (llms) have demonstrated remarkable capabilities across a wide range of natural language processing (nlp) tasks, including text classification, sentiment analysis, translation, and question-answering @xcite . these foundational models exhibit immense potential in tackling a diverse array of nlp tasks, spanning from natural language understanding (nlu) to natural language generation (nlg), and even laying the groundwork for artificial general intelligence (agi) @xcite . in the world of advanced llms, @xcite as an ai model developed by @xcite has become one of the most popular and widely used models, setting new records for performance and flexibility in many applications. according to the latest available data, chatgpt (2023) currently has over 100 million users and the website currently generates 1 billion visitors per month @xcite . while chatgpt has brought numerous benefits such as it allows us to obtain information more effectively, improves people's writing skills etc., however, it has also introduced considerable risks @xcite . a major risk associated with the growing dependence on chatgpt is the escalation of plagiarism in academic writing @xcite , which subsequently compromises the integrity and purpose of assignments and examinations. thanks to its advanced training process and access to abundant pre-training data sets, chatgpt is capable of resembling human-like language when provided with a prompt @xcite . it even exceeds human performance in some academic writing while maintaining authenticity and richness. furthermore, humans are unable to accurately distinguish between human generated text (hgt) and machine generated text (mgt), regardless of their familiarity with chatgpt @xcite . these factors present significant challenges in maintaining educational integrity and challenge the current paradigm of how teachers teach. to reduce potential plagiarism caused by the use of llms, researchers have developed various ai-generated text classifiers or tools such as log-likelihood @xcite , roberta-qa (hc3) @xcite , gptzero (2023), openai classifier (openai, 2023b), detectgpt @xcite , and turntin @xcite . figure 1 lists popular llms used for text generation and ai-generated text detection tools. existing approaches for detecting text generated by language models have several limitations as highlighted in ## related work the field of ai-generated text detection has garnered significant interest, but only a few models and tools have achieved widespread adoption. in this section, we discuss state-of-the-art approaches, the datasets used for training their classifiers, and their limitations. ## our method in this section, we present our approach to effectively compare and detect plagiarism in student responses. our method utilizes an advanced paraphrasing model, a state-of-the-art language model, and a contrastive loss function to deliver a comprehensive and transparent evaluation system. figure 2 shows the different components of our proposed model architecture. ## experimental evaluation in our experimental evaluation, we aim to measure the accuracy of our approach in detecting similarities between human and machine-generated answers. we use the human chatgpt comparison corpus (hc3) dataset, which contains human and chatgpt-generated answers to the same questions. ## conclusion in conclusion, this research presents a novel and effective method for detecting machine-generated text in academic settings, offering a valuable contribution to the field of plagiarism detection. by leveraging a comprehensive comparison technique, our approach provides more accurate and explainable evaluations compared to existing methods. the sentence level quantifiable metrics facilitate easier interpretation for human evaluators, mitigating the black-box nature of existing ai text detection methods. our model is adaptable to various nlg models, including cutting-edge llms like bardai and character.ai, ensuring its relevance and effectiveness as technology continues to evolve. this adaptability makes our approach a significant asset in maintaining academic integrity in the face of rapidly advancing natural language processing technologies. future research directions include collecting additional unbiased datasets for evaluation and comparing the performance of our model with other detection tools. we also plan to explore the incorporation of different algorithms at the sentence level, assembling them to achieve even better performance. moreover, we plan to employ stylometry techniques to identify each student's unique writing style as more data from their responses are collected. this process will create a distinct signature based on the student's writing patterns, making it increasingly easy to detect plagiarism in future submissions. these efforts will further refine our model and contribute to the ongoing pursuit of robust, transparent, and adaptable plagiarism detection methods in academia.