Mirror Minds : An Empirical Study on Detecting LLM -Generated Text via LLM s (2025)

Abstract: The use of large language models (LLMs) is inevitable in text generation. LLMs are intelligent and slowly replacing the search engines. LLMs became the de facto choice for conversation, knowledge extraction, and brain storming. This study focuses on a question: ‘Can we utilize the generative capabilities of LLMs to detect AI-generated content?’ We present a methodology and empirical results on four publicly available data sets. The result shows, with 90% accuracy it is possible to detect AI-generated content by a zero-shot detector utilizing multiple LLMs.

## introduction the rapid advancement of large language models (llms) has elevated their text generation capabilities to levels comparable to human writing @xcite @xcite . llms have become integral to various aspects of daily life and are increasingly pivotal in numerous professional workflows @xcite . they aid in tasks such as creating advertising slogans @xcite , composing news articles @xcite , and generating stories @xcite . additionally, the influence of llms is profoundly shaping the development of many sectors and disciplines, including education @xcite , law @xcite , and medicine @xcite . however, the remarkable proficiency of generative language models in producing text has simultaneously heightened worries about their potential misuse in fields such as phishing, spreading misinformation, and academic dishonesty. regrettably, humans are only marginally better than chance at distinguishing between ai-generated text and text written by people @xcite . consequently, we aim to create an automated system that can accurately identify ai-generated texts to help prevent their harmful applications. llms no longer require task-specific learning or alignment between task inputs and desired outputs because they have already acquired most necessary knowledge during pre-training. instruction tuning helps to better align the model with the anticipated responses for user tasks. motivated by this, we do an empirical study on a question that "can a llm detect the content written by ai?". we developed a three-step methodology to identify ai-generated texts. in the first step, we generate a contextually relevant query from the input through large language model. essentially, we aim to create an input prompt that logically follows the given text. the process of generating this prompt is very fast (∼2 seconds) due to the nature of zero-shot inference in llms. once the prompt is generated, we pass it to two different large language models, each of which produces a response of approximately the same length as the input text. these generated texts are then compared with original input for the final classification. we use bleu (bilingual evaluation understudy) @xcite and meteor @xcite score as comparative measures. we tested our methodology on four publicly available datasets, and the results show that our pipeline can detect ai-generated text with an accuracy of approximately 90%. ## related work earlier detection methods primarily relied on feature-based approaches, such as analyzing the frequency of rare bigrams @xcite , n-gram frequencies @xcite , or topk word patterns as used in gltr @xcite . however, as machine-generated text becomes increasingly sophisticated, a shift toward trained-based methods has emerged. for exmaple, recent literature to detect ai-generated texts include guo et al. ( 2023 ), who fine-tuned roberta to identify texts in english and chinese as either ai-generated or human-written, facing challenges with overfitting that degraded performance on outof-domain data. also, openai text classifier (ope-nai, jan 2023) and gptzero (tian, 2023) that detects ai-generated text by training on the input data. a notable challenge with these models is the need for periodic retraining to keep pace with updates and advancements in new large language models (llms). another approach, detectgpt by @xcite , employs a zero-shot classifier that assumes ai-generated texts exhibit lower model probabilities compared to originals, but struggles with short texts due to its need for lengthy inputs to understand context. additionally, @xcite explored using watermarks in ai-generated texts to aid detection, though this method can compromise text readability and watermark removal presents further difficulties. in a very recent times, dna-gpt @xcite proposed which provides a distinctive and effective approach to identifying gpt-generated text by leveraging the inherent differences in continuation patterns between human-written and aigenerated content. ## conclusion while large language models (llms) have demonstrated impressive capabilities in generative tasks, mitigating their potential misuse remains crucial. this paper provides an empirical study by introducing a simple yet effective method for detecting ai-generated text. our zero-shot detector leverages the extensive knowledge acquired by llms during pre-training, enabling them to identify their own generated outputs. note that this methodology does not involve any training for classification; instead, it classifies input text using zero-shot inference. the process begins by extracting the main context of the input text, using the llm to generate a corresponding response. this approach facilitates the detection of ai-generated text by comparing the similarity between the input and the generated text. instruction tuning refines the model's alignment with the user expected responses in text detection tasks. we evaluated our method across four publicly available datasets, which cover responses generated by different llms and including both in-domain and out-of-domain (ood) data. the results confirm that our approach effectively detects texts produced by llms. future efforts will focus on enhancing detection capabilities for short texts and extending the application to other areas such as fake news detection.