Text Fluoroscopy: Detecting LLM -Generated Text through Intrinsic Features (2024)

Abstract: Large language models (LLMs) have revolutionized the domain of natural language processing because of their excellent performance on various tasks. Despite their impressive capabilities, LLMs also have the potential to generate texts that pose risks of misuse. Consequently, detecting LLM-generated text has become increasingly important.Previous LLM-generated text detection methods use semantic features, which are stored in the last layer. This leads to methods that overfit the training set domain and exhibit shortcomings in generalization. Therefore, We argue that utilizing intrinsic features rather than semantic features for detection results in better performance.In this work, we design Text Fluoroscopy, a black-box method with better generalizability for detecting LLM-generated text by mining the intrinsic features of the text to be detected. Our method captures the text’s intrinsic features by identifying the layer with the largest distribution difference from the last and first layers when projected to the vocabulary space.Our method achieves 7.36% and 2.84% average improvement in detection performance compared to the baselines in detecting texts from different domains generated by GPT-4 and Claude3, respectively.

## introduction large language models (llms) such as palm @xcite , chatgpt (ope-nai, 2022), llama @xcite , and gpt-4 @xcite demonstrate remarkable advancements in language capabilities. llms have significantly impacted the field of natural language processing, enabling proficient text generation for diverse tasks, including emails, news, and academic papers. with the advent * corresponding author. however, if misused, llms such as chatgpt have the potential to act as a "weapon of mass @xcite . for example, the advanced writing capabilities of llms pose a significant threat to democracy, as they facilitate the creation of automated bots on social networks that can influence political decisions during election campaigns @xcite . moreover, the use of chatgpt by students in educational institutions has led to instances of academic dishonesty, with essays being generated by these models, as reported by various news outlets @xcite . therefore, it is crucial and urgent to detect llmgenerated texts. previous methods for detecting llm-generated text can be classified into two categories. the first category relies on the features of the last layer in the language model, e.g., bert @xcite @xcite , which can be seen as the semantic features @xcite . however, semantic features in human-created and llm-generated text can be remarkably similar, especially when the topics are more narrowly defined, affecting detection quality and generalization. the second category relies on linguistic features @xcite @xcite , which are expressed as differences in the frequency of words and grammatical patterns. however, experimental results show that linguistic features are more fragile to paraphrase attacks than semantic features @xcite . guided by the above analysis, it is evident that overly abstract semantic features and overly simple linguistic features can adversely affect detection quality and robustness. consequently, we can infer that the features of the first and last layers are encoder text fluoroscopy a man invents time travel in order to find a cure for his sick wife… . . . . . . ## methods the goal of an llm-generated text detection task is to ascertain whether a given text is generated by llms. let x be the text to be detected. formally, @xmath0, x 1 , . . . , x t-1 ) consists of t tokens. we leverage a pre-trained language model as the encoder to extract intrinsic features. pre-trained language models consist of an embedding layer, n stacked transformer layers, and an affine layer ϕ(•) for predicting the distribution of the next word. first, the embedding layer embeds the tokens then h 0 would be processed by each of the transformer layers successively. we denote the output of the j-th layer as h j , (0 ≤ @xmath1. then, the vocabulary head ϕ(•) predicts the probability of the next token x t over the vocabulary set x . the method based on semantic features uses the feature h (n ) t to the classifier. instead of using the features stored in the last layer, our method identifies the layer with the largest distribution difference from the last and first layers when projected to the vocabulary space. although the vocabulary head ϕ(•) is only trained on the last layer, it is appropriate to use this head on the middle layers. we follow previous work @xcite @xcite , where they apply the vocabulary head directly to the hidden states of the middle layers for early exit. the rationality of the process has also been given in previous work. since the residual connections in language models @xcite make hidden representations evolve gradually without abrupt changes. this smooth transition and stability mean that directly applying the vocabulary head trained on the final layer to the middle layers can still yield reasonable prediction results, thus eliminating the need for additional training of the vocabulary head. we first predict the probability of the next token x t over the vocabulary set x for every layer. for j-th layer, we predict the probability of the next token x t over the vocabulary set x , where j ⊂ {1, . . . , n -1} is a set of candidate layers, the probability of the next token x t over the vocabulary set x for the 0-th and n -th layers are denoted as q 0 (x t | @xmath2, respectively. then, we use kullback-leibler divergence (kl divergence) to calculate the difference between the distributions. we calculate the difference between the distributions q and we select the layer with the largest kl divergence from the 0-th and nth layers, denoted the m -th layer (0 < @xmath3. discussions about the kl divergence and selection of layers are shown in appendix a. after determining m , we use the features of the m layer for classification. ## conclusion in this paper, we design text fluoroscopy, a blackbox method for detecting llm-generated text through intrinsic features. our method captures the intrinsic features by identifying the layer with the largest distribution difference from the first and last layers when projected to the vocabulary space. compared with previous methods, we reduce the impact of semantic features on the detection process to achieve better detection quality and generalization. our method can effectively detect llmgenerated texts and is more robust in real-world scenarios. we aspire that text fluoroscopy will inspire future research in llm-generated text detection and offer insightful references for identifying content generated by lms in other fields.